Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-20983

Thor startup per job

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Scheduled
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Thor
    • Labels:
    • Compatibility:
      Minor

      Description

      There are a lot of reasons why it would be preferable to start Thor "on demand" for each job rather than leave it running waiting for next job. These include:

      1. Can run thor as a user corresponding to the job owner rather than always using hpcc user
      2. Can more easily size thor according to the job needs
      3. Cloud-friendly - not paying for machines while idle
      4. Recovery from last spill may be easier to implement
      5. Likely to be more reliable
      6. Isolates jobs from side effects of previous jobs (e.g. Java VM or Python artefacts, resource leaks from C++ code (or platform bugs) etc
      7. etc

      Main potential downside is that the startup time needs to be short enough to be insignificant for overall thor job times. We COULD consider continuing without a restart if there is another job waiting and some conditions were met, though you may lose some of the above advantages if you do.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mckellyln Mark Kelly
                Reporter:
                richardkchapman Richard Chapman
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated: