Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-8707

Allow Thor to manage multiple jobs in a single cluster


    • Type: Improvement
    • Status: Accepted
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Thor
    • Labels:
    • Compatibility:


      There are two roadmap items which rely on the following items being done prior to starting:

      1) To create and store more partition points per output dataset than # of slaves the cluster is currently using.
      2) To repartition input datasets efficiently where possible, based on 1)
      3) Get thor dynamically choosing numbers of slave processors in-place of current static configuration
      4) Allow Thor to run multiple jobs using subsets of the slave 'pool'.
      5) Allow Thor to run concurrent jobs on same logical cluster, in most basic case will be equivalent to multi-thor setups we have now.

      These are shown as sub tasks below but need more information to be added.

      The two roadmap items which are reliant on them are...

      Better dynamic spilling - https://track.hpccsystems.com/browse/HPCC-8685
      MPI Migration for Thor - https://track.hpccsystems.com/browse/HPCC-8706
      Dynamic sizing of clusters - https://track.hpccsystems.com/browse/HPCC-9960


          Issue Links



              • Assignee:
                jakesmith Jake Smith
                lorraineachapman Lorraine Chapman
              • Votes:
                0 Vote for this issue
                2 Start watching this issue


                • Created: