Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-8707

Allow Thor to manage multiple jobs in a single cluster

    Details

    • Type: Improvement
    • Status: Accepted
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Thor
    • Labels:
    • Compatibility:
      Major

      Description

      There are two roadmap items which rely on the following items being done prior to starting:

      1) To create and store more partition points per output dataset than # of slaves the cluster is currently using.
      2) To repartition input datasets efficiently where possible, based on 1)
      3) Get thor dynamically choosing numbers of slave processors in-place of current static configuration
      4) Allow Thor to run multiple jobs using subsets of the slave 'pool'.
      5) Allow Thor to run concurrent jobs on same logical cluster, in most basic case will be equivalent to multi-thor setups we have now.

      These are shown as sub tasks below but need more information to be added.

      The two roadmap items which are reliant on them are...

      Better dynamic spilling - https://track.hpccsystems.com/browse/HPCC-8685
      MPI Migration for Thor - https://track.hpccsystems.com/browse/HPCC-8706
      Dynamic sizing of clusters - https://track.hpccsystems.com/browse/HPCC-9960

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jakesmith Jake Smith
                Reporter:
                lorraineachapman Lorraine Chapman
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: