Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-9462

Add cluster file management options


    • Type: New Feature
    • Status: Scheduled
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: DFU Server
    • Labels:
    • Compatibility:


      Add the facility to copy all files and super files from one cluster to another, within the same environment and foreign.

      There's also a case for a more flexible way to copy/spray multiple files to a target cluster, but I think that's slightly different.

      In the case of an entire cluster->cluster copy, I think there are special considerations:

      • If cluster widths are the same, should DFU employ a rsync or similar to efficiently copy everything across?
      • Ignore some files during copy ? e.g. 'hpccinternal::' and other job owned files?
      • Should persist files be copied ? if cluster widths are different than I think won't make sense to.
      • There should be options to ignore existing files that match (to facilitate syncing new files and/or continue from a previously stopped cluster copy)
      • This is going to take a considerable amount of time and block other sprays in the meantime..
        DFUServer runs jobs in series, it may make sense if source and destination are the same, I think there should at least be the option of running multiple DFU jobs in parallel. And there should perhaps be DFU job priorities, where these bulk copies were by default a lower priority than user / query sprays..

      Richard Chapman Gavin Halliday Tony Kirk





            • Assignee:
              attilavamos Attila Vamos
              jakesmith Jake Smith
            • Votes:
              3 Vote for this issue
              9 Start watching this issue


              • Due: