Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-9462

Add cluster file management options

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Scheduled
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: DFU Server
    • Labels:
      None
    • Compatibility:
      Minor

      Description

      Add the facility to copy all files and super files from one cluster to another, within the same environment and foreign.

      There's also a case for a more flexible way to copy/spray multiple files to a target cluster, but I think that's slightly different.

      In the case of an entire cluster->cluster copy, I think there are special considerations:

      • If cluster widths are the same, should DFU employ a rsync or similar to efficiently copy everything across?
      • Ignore some files during copy ? e.g. 'hpccinternal::' and other job owned files?
      • Should persist files be copied ? if cluster widths are different than I think won't make sense to.
      • There should be options to ignore existing files that match (to facilitate syncing new files and/or continue from a previously stopped cluster copy)
      • This is going to take a considerable amount of time and block other sprays in the meantime..
        DFUServer runs jobs in series, it may make sense if source and destination are the same, I think there should at least be the option of running multiple DFU jobs in parallel. And there should perhaps be DFU job priorities, where these bulk copies were by default a lower priority than user / query sprays..

      Richard Chapman Gavin Halliday Tony Kirk

      thoughts?

        Attachments

          Activity

            People

            • Assignee:
              attilavamos Attila Vamos
              Reporter:
              jakesmith Jake Smith
            • Votes:
              3 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Due:
                Created:
                Updated: