Type: New Feature
Affects Version/s: None
Fix Version/s: None
Component/s: DFU Server
Add the facility to copy all files and super files from one cluster to another, within the same environment and foreign.
There's also a case for a more flexible way to copy/spray multiple files to a target cluster, but I think that's slightly different.
In the case of an entire cluster->cluster copy, I think there are special considerations:
- If cluster widths are the same, should DFU employ a rsync or similar to efficiently copy everything across?
- Ignore some files during copy ? e.g. 'hpccinternal::' and other job owned files?
- Should persist files be copied ? if cluster widths are different than I think won't make sense to.
- There should be options to ignore existing files that match (to facilitate syncing new files and/or continue from a previously stopped cluster copy)
- This is going to take a considerable amount of time and block other sprays in the meantime..
DFUServer runs jobs in series, it may make sense if source and destination are the same, I think there should at least be the option of running multiple DFU jobs in parallel. And there should perhaps be DFU job priorities, where these bulk copies were by default a lower priority than user / query sprays..