Details
-
New Feature
-
Status: New
-
Not specified
-
Resolution: Unresolved
-
None
-
None
-
None
Description
When copying files on cloud storage it is likely to be much more efficient to use the azure/aws apis than to have a process reading a file and writing to a target. It is likely to scale much better, cost less and be more resilient.
E.g.
https://docs.microsoft.com/en-us/rest/api/storageservices/copy-blob
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
This should be able to use the new alias section of the data plane to provide a mechanism for getting at the information needed to call the apis
Attachments
Issue Links
- relates to
-
HPCC-28147 Provide azure copy api through filehooks
-
- Resolved
-
1.
|
Basic copying with API |
|
Resolved | Shamser Ahmed |
2.
|
Copy files with storage striped across containers/file shares |
|
Resolved | Shamser Ahmed |
3.
|
Support managed identity authentication |
|
New | Unassigned |
4.
|
Copy files striped across storage accounts |
|
Resolved | Shamser Ahmed |
5.
|
Spray files from 1 to many parts |
|
New | Unassigned |
6.
|
Spray files from many to one parts |
|
New | Unassigned |
7.
|
Use multiple threads for copying large files |
|
New | Unassigned |
8.
|
Refactor to allow plugable copy apis. |
|
Resolved | Shamser Ahmed |
9.
|
Allow azure-specific code to be conditionally removed |
|
Resolved | Unassigned |
10.
|
Consider using api copy for helper dlls when deploying to roxie |
|
Accepted | Shamser Ahmed |
11.
|
Improve copy progress tracking |
|
New | Unassigned |
12.
|
Document api copying configuration |
|
New | Unassigned |
13.
|
Add api options to terraform module |
|
New | Unassigned |
14.
|
Support aws api |
|
New | Unassigned |