[HPCC-23383] Rudimentary S3 and Azure Blob reading - HPCC

XML

Word

Printable

Rudimentary S3 and Azure Blob reading

relates to

HPCC-23412 Reading/writing logical files to S3/azure

HPCC-23427 Search for situations where IFiles are created, destroyed and then created

HPCC-25249 Review all disk buffer sizes

HPCC-24091 Archive plugin cores if opening zip file with no subfile

HPCC-23188 Investigate supporting Azure blobs

HPCC-23375 More complete storage plane solution

(1 relates to)

Progress

1.	Move AWS library to correct place in build folders	Resolved	Unassigned
2.	Add appropriate Azure libraries in submodule	Resolved	Unassigned
3.	Implement S3 IFile hook	Accepted	Gavin Halliday
4.	Implement Azure Blob IFile hook	Resolved	Gavin Halliday
5.	Avoid creating a directory unless a file create fails.	Accepted	Gavin Halliday
6.	Fix buffering of index write	Resolved	Gavin Halliday
7.	Compression code is checking size of output file when overwriting	Resolved	Gavin Halliday
8.	~CompressedFile performs two writes	Accepted	Gavin Halliday
9.	Do not rename outputs files when writing to blobs	New	Unassigned
10.	Should ~Shared have noexcept(false)?	Resolved	Unassigned
11.	Allow data data to go in a different location to hpcc data	Resolved	Gavin Halliday
12.	Check containers are created if they do not exists	Accepted	Gavin Halliday
13.	check uses of getRemotePath() and do not prefix with //ip/	Accepted	Gavin Halliday
14.	Use asynchronous writes for output	Accepted	Gavin Halliday
15.	Add support for secrets to the system	Resolved	Gavin Halliday
16.	Think through implications of blobs for buffering	New	Unassigned
17.	Allow noSeek build indexes to be defaulted on for a workunit	Resolved	Gavin Halliday
18.	Add asynchronous read ahead for blobs	Accepted	Gavin Halliday
19.	Ensure credentials are not leaked from filenames	Accepted	Gavin Halliday
20.	Cache the client object for a storage account	Accepted	Gavin Halliday
21.	Protect azure blob writing larger buffer sizes	New	Unassigned