Details
-
Improvement
-
Status: Accepted
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
-
Minor
-
Not applicable
-
Description
Fido has close to 1.5 PB of data that will be migrated to Azure / Blob Storage / File System.
We are on a 20 node 40 way cluster.
#1 Spray files in csv / tsv format
#2 Stage files in compressed Thor format
#3 Dimensional model data core business logic, in dims and facts (some facts are partitioned by year month in its super file)
End goal is to have all of these moved to the Azure platform as part of Lift and Shift.
Expectation is: If I'm able to get the Dali / DFU metadata information exported as an XML or JSON for each file that has the file parts information, then I will be able migrate easily.
This will also allow me to bring data from different underlying Legacy fileparts (some in 20 / 40 / 60 way).
Happy to share additional details when needed.