Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-24800

Ability to tie Dali / DFU meta information to the file parts.

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Accepted
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Dali, DFU Server, Thor
    • Labels:
      None
    • Compatibility:
      Minor
    • Roadmap:
      Not applicable
    • Roadmap Summary:

      Description

      Fido has close to 1.5 PB of data that will be migrated to Azure / Blob Storage / File System.

      We are on a 20 node 40 way cluster.

      #1 Spray files in csv / tsv format 

      #2 Stage files in compressed Thor format 

      #3 Dimensional model data core business logic, in dims and facts (some facts are partitioned by year month in its super file)

      End goal is to have all of these moved to the Azure platform as part of Lift and Shift.

      Expectation is: If I'm able to get the Dali / DFU metadata information exported as an XML or JSON for each file that has the file parts information, then I will be able migrate easily.

      This will also allow me to bring data from different underlying Legacy fileparts (some in 20 / 40 / 60 way).

      Happy to share additional details when needed.

       

        Attachments

          Activity

            People

            Assignee:
            anybody Available for anyone
            Reporter:
            rnagarajan Raju
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: