Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-27148

Allow files to be striped over storage accounts

    XMLWordPrintable

Details

    • New Feature
    • Status: Resolved
    • Not specified
    • Resolution: Duplicate
    • None
    • None
    • cloud, DFS
    • None

    Description

      There is likely to be a need to stripe files over different storage accounts (on azure) to boost the performance.

      The helm charts already allow a plane to configure the number of devices for a storage plane (numDevices property).  If this is >1 then the helm charts mount a set of pvcs:

       

      <pvc>-<n> at <prefix>/d<num>
      

      e.g.

       

      datapvc-1 at /var/lib/HPCCSystems/primary/d1
      datapvc-2 at /var/lib/HPCCSystems/primary/d2
      datapvc-3 at /var/lib/HPCCSystems/primary/d3
      

      It should be possible to add some code similar to the file part suffix to add a device directory at the head of the filename.  If it causes complications because it needs to be done too early, then one suggestion is to use a placeholder e.g., $d$ which gets replaced later on.  It needs to be modulo the number of devices.

      NOTE: The device for the file part should be based on a hash of the logical filename and the part number, so that single part files are distributed over the different storage accounts.  (It could be stored in the meta data to avoid having to recalculate when accessing).

       

      NOTE: This feature is also long-requested for the bare-metal version of the system.  See HPCC-10761.

      jakesmith richardkchapman

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ghalliday Gavin Halliday
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: