Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-10221

backupnode file scanner inefficient

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Timed Out
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: BackUpNode
    • Labels:
      None
    • Compatibility:
      Minor

      Description

      Backupnode begins by scanning files for files that match group it's dealing with then building up a files part list.

      The file scanning portion is implemented such that backupnode fetch all logical files from Dali to check whether it's a match.
      It seems to want to do this way, so it can check the IP in every group of every file it checks, is indeed in the group (of IPS's) that it's supposed to be backing up.

      There is also a lot Group lookups involved here.
      There is some caching.. if the check-all-my-IP's-in-group succeeds then it avoids looking up that same named group again.

      I don't see why this is necessary, after establishing the named group you're backing up, matches the named group in Dali, it should be able to do a simple (server-side) fullfilment to get list of files for the named cluster only.

      Both of these efficiencies means on large systems this file part build process is taking a lot longer than it should.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jakesmith Jake Smith
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: