Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-16285

Crazy remote directory scan for every size() call

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 7.0.0
    • Component/s: dafilesrv
    • Labels:
      None
    • Compatibility:
      Minor

      Description

      No idea why, but dafilesrv clients obtains the size() of remote files, by sending a getdir with an exact mask for the file it wants the size of.

      This is 'ok' if remote directory is smallish. It is horrendous if there's a lot of size() calls and the remote dir(s) are not trivially small.

      On the server-side, it ends up creating a directory iterator, serializing that iterator back to the client, which serializes more info that is required albeit 1 files worth.

      All in all if there are lot of these requests in directories that are not trivially small it's expensive.

      There's a very old comment in the client code for CRemoteFile::size() that says:

      // do this by using dir call (could be improved with new function but this not too bad)

      We should add a new dafilesrv command to do this.
      There's already a RFCsize, but that's for when you have already opened the file and have a handle.
      There should be a RFCfilesize or similar, which passes the filename and performs a stat() on it to retrieve the size.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mckellyln Mark Kelly
                Reporter:
                jakesmith Jake Smith
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: