Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-24630

KeyedJoin from Thor accessing a 1-way on same cluster performs poorly

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.10.16
    • Component/s: Thor
    • Labels:
      None

      Description

      Keyed Join detects which slaves have parts locally, and (by default) sends all requests to be fulfilled by that local slave.
      If the index is 1-way (or much smaller than the cluster size if overlapping), then it sending all requests to be remotely fulfilled like this causes inefficiency.
      It it probably better to read them directly (as if all parts were local), especially if the key is small.
      That way the jhtree key node will service a lot of the requests, with the upshot that the speed is greatly increased.

      For a large small width key it may still make sense to service remotely, but I suspect it is better to turn off remote keyed lookups where the key is significantly smaller than the cluster.

      Normally the only case that is true, is where the key is 1-way, but it can also happen if there are clusters of different sizes that overlap the cluster they query is running on, and the key has been built on one of those smaller overlapping clusters.

      The workaround if this is an issue for now, is to use either:

      #option('remoteKeyedLookup', false);
      

      or to add:

      HINT(remoteKeyedLookup(false))
      

      to the keyed JOIN's affected.

        Attachments

          Activity

            People

            • Assignee:
              jakesmith Jake Smith
              Reporter:
              jakesmith Jake Smith
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: