Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-19917

Change index merging to only sort by keyed fields

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 7.0.0
    • Component/s: JHTree, Roxie
    • Labels:
    • Compatibility:
      Major

      Description

      Currently the key merging code compares all fields when merging rows from multiple sources.  It currently does this using a memcmp on the maximum length of the key record.  This has several problems:

      • It could access uninitialised data.
      • The payload may not be sorted in a memcmp order - so may return rows out of order
      • If field translation is in use there is no guarantee that the rows will be ordered

      I propose changing the comparison so that it only compared the key fields.  If they are identical the row from the lower part is returned.

       

      I think the key merger is currently used in the following situations:

      a) Smart stepping.  This should be unaffected by the change.

      b) Merging results from local indexes.  This will potentially change.

      c) Keyed join in thor.  I suspect this is for convenience.  For global keys it may be more efficient to use a key manager that does not compare - and takes rows from parts in order.

       

      Jake Smith Richard Chapman

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                richardkchapman Richard Chapman
                Reporter:
                ghalliday Gavin Halliday
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: