Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-19917

Change index merging to only sort by keyed fields

    XMLWordPrintable

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • 7.0.0
    • JHTree, Roxie
    • Major

    Description

      Currently the key merging code compares all fields when merging rows from multiple sources.  It currently does this using a memcmp on the maximum length of the key record.  This has several problems:

      • It could access uninitialised data.
      • The payload may not be sorted in a memcmp order - so may return rows out of order
      • If field translation is in use there is no guarantee that the rows will be ordered

      I propose changing the comparison so that it only compared the key fields.  If they are identical the row from the lower part is returned.

       

      I think the key merger is currently used in the following situations:

      a) Smart stepping.  This should be unaffected by the change.

      b) Merging results from local indexes.  This will potentially change.

      c) Keyed join in thor.  I suspect this is for convenience.  For global keys it may be more efficient to use a key manager that does not compare - and takes rows from parts in order.

       

      jakesmith richardkchapman

       

      Attachments

        Issue Links

          Activity

            People

              richardkchapman Richard Chapman
              ghalliday Gavin Halliday
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: