Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-15928

Generate a full compare function for all non-stable sorts

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Accepted
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Code Generator
    • Labels:
      None
    • Compatibility:
      Minor

      Description

      In order to ensure results are deterministic from run to run we have tended to default to using STABLE sorts. Unfortunately this has a couple of consequences:

      • Stable sorts typically use more memory
      • It is almost impossible to optimize ordering since the ordering from previous activities theoretically feeds into the next one.
      • Some activities have to jump through hoops to ensure the ordering is preserved.

      This proposal is for any comparison function to always compare all the fields in the rows - UNLESS STABLE has been specified.

      This means that Thor can use an unstable sort unless STABLE has been explicitly specified. It will provide a way for the new lookup join code to ensure the order of the right is consistent.

      Need to be very careful about how this sort order is tracked to ensure that the second sort in the sequence:

      s1 := SORT(a,b,c,LOCAL)
      s2 := SORT(a,b,LOCAL)

      can be removed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ghalliday Gavin Halliday
                Reporter:
                ghalliday Gavin Halliday
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: