Type: New Feature
Affects Version/s: None
Fix Version/s: None
Component/s: Code Generator
In order to ensure results are deterministic from run to run we have tended to default to using STABLE sorts. Unfortunately this has a couple of consequences:
- Stable sorts typically use more memory
- It is almost impossible to optimize ordering since the ordering from previous activities theoretically feeds into the next one.
- Some activities have to jump through hoops to ensure the ordering is preserved.
This proposal is for any comparison function to always compare all the fields in the rows - UNLESS STABLE has been specified.
This means that Thor can use an unstable sort unless STABLE has been explicitly specified. It will provide a way for the new lookup join code to ensure the order of the right is consistent.
Need to be very careful about how this sort order is tracked to ensure that the second sort in the sequence:
s1 := SORT(a,b,c,LOCAL)
s2 := SORT(a,b,LOCAL)
can be removed.