Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-13268

Optimize DEDUP(ds, somethingcomplex, ALL)


    • Type: Bug
    • Status: Accepted
    • Priority: Not specified
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Code Generator
    • Labels:


      There are several situations where it would be better to project a complex expression into a dataset and then use it in a subsequent operation:

      DEDUP(ds, complexfunc(ds), ALL);


      p := TABLE(ds, { ds, extra := complexfunc(ds); });
      DEDUP(p, extra, ALL);

      It isn't always best though - things to bear in mind:

      • Projecting the expression will use more memory which may cause thor to spill.
      • If the complex expression is not the leading component it may not be evaluated at all for some operations, so projecting it will evaluate it unnecessarily. There would need to be an option to prevent on a per activity basis the optimization.
      • If there is only 1 input record it may be evaluated when it wouldn't be otherwise.

      Possible examples:

      • SORT - especially for leading components.
      • DEDUP - especially , ALL, but non-all might also benefit.
      • TABLE grouping criteria - if a hash aggregate isn't used.




            • Assignee:
              anybody Available for anyone
              ghalliday Gavin Halliday
            • Votes:
              0 Vote for this issue
              1 Start watching this issue


              • Created: