Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-13268

Optimize DEDUP(ds, somethingcomplex, ALL)

    Details

    • Type: Bug
    • Status: Accepted
    • Priority: Not specified
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Code Generator
    • Labels:

      Description

      There are several situations where it would be better to project a complex expression into a dataset and then use it in a subsequent operation:

      DEDUP(ds, complexfunc(ds), ALL);
      

      becomes

      p := TABLE(ds, { ds, extra := complexfunc(ds); });
      DEDUP(p, extra, ALL);
      

      It isn't always best though - things to bear in mind:

      • Projecting the expression will use more memory which may cause thor to spill.
      • If the complex expression is not the leading component it may not be evaluated at all for some operations, so projecting it will evaluate it unnecessarily. There would need to be an option to prevent on a per activity basis the optimization.
      • If there is only 1 input record it may be evaluated when it wouldn't be otherwise.

      Possible examples:

      • SORT - especially for leading components.
      • DEDUP - especially , ALL, but non-all might also benefit.
      • TABLE grouping criteria - if a hash aggregate isn't used.

        Attachments

          Activity

            People

            • Assignee:
              anybody Available for anyone
              Reporter:
              ghalliday Gavin Halliday
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: