Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-16649

Code optimizer, is moving PROJECT's above DISTRIBUTE's sometimes where PROJECT is significantly increasing size of records.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.4.0
    • Component/s: Code Generator
    • Labels:
      None

      Description

      I don't know what the optimizer semantics are, but I've seen a case of a DISKREAD->DISTRIBUTE->PROJECT, where the the diskread records were relatively small, but the project records were very large.

      Under those circumstances, it is not good to move the project above the distribute (which then gets folded into the diskread), as it causes a lot of additional work for distribute, i.e. a lot of additional network traffic, and consequently a slow down.

      This is a code extract of the code involved:

          redist_input := distribute(pIn_file,skew(0));
      
          quad_keys := project(redist_input,                    transform(mapview64.M_PERIL_TAGGING_UTILS.DenormRec,
                  quad_keys := mapview.M_QuadkeyUtil.getQuadkeyForWGS84GEOMBbox(left.geom,18);
                  this := left;
                  self.peril_type := pIn_peril_type;                        
                  self.quad_keys := project(quad_keys, transform(mapview64.M_PERIL_TAGGING_UTILS.quadKeyIdRec,
                          self.quad_key := left.quad_key;
                          self:=this;     
                          self.peril_type := pIn_peril_type;
                      )
                  );
                  self.quad_key_count := count(self.quad_keys);
                  self:=left;
              ),
              LOCAL
          ).quad_keys;
      

        Attachments

          Activity

            People

            • Assignee:
              ghalliday Gavin Halliday
              Reporter:
              jakesmith Jake Smith
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: