Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-18893

RHS of local join truncated if grouped and already sorted


    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.4.0, 6.4.2, 6.4.4
    • Fix Version/s: 6.4.8
    • Component/s: Thor
    • Labels:
    • Environment:
      LnP (formerly USLM) Dev


      Jake Smith Sorry this is so sloppy.  It's pasted from an email and lost the images.  

      From new (W

      That is worrying and can never be right.
      The Join should read the entire RHS.
      The fact it's read apparently 1 per node suggests it's though it hit end of input.
      My guess is it's something to do with the fact that's a grouped dataset and the fact that is marked as already sorted.

      The removal of the 'sorted' my not be significant, it looks like the new one without it, output the RHS dataset as a named dataset:

      vs the one that hit problems, which had a spill write, spill read into RHS of the problematic join.
      It may well be that that 'fixed' the issue.

      I'm hoping I have enough info. to reproduce the issue in a self-contained test case.
      In the meantime, please go ahead and open a JIRA.


      On 12/12/17 19:52, Cella, Joseph R. (LNG-DAY) wrote:

      Jake – this was separate from the TBB malloc MP Link issues.  Have you had a chance to look at this yet?


      From: Cella, Joseph R. (LNG-DAY)
      Sent: Thursday, December 07, 2017 2:04 PM
      To: Cobbett-Smith, Jacob (RIS-HBE)
      Cc: Crain, David B. (LNG-DAY) (David.Crain@lexisnexis.com); Wagner III, Russell (RIS-BCT)
      Subject: issue - sorted(...,field,local)




      Got some interesting behavior with which I need your help determining what to do.  Dave Crain put together a few nice screenshots in the below email. 


      First, a little explanation of Dave’s terminology:


      “Old Dev” has ECLWatch at (internal_5.4.10-1)


      “New Dev” has ECLWatch at (internal_6.4.4-1)


      The “interesting” thing is the right side coming into local join on the second and third screenshots below.  There is different behavior between the two builds.  It appears to only be taking one record per thor node with the build. 


      The relevant code (from the Properties of the screenshots are) are:


      Screenshot 1 - DCrain_Tools.ExtractLbuInfo2(70,19):


      (Ignore the highlighting for now…)


         uniqueDocInfo := dedup(sort(group(sorted(docIDTable,doc_id, local),


                                     head_title, meta_title, sourcename),

                                head_title, meta_title, sourcename);


      I’ll throw thes line in to transition to the next attribute:


         export extractDocInfo := uniqueDocInfo;


        result := module

          export action := outAction();

          export extractDocInfo := uniqueDocInfo;





      extractLpInfo := DCrain_Tools.ExtractLbuInfo2(info, selectedNodes, inDocInfo);

      extractedInfo := sorted(distributed(extractLpInfo.extractDocInfo),doc_id,local);



      Screenshots 2 and 3 – line 406 from the BWR:


      updDocsPull := join(distDocsPull, extractedInfo,

                        left.doc_id = right.doc_id,


                          self.lp_doc_id            := left.doc_id,

                          self.lp_lni               := left.lni,

                          self.lp_pcsi              := left.rci,

                          self.lp_hierarchy_nominal := left.hierarchy_nominal,

                          self.lp_sourcename        := right.sourcename,

                          self.lp_meta_title        := right.meta_title,

                          self.lp_head_title        := right.head_title,

                          self                      := [],




      Looking at screenshot 3, only 50 records are selected, which is the problem. 


      Getting back to the highlighting, I had Dave remove the sorted(…,local) and the result matches the 5.4 build results (the workunit on “new Dev” is W20171207-092553).


      That said, I think the sorted(…,local)statement is accurate and the data is locally sorted by the doc_id field.  This was not the result of a global sort(…,doc_id), but my opinion is that should not be required to declare  sorted(…,local).


      My first question is:  Is sorted(…,local) an acceptable statement?  If not, the compiler should probably not accept it. 


      Assuming it is acceptable, my second question is:  Is there something incorrect in the build for which a JIRA needs to be opened?



      I’ve attached the archives for the two workunits Dave mentions below.  Let me know if you need the archive from the workunit without sorted(…,local)  or any other info.




      From: Crain, David B. (LNG-DAY)
      Sent: Thursday, December 07, 2017 9:01 AM
      To: Cella, Joseph R. (LNG-DAY)
      Cc: Crain, David B. (LNG-DAY)
      Subject: Problem in New Dev...




      Per our conversation, here is the information you requested…

      In Old Dev, WU for comparison: W20171207-074923


      In New Dev, WU where I encounter the problem: W20171207-075159


      In Graph 1, sub-graph 230 from both WU’s look good:



      It is when we go to sub-graph 235 that we have a problem…


      Graph from Old Dev:



      Graph from New Dev:




      Let me know if there is any other information you need…

      David B. Crain

      System Consultant
      Global Content Systems Development (GCSD), *C*ontent *E*nrichment *S*ystems

      937.247.8602 (Office)
      937.239.3407 (Cell)




          Issue Links



              • Assignee:
                jakesmith Jake Smith
                joecella Joe Cella
              • Votes:
                0 Vote for this issue
                4 Start watching this issue


                • Created: