Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-22146

Unexpected JOIN LIMIT behavior

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.2.22
    • Component/s: Documentation
    • Labels:
      None

      Description

      We're seeing some unexpected behavior with the way default LIMITs are applied to keyed JOINs. The specific case that we had was with a JOIN, KEEP(10000) similar to j4 that we added a non-keyed condition to, that then failed, similar to j2. This code seems to behave the same between 7.2 and 6.4 so nothing new just the first time I've ran into an error because of it. 

      layout := {
      	unsigned proxid,
      	unsigned rcid,
      };
      
      ds := dataset([{3,1},{3,2}], layout);
      ds1 := dataset([{3,1}], layout);
      ds10k := normalize(ds1, 10001, transform(layout, self.rcid := counter, self.proxid := left.proxid));
      
      keyname := '~temp::bipv2::dustin::limit_keep';
      
      k := index(ds10k, {proxid}, {ds10k}, keyname);
      
      
      inDs := nofold(dataset([{3,0}], layout));
      j := join(inDs, k, left.proxid = right.proxid and (left.rcid = right.rcid), transform(layout, self := right), keep(10000), limit(10000));
      j2 := join(inDs, k, left.proxid = right.proxid and (left.rcid = right.rcid), transform(layout, self := right), keep(10000));
      j3 := join(inDs, k, left.proxid = right.proxid, transform(layout, self := right), keep(10000), limit(10000));
      j4 := join(inDs, k, left.proxid = right.proxid, transform(layout, self := right), keep(10000));
      j5 := join(inDs, k, left.proxid = right.proxid, transform(layout, self := right));
      
      fnOut := '~temp::bipv2::dustin::limit_keep_out';
      
      sequential(
      	// buildindex(k, overwrite);
      	// output(j); // error "JOIN limit exceeded(10000)"
      	// output(j2); // error "more than 10000 match candidates in keyed join"
      	// output(j3); // error "JOIN limit exceeded(10000)"
      	// output(j4); // works
      	output(j5); // error "more than 10000 match candidates in keyed join"
      );
      	
       

        Attachments

          Activity

            People

            • Assignee:
              jamesdefabia Jim DeFabia
              Reporter:
              dustinskaggs Dustin Skaggs
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved: