Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-3303

LOCAL JOIN fails with "Out of memory"

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 3.8.4
    • Component/s: None
    • Labels:

      Description

      A local join with many matches fails with message: "Out of memory, allocating row array, had 9830395, trying to allocate 9830396 elements". If LOCAL is not specified, the join completes successfully. It also works if NOSORT is specified. Occasionally, the LOCAL JOIN will work without an error, but then if run again, it will fail. I ran this with build 3.8.2. On the 702 build, this gives a "group too large" error. Here is some code that will illustrate the issue.

      layoutLeft := {
      	unsigned field1,
      	unsigned match_field,
      };
      
      layoutRight := {
      	unsigned match_field,
      	string3 s1 := 'abc',
      };
      
      dsL := dataset([{1,1}], layoutLeft);
      dsL_dist := distribute(dsL, match_field);
                        
      dsR := dataset([{1}], layoutRight);
      // make a dataset with 30 million matches and distribute the same as left dataset
      dsR_dist := normalize(distribute(dsR, match_field),
                        30000000,
                        transform(layoutRight,
                          self := left));
      
      writeFiles := parallel(
      	output(dsL_dist,, '~dustin::delete::dsL', overwrite),
      	output(dsR_dist,, '~dustin::delete::dsR', overwrite)
      );
      
      // read from disk to avoid any compiler optimizations of inline datasets
      dsL_read := distributed(dataset('~dustin::delete::dsL', layoutLeft, thor));
      dsR_read := distributed(dataset('~dustin::delete::dsR', layoutRight, thor));
      
      j1 := join(dsL_read, dsR_read,
                 left.match_field = right.match_field
                 , skew(1) // I know my test data is skewed; suppress error
                 ,local); // fails "Out of memory, allocating row array,..."
                 // ,local, keep(1)); // also fails "Out of memory"
                 // ); // global join: works
                 // ,nosort); // global join: works
                 // ,nosort, local); // works
      
      sequential(
      	writeFiles,
      	output(j1,, '~dustin::delete::local_join_test', overwrite)
      );
      

        Attachments

          Activity

            People

            • Assignee:
              jakesmith Jake Smith
              Reporter:
              dustinskaggs Dustin Skaggs
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: