Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-9959

Java example works in hthor, not in thor

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Not specified
    • Resolution: Invalid
    • Affects Version/s: 4.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:
      4.0.0-9 running on RHEL6

      Description

      This simple example works in hthor:

      IMPORT java;
       
      INTEGER add1(INTEGER val) := IMPORT(java, 'JavaCat.add1:(I)I');
       
      add1(10);
      

      (I could not force this code to execute in thor, even with #OPTION('pickBestEngine',FALSE); added.)

      If you expand that example to something like this:

      IMPORT java;
       
      INTEGER add1(INTEGER val) := IMPORT(java, 'JavaCat.add1:(I)I');
      
      DataRec := RECORD
          INTEGER     i
      END;
      
      DataRec MakeDataRec() := TRANSFORM
          SELF.i := RANDOM() % 32768;
      END;
      
      d1 := DISTRIBUTE(DATASET(10000,MakeDataRec()));
      
      d2 := PROJECT
          (
              d1,
              TRANSFORM
                  (
                      DataRec,
                      SELF.i := add1(LEFT.i)
                  ),
              LOCAL
          );
      
      OUTPUT(d2);
      

      It dies. Here is the job log from the thor master:

      00000109 2013-09-09 12:56:56.427 57824 57824 "Processing wuid=W20130909-125655, graph=graph1 from agent: 10.210.40.10"
      0000010A 2013-09-09 12:56:56.428 57824 57824 ",Progress,Thor,Start,mythor,W20130909-125655,graph1,dcamper,mythor,thor.thor"
      0000010B 2013-09-09 12:56:56.435 57824 57824 "Saving dll: /var/lib/HPCCSystems/queries/mythor/V1450585719_libW20130909-125655.so"
      0000010C 2013-09-09 12:56:56.435 57824 57824 "Started wuid=W20130909-125655, user=dcamper, graph=graph1
      "
      0000010D 2013-09-09 12:56:56.435 57824 57824 "Query /var/lib/HPCCSystems/queries/mythor/V1450585719_libW20130909-125655.so loaded"
      0000010E 2013-09-09 12:56:56.439 57824 57824 "CRC allocator OFF"
      0000010F 2013-09-09 12:56:56.439 57824 57824 "Packed allocator OFF"
      00000110 2013-09-09 12:56:56.440 57824 57824 "Global memory size = 24051 MB, memory spill at = 80%, large mem size = 18038 MB"
      00000111 2013-09-09 12:56:56.440 57824 57824 "maxActivityCores = [unbound]"
      00000112 2013-09-09 12:56:56.440 57824 57824 "Ecl plugin /opt/HPCCSystems/plugins//javaembed ignored"
      00000113 2013-09-09 12:56:56.539 57824 57824 "allocateMPTag: tag = 65537"
      00000114 2013-09-09 12:56:56.539 57824 57824 "allocateMPTag: tag = 65538"
      00000115 2013-09-09 12:56:56.539 57824 57824 "allocateMPTag: tag = 65539"
      00000116 2013-09-09 12:56:56.539 57824 57824 "allocateMPTag: tag = 65540"
      00000117 2013-09-09 12:56:56.539 57824 57824 "allocateMPTag: tag = 65541"
      00000118 2013-09-09 12:56:56.540 57824 57824 "Graph graph1 created"
      00000119 2013-09-09 12:56:56.540 57824 57824 "Running graph=graph1"
      0000011A 2013-09-09 12:56:56.545 57824 57824 "temp directory cleared"
      0000011B 2013-09-09 12:56:56.548 57824 57824 "Running graph [global] :   <graph>
         <node id="2" label="Inline Dataset">
          <att name="definition" value="C:\Users\lordgrey\AppData\Local\Temp\TFR2B61.tmp(13,18)"/>
          <att name="_kind" value="166"/>
          <att name="ecl" value="DATASET(10000, makedatarec());&#10;"/>
          <att name="recordSize" value="8"/>
          <att name="recordCount" value="10000..10000[few]"/>
         </node>
         <node id="3" label="Hash Distribute">
          <att name="definition" value="C:\Users\lordgrey\AppData\Local\Temp\TFR2B61.tmp(13,1)"/>
          <att name="name" value="d1"/>
          <att name="_kind" value="24"/>
          <att name="ecl" value="DISTRIBUTE(HASH32(i, internal));&#10;"/>
          <att name="recordSize" value="8"/>
          <att name="recordCount" value="10000..10000[few]"/>
         </node>
         <node id="4" label="Firstn">
          <att name="_kind" value="12"/>
          <att name="ecl" value="CHOOSEN(100);&#10;"/>
          <att name="recordSize" value="8"/>
          <att name="recordCount" value="100..100[group]"/>
         </node>
         <node id="5" label="Project">
          <att name="definition" value="C:\Users\lordgrey\AppData\Local\Temp\TFR2B61.tmp(15,1)"/>
          <att name="name" value="d2"/>
          <att name="_kind" value="7"/>
          <att name="ecl" value="PROJECT(TRANSFORM({ integer8 i },SELF.i := add1(LEFT.i);), local);&#10;"/>
          <att name="recordSize" value="8"/>
          <att name="recordCount" value="100..100[group]"/>
         </node>
         <node id="6" label="Output&#10;Result #1">
          <att name="definition" value="C:\Users\lordgrey\AppData\Local\Temp\TFR2B61.tmp(1,1)"/>
          <att name="name" value="tfr2b61"/>
          <att name="definition" value="C:\Users\lordgrey\AppData\Local\Temp\TFR2B61.tmp(26,1)"/>
          <att name="_kind" value="21"/>
          <att name="ecl" value="OUTPUT(..., workunit);&#10;"/>
          <att name="recordSize" value="8"/>
         </node>
         <att name="rootGraph" value="1"/>
         <edge id="2_0" source="2" target="3"/>
         <edge id="3_0" source="3" target="4"/>
         <edge id="4_0" source="4" target="5"/>
         <edge id="5_0" source="5" target="6"/>
        </graph>
       - graph(graph1, 1)"
      0000011C 2013-09-09 12:56:56.548 57824 57824 "CONNECTING (id=2, idx=0) to (id=3, idx=0) - activity(hashdistribute, 3)"
      0000011D 2013-09-09 12:56:56.548 57824 57824 "allocateMPTag: tag = 65542"
      0000011E 2013-09-09 12:56:56.548 57824 57824 "CONNECTING (id=3, idx=0) to (id=4, idx=0) - activity(firstn, 4)"
      0000011F 2013-09-09 12:56:56.548 57824 57824 "allocateMPTag: tag = 65543"
      00000120 2013-09-09 12:56:56.549 57824 57824 "CONNECTING (id=4, idx=0) to (id=5, idx=0) - activity(project, 5)"
      00000121 2013-09-09 12:56:56.549 57824 57824 "CONNECTING (id=5, idx=0) to (id=6, idx=0) - activity(workunitwrite, 6)"
      00000122 2013-09-09 12:56:56.549 57824 57824 "allocateMPTag: tag = 65544"
      00000123 2013-09-09 12:56:56.549 57824 57824 "Query dll: /var/lib/HPCCSystems/queries/mythor/V1450585719_libW20130909-125655.so"
      00000124 2013-09-09 12:56:56.549 57824 57824 "Loading query for serialization to slaves took 0 ms"
      00000125 2013-09-09 12:56:56.723 57824 57824 "Serialization of query init info (44533 bytes) to slaves took 167 ms"
      00000126 2013-09-09 12:56:56.732 57824 57824 ",Progress,Thor,StartSubgraph,mythor,W20130909-125655,1,1,mythor,thor.thor"
      00000127 2013-09-09 12:56:56.732 57824 57824 "allocateMPTag: tag = 65545"
      00000128 2013-09-09 12:56:56.738 57824 57824 "sendGraph took 6 ms - graph(graph1, 1)"
      00000129 2013-09-09 12:56:56.738 57824 57824 "Processing graph - graph(graph1, 1)"
      0000012A 2013-09-09 12:56:56.746 57824 58093 "HashDistributeActivityMaster::process - activity(hashdistribute, 3)"
      0000012B 2013-09-09 12:56:56.746 57824 58093 "HashDistributeActivityMaster::process exit - activity(hashdistribute, 3)"
      0000012C 2013-09-09 12:56:56.759 57824 57967 "ERROR: 0: Graph[1], project[5]: SLAVE 10.210.40.18:20100: javaembed: Failed to resolve class name JavaCat, "
      0000012D 2013-09-09 12:56:56.759 57824 57967 "INFORM [EXCEPTION]"
      0000012E 2013-09-09 12:56:56.759 57824 57967 "ERROR: 0: Graph[1], project[5]: SLAVE 10.210.40.18:20100: javaembed: Failed to resolve class name JavaCat, "
      0000012F 2013-09-09 12:56:56.760 57824 57967 "Posting exception: Graph[1], project[5]: SLAVE 10.210.40.18:20100: javaembed: Failed to resolve class name JavaCat,  to agent 10.210.40.10 for workunit(W20130909-125655)"
      00000130 2013-09-09 12:56:56.774 57824 57967 "INFORM [EXCEPTION]"
      00000131 2013-09-09 12:56:56.774 57824 57967 "Abort condition set - activity(workunitwrite, 6)"
      00000132 2013-09-09 12:56:56.775 57824 57967 "Abort condition set - activity(project, 5)"
      00000133 2013-09-09 12:56:56.775 57824 57967 "Abort condition set - activity(firstn, 4)"
      00000134 2013-09-09 12:56:56.775 57824 57967 "Abort condition set - activity(hashdistribute, 3)"
      00000135 2013-09-09 12:56:56.775 57824 57967 "Abort condition set - activity(inlinetable, 2)"
      00000136 2013-09-09 12:56:56.776 57824 57824 "WARNING: Graph wait cancelled, aborted=true - graph(graph1, 1)"
      00000137 2013-09-09 12:56:56.781 57824 57824 ",Timing,ThorGraph,mythor,W20130909-125655,1,1,1,57,SUCCESS,mythor,thor.thor"
      00000138 2013-09-09 12:56:56.783 57824 57824 "Graph Done - graph(graph1, 1)"
      00000139 2013-09-09 12:56:56.783 57824 57967 "ERROR: 0: Graph[1], SLAVE 10.210.40.18:20100: Graph[1], project[5]: javaembed: Failed to resolve class name JavaCat, "
      0000013A 2013-09-09 12:56:56.783 57824 57967 "INFORM [EXCEPTION]"
      0000013B 2013-09-09 12:56:56.784 57824 57967 "ERROR: 0: Graph[1], SLAVE 10.210.40.18:20100: Graph[1], project[5]: javaembed: Failed to resolve class name JavaCat, "
      0000013C 2013-09-09 12:56:56.784 57824 57967 "INFORM [EXCEPTION]"
      0000013D 2013-09-09 12:56:56.785 57824 57824 "PU=  0% MU=  2% MAL=3785959984 MMP=3784790016 SBK=1169968 TOT=3697304K RAM=1237732K SWP=812K - graph(graph1, 1)"
      0000013E 2013-09-09 12:56:56.786 57824 57824 "CLEARING TIMEOUT [EXCEPTION]"
      

      We've been having a great deal of trouble calling Java code from Thor processes, and I think this may be a decent example of our problems. In our case, however, we're seeing core dumps (provided upon request!).

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              dcamper Dan S. Camper
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: