Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-16554

Query Alias referencing non existing query seg faults WsECL

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.4.6
    • Fix Version/s: 6.2.0
    • Component/s: Dali, Roxie, WS-ECL
    • Labels:
      None
    • Environment:
      Centos 5.9, 20 node roxie, dali and eclwatch are VMs

      Description

      Friday, we had a server fault. When the server was rebooted the 2 VMs for the dali and eclwatch were reboted as well (clean reboot).

      After that point, various issues were seen, as if queries on the roxie had jumped back in time (a few months) And the roxie showed duplicate queries with different version numbers.

      Queries were deleted and recompiled, and eventually this was fixed, but now the WsECL page doesn't display any active queries in the cluster. Eclwatch shows them as all active, ecl query list on the ROD shows them as active, and the roxie cluster itself shows no issues.

      in WsECL, the option "All Queries" shows all the queries, but "Active Queries Shows Nothing.

      Monitoring the logs while running WsECL, when the option to expect the roxie for active queries is clicked, the following appears.

      /var/log/HPCCSystems/esp_cert/esp.log

      00000228 2016-11-01 10:15:43.340 29567 29698 "HTTP First Line: GET /esp/navdata?queryset=roxie_cert128&AliasList HTTP/1.1"
      00000229 2016-11-01 10:15:43.340 29567 29698 "GET /esp/navdata, from dlucas@172.23.222.36"
      0000022A 2016-11-01 10:15:43.348 29567 29698 "================================================"
      0000022B 2016-11-01 10:15:43.352 29567 29698 "Signal: 11 Segmentation fault"
      0000022C 2016-11-01 10:15:43.352 29567 29698 "Fault IP: 00007FB20CC03E0F"
      0000022D 2016-11-01 10:15:43.352 29567 29698 "Accessing: 0000000000000000"
      0000022E 2016-11-01 10:15:43.352 29567 29698 "Registers:"
      0000022F 2016-11-01 10:15:43.352 29567 29698 "EAX:0000000000000000 EBX:0000000000000000 ECX:00007FB220000048 EDX:0000000000000000 ESI:0000000000000000 EDI:0000000000000000"
      00000230 2016-11-01 10:15:43.352 29567 29698 "CS:EIP:0033:00007FB20CC03E0F"
      00000231 2016-11-01 10:15:43.352 29567 29698 " ESP:00007FB208D1B930 EBP:00007FB22000EF86"
      00000232 2016-11-01 10:15:43.352 29567 29698 "Stack[00007FB208D1B930]: 0000000000B4F330 59E99BD100000000 00007FB259E99BD1 200118C000007FB2 00007FB2200118C0 0000002D00007FB2 000008000000002D 20010C1800000800"
      00000233 2016-11-01 10:15:43.352 29567 29698 "Stack[00007FB208D1B950]: 00007FB220010C18 200082D000007FB2 00007FB2200082D0 00B4F33000007FB2 0000000000B4F330 2000EF8600000000 00007FB22000EF86 2000751000007FB2"
      00000234 2016-11-01 10:15:43.352 29567 29698 "Stack[00007FB208D1B970]: 00007FB220007510 0CC07D8400007FB2 00007FB20CC07D84 2000831000007FB2 00007FB220008310 0000000000007FB2 0000000000000000 006206D000000000"
      00000235 2016-11-01 10:15:43.352 29567 29698 "Stack[00007FB208D1B990]: 00000000006206D0 59E9928C00000000 00007FB259E9928C 2000768000007FB2 00007FB220007680 08D1B9E000007FB2 00007FB208D1B9E0 0000003700007FB2"
      00000236 2016-11-01 10:15:43.352 29567 29698 "Stack[00007FB208D1B9B0]: 0000000000000037 200082D000000000 00007FB2200082D0 0000000000007FB2 0000000000000000 0000000000000000 0000000000000000 5BF4009700000000"
      00000237 2016-11-01 10:15:43.352 29567 29698 "Stack[00007FB208D1B9D0]: 00007FB25BF40097 0000000000007FB2 0000000000000000 2001324000000000 00007FB220013240 0000004600007FB2 0000008000000046 200112C000000080"
      00000238 2016-11-01 10:15:43.352 29567 29698 "Stack[00007FB208D1B9F0]: 00007FB2200112C0 5BF4009700007FB2 00007FB25BF40097 200085D800007FB2 00007FB2200085D8 0000000000007FB2 0000000000000000 2000751000000000"
      00000239 2016-11-01 10:15:43.352 29567 29698 "Stack[00007FB208D1BA10]: 00007FB220007510 00B70F6000007FB2 0000000000B70F60 0000000000000000 0000000000000000 200082D000000000 00007FB2200082D0 0000000100007FB2"
      0000023A 2016-11-01 10:15:43.352 29567 29698 "Backtrace:"
      0000023B 2016-11-01 10:15:43.353 29567 29698 " /opt/HPCCSystems/lib/libjlib.so(_Z16printStackReportv+0x32) [0x7fb259e49542]"
      0000023C 2016-11-01 10:15:43.353 29567 29698 " /opt/HPCCSystems/lib/libjlib.so(_Z13excsighandleriP7siginfoPv+0x9ca) [0x7fb259e4a26a]"
      0000023D 2016-11-01 10:15:43.353 29567 29698 " /usr/lib/jvm/jre/lib/amd64/server/libjvm.so(+0x89a6c4) [0x7fb2426586c4]"
      0000023E 2016-11-01 10:15:43.353 29567 29698 " /usr/lib/jvm/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x95) [0x7fb24265ef65]"
      0000023F 2016-11-01 10:15:43.353 29567 29698 " /usr/lib/jvm/jre/lib/amd64/server/libjvm.so(+0x897423) [0x7fb242655423]"
      00000240 2016-11-01 10:15:43.353 29567 29698 " /lib64/libpthread.so.0(+0xf7e0) [0x7fb2587237e0]"
      00000241 2016-11-01 10:15:43.353 29567 29698 " /opt/HPCCSystems/lib/libws_ecl.so(_ZN13CWsEclBinding13getQueryNamesEP13IPropertyTreePKcS3_R11StringArray+0x7f) [0x7fb20cc03e0f]"
      00000242 2016-11-01 10:15:43.353 29567 29698 " /opt/HPCCSystems/lib/libws_ecl.so(_ZN13CWsEclBinding13getDynNavDataER11IEspContextP11IPropertiesR13IPropertyTree+0x244) [0x7fb20cc07d84]"
      00000243 2016-11-01 10:15:43.353 29567 29698 " /opt/HPCCSystems/lib/libesphttp.so(_ZN19CEspApplicationPort13getDynNavDataER11IEspContextP11IPropertiesR12StringBufferS5_Rb+0x6e) [0x7fb25bf31dae]"
      00000244 2016-11-01 10:15:43.353 29567 29698 " /opt/HPCCSystems/lib/libesphttp.so(_ZN14CEspHttpServer15onGetDynNavDataEP12CHttpRequestP13CHttpResponse+0x7a) [0x7fb25bf01f2a]"
      00000245 2016-11-01 10:15:43.353 29567 29698 " /opt/HPCCSystems/lib/libesphttp.so(_ZN14CEspHttpServer14processRequestEv+0x1095) [0x7fb25bf048f5]"
      00000246 2016-11-01 10:15:43.353 29567 29698 " /opt/HPCCSystems/lib/libesphttp.so(_ZN11CHttpThread9onRequestEv+0x171) [0x7fb25beffa71]"
      00000247 2016-11-01 10:15:43.353 29567 29698 " /opt/HPCCSystems/lib/libesphttp.so(_ZN18CEspProtocolThread3runEv+0x21) [0x7fb25bf33ea1]"
      00000248 2016-11-01 10:15:43.353 29567 29698 " /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread5beginEv+0x2f) [0x7fb259ef1cbf]"
      00000249 2016-11-01 10:15:43.353 29567 29698 " /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread11_threadmainEPv+0x1c) [0x7fb259ef0b0c]"
      0000024A 2016-11-01 10:15:43.353 29567 29698 " /lib64/libpthread.so.0(+0x7aa1) [0x7fb25871baa1]"
      0000024B 2016-11-01 10:15:43.353 29567 29698 " /lib64/libc.so.6(clone+0x6d) [0x7fb25846893d]"
      0000024C 2016-11-01 10:15:43.353 29567 29698 "ThreadList:
      7FB255713700 140403914389248 29568: CMPNotifyClosedThread
      7FB254D12700 140403903899392 29569: CSocketBaseThread
      7FB254311700 140403893409536 29570: MP Connection Thread
      7FB253910700 140403882919680 29572: CMemoryUsageReporter
      7FB24AEB4700 140403737839360 29573: unknown
      7FB211EF9700 140402781820672 29582: unknown
      7FB210CF2700 140402762917632 29583: CDaliPublisherClient
      7FB20E3BB700 140402719700736 29584: unknown
      7FB20CBF0700 140402694752000 29585: unknown
      7FB20BF21700 140402681321216 29589: CSocketBaseThread
      7FB20AB1F700 140402660341504 29603: Member of thread pool: CDaliPublisherClientMessages
      7FB20B520700 140402670831360 29635: CEspProtocolThread
      7FB208D1C700 140402628871936 29698: CEspProtocolThread
      "
      00000002 2016-11-01 10:15:48.486 29706 29706 "Esp starting internal_5.4.6-1"

        Attachments

          Activity

            People

            • Assignee:
              afishbeck Anthony Fishbeck
              Reporter:
              lucasdj Dominic Lucas
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: