Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-18689

segfault possibly due to mismatched ESPs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.2.22
    • Fix Version/s: 6.4.6
    • Component/s: ESP
    • Labels:
      None
    • Environment:
      Dataland Staging

      Description

       

      Core

      10.241.12.204:/var/lib/HPCCSystems/staging_esp/core.10721

       

      Log

       

      10.241.12.204:/var/log/HPCCSystems/staging_esp/esp.log
      00259D6E 2017-11-06 13:56:20.335 10721 54106 "Signal:    11 Segmentation fault"
      00259D6F 2017-11-06 13:56:20.335 10721 54106 "Fault IP:  000000000040E636"
      00259D70 2017-11-06 13:56:20.335 10721 54106 "Accessing: 0000000000000028"
      00259D71 2017-11-06 13:56:20.335 10721 54106 "Registers:"
      00259D72 2017-11-06 13:56:20.335 10721 54106 "EAX:0000000000000000  EBX:00007F9C9E9AAD70  ECX:0000000000000009  EDX:0000000000000009  ESI:00007F9C9E9AAD70  EDI:00007F9C9E9AAD70"
      00259D73 2017-11-06 13:56:20.335 10721 54106 "CS:EIP:0033:000000000040E636"
      00259D74 2017-11-06 13:56:20.335 10721 54106 "   ESP:00007F9C04FFB408  EBP:0000000000F223B8"
      00259D75 2017-11-06 13:56:20.335 10721 54106 "Stack[00007F9C04FFB408]: 00007F9D0A67AF7D 4D67B3B300007F9D 00007F9C4D67B3B3 00F223B800007F9C 0000000000F223B8 9E9AAD7000000000 00007F9C9E9AAD70 0A67B7F600007F9C"
      00259D76 2017-11-06 13:56:20.335 10721 54106 "Stack[00007F9C04FFB428]: 00007F9D0A67B7F6 0000000700007F9D 0000000000000007 00F223B800000000 0000000000F223B8 9F0CF18000000000 00007F9C9F0CF180 0C6B6A8000007F9C"
      00259D77 2017-11-06 13:56:20.335 10721 54106 "Stack[00007F9C04FFB448]: 00007F9D0C6B6A80 C729555000007F9D 00007F9CC7295550 04FFB4A000007F9C 00007F9C04FFB4A0 C7568F4000007F9C 00007F9CC7568F40 C729556700007F9C"
      00259D78 2017-11-06 13:56:20.335 10721 54106 "Stack[00007F9C04FFB468]: 00000000C7295567 C639C87000000000 00007F9CC639C870 0000000100007F9C 0000000800000001 0C8DA33000000008 00007F9D0C8DA330 0000000100007F9D"
      00259D79 2017-11-06 13:56:20.335 10721 54106 "Stack[00007F9C04FFB488]: 0000000100000001 00F223B800000001 0000000000F223B8 9E9AAD7000000000 00007F9C9E9AAD70 0000000700007F9C 0000000000000007 0000000000000000"
      00259D7A 2017-11-06 13:56:20.335 10721 54106 "Stack[00007F9C04FFB4A8]: 0000000000000000 0000006E00000000 000000000000006E 0000000000000000 0000000000000000 04FF000000000000 00007F9C04FF0000 0000000000007F9C"
      00259D7B 2017-11-06 13:56:20.335 10721 54106 "Stack[00007F9C04FFB4C8]: 0000000000000000 C651E4B000000000 00007F9CC651E4B0 C41B507000007F9C 00007F9CC41B5070 0000007C00007F9C 00007F9C0000007C 0000000000007F9C"
      00259D7C 2017-11-06 13:56:20.335 10721 54106 "Stack[00007F9C04FFB4E8]: 0000000000000000 4D66235100000000 00007F9C4D662351 0C6A339C00007F9C 00007F9D0C6A339C FFFFFFFF00007F9D 00000006FFFFFFFF C651E4B000000006"
      00259D7D 2017-11-06 13:56:20.335 10721 54106 "Backtrace:"
      00259D7E 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libjlib.so(+0xcf048) [0x7f9d0a5dd048]"
      00259D7F 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libjlib.so(_Z13excsighandleriP9siginfo_tPv+0x33c) [0x7f9d0a5decbc]"
      00259D80 2017-11-06 13:56:20.337 10721 54106 "  /usr/lib/jvm/jre/lib/amd64/server/libjvm.so(+0x7fd3eb) [0x7f9ce67f03eb]"
      00259D81 2017-11-06 13:56:20.337 10721 54106 "  /usr/lib/jvm/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x1a1) [0x7f9ce67f4f71]"
      00259D82 2017-11-06 13:56:20.337 10721 54106 "  /usr/lib64/libpthread.so.0(+0xf370) [0x7f9d08e45370]"
      00259D83 2017-11-06 13:56:20.337 10721 54106 "  esp(_ZNK9HashTable18getHashFromElementEPKv+0x6) [0x40e636]"
      00259D84 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libjlib.so(_ZNK14SuperHashTable11doFindExactEPKv+0x2d) [0x7f9d0a67af7d]"
      00259D85 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libjlib.so(_ZNK14SuperHashTable4nextEPKv+0x16) [0x7f9d0a67b7f6]"
      00259D86 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libesphttp.so(_ZN10SecHandler25validateSecFeaturesAccessER11MapStringToI14SecAccessFlagsS1_15MappingStringToIS1_S1_EEb+0x130) [0x7f9d0c6b6a80]"
      00259D87 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libesphttp.so(ZN14EspHttpBinding19onFeaturesAuthorizeER11IEspContextR11MapStringToI14SecAccessFlagsS3_15MappingStringToIS3_S3_EEPKcS9+0x2b) [0x7f9d0c67a94b]"
      00259D88 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libws_workunits.so(ZN12ws_workunits23CWsWorkunitsSoapBinding14processRequestEP11IRpcMessageS2+0xa6dc) [0x7f9c4d48a2d6]"
      00259D89 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libesphttp.so(ZN12CSoapService14processRequestER12ISoapMessageS1+0x7cf) [0x7f9d0c6ab80f]"
      00259D8A 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libesphttp.so(_ZN16CHttpSoapBinding17HandleSoapRequestEP12CHttpRequestP13CHttpResponse+0x290) [0x7f9d0c69fe00]"
      00259D8B 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libesphttp.so(_ZN16CHttpSoapBinding13onSoapRequestEP12CHttpRequestP13CHttpResponse+0x20) [0x7f9d0c6a0df0]"
      00259D8C 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libesphttp.so(_ZN14CEspHttpServer14processRequestEv+0x8df) [0x7f9d0c6883bf]"
      00259D8D 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libesphttp.so(_ZN11CHttpThread9onRequestEv+0x154) [0x7f9d0c683c24]"
      00259D8E 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libesphttp.so(_ZN18CEspProtocolThread3runEv+0x21) [0x7f9d0c6b4eb1]"
      00259D8F 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread5beginEv+0x2c) [0x7f9d0a67c22c]"
      00259D90 2017-11-06 13:56:20.337 10721 54106 "  /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread11_threadmainEPv+0x1e) [0x7f9d0a67dbfe]"
      00259D91 2017-11-06 13:56:20.337 10721 54106 "  /usr/lib64/libpthread.so.0(+0x7dc5) [0x7f9d08e3ddc5]"
      00259D92 2017-11-06 13:56:20.337 10721 54106 "  /usr/lib64/libc.so.6(clone+0x6d) [0x7f9d08b6c73d]"
       
      

       

      Kevin Wang mentioned:

      ESP crashed when authorizing WU feature access for: SOAP method <WUResult> from vchikte_prod@172.25.241.220

       

      Tony Kirk mentioned:

      I see Vani has workunits on Dev (6.4.4-1) and Sta clusters (6.2.22-1).  Could there be some interface conflict because of mixed environment, her hitting Staging ESP for Dev workunits from 6.4.x IDE, etc?

       

      Rodrigo Pastrana mentioned:

      This may be caused by the mismatch in ESPs (as Tony mentioned), but regardless we need to look deeper and prevent this issue in the future.

        Attachments

          Activity

            People

            • Assignee:
              rpastrana Rodrigo Pastrana
              Reporter:
              rwagner42 Russell Wagner
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: