Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-26807

Inquiry History Build(Unexpected process termination)

    XMLWordPrintable

Details

    Description

      The builds on weekends have been failing with “unexpected process termination” ever since we upgraded to 7.12.58-1. The jobs seem to fail every time on graph8.

      We were on OSS 7.6.40-1 before the upgrade.

      http://alpha_prod_thor_esp.risk.regn.net:8010/#/stub/Main

      A ticket has been created for this - IOG-17084

      This is the most recent one W20211106-030029.

      I have attached the core file and we also have a backtrace on the slave.

      00044E9D 2021-11-07 07:30:05.950 1858457 1858457 "Aborting slave graph - graph(graph8, 2056) : Unexpected process termination (ep:10.194.112.31:20100)"
      00044E9E 2021-11-07 07:30:05.950 1858457 1858457 " - graph(graph8, 2056) : Unexpected process termination (ep:10.194.112.31:20100)"
      00044E9F 2021-11-07 07:30:05.950 1858457 1858457 " - graph(graph8, 2056) : Unexpected process termination (ep:10.194.112.31:20100)"
      00044EA0 2021-11-07 07:30:05.950 1858457 1858457 "Graph Done - graph(graph8, 2056)"
      00044EA1 2021-11-07 07:30:06.103 1858457 1781480 "INFORM [EXCEPTION]"
      00044EA2 2021-11-07 07:30:06.104 1858457 1858509 "ERROR: 10056: Watchdog has lost connectivity with Thor slave: 10.194.112.31:20100 (Process terminated or node down?)"
      00044EA3 2021-11-07 07:30:06.163 1858457 1858509 "INFORM [EXCEPTION]"
      00044EA4 2021-11-07 07:30:06.163 1858457 1858457 "ERROR: 4: Graph graph8[2074], spillread[2077]: Unexpected process termination (ep:10.194.112.31:20100), - caused by (4, Unexpected process termination (ep:10.194.112.31:20100))"
      00044EA5 2021-11-07 07:30:06.163 1858457 1858457 "INFORM [EXCEPTION]"
      00044EA6 2021-11-07 07:30:06.267 1858457 1858457 "ERROR: 4: thgraphmaster.cpp(1449) : broadcastSendAsync : Unexpected process termination (ep:10.194.112.31:20100)"
      00044EA7 2021-11-07 07:30:06.267 1858457 1858457 "ERROR: 4: thgraphmaster.cpp(1830) : Unexpected process termination (ep:10.194.112.31:20100)"
      00044EA8 2021-11-07 07:30:07.291 1858457 1858457 "CMPServer::flush(67659) discarded 21 buffers"
      00044EA9 2021-11-07 07:30:50.928 1858457 1858457 "Job ended : graph8"
      00044EAA 2021-11-07 07:30:50.928 1858457 1858457 "CGraphExecutor exiting, waiting on graph pool"
      00044EAB 2021-11-07 07:30:50.928 1858457 1858457 "CGraphExecutor graphPool finished"
      00044EAC 2021-11-07 07:30:50.931 1858457 1858457 "Unloading dll /opt/HPCCSystems/plugins//libstringlib.so"
      00044EAD 2021-11-07 07:30:50.932 1858457 1858457 "Unloading dll /opt/HPCCSystems/plugins//libtimelib.so"
      00044EAE 2021-11-07 07:30:50.932 1858457 1858457 "Unloading dll /opt/HPCCSystems/plugins//libfileservices.so"
      00044EAF 2021-11-07 07:30:50.941 1858457 1858457 ",Progress,Thor,Stop,thor400_112,W20211106-030029,graph8,InsDataops,thor400_112,thor400_112.thor,thor400_112_eclcc.thor"
      00044EB0 2021-11-07 07:30:50.944 1858457 1858457 "================================================"
      00044EB1 2021-11-07 07:30:50.944 1858457 1858457 "Program: 10.194.112.101:/mnt/disk1/HPCCSystems/bin/thormaster_lcr"
      00044EB2 2021-11-07 07:30:50.944 1858457 1858457 "Signal: 11 Segmentation fault"
      00044EB3 2021-11-07 07:30:50.944 1858457 1858457 "Fault IP: 00007F237C5EBEBA"
      00044EB4 2021-11-07 07:30:50.945 1858457 1858457 "Accessing: 0000000000000000"
      00044EB5 2021-11-07 07:30:50.945 1858457 1858457 "Backtrace:"
      00044EB6 2021-11-07 07:30:50.945 1858457 1858457 " /opt/HPCCSystems/lib/libgraphmaster_lcr.so(_ZN12CMasterGraphD1Ev+0x3a) [0x7f237c5ebeba]"
      00044EB7 2021-11-07 07:30:50.945 1858457 1858457 " /opt/HPCCSystems/lib/libgraphmaster_lcr.so(_ZN12CMasterGraphD0Ev+0x9) [0x7f237c5ebfd9]"
      00044EB8 2021-11-07 07:30:50.945 1858457 1858457 " /opt/HPCCSystems/lib/libjlib.so(_ZNK10CInterface7ReleaseEv+0x31) [0x7f23769c7811]"
      00044EB9 2021-11-07 07:30:50.945 1858457 1858457 " /mnt/disk1/var/lib/HPCCSystems/thor400_112/thormaster_thor400_112() [0x40bb1c]"
      00044EBA 2021-11-07 07:30:50.945 1858457 1858457 " /opt/HPCCSystems/lib/libjlib.so(_ZNK16CSimpleInterface7ReleaseEv+0x1a) [0x7f23769c77ca]"
      00044EBB 2021-11-07 07:30:50.945 1858457 1858457 " /mnt/disk1/var/lib/HPCCSystems/thor400_112/thormaster_thor400_112() [0x4138e9]"
      00044EBC 2021-11-07 07:30:50.945 1858457 1858457 " /mnt/disk1/var/lib/HPCCSystems/thor400_112/thormaster_thor400_112() [0x4139f9]"
      00044EBD 2021-11-07 07:30:50.945 1858457 1858457 " /opt/HPCCSystems/lib/libjlib.so(_ZNK16CSimpleInterface7ReleaseEv+0x1a) [0x7f23769c77ca]"
      00044EBE 2021-11-07 07:30:50.945 1858457 1858457 " /mnt/disk1/var/lib/HPCCSystems/thor400_112/thormaster_thor400_112() [0x418305]"
      00044EBF 2021-11-07 07:30:50.945 1858457 1858457 " /mnt/disk1/var/lib/HPCCSystems/thor400_112/thormaster_thor400_112() [0x4096d4]"
      00044EC0 2021-11-07 07:30:50.945 1858457 1858457 " /usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f2374b7d505]"
      00044EC1 2021-11-07 07:30:50.945 1858457 1858457 " /mnt/disk1/var/lib/HPCCSystems/thor400_112/thormaster_thor400_112() [0x40a230]"
      00044EC2 2021-11-07 07:30:50.945 1858457 1858457 "Registers:"
      00044EC3 2021-11-07 07:30:50.945 1858457 1858457 "EAX:00000000011B80F0 EBX:000000000944C360 ECX:0000000000C6C100 EDX:00007F237C80B870 ESI:0000000000010C42 EDI:00000000010DDF90"
      00044EC4 2021-11-07 07:30:50.945 1858457 1858457 "R8 :0000000000B52A90 R9 :0000000000C6C1E0 R10:0000000000000000 R11:00007F2374CE9610"
      00044EC5 2021-11-07 07:30:50.945 1858457 1858457 "R12:0000000000000008 R13:0000000000B19E48 R14:00000000006241D0 R15:0000000000D446A0"
      00044EC6 2021-11-07 07:30:50.945 1858457 1858457 "CS:EIP:0033:00007F237C5EBEBA"
      00044EC7 2021-11-07 07:30:50.945 1858457 1858457 " ESP:00007FFD8C3E6E40 EBP:000000000944C360"
      00044EC8 2021-11-07 07:30:50.945 1858457 1858457 "Stack[00007FFD8C3E6E40]: 000000000944C360 00B19DC000000000 0000000000B19DC0 0000000800000000 0000000000000008 7C5EBFD900000000 00007F237C5EBFD

      Attachments

        Issue Links

          Activity

            People

              jakesmith Jake Smith
              gravlebl Brenda Gravley
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: