Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-24643

dali pod is getting CrashLoopBackOff

    XMLWordPrintable

Details

    • Bug
    • Status: Accepted
    • Major
    • Resolution: Unresolved
    • 7.10.12
    • None
    • Azure, Dali
    • None
    • aks
    • Point

    Description

      This is related to HPCC-24641. I upgraded the helm chart of hpcc. I used the following command:

      helm upgrade tlh-cluster hpcc/hpcc --set global.image.version=latest -f examples/azure/values-data-default.yaml
      

      where "examples/azure/values-data-default.yaml" contains the following:

      # Overrides for azure storage in hpcc.
      # NB: Using the default "azurefile" storage class results in automatically
      # provisioned Persistent Volume's with a reclaimPolicy of "Delete"
      
      storage:
        dllStorage:
          existingClaim: "azure-default-dll-disk"
      
        daliStorage:
          existingClaim: "azure-default-dali-disk"
      
        dataStorage:
          existingClaim: "azure-default-disk"
      

      In HPCC-24641, I had "existingClaim" only on dataStorage and the other 2 I didn't change. But, I was getting an error when I ran ecl (see HPCC-24641), so I changed "values-data-default.yaml" to the above. Now, dali crashes. I did the following command to get the contents of the dali log:

      kubectl logs -f mydali-7d9699df6f-mh29t
      

      Here is the contents of the log:

      00000000 USR 2020-08-20 14:33:41.667     1     1 "Build community_7.10.14-1Debug[community_7.10.14-1]"
      00000001 PRG 2020-08-20 14:33:41.669     1     1 "Server Version = 3.16, required minimum client version 1.5"
      00000002 USR 2020-08-20 14:33:41.670     1     1 "ERROR: 13: daserver.cpp(713) : Exception : CFile::open /var/lib/HPCCSystems/dalistorage/dalicoven.xml, Permission denied"
      00000003 USR 2020-08-20 14:33:41.671     1     9 "================================================"
      00000004 USR 2020-08-20 14:33:41.672     1     9 "Program:   10.244.0.58:/opt/HPCCSystems/bin/daserver"
      00000005 USR 2020-08-20 14:33:41.672     1     9 "Signal:    11 Segmentation fault"
      00000006 USR 2020-08-20 14:33:41.673     1     9 "Fault IP:  0000000000000000"
      00000007 USR 2020-08-20 14:33:41.673     1     9 "Accessing: 0000000000000000"
      00000008 PRG 2020-08-20 14:33:41.674     1     9 "Backtrace:"
      00000009 PRG 2020-08-20 14:33:41.675     1     9 "  /opt/HPCCSystems/lib/libjlib.so(_Z16printStackReportx+0x5e) [0x7f163beb7c2d]"
      0000000A PRG 2020-08-20 14:33:41.676     1     9 "  /opt/HPCCSystems/lib/libjlib.so(_Z13excsighandleriP9siginfo_tPv+0x3c6) [0x7f163beb6dc2]"
      0000000B PRG 2020-08-20 14:33:41.677     1     9 "  /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f163b19b3c0]"
      0000000C USR 2020-08-20 14:33:41.678     1     9 "Registers:"
      0000000D USR 2020-08-20 14:33:41.679     1     9 "EAX:00007F1630000B60  EBX:0000000000000000  ECX:000000000000006E  EDX:0000000000000000  ESI:00005601F6A4B680  EDI:00007F1630000B60"
      0000000E USR 2020-08-20 14:33:41.680     1     9 "R8 :00007F1636B9D2C8  R9 :00000000FFFFFFFF  R10:0000000000000000  R11:0000000000000246"
      0000000F USR 2020-08-20 14:33:41.680     1     9 "R12:00007FFCABAACFDE  R13:00007FFCABAACFDF  R14:00007FFCABAAD090  R15:00007F1636B9D4C0"
      00000010 PRG 2020-08-20 14:33:41.681     1     1 "Backtrace:"
      00000011 PRG 2020-08-20 14:33:41.684     1     1 "  /opt/HPCCSystems/lib/libjlib.so(_Z16printStackReportx+0x5e) [0x7f163beb7c2d]"
      00000012 PRG 2020-08-20 14:33:41.684     1     1 "  /opt/HPCCSystems/lib/libjlib.so(_Z20raiseAssertExceptionPKcS0_j+0xc9) [0x7f163beb67b5]"
      00000013 PRG 2020-08-20 14:33:41.685     1     1 "  /opt/HPCCSystems/lib/libLdapSecurity.so(_ZN13ReadWriteLockD2Ev+0x8e) [0x7f163c61450e]"
      00000014 PRG 2020-08-20 14:33:41.686     1     1 "  /opt/HPCCSystems/lib/libjlib.so(_ZN14CLogMsgManagerD1Ev+0x270) [0x7f163bf12c9c]"
      00000015 PRG 2020-08-20 14:33:41.686     1     1 "  /opt/HPCCSystems/lib/libjlib.so(_ZN14CLogMsgManagerD0Ev+0x1c) [0x7f163bf12d6a]"
      00000016 PRG 2020-08-20 14:33:41.689     1     1 "  /opt/HPCCSystems/lib/libjlib.so(+0x1ffb81) [0x7f163bf1ab81]"
      00000017 PRG 2020-08-20 14:33:41.691     1     1 "  /opt/HPCCSystems/lib/libjlib.so(_ZN9InitTable4exitEPv+0x174) [0x7f163bf03eec]"
      00000018 PRG 2020-08-20 14:33:41.692     1     1 "  /opt/HPCCSystems/lib/libjlib.so(_Z17ExitModuleObjectsPv+0x73) [0x7f163bf03946]"
      00000019 PRG 2020-08-20 14:33:41.693     1     1 "  /opt/HPCCSystems/lib/libjlib.so(_Z17ExitModuleObjectsv+0x12) [0x7f163bf03980]"
      0000001A PRG 2020-08-20 14:33:41.693     1     1 "  /lib/x86_64-linux-gnu/libc.so.6(+0x49a27) [0x7f163afdda27]"
      0000001B PRG 2020-08-20 14:33:41.702     1     1 "  /lib/x86_64-linux-gnu/libc.so.6(on_exit+0) [0x7f163afddbe0]"
      0000001C PRG 2020-08-20 14:33:41.703     1     1 "  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfa) [0x7f163afbb0ba]"
      0000001D PRG 2020-08-20 14:33:41.704     1     1 "  daserver(+0x100ce) [0x5601f4d0b0ce]"
      

      Attachments

        Activity

          People

            jakesmith Jake Smith
            tlhumphrey2 Timothy Humphrey
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: