Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-24548

HPCC Pods hanging with istio envoy (sidecar) running

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 7.6.x, 7.8.x, 7.10.x
    • 8.0.0
    • cloud
    • None

    Description

      Problem:

      Whenever Istio Envoy is enabled on HPCC-Systems clusters in the Google Cloud Platform it will create pods with two containers. One container is the process for the job, while the other process is for sidecar. Sidecar is essentially a logging mechanism. However, when the main process is finished, the sidecar process will continue to run, which means that the job will never finish.

      How to fix:

      Introduce a messaging/signaling mechanism to indicate to the job to terminate

      (I believe HPCC-Platform has the infrastructure that supports this, however, it only accepts one parameter, and adding another one will break the code)

      Downsides to this method:

      **As discussed in this this thread

      • Repetitive lifecycle logic must be rewritten in each instance a sidecar is deployed.
      • Third-party containers typically require a wrapper to add this behavior, normally provided via an entry point wrapper script implemented in the k8s container spec. This adds undesirable overhead and introduces repetition between the k8s and upstream container image specs.
      • The wrapping typically requires the presence of a shell in the container image, so this pattern does not work for minimal containers which ship without a toolchain.

      Evidence:

      **As depicted in the attachment, the pod "compile-w20200805-012943-kwdkc" is experiencing this issue. The job has already finished and therefore under the ready column it displays 1/2. This means only the sidecar container is running. This job will never finish.

      Attachments

        Issue Links

          Activity

            People

              jakesmith Jake Smith
              shroud Jeff Mao
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: