Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-21996

Performance monitoring can get stuck and cause parent process problems, if child process (lsblk) never finishes

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Scheduled
    • Priority: Not specified
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: dafilesrv
    • Labels:
      None

      Description

      Marking component dafilesrv, but I presume could happen in any process that starts performance monitor.

      This was the stack observed:

      Thread 1 (Thread 0x7f1238a47140 (LWP 84667)):
      #0  0x00007f12325506fd in read () from /lib64/libpthread.so.0
      #1  0x00007f1233e25fdc in CLinuxPipeProcess::read (this=0x22c7a00, sz=8192, buf=0x22ca1e0) at /mnt/disk1/jenkins/workspace/LN-with-Plugins-Spark-7.2.x-Nightly-Bu
      ild/LN/centos-7.0-x86_64/HPCC-Platform/system/jlib/jthread.cpp:2111
      #2  0x00007f1233d9d3b0 in readSimpleStream<StringBuffer> (target=..., stream=..., readChunkSize=readChunkSize@entry=8192) at /mnt/disk1/jenkins/workspace/LN-with
      -Plugins-Spark-7.2.x-Nightly-Build/LN/centos-7.0-x86_64/HPCC-Platform/system/jlib/jio.cpp:1459
      #3  0x00007f1233d70cf7 in OsDiskInfo::initMajorMinor (this=this@entry=0x22c79a0) at /mnt/disk1/jenkins/workspace/LN-with-Plugins-Spark-7.2.x-Nightly-Build/LN/cen
      tos-7.0-x86_64/HPCC-Platform/system/jlib/jdebug.cpp:1865
      #4  0x00007f1233d71042 in OsDiskInfo::OsDiskInfo (this=0x22c79a0) at /mnt/disk1/jenkins/workspace/LN-with-Plugins-Spark-7.2.x-Nightly-Build/LN/centos-7.0-x86_64/
      HPCC-Platform/system/jlib/jdebug.cpp:1813
      #5  0x00007f1233d7110d in operator() (__closure=<optimized out>) at /mnt/disk1/jenkins/workspace/LN-with-Plugins-Spark-7.2.x-Nightly-Build/LN/centos-7.0-x86_64/H
      PCC-Platform/system/jlib/jdebug.cpp:1917
      #6  querySingleton<OsDiskInfo, queryGlobalOsDiskInfo()::__lambda0> (singleton=..., cs=..., factory=...) at /mnt/disk1/jenkins/workspace/LN-with-Plugins-Spark-7.2
      .x-Nightly-Build/LN/centos-7.0-x86_64/HPCC-Platform/system/jlib/jmutex.hpp:979
      #7  query<queryGlobalOsDiskInfo()::__lambda0> (this=0x7f12340dbec0 <globalOsDiskInfo>, factory=...) at /mnt/disk1/jenkins/workspace/LN-with-Plugins-Spark-7.2.x-N
      ightly-Build/LN/centos-7.0-x86_64/HPCC-Platform/system/jlib/jmutex.hpp:999
      #8  queryGlobalOsDiskInfo () at /mnt/disk1/jenkins/workspace/LN-with-Plugins-Spark-7.2.x-Nightly-Build/LN/centos-7.0-x86_64/HPCC-Platform/system/jlib/jdebug.cpp:
      1917
      #9  0x00007f1233d7395f in CExtendedStats (printklog=true, this=0x22c7720) at /mnt/disk1/jenkins/workspace/LN-with-Plugins-Spark-7.2.x-Nightly-Build/LN/centos-7.0
      -x86_64/HPCC-Platform/system/jlib/jdebug.cpp:2274
      #10 CMemoryUsageReporter (printklog=true, _hook=0x7fffdb033c60, _traceMode=<optimized out>, _interval=600000, this=0x22c75f0) at /mnt/disk1/jenkins/workspace/LN-
      with-Plugins-Spark-7.2.x-Nightly-Build/LN/centos-7.0-x86_64/HPCC-Platform/system/jlib/jdebug.cpp:2633
      #11 startPerformanceMonitor (interval=600000, traceMode=<optimized out>, hook=0x7fffdb033c60) at /mnt/disk1/jenkins/workspace/LN-with-Plugins-Spark-7.2.x-Nightly
      -Build/LN/centos-7.0-x86_64/HPCC-Platform/system/jlib/jdebug.cpp:2953
      #12 0x0000000000403949 in main ()
      

      It should also be noted, that although the perf. monitor starts it's own thread to call the pipe process from, this stack indicates that it is creating a PIPE at construction time, normally not a major problem, but prob. would be better if delayed and ran from thread as it normally does.

        Attachments

          Activity

            People

            Assignee:
            mckellyln Mark Kelly
            Reporter:
            jakesmith Jake Smith
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated: