Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-16653

JBuff: Exceeded maximum size creating ZAP file.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0.0
    • Component/s: ESP
    • Labels:
      None

      Description

      Trying to create a ZAP file that includes slave logs hit this error:

      Exception(s) occurred:
      Reporter: WsWorkunits::WUCREATEZAPINFO()
      Code Message
      -10 2016-11-23 17:33:56 GMT: Jbuff: Exceeded maximum size (4294747905,256091)

      Looks like the code, reads everything from the slave log into a memory buffer, before then writing it to a file.
      Here is the stack trace in the esp log:

      0004B284 2016-11-23 12:33:56.521 30645 18154 "Jbuff: Exceeded maximum size (-10,4294747905,250k)"
      0004B285 2016-11-23 12:33:56.521 30645 18154 "Backtrace:"
      0004B286 2016-11-23 12:33:56.583 30645 18447 "TxSummary[activeReqs=9;user=akayttala@10.145.175.67;req=SOAP wssmc.Activity v1.12;total=435ms;]"
      0004B287 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libjlib.so(+0xe3868) [0x7fcdf89d9868]"
      0004B288 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libjlib.so(_Z23createOutOfMemExceptionimmbPKc+0xcd) [0x7fcdf89c515d]"
      0004B289 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libjlib.so(_Z22RaiseOutOfMemExceptionimmbPKc+0x3f) [0x7fcdf89c522f]"
      0004B28A 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libjlib.so(_ZN12MemoryBuffer6appendEjPKv+0x3c) [0x7fcdf89c5d6c]"
      0004B28B 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libws_workunits.so(_ZN12ws_workunits8WsWuInfo23getWorkunitThorSlaveLogEPKcS2_S2_S2_iR12MemoryBufferb+0x376) [0x7fcda4984776]"
      0004B28C 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libws_workunits.so(_ZN14CWsWorkunitsEx19addThorSlaveLogfileER5OwnedI14IConstWorkUnitERN12ws_workunits8WsWuInfoEPKc+0x4d4) [0x7fcda49711c4]"
      0004B28D 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libws_workunits.so(_ZN14CWsWorkunitsEx17onWUCreateZAPInfoER11IEspContextR26IEspWUCreateZAPInfoRequestR27IEspWUCreateZAPInfoResponse+0x412) [0x7fcda4972812]"
      0004B28E 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libws_workunits.so(_ZN12ws_workunits23CWsWorkunitsSoapBinding17onGetInstantQueryER11IEspContextP12CHttpRequestP13CHttpResponsePKcS8_+0x3595) [0x7fcda47f09f9]"
      0004B28F 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libesphttp.so(_ZN14EspHttpBinding5onGetEP12CHttpRequestP13CHttpResponse+0x1d8) [0x7fcdfaa9c968]"
      0004B290 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libws_workunits.so(_ZN25CWsWorkunitsSoapBindingEx5onGetEP12CHttpRequestP13CHttpResponse+0x6d7) [0x7fcda4991d37]"
      0004B291 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libesphttp.so(_ZN14CEspHttpServer14processRequestEv+0x478) [0x7fcdfaaa62b8]"
      0004B292 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libesphttp.so(_ZN11CHttpThread9onRequestEv+0x154) [0x7fcdfaaa1d64]"
      0004B293 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libesphttp.so(_ZN18CEspProtocolThread3runEv+0x21) [0x7fcdfaad2491]"
      0004B294 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread5beginEv+0x2c) [0x7fcdf8a7f8fc]"
      0004B295 2016-11-23 12:33:56.583 30645 18154 "  /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread11_threadmainEPv+0x1e) [0x7fcdf8a8129e]"
      0004B296 2016-11-23 12:33:56.583 30645 18154 "  /lib64/libpthread.so.0(+0x7aa1) [0x7fcdf72adaa1]"
      0004B297 2016-11-23 12:33:56.583 30645 18154 "  /lib64/libc.so.6(clone+0x6d) [0x7fcdf6ffaaad]"
      

      The code should be changed, so that it does not use significant memory and instead streams to disk.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wangkx Kevin Wang
                Reporter:
                jakesmith Jake Smith
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: