Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-12409

Distribute can suffer severe slow down if producer producing just ahead of fast consumer

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 5.0.4
    • Thor
    • None

    Description

      The distribute send side, when full, queues the largest partial buckets if there is some room. If the produce keeps pace or is just ahead of the consumer, the net effect is to create ever increasingly small partial buckets to queue. In the worst case it degrades to having buckets with a single row to send, with severe performance consequences.

      The sender side should not have been queuing partial buckets to a particular target just so it filled potential space in the send buffer, if there was already a write handler actively sending to that target. Restricting the queued candidates to writers that are inactive, prevents the pathological above.

      Attachments

        Activity

          People

            jakesmith Jake Smith
            jakesmith Jake Smith
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: