Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-28524

Index write should not use a spilling lookahead stream unconditionally

    XMLWordPrintable

Details

    • Improvement
    • Status: Resolved
    • Not specified
    • Resolution: Fixed
    • None
    • 8.10.6
    • Thor
    • None

    Description

      Currently, I believe unintentionally, the index write employs a lookahead stream with spilling enabled unconditionally.
      Other sinks conditionally base whether it can spill or not on ::canStall(input)

      This is potentially going to mean that the index write uses excessibe unwanted spilling.
      It will only do so, if the indexwrite output (write to disk) doesn't keep up with the streamed (merged) output of the sort.
      This may not have been noticed particularly before because:
      1) local [bare-metal] disks were ample big enough to cope with this unwanted extra spilling.
      2) The write to [local bare-metal] disk, was fast enough (and faster than the merging sort stream) to avoid the needed for the lookahead stream to start spilling.

      This issue was noticed when running on a K8s system, where the index write output (writes) may have been slower than the local sort merger stream.

      Attachments

        Activity

          People

            jakesmith Jake Smith
            jakesmith Jake Smith
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: