Details
-
Improvement
-
Status: Resolved
-
Not specified
-
Resolution: Fixed
-
None
-
None
Description
Currently, I believe unintentionally, the index write employs a lookahead stream with spilling enabled unconditionally.
Other sinks conditionally base whether it can spill or not on ::canStall(input)
This is potentially going to mean that the index write uses excessibe unwanted spilling.
It will only do so, if the indexwrite output (write to disk) doesn't keep up with the streamed (merged) output of the sort.
This may not have been noticed particularly before because:
1) local [bare-metal] disks were ample big enough to cope with this unwanted extra spilling.
2) The write to [local bare-metal] disk, was fast enough (and faster than the merging sort stream) to avoid the needed for the lookahead stream to start spilling.
This issue was noticed when running on a K8s system, where the index write output (writes) may have been slower than the local sort merger stream.