Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-16694

Delay spilling inter-subgraph spills

    Details

    • Type: New Feature
    • Status: Scheduled
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 7.2.0
    • Component/s: Code Generator, Roxie, Thor
    • Labels:
      None

      Description

      Currently datasets that are shared between subgraphs are spilled to disk in Thor. However this can be very inefficient if the Thor is not running low on memory since it involves extra desrialize/compress/diskwrite/diskread/decompress stages that could be avoided.

      The feature could be implemented within Thor using the current helpers, but it would be inefficient if the disk read activity projects the input because the rows will need to be serialized before the transform can be called.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jakesmith Jake Smith
                Reporter:
                ghalliday Gavin Halliday
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: