Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-14540

Combine functionality of balanced and unbalanced splitter to allow it to use more memory and dynamically spill

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Timed Out
    • Affects Version/s: None
    • Fix Version/s: 8.0.0
    • Component/s: Thor
    • Labels:
      None
    • Compatibility:
      Minor

      Description

      The code generator currently flags splitter as balanced or unbalanced.
      If it is balanced, it uses a limited amount of memory and expects all consumers to keep up, if they fall behind the splitter blocks until they catchup to keep the amount it buffers to minimum.
      If the splitter is marked as unbalanced, it will start paging to disk if consumers fall behind.

      Change splitter so that it can use a growing amount of memory and allow memory pressure to cause it to start failing over to a disk based scheme.

        Attachments

          Activity

            People

            • Assignee:
              jakesmith Jake Smith
              Reporter:
              jakesmith Jake Smith
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: