Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-15695

Smart lookup, fix a few bugs + add a few optimizations

    XMLWordPrintable

    Details

    • Type: Regression
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.0.2
    • Component/s: Thor
    • Labels:
      None

      Description

      The default spill cost was uninitialized since it's introduction (5.0.10), which meant spurious low values could cause OOM in smart join, which should have instead caused spilling.

      Other fixes:
      1) Keep rhs rows spillable until after fully prepared HT
      Could cause spurious OOM
      2) Protect derived addRHSRow(s) from channel threads
      Multiple threads could have both triggered overflowFile
      3) Ensure overflow non-local rows re. pruned
      4) Remove shrink() code from collector::spillRows()
      Could cause RoxieMem Invalid pointer on rows
      5) Initialize defaultMaxSpillCost
      Could mean spill callbacks were not called

      Optimizations:
      1) Move broadcast send to others crit around send only
      2) Batch up addRHSRows to avoid some contention
      3) Ensure rightAllocator is unique per channel

        Attachments

          Activity

            People

            • Assignee:
              jakesmith Jake Smith
              Reporter:
              jakesmith Jake Smith
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: