Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-14442

Extend memory manager to allow shared global memory


    • Type: Bug
    • Status: Resolved
    • Priority: Not specified
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.0.0
    • Component/s: Memory Manager
    • Labels:


      Thor now supports multiple slaves within a single process.
      To efficiently implement the lookup join optimizations it needs to be able to allocate rows from a "global" memory space, which effectively reduces the memory available to each of the slave processes.

      In particular we want the following characteristics:

      Some issues we are trying to solve:

      1. If we spill some rows, then we want to spill rows that have the lowest impact (/cost) for the query.
      2. Some operation want to share memory between all the virtual slaves in a process (e.g., Lookup Join)
      3. We would like some dynamism between slaves to allow them to cope with some skew.
      4. We would also like to support some dynamism between thors. Ideally this would also be under some user control and allow higher "skew" than we would allow between slaves. (E.g., normal, max)
      5. Priority of callbacks tend to change over time (e.g., lower cost once sorted.)
      6. Number of callbacks will increase as more spills to disk are delayed.
      7. One virtual slave should not cause other virtual slaves to starve (from assumption B)
      8. Virtual slaves need to coordinate which rows should be spilled (from assumption C)
      9. Delay spill to disk (spills created as activities as the resourcing in eclcc)
      10. Avoid contention on allocators between threads.




            • Assignee:
              ghalliday Gavin Halliday
              ghalliday Gavin Halliday
            • Votes:
              0 Vote for this issue
              2 Start watching this issue


              • Created: