Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-15251

Potential sort improvements

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Unresourced
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: JLib, Memory Manager, Thor
    • Labels:
      None

      Description

      A summary of some ideas that came up during the offsite:

      • Memory manager should have an option to allocate twice the virtual address space. Half would not be used, but it should help avoid fragmentation problems. (There may still be an issue with elements from multiple os (huge) pages being allocated, and that may complicate the issue)
      • The stable sort should reserve memory for the extra sort array, but not allocate a pointer for it.
      • Once allocations have got above a certain size (e.g., a huge page), should allocations be rounded up to a multiple (and preferably aligned on a huge page boundary)
      • Add a function to allocare a pointer passing in memory that has previously been reserved.
      • Could read rows 1 page at a time, and 1st stage of processing a full sort:
      • allocate extra
      • clone to extra
      • free all source blocks, and allocate a big block (atomically)
      • sort.
      • Merge sort could
      • sort each block as it is read (which could be executed in parallel)
      • potentially merge sort from blocks to a combine block as a 1st stage.
      • spill could perform a merge from the sorted subblocks rather than sorting the entire dataset first. It may well save time since the processing will overlap with spilling.

      Jake Smith Richard Chapman Mark Kelly for discussion.

        Attachments

          Activity

            People

            • Assignee:
              ghalliday Gavin Halliday
              Reporter:
              ghalliday Gavin Halliday
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: