Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-27887 Produce new index format that can be compared compressed
  3. HPCC-28469

Investigate leaf compressing every N items and decompressing on demand

    XMLWordPrintable

Details

    • Sub-task
    • Status: New
    • Not specified
    • Resolution: Unresolved
    • None
    • None
    • JHTree
    • None

    Description

      There are many different scenarios to investigate

      • indexes with very little payload (e.g. just a blobid/filepos).
        Likely to be worth storing uncompressed. What features in the keyed compression help?
      • changing the leaf compression so N records at a time are compressed.
        Decompressing on demand, and only when the record is known to be needed. Will require some refactoring in jhtree, but probably not too painful. Would want to cache decompressed blocks but not impose too large an overhead. How does N affect the compression ratio?
      • Compressing with a maximum size decompressed.
        May help with fragmentation, especially if roxiemem is used for the temporary

      Attachments

        Activity

          People

            Unassigned Unassigned
            ghalliday Gavin Halliday
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: