Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-27887

Produce new index format that can be compared compressed

    XMLWordPrintable

Details

    • Improvement
    • Status: Scheduled
    • Not specified
    • Resolution: Unresolved
    • None
    • None
    • JHTree
    • None

    Description

      Currently the page decompression takes quite a long time, and means that fewer pages can be cached in memory.  If they could be compared in place it may provide significant benefits.

      It would cause backward compatibility issues, and complicate the work of any other code that tries to directly read the index format.

       

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HPCC-28406 Publish uncompressed size and offset of branches for indexes

          • Not specified - Priority has not been specified
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HPCC-29159 Add an option to provide the default index compression format

          • Not specified - Priority has not been specified
          • Resolved
          1.
          		 Create a new index format for non-leaf nodes. Sub-task Resolved Gavin Halliday
          2.
          		 Introduce a inplace compression for leaf nodes Sub-task Resolved Unassigned
          3.
          		 Add stats for number of leaf/branch/blob nodes in an index write Sub-task Resolved Gavin Halliday
          4.
          		 Add number of leaf, blob and branch nodes to published meta Sub-task Resolved Unassigned
          5.
          		 add a function to calculate index sizes for a query/queryset Sub-task Accepted Anthony Fishbeck
          6.
          		 Investigate leaf compressing every N items and decompressing on demand Sub-task New Unassigned
          7.
          		 Integrate index cache with roxiemem Sub-task New Unassigned
          8.
          		 Extend index read stats to include disk time and #from page cache Sub-task Resolved Gavin Halliday
          9.
          		 Publish extended information for indexes in eclwatch Sub-task Resolved Gavin Halliday
          10.
          		 Default FILEPOSITION(FALSE) on indexes Sub-task New Unassigned
          11.
          		 Store the maximum row length in the header when building an index Sub-task Resolved Richard Chapman
          12.
          		 Display details of index on file information page Sub-task Resolved Jeremy Clements
          13.
          		 refactor in preparation for new index formats Sub-task Resolved Gavin Halliday
          14.
          		 Add original blob size details to file meta Sub-task Resolved Gavin Halliday
          15.
          		 Publish extended index information from Thor Sub-task Resolved Gavin Halliday
          16.
          		 Refactor to allow nodes to support additional compression methods Sub-task Resolved Richard Chapman
          17.
          		 Refactor to support nodes types that can search without decompressing Sub-task Resolved Richard Chapman
          18.
          		 Refactor jhtree class structure to support additional node types Sub-task Resolved Richard Chapman
          19.
          		 Add POC compression format to act as template for new index layouts Sub-task Resolved Richard Chapman
          20.
          		 Inplace compression improvements Sub-task New Unassigned
          21.
          		 inplace indexes - minor code optimizations Sub-task New Unassigned
          22.
          		 Support uncompressed payloads Sub-task Resolved Gavin Halliday
          23.
          		 Store the inline payload last Sub-task Resolved Gavin Halliday
          24.
          		 Extend dumpkey to support recoding/analysing existing indexes Sub-task Resolved Richard Chapman
          25.
          		 Fix inplace compression issues revealed by defaulting on Sub-task Resolved Gavin Halliday
          26.
          		 Minor refactoring of inplace index code Sub-task Resolved Gavin Halliday
          27.
          		 Support non-incremental compression schemes Sub-task Resolved Gavin Halliday
          28.
          		 Check that increment in payload size cannot cause internal consistency problems Sub-task New Unassigned
          29.
          		 Document the new compression format options Sub-task Resolved Jim DeFabia
          30.
          		 Reuse the compressor when building and index Sub-task New Unassigned
          31.
          		 Expand inplace payload on demand Sub-task New Unassigned
          32.
          		 Reuse temporary buffers when building an index Sub-task New Unassigned
          33.
          		 Improve the block-compressed serialization Sub-task New Unassigned
          34.
          		 Support other non-incremental compression Sub-task New Unassigned
          35.
          		 Optimize option with range of values Sub-task Resolved Gavin Halliday
          36.
          		 Avoid storing duplicate trailing keyed components Sub-task Resolved Gavin Halliday
          37.
          		 Create test cases for inplace indexes Sub-task Resolved Gavin Halliday
          38.
          		 Optimize TLK and fixed length file positions Sub-task Resolved Gavin Halliday
          39.
          		 Investigate dynamic payload lengths Sub-task New Unassigned
          40.
          		 Rename variable to make code clearer Sub-task Resolved Gavin Halliday
          41.
          		 Optimize the inplace keybuild code Sub-task Resolved Gavin Halliday
          42.
          		 Optimize lookup of a range option Sub-task New Unassigned
          43.
          		 Compress blobs using lz4 Sub-task New Unassigned
          44.
          		 Publish accurate in-memory index sizes Sub-task Resolved Gavin Halliday
          45.
          		 Avoid storing compressed data in memory Sub-task Resolved Gavin Halliday

          Activity

            People

              ghalliday Gavin Halliday
              ghalliday Gavin Halliday
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: