Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-27887

Produce new index format that can be compared compressed

    XMLWordPrintable

Details

    • Improvement
    • Status: Scheduled
    • Not specified
    • Resolution: Unresolved
    • None
    • None
    • JHTree
    • None

    Description

      Currently the page decompression takes quite a long time, and means that fewer pages can be cached in memory.  If they could be compared in place it may provide significant benefits.

      It would cause backward compatibility issues, and complicate the work of any other code that tries to directly read the index format.

       

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HPCC-28406 Publish uncompressed size and offset of branches for indexes

          • Not specified - Priority has not been specified
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HPCC-29159 Add an option to provide the default index compression format

          • Not specified - Priority has not been specified
          • Resolved
          1.
          		 Create a new index format for non-leaf nodes. Sub-task Resolved Gavin Halliday
          2.
          		 Introduce a inplace compression for leaf nodes Sub-task New Unassigned
          3.
          		 Add stats for number of leaf/branch/blob nodes in an index write Sub-task Resolved Gavin Halliday
          4.
          		 Add number of leaf, blob and branch nodes to published meta Sub-task Resolved Unassigned
          5.
          		 add a function to calculate index sizes for a query/queryset Sub-task Accepted Anthony Fishbeck
          6.
          		 Investigate leaf compressing every N items and decompressing on demand Sub-task New Unassigned
          7.
          		 Integrate index cache with roxiemem Sub-task New Unassigned
          8.
          		 Extend index read stats to include disk time and #from page cache Sub-task Resolved Gavin Halliday
          9.
          		 Publish extended information for indexes in eclwatch Sub-task Resolved Gavin Halliday
          10.
          		 Default FILEPOSITION(FALSE) on indexes Sub-task New Unassigned
          11.
          		 Store the maximum row length in the header when building an index Sub-task Resolved Richard Chapman
          12.
          		 Display details of index on file information page Sub-task Resolved Jeremy Clements
          13.
          		 refactor in preparation for new index formats Sub-task Merge Pending Gavin Halliday
          14.
          		 Add original blob size details to file meta Sub-task Resolved Gavin Halliday
          15.
          		 Publish extended index information from Thor Sub-task Resolved Gavin Halliday
          16.
          		 Refactor to allow nodes to support additional compression methods Sub-task Resolved Richard Chapman
          17.
          		 Refactor to support nodes types that can search without decompressing Sub-task Resolved Richard Chapman
          18.
          		 Refactor jhtree class structure to support additional node types Sub-task Resolved Richard Chapman
          19.
          		 Add POC compression format to act as template for new index layouts Sub-task Resolved Richard Chapman
          20.
          		 Inplace compression improvements Sub-task New Unassigned
          21.
          		 inplace indexes - minor code optimizations Sub-task New Unassigned
          22.
          		 Support uncompressed payloads Sub-task New Unassigned
          23.
          		 Store the inline payload last Sub-task New Unassigned
          24.
          		 Extend dumpkey to support recoding/analysing existing indexes Sub-task Resolved Richard Chapman
          25.
          		 Fix inplace compression issues revealed by defaulting on Sub-task Resolved Gavin Halliday
          26.
          		 Minor refactoring of inplace index code Sub-task New Unassigned

          Activity

            People

              ghalliday Gavin Halliday
              ghalliday Gavin Halliday
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: