Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-24501

Look at using the much faster LZ4 as a index key data compressor/decompressor

    XMLWordPrintable

Details

    • Improvement
    • Status: Resolved
    • Not specified
    • Resolution: Duplicate
    • None
    • None
    • JHTree
    • None

    Description

      LZ4 is significantly faster than LZW, both to compress and decompress.
      Unittests added in HPCC-24470 give a ~7.5x decompression speed increase.

      Roxie, in particular, could benefit from this, as it spends a lot of time reading indexes, and hence bottlenecking on decompressing index key pages.

      We could enable this via a flag on build, but it would mean new keys with this new compression, would not be backward compatible, so for the foreseeable future we would not want to change the default.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jakesmith Jake Smith
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: