Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-24501

Look at using the much faster LZ4 as a index key data compressor/decompressor

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Not specified
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: JHTree
    • Labels:
      None

      Description

      LZ4 is significantly faster than LZW, both to compress and decompress.
      Unittests added in HPCC-24470 give a ~7.5x decompression speed increase.

      Roxie, in particular, could benefit from this, as it spends a lot of time reading indexes, and hence bottlenecking on decompressing index key pages.

      We could enable this via a flag on build, but it would mean new keys with this new compression, would not be backward compatible, so for the foreseeable future we would not want to change the default.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jakesmith Jake Smith
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: