Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-12546

IExpander should allow incremental expansion expand(buffer, maxsize)

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Timed Out
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: JLib
    • Labels:

      Description

      Currently the IExpander can only expand a complete block at a time. This isn't very good for decompressing rows since it means the whole block of rows must be decompressed before any rows can be created.

      That uses a larger block of memory when smaller blocks could do.
      It will lead to large blocks being reallocated on the heap.

      You could either allow expansion to be staged - e.g., expand x bytes. Or you could have a new kind of row deserializer which was fed the bytes as they were generated, and used them to create rows.

      Expanding in stages is probably less efficient than complete expansion (because internal state would need to be saved), but is likely to be worth it if the number of bytes expanded each time is still reasonably large so the overhead is minimized.

      A new deserializer would be a bit tricky (but interesting), and would probably need to receive blocks of bytes to be relatively efficient. State would need to be saved in the deserializer. On reflection though, it may have little benefit over the partial expansion scheme - which would also have the advantage of being able to simply create rows on demand, so I would go with the simpler partial expansion implementation.

      Jake Smith a good idea?

        Attachments

          Activity

            People

            • Assignee:
              anybody Available for anyone
              Reporter:
              ghalliday Gavin Halliday
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: