Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-19874

Filter activities are treated as 'fast through' but if sparse can cause downstream slowness due to lack of look aheads

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.4.22
    • Component/s: Thor
    • Labels:
      None

      Description

      Filters are marked as fast through at the moment, with the expectation they output records without delay, but if the filter doesn't match often, the output will be slow.
      FastThrough is used downstream to determine whether a lookahead is needed in various cases, e.g. by a CHOOSEN.
      In CHOOSEN's case, if there's no lookahead and the input count is slow and small it will mean there's a big delay before it even looks at the next part, etc., so the operation becomes a serial process, making it a lot slower than it would be if a lookahead had been involved and allowed the filtered read to be parallelized.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jakesmith Jake Smith
                Reporter:
                jakesmith Jake Smith
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: