Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-19201

Spark remote read field pruning

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0.0
    • Component/s: Embedded Languages
    • Labels:
      None

      Description

      The current implementation uses the definition for reading the data as the output definition.  The output definition should be pruned to include only the fields requested by the Spark application.  NOTE: the field list must be supplied to the HpccFile getRDD(...) method so that the dependency DAG can be constructed properly by Spark. 

        Attachments

          Activity

            People

            • Assignee:
              rpastrana Rodrigo Pastrana
              Reporter:
              johnholt John Holt
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: