Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-19374

Spark use of Keys, keyed lookups

    Details

    • Type: Improvement
    • Status: Scheduled
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Spark-HPCC
    • Compatibility:
      Minor

      Description

      The HpccFile object will be modified to create a serializable object holding the file part map and the high level index block.  The Spark infrastructure will be responsible for replicating and distributing the object as required.

      There will be a readRows(...) method which will return an Iterator<Row> object.

      The preferred implementation will take a Column argument for the join expression (see Dataset<Row> in the spark sql package.  

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rpastrana Rodrigo Pastrana
                Reporter:
                johnholt John Holt
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated: