I'm getting questions about "push down" filtering with the Spark HPCC Connector.
I'm trying to get clarification from the engineer here at the client.
This page shows:
|spark.sql.parquet.filterPushdown||true||Enables Parquet filter push-down optimization when set to true.|
Is this relevant to the Spark HPCC Connector?
Does the Connector support "filterPushdown"? If so, I'd love to get some discussion/notes on that.
The example from the engineer here -
Push Down model
File Filter model -
In which of these examples would the filter (f1 = "...") be passed down to HPCC (so that the entire file is not passed up to Spark)?
More on "push down":