7.0.0 provides a compile-time option to integrate SPARK environment into the HPCC cluster.
The HPCC install package will include the SPARK install binary and the spark-hpcc.jar library along with spark environment configuration files.
- Spark will be installed here: /opt/HPCCSystems/externals/spark
- Spark-hpcc is installed here: /opt/HPCCSystems/jars/spark-hpccXYZ.jar
Upon startup of the platform, the spark cluster will be configured to have a master node on the eclwatch node, and a slave node on all Thor nodes, and the spark-hpcc library will be added to the spark env path.