Uploaded image for project: 'HPCC'
  1. HPCC
  2. HPCC-20344

Document HPCC + SPARK integration

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0.0
    • Component/s: Documentation
    • Labels:

      Description

      7.0.0 provides a compile-time option to integrate SPARK environment into the HPCC cluster.

      The HPCC install package will include the SPARK install binary and the spark-hpcc.jar library along with spark environment configuration files.

      • Spark will be installed here: /opt/HPCCSystems/externals/spark
      • Spark-hpcc is installed here: /opt/HPCCSystems/jars/spark-hpccXYZ.jar

      Upon startup of the platform, the spark cluster will be configured to have a master node on the eclwatch node, and a slave node on all Thor nodes, and the spark-hpcc library will be added to the spark env path.

      Jim DeFabia Greg Panagiotatos Michael Gardner James McMullan

       

        Attachments

          Activity

            People

            • Assignee:
              g-pan Greg Panagiotatos
              Reporter:
              rpastrana Rodrigo Pastrana
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: