ForstXML libraries conflict when using PySpark integrated to HPCC

Trying to use PySpark results in an error. Below some excerptions from PySpark shell:

>>> df.show()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/HPCCSystems/externals/spark-hadoop/python/pyspark/sql/dataframe.py", line 378, in show
    print(self._jdf.showString(n, 20, vertical))
  File "/opt/HPCCSystems/externals/spark-hadoop/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/opt/HPCCSystems/externals/spark-hadoop/python/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/opt/HPCCSystems/externals/spark-hadoop/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o39.showString.
: java.lang.ExceptionInInitializerError
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
………
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.11.0
        at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
………
 

As I see from this log the system has inappropriate version of “ForestXML” library. And I found two different versions of this library in two places both used in Spark:

1. The jar file /opt/HPCCSystems/jars/spark/dfsclient-7.12.30-1-jar-with-dependencies.jar contains this library.

File DEPENDENCIES within this Jar includes the following description:

From: 'FasterXML' (http://fasterxml.com/)

  - Jackson-annotations (http://github.com/FasterXML/jackson) com.fasterxml.jackson.core:jackson-annotations:bundle:2.11.0

    License: The Apache Software License, Version 2.0  (http://www.apache.org/licenses/LICENSE-2.0.txt)

  - Jackson-core (https://github.com/FasterXML/jackson-core) com.fasterxml.jackson.core:jackson-core:bundle:2.11.0

    License: The Apache Software License, Version 2.0  (http://www.apache.org/licenses/LICENSE-2.0.txt)

  - jackson-databind (http://github.com/FasterXML/jackson) com.fasterxml.jackson.core:jackson-databind:bundle:2.11.0

    License: The Apache Software License, Version 2.0  (http://www.apache.org/licenses/LICENSE-2.0.txt)

1. Partial Listing of /opt/HPCCSystems/externals/spark-hadoop/jars/ directory:

[msv@hpcc-1 ~]$ cd /opt/HPCCSystems/externals/spark-hadoop/jars/

[msv@hpcc-1 jars]$ ls -al jackson

rw-rr-. 1 root root   45823 Feb 24 14:16 jackson-annotations-2.6.7.jar

rw-rr-. 1 root root  256281 Feb 24 14:16 jackson-core-2.6.7.jar

rw-rr-. 1 root root  229445 Feb 24 14:16 jackson-core-asl-1.9.13.jar

rw-rr-. 1 root root 1151704 Feb 24 14:16 jackson-databind-2.6.7.1.jar

rw-rr-. 1 root root  315564 Feb 24 14:16 jackson-dataformat-yaml-2.6.7.jar

rw-rr-. 1 root root   18031 Feb 24 14:16 jackson-jaxrs-1.9.13.jar

rw-rr-. 1 root root  770013 Feb 24 14:16 jackson-mapper-asl-1.9.13.jar

rw-rr-. 1 root root   31908 Feb 24 14:16 jackson-module-jaxb-annotations-2.6.7.jar

rw-rr-. 1 root root   42090 Feb 24 14:16 jackson-module-paranamer-2.7.9.jar

rw-rr-. 1 root root  507691 Feb 24 14:16 jackson-module-scala_2.11-2.6.7.1.jar

rw-rr-. 1 root root   26671 Feb 24 14:16 jackson-xc-1.9.13.jar

rw-rr-. 1 root root   47689 Feb 24 14:16 json4s-jackson_2.11-3.5.3.jar

rw-rr-. 1 root root 1034169 Feb 24 14:17 parquet-jackson-1.10.1.jar

[msv@hpcc-1 jars]$