Prerequisites
Download and install Apache Spark from https://spark.apache.org/.
Download and install Java from http://java.com/en/download/manual.jsp.
Native Libraries
The native library files should be placed in a folder, e.g. /opt/matchithub.lib or /usr/local/lib64, that can be synced with all the processing nodes. Execute the following steps:
cd matchithub-spark/libsudo cp lib*.so /usr/local/lib64
Use rsync to sync the /usr/local/lib64 folder with all processing nodes in the cluster. Ensure that spark-env.sh contains these lines:
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib64SPARK_YARN_USER_ENV="LD_LIBRARY_PATH=$LD_LIBRARY_PATH"
Running Jobs
Each of the sample application folders contains a run.sh example script showing a spark-submit command.