Solved — Apache Spark driverState FAILED when using REST API

Recently I encountered a continuos error when submitting a new job to Apache Sprak via the REST API. All jobs submissions happen successfully and return a good result in first instance.

{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20240427093303-0006",
  "serverSparkVersion" : "3.5.1",
  "submissionId" : "driver-20240427093303-0006",
  "success" : true
}

While the above looks to be promissing the actual job never starts / completed. When we call the driver submission status REST API for the specific driver submission ID we get the below shown JSON response when using this cURL command

curl http://10.0.0.50:6066/v1/submissions/status/driver-20240427093303-0006

The above command to retrieve the Apache Spark driver submission status results in the below response value which indicated that the DriverState is FAILED

{
  "action" : "SubmissionStatusResponse",
  "driverState" : "FAILED",
  "serverSparkVersion" : "3.5.1",
  "submissionId" : "driver-20240427093303-0006",
  "success" : true,
  "workerHostPort" : "10.0.0.50:7078",
  "workerId" : "worker-20240426100006-10.0.0.50-7078"
}

Even though the above is not giving that much information it is a good start for the investigation as we now know that the driver is the next component we need to start looking into. The stdout and stderr for drivers can be found (in my case) at /opt/spark/work/{driver-ID}/stderr where we found the below;

Traceback (most recent call last):
  File "/home/opc/scripts/job0/sparkjob.py", line 12, in <module>
    .appName("CaesarCipherDecipher") \
  File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 497, in getOrCreate
  File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 515, in getOrCreate
  File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 215, in __init__
  File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 296, in _do_init
  File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 421, in _initialize_context
  File "/opt/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1588, in __call__
  File "/opt/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py"…