Solved — Apache Spark driverState FAILED when using REST API

Recently I encountered a continuos error when submitting a new job to Apache Sprak via the REST API. All jobs submissions happen successfully and return a good result in first instance.

{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20240427093303-0006",
  "serverSparkVersion" : "3.5.1",
  "submissionId" : "driver-20240427093303-0006",
  "success" : true
}

While the above looks to be promissing the actual job never starts / completed. When we call the driver submission status REST API for the specific driver submission ID we get the below shown JSON response when using this cURL command

curl http://10.0.0.50:6066/v1/submissions/status/driver-20240427093303-0006

The above command to retrieve the Apache Spark driver submission status results in the below response value which indicated that the DriverState is FAILED

{
  "action" : "SubmissionStatusResponse",
  "driverState" : "FAILED",
  "serverSparkVersion" : "3.5.1",
  "submissionId" : "driver-20240427093303-0006",
  "success" : true,
  "workerHostPort" : "10.0.0.50:7078",
  "workerId" : "worker-20240426100006-10.0.0.50-7078"
}

Even though the above is not giving that much information it is a good start for the…