Spark job on Kubernetes Under Resource Starvation Wait Indefinitely For SPARK_MIN_EXECUTORS

Question

I am using Spark 3.0.1 and working on a project spark deployment on Kubernetes where Kubernetes acting cluster manager for spark job and spark submits the job using client mode. In case Cluster does not have sufficient resource (CPU/ Memory ) for minimum number of executors , the executors goes in Pending State for indefinite time until the resource gets free.

Suppose, Cluster Configurations are:

total Memory=204Gi
used Memory=200Gi
free memory= 4Gi
SPARK.EXECUTOR.MEMORY=10G
SPARK.DYNAMICALLOCTION.MINEXECUTORS=4
SPARK.DYNAMICALLOCATION.MAXEXECUTORS=8

Here job should not be submitted as executors allocated are less than MIN_EXECUTORS.

How can driver abort the job in this scenario?

question from:https://stackoverflow.com/questions/66057902/spark-job-on-kubernetes-under-resource-starvation-wait-indefinitely-for-spark-mi

Answer 1

Firstly would like to mention that, spark dynamic allocation not supported for kubernetes yet(as of version 3.0.1), its in pipeline for future release Link

while for the requirement you have posted, you could address by running a resource monitor code snippet before the job initialized and terminate the initialization pod itself with error.

if you want to run this from CLI you could use kubectl describe nodes/ kube-capacity utility to monitor the resources