pyspark.SparkContext.setInterruptOnCancel

SparkContext.setInterruptOnCancel(interruptOnCancel: bool) → None

Set the behavior of job cancellation from jobs started in this thread.

New in version 3.5.0.

Parameters

interruptOnCancelbool

If true, then job cancellation will result in Thread.interrupt() being called on the job’s executor threads. This is useful to help ensure that the tasks are actually stopped in a timely manner, but is off by default due to HDFS-1208, where HDFS may respond to Thread.interrupt() by marking nodes as dead.

See also

SparkContext.addJobTag()

SparkContext.removeJobTag()

SparkContext.cancelAllJobs()

SparkContext.cancelJobGroup()

SparkContext.cancelJobsWithTag()

previous

pyspark.SparkContext.setCheckpointDir

next

pyspark.SparkContext.setJobDescription