Failure Policy
JobSet provides failure policy API to control how your workload behaves in response to child Job failures.
The failurePolicy is defined by a set of rules. For any job failure, the rules are evaluated in order, and the first matching rule’s action is executed. If no rule matches, the default action is RestartJobSet, which counts towards the maxRestarts limit.
Failure Policy Actions
FailJobSet
This action immediately marks the entire JobSet as failed.
In this example, the JobSet is configured to fail immediately if the leader job fails.
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: failjobset-action-example
spec:
failurePolicy:
maxRestarts: 3
rules:
# The JobSet will fail immediately when the leader job fails.
- action: FailJobSet
targetReplicatedJobs:
- leader
replicatedJobs:
- name: leader
replicas: 1
template:
spec:
# Set backoff limit to 0 so job will immediately fail if any pod fails.
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
containers:
- name: leader
image: bash:latest
command:
- bash
- -xc
- |
echo "JOB_COMPLETION_INDEX=$JOB_COMPLETION_INDEX"
if [[ "$JOB_COMPLETION_INDEX" == "0" ]]; then
for i in $(seq 10 -1 1)
do
echo "Sleeping in $i"
sleep 1
done
exit 1
fi
for i in $(seq 1 1000)
do
echo "$i"
sleep 1
done
- name: workers
replicas: 1
template:
spec:
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
containers:
- name: worker
image: bash:latest
command:
- bash
- -xc
- |
sleep 1000
RestartJobSet
This action restarts the entire JobSet by recreating its child jobs. The number of restarts before failure is limited by failurePolicy.maxRestarts. This is the default action if no other rules match a failure.
In this example, the JobSet will restart up to 3 times if the leader job fails.
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: restartjobset-action-example
spec:
failurePolicy:
maxRestarts: 3
rules:
# The JobSet will restart when the leader job fails.
- action: RestartJobSet
targetReplicatedJobs:
- leader
replicatedJobs:
- name: leader
replicas: 1
template:
spec:
# Set backoff limit to 0 so job will immediately fail if any pod fails.
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
containers:
- name: leader
image: bash:latest
command:
- bash
- -xc
- |
echo "JOB_COMPLETION_INDEX=$JOB_COMPLETION_INDEX"
if [[ "$JOB_COMPLETION_INDEX" == "0" ]]; then
for i in $(seq 10 -1 1)
do
echo "Sleeping in $i"
sleep 1
done
exit 1
fi
for i in $(seq 1 1000)
do
echo "$i"
sleep 1
done
- name: workers
replicas: 1
template:
spec:
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
containers:
- name: worker
image: bash:latest
command:
- bash
- -xc
- |
sleep 1000
RestartJobSetAndIgnoreMaxRestarts
This action is similar to RestartJobSet, but the restart does not count towards the maxRestarts limit.
In this example, the JobSet will restart an unlimited number of times if the leader job fails. However, workers jobs will be covered by the default RestartJobSet action and only restart up to maxRestarts times.
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
# rjimr stands for "restartjobsetandignoremaxrestarts"
name: rjimr-action-example
spec:
failurePolicy:
maxRestarts: 3
rules:
# The JobSet will restart an unlimited number of times
# when the leader job fails.
- action: RestartJobSetAndIgnoreMaxRestarts
targetReplicatedJobs:
- leader
replicatedJobs:
- name: leader
replicas: 1
template:
spec:
# Set backoff limit to 0 so job will immediately fail if any pod fails.
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
containers:
- name: leader
image: bash:latest
command:
- bash
- -xc
- |
echo "JOB_COMPLETION_INDEX=$JOB_COMPLETION_INDEX"
if [[ "$JOB_COMPLETION_INDEX" == "0" ]]; then
for i in $(seq 10 -1 1)
do
echo "Sleeping in $i"
sleep 1
done
exit 1
fi
for i in $(seq 1 1000)
do
echo "$i"
sleep 1
done
- name: workers
replicas: 1
template:
spec:
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
containers:
- name: worker
image: bash:latest
command:
- bash
- -xc
- |
sleep 1000
RestartJob
This action restarts by recreating the individual failed job, rather than recreating all jobs in the JobSet. The number of restarts before failure is limited by failurePolicy.maxRestarts. The maximum number of replicas per ReplicatedJob is 1024 when using this action.
In this example, failures in the leader Jobs force all Jobs to be recreated, whereas failures in the worker Jobs only recreate the failed worker Job. This happens up to 3 times, no matter the type of failure.
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: js
spec:
failurePolicy:
maxRestarts: 2
rules:
# If any Job from replicatedJob `rj-leader` fails, restart the JobSet by recreating ALL Jobs
# Count the restart towards `jobSet.spec.failurePolicy.maxRestarts`
- name: r_leader
action: RestartJobSet
targetReplicatedJobs:
- rj-leader
# If any Job from replicatedJob `rj-worker` fails, restart the JobSet by recreating only the failed Job
# Count the restart towards `jobSet.spec.failurePolicy.maxRestarts`
- name: r_worker
action: RestartJob
targetReplicatedJobs:
- rj-worker
replicatedJobs:
- name: rj-leader
replicas: 1
template:
spec:
parallelism: 2
completions: 2
backoffLimit: 0 # Job fails if any Pod fails
template:
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 0 # Useful for testing
containers:
- name: leader
image: busybox:1.37
command: ["/bin/sh", "-c"]
# Fail a leader Pod with `kubectl exec ${POD_NAME:?} -c leader -- touch /tmp/fail`
args:
- |
echo "Start"
while true; do
if [ -f /tmp/fail ]; then
echo "Exiting 1"
exit 1
elif [ -f /tmp/succeed ]; then
echo "Exiting 0"
exit 0
fi
sleep 1
done
- name: rj-worker
replicas: 1
template:
spec:
parallelism: 2
completions: 2
backoffLimit: 0 # Job fails if any Pod fails
template:
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 0 # Useful for testing
containers:
- name: worker
image: busybox:1.37
command: ["/bin/sh", "-c"]
# Fail a worker Pod with `kubectl exec ${POD_NAME:?} -c worker -- touch /tmp/fail`
args:
- |
echo "Start"
while true; do
if [ -f /tmp/fail ]; then
echo "Exiting 1"
exit 1
elif [ -f /tmp/succeed ]; then
echo "Exiting 0"
exit 0
fi
sleep 1
doneRestartJobAndIgnoreMaxRestarts
This action is similar to RestartJob, but the individual job restart does not count towards the maxRestarts limit. The maximum number of replicas per ReplicatedJob is 1024 when using this action.
In this example, failures in the leader Jobs force all Jobs to be recreated, whereas failures in the worker Jobs only recreate the failed worker Job. The failures are not counted towards the maxRestarts limit.
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: js
spec:
failurePolicy:
maxRestarts: 2
rules:
# If any Job from replicatedJob `rj-leader` fails, restart the JobSet by recreating ALL Jobs
# Do NOT count the restart towards `jobSet.spec.failurePolicy.maxRestarts`
- name: r_leader
action: RestartJobSetAndIgnoreMaxRestarts
targetReplicatedJobs:
- rj-leader
# If any Job from replicatedJob `rj-worker` fails, restart the JobSet by recreating only the failed Job
# Do NOT count the restart towards `jobSet.spec.failurePolicy.maxRestarts`
- name: r_worker
action: RestartJobAndIgnoreMaxRestarts
targetReplicatedJobs:
- rj-worker
replicatedJobs:
- name: rj-leader
replicas: 1
template:
spec:
parallelism: 2
completions: 2
backoffLimit: 0 # Job fails if any Pod fails
template:
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 0 # Useful for testing
containers:
- name: leader
image: busybox:1.37
command: ["/bin/sh", "-c"]
# Fail a leader Pod with `kubectl exec ${POD_NAME:?} -c leader -- touch /tmp/fail`
args:
- |
echo "Start"
while true; do
if [ -f /tmp/fail ]; then
echo "Exiting 1"
exit 1
elif [ -f /tmp/succeed ]; then
echo "Exiting 0"
exit 0
fi
sleep 1
done
- name: rj-worker
replicas: 1
template:
spec:
parallelism: 2
completions: 2
backoffLimit: 0 # Job fails if any Pod fails
template:
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 0 # Useful for testing
containers:
- name: worker
image: busybox:1.37
command: ["/bin/sh", "-c"]
# Fail a worker Pod with `kubectl exec ${POD_NAME:?} -c worker -- touch /tmp/fail`
args:
- |
echo "Start"
while true; do
if [ -f /tmp/fail ]; then
echo "Exiting 1"
exit 1
elif [ -f /tmp/succeed ]; then
echo "Exiting 0"
exit 0
fi
sleep 1
doneTargeting Specific Failures
You can make your failure policy rules more granular by using targetReplicatedJobs and onJobFailureReasons.
Targeting Replicated Jobs
The targetReplicatedJobs field allows you to apply a rule only to failures originating from specific replicated jobs. All the examples above use this field.
Targeting Job Failure Reasons
The onJobFailureReasons field allows you to trigger a rule based on the reason for the job failure. This is powerful for distinguishing between different kinds of errors. Valid reasons include BackoffLimitExceeded, DeadlineExceeded, and PodFailurePolicy.
Example: Handling BackoffLimitExceeded
This example configures the JobSet to perform unlimited restarts only when the leader job fails because its backoffLimit was exceeded.
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: onjobfailurereasons-present-example
spec:
failurePolicy:
maxRestarts: 3
rules:
# The JobSet will restart an unlimited number of times when the
# leader job fails with the failure reason BackoffLimitExceeded.
- action: RestartJobSetAndIgnoreMaxRestarts
targetReplicatedJobs:
- leader
onJobFailureReasons:
- BackoffLimitExceeded
replicatedJobs:
- name: leader
replicas: 1
template:
spec:
# Set backoff limit to 0 so job will immediately fail if any pod fails.
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
containers:
- name: leader
image: bash:latest
command:
- bash
- -xc
- |
echo "JOB_COMPLETION_INDEX=$JOB_COMPLETION_INDEX"
if [[ "$JOB_COMPLETION_INDEX" == "0" ]]; then
for i in $(seq 10 -1 1)
do
echo "Sleeping in $i"
sleep 1
done
exit 1
fi
for i in $(seq 1 1000)
do
echo "$i"
sleep 1
done
- name: workers
replicas: 1
template:
spec:
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
containers:
- name: worker
image: bash:latest
command:
- bash
- -xc
- |
sleep 1000
Example: Handling PodFailurePolicy
This example shows how to use Kubernetes’ podFailurePolicy in conjunction with JobSet’s failurePolicy. The JobSet will restart an unlimited number of times only if the leader job fails due to its pod failure policy being triggered (in this case, a container exiting with code 1).
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: onjobfailurereasons-podfailurepolicy-example
spec:
failurePolicy:
maxRestarts: 3
rules:
# The JobSet will restart an unlimited number of times
# when the leader job fails with a failure reason matching
# the pod failure policy.
- action: RestartJobSetAndIgnoreMaxRestarts
targetReplicatedJobs:
- leader
onJobFailureReasons:
- PodFailurePolicy
replicatedJobs:
- name: leader
replicas: 1
template:
spec:
# Set backoff limit to 0 so job will immediately fail if any pod fails.
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
restartPolicy: Never
containers:
- name: leader
image: bash:latest
command:
- bash
- -xc
- |
echo "JOB_COMPLETION_INDEX=$JOB_COMPLETION_INDEX"
if [[ "$JOB_COMPLETION_INDEX" == "0" ]]; then
for i in $(seq 10 -1 1)
do
echo "Sleeping in $i"
sleep 1
done
exit 1
fi
for i in $(seq 1 1000)
do
echo "$i"
sleep 1
done
podFailurePolicy:
rules:
- action: FailJob
onPodConditions: []
onExitCodes:
containerName: leader
operator: In
values: [1]
- name: workers
replicas: 1
template:
spec:
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
containers:
- name: worker
image: bash:latest
command:
- bash
- -xc
- |
sleep 1000
Example: Handling Host Maintenance Events
This example models how to handle a host maintenance event. A pod eviction due to maintenance triggers a DisruptionTarget pod condition, which in turn triggers the job’s podFailurePolicy. The JobSet’s failure policy rule matches on the PodFailurePolicy reason and restarts the JobSet without counting it against maxRestarts. Any other type of failure in the leader job will trigger a normal restart that counts against maxRestarts (which is 0 in this case, meaning any other failure will cause the JobSet to fail immediately).
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: host-maintenance-event-model
spec:
failurePolicy:
maxRestarts: 0
rules:
# The JobSet will restart an unlimited number of times when failure matches the pod failure policy.
- action: RestartJobSetAndIgnoreMaxRestarts
onJobFailureReasons:
- PodFailurePolicy
# The JobSet is restarted as normal when the leader job fails and the above rule is not matched.
- action: RestartJobSet
targetReplicatedJobs:
- leader
replicatedJobs:
- name: leader
replicas: 1
template:
spec:
# Set backoff limit to 0 so job will immediately fail if any pod fails.
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
restartPolicy: Never
containers:
- name: leader
image: bash:latest
command:
- bash
- -xc
- |
echo "JOB_COMPLETION_INDEX=$JOB_COMPLETION_INDEX"
if [[ "$JOB_COMPLETION_INDEX" == "0" ]]; then
for i in $(seq 120 -1 1)
do
echo "Sleeping in $i"
sleep 1
done
exit 1
fi
for i in $(seq 1 1000)
do
echo "$i"
sleep 1
done
# This failure policy is triggered when a node undergoes host maintenace.
# In such a case, the pods are evicted and the job will fail with a condition
# of type DisruptionTarget.
podFailurePolicy:
rules:
- action: FailJob
onPodConditions:
- type: DisruptionTarget
- name: workers
replicas: 1
template:
spec:
backoffLimit: 0
completions: 2
parallelism: 2
template:
spec:
containers:
- name: worker
image: bash:latest
command:
- bash
- -xc
- |
sleep 1000
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.