---
title: "Troubleshoot head node eviction"
description: "Troubleshoot head node eviction and configure PodDisruptionBudget for Anyscale on Kubernetes."
---

# Troubleshoot head node eviction

Head nodes may be evicted due to pod preemption by higher-priority pods, causing cluster disruption.

## Symptoms

Cluster logs show messages indicating head node eviction:

```text
Instance k-ecb7f6f7c200c0000 (node IP: <node-ip>) will be terminated soon
(reason: pod eviction/deletion; for more details, check Kubernetes events for
pod/k-ecb7f6f7c200c0000 or node/<node-name>).
```

Or preemption messages:

```text
Terminating instance k-d403ea0c7bf910000 due to an unexpected failure: Pod was disrupted
(reason: PreemptionByScheduler, details: default-scheduler: preempting to accommodate a higher priority pod).
```

## Root cause

The head node is being preempted by Kubernetes to accommodate higher-priority pods. A PodDisruptionBudget (PDB) on the head node can help prevent this. See [PodDisruptionBudget in the Kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#poddisruptionbudget-is-supported-but-not-guaranteed).

The `workloads.enableAnyscaleRayHeadNodePDB` Helm parameter defaults to `true`, so this issue typically appears only after someone disables PDB protection. See [High availability](/clouds/kubernetes/helm-ref.md#user-ha).

## Solution

Re-enable PodDisruptionBudget for Anyscale head nodes through the Helm chart configuration.

### Configure PDB through Helm

Set the `workloads.enableAnyscaleRayHeadNodePDB` flag when installing or upgrading the Anyscale operator:

```bash
helm upgrade <release-name> anyscale/anyscale-operator \
  --set workloads.enableAnyscaleRayHeadNodePDB=true \
  --namespace <namespace>
```

### Verify PDB configuration

Check that the PodDisruptionBudget was created:

```bash
kubectl get poddisruptionbudgets -n <namespace> -o yaml
```

Example output showing a properly configured PDB:

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: anyscale-ray-head-nodes
  namespace: <namespace>
  labels:
    app.kubernetes.io/instance: <release-name>
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: anyscale-operator
spec:
  maxUnavailable: 0
  selector:
    matchLabels:
      ray-node-type: head
  unhealthyPodEvictionPolicy: AlwaysAllow
```

## Prevention

To prevent head node eviction, do the following:

1.  Keep `workloads.enableAnyscaleRayHeadNodePDB` enabled in your Helm values, or re-enable it as shown above.
2.  Schedule head nodes on on-demand node pools rather than spot instances.
3.  Configure appropriate pod priorities for Anyscale workloads.
4.  Monitor cluster events for eviction warnings.

## Related resources

-   [Kubernetes PodDisruptionBudget documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#poddisruptionbudget-is-supported-but-not-guaranteed)
-   For high-availability Helm parameters, see [High availability](/clouds/kubernetes/helm-ref.md#user-ha).
-   For guidance on temporarily disabling PDB to allow node upgrades, see [Upgrade Kubernetes nodes with Anyscale services running](/kb/kubernetes-node-upgrades.md).

---

Previous: [VS Code not loading in workspace over VPN (Chrome)](/kb/vscode-vpn-chrome-loading.md) | Next: [Troubleshoot GPU visibility in pods](/kb/k8s-gpu-visibility.md)