LowCodeStacks™
Sign in

AKS upgrade failed: fix the pod disruption budget and capacity blockers

When an AKS upgrade failed, the cause is almost always a pod disruption budget that allows zero disruptions, two budgets on one pod, or no capacity for surge nodes. Read the UpgradeFailed message, fix the budget or the surge settings, and reconcile the cluster, with the undrainable-node and automatic-budget options for next time.

Apps & integrationFix a problem7 min read

Checked against Microsoft Learn · 5 sourcesPosted by the Maker DeskHow we write guides

An Azure Kubernetes Service upgrade fails, the cluster drops into a Failed state, and the message mentions a pod disruption budget or a SKU that isn't available. The upgrade didn't break anything by itself; it stopped because something stopped it draining a node or adding one. This guide reads the two kinds of UpgradeFailed, fixes each, gets the cluster back to Succeeded, and sets the controls that stop it happening at the next upgrade. The timing matters more than it used to: from 1 June 2027, AKS upgrades clusters that linger on unsupported versions itself, outside your maintenance window and over your disruption budgets.

Read the error

Open the failed operation in the cluster's Activity log, or run az aks show --query provisioningState. The message tells you which of two problems you have.

A drain blocked by a pod disruption budget (PDB):

text
Code: UpgradeFailed
Message: Drain node aks-<pool>-xxxx-vmssxxxxxx failed when evicting pod <pod> failed with Too Many Requests error.
This error is often caused by a restrictive Pod Disruption Budget (PDB) policy. ...
Original error: Cannot evict pod as it would violate the pod's disruption budget.
PDB debug info: <namespace>/<pod> blocked by pdb <pdb-name> with 0 unready pods.

A variant names a different cause: KubernetesAPICallFailed with "Evict blocked by conflicting disruption budgets" or "This pod has more than one PodDisruptionBudget". Kubernetes can't evict a pod covered by two budgets, whatever their values.

No capacity for surge nodes: SKUNotAvailable, AllocationFailed or OverconstrainedAllocationRequest. AKS upgrades by adding surge nodes, moving pods to them and removing the old ones; if the region or your quota can't supply the surge nodes, the upgrade can't start on that pool. A SubnetIsFull error is the same problem one layer down: no IP addresses for the new nodes.

To upgrade, AKS drains each node, which means evicting its pods. A PDB limits how many pods of a workload may be unavailable at once. kubectl get events | grep -i drain shows the evictions AKS tried and the "Eviction blocked by Too Many Requests (usually a pdb)" warnings.

Fix a blocked drain

Look at the budget the message names:

bash
kubectl get pdb -n <namespace>
# NAME        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
# nginx-pdb   2               N/A               0

ALLOWED DISRUPTIONS must be 1 or more for a drain to proceed. It's 0 when minAvailable equals the running replicas, when maxUnavailable is 0, or when the deployment has fewer healthy replicas than the budget assumes. Three ways out, in order of preference:

  1. Let the pods drain. Raise the replica count, or loosen the budget (maxUnavailable: 1, or minAvailable below the replica count), then rerun the upgrade to the same version; that triggers the reconciliation.
  2. Back up, delete and reapply the budget if you can't edit it: kubectl get pdb <name> -n <ns> -o yaml > backup.yaml, kubectl delete pdb <name> -n <ns>, upgrade, kubectl apply -f backup.yaml.
  3. Scale the workload to zero (kubectl scale --replicas=0 deployment/<name>) or delete the stuck pods, after backing up the Deployment or StatefulSet, when the other two aren't options.

For the two budgets on one pod variant, compare each PDB's .spec.selector with the pod's labels (kubectl get pod <pod> --show-labels), delete one so every pod matches at most one budget, confirm the pods are healthy, and rerun. Avoid overlapping selectors in a namespace.

Then rerun:

bash
az aks upgrade --name <cluster> --resource-group <rg> --kubernetes-version <version>
# "Cluster currently in failed state. Proceeding with upgrade to existing version ... to attempt resolution"

Fix a capacity failure

LeverWhat it does
Lower maxSurgeFewer surge nodes needed at once. Microsoft suggests 33% for production, 50% for dev and test; set a count or a percentage with az aks nodepool update --max-surge
Use maxUnavailable insteadUpgrade by draining existing nodes rather than adding new ones; the way out when capacity is simply unavailable, at the cost of less headroom during the upgrade
Check quota and the subnetSurge nodes need vCPU quota for the SKU and IP addresses: (nodes + maxSurge) × (1 + maxPods) with Azure CNI. Request quota, expand the subnet, or lower maxPods
Security patches without surgeFor OS security updates only, use security-patch reimaging, which needs no surge nodes
ZonesSurge nodes land in zones AKS picks; set surge to a multiple of three to stay balanced, and remember zone-bound disks can't follow a pod to another zone

Both failure modes leave the cluster Failed. Fix the cause, then rerun the same upgrade or run az aks update -g <rg> -n <cluster> with no other arguments to trigger reconciliation.

Reconcile, then prevent the next one

Before every upgrade: in the portal, Settings > Upgrades > Upgrade version, pick the target and select Check next to Upgrade readiness. It runs the validations AKS applies anyway, deprecated APIs, a valid version path, misconfigured PDBs (maxUnavailable: 0), quota for surge nodes, subnet addresses, expired credentials and resource locks, before you commit. Check the release notes for API breaking changes too; kubent finds deprecated API usage in the cluster.

Settings that make drains succeed (AKS Standard; AKS Automatic manages them for you):

  • Pod disruption budgets with headroom: maxUnavailable of at least 1, enough replicas to honour it, and one budget per pod. Test them in staging.
  • --drain-timeout: how long AKS waits for a node to drain, 30 minutes by default; extend it for workloads that shut down slowly.
  • Undrainable-node behaviour: az aks nodepool update --undrainable-node-behavior Cordon --max-blocked-nodes 2. A node that won't drain is cordoned and labelled kubernetes.azure.com/upgrade-status=Quarantined while the rest of the upgrade continues; afterwards you delete the blocking PDB, remove the label and reconcile. Without it (Schedule), AKS deletes the blocked node and surges a replacement.
  • Automatic PDB management (preview): an extension that creates budgets for deployments without one and, when a budget blocks a drain, temporarily scales the deployment up so eviction is allowed, then scales it back.
  • Force upgrade (--enable-force-upgrade --upgrade-override-until <time>) bypasses PDBs entirely and can drain everything at once. Last resort, after you've tried to fix the budgets, and it overrides the undrainable-node settings while it's on.
  • Planned maintenance windows of at least four hours, in a quiet period, with node soak time if you want to stagger nodes.

Why not to wait. Kubernetes versions leave support on a fixed cadence, and from 1 June 2027 a cluster still on an unsupported version 60 days after its platform-support grace period is upgraded by AKS itself. That platform-initiated upgrade ignores your maintenance window, skips the deprecated-API check and can override a PDB that blocks a drain. Everything above is how you keep the choice of timing yours. For most production clusters Microsoft now recommends AKS Automatic, which manages upgrade behaviour, node images and the checks by default; AKS Standard stays for teams that need the manual controls.

Sources

Did this fix it?

Comments

No comments yet. Share what worked for you.

Sign in to comment. Comments show under a display name you choose, never your email.