Cluster metrics show an eight-worker distributed training job stalled: seven pods report Running, one sits Pending, and the job reports healthy activity. Allocated GPUs sit idle without running an all-reduce cycle.
The default kube-scheduler binds pods individually without evaluating collective footprint. Gang scheduling supplies the atomic placement decision the default binder never makes; retrying a stalled job cannot substitute for it. Kubernetes 1.37 brought gang scheduling in-tree as a beta capability behind the GenericWorkload gate as part of the wider Kubernetes retrofit for agent and GPU workloads. This article traces what the in-tree model guarantees, and where that guarantee stops across group APIs, cluster-scoped preemption, and device reclaim.
What is gang scheduling in Kubernetes, and why does placing pods one at a time deadlock a distributed training job?
Gang scheduling in Kubernetes is an all-or-nothing placement contract: a pod group is evaluated against a single cluster snapshot, ensuring either minCount pods (the minimum subset that still does useful work) find capacity or none are bound. It applies only to pods referencing a PodGroup. Sequential pod-at-a-time binding creates locally optimal placements that stall distributed jobs when remaining pods cannot fit.
Kube-scheduler pops a pod from its queue, filters feasible nodes, scores them, and writes a binding. Each decision is optimal for that single pod at that instant, but nothing asks whether the rest of the group can land.
This creates a partial-placement deadlock. If a job requires eight workers with one GPU each, the scheduler might place seven workers before capacity runs out and then never schedule the last worker. Infrastructure accounting metrics show high placement counts while the job achieves zero compute execution. Workers cannot train without full rank membership, yet will not release their accelerators.
Retry cycles often make this worse. Failed groups return to the unschedulable queue while bound pods hold quota, causing head-of-line blocking for other workloads. Most pod-level alerts miss this because pods show Scheduled and jobs report Running. As documented in the PodGroup API, PodGroupInitiallyScheduled never reverts to False if placed pods drop later.
Layout constraints make the same failure more likely. Under TopologyAwareWorkloadScheduling rules, jobs often specify co-location within an NVLink domain. When capacity is fragmented, eight scattered GPUs cannot satisfy an interconnected gang, as covered when you check what your release actually turns on. Queue controllers like Kueue manage quota admission, but physical fit belongs to the scheduler. Whatever decides fit has to be declared somewhere first.
What is the Workload API, and how does a PodGroup differ from the Workload that declares it?
A Workload is an API-server policy template declaring one or more podGroupTemplates, while a PodGroup is the concrete runtime object that the scheduler evaluates and admits atomically. The in-tree Job controller generates PodGroups from the Workload, and member pods reference that PodGroup via spec.schedulingGroup.podGroupName. A PodGroup is not a Workload, and neither is a Deployment with a replica count.
This separation exists because pods and runtime groupings have distinct lifecycles. Upstream design documentation notes that pods outlive initial gang attempts, and jobs often require multiple groups over time. The Workload acts as an enduring template with lifecycle ownership and garbage collection, while the PodGroup’s status.conditions reveal why an admission succeeded or stalled, making the PodGroup API the first place to look for placement diagnostics.
Do not confuse these names with third-party tools. The in-tree PodGroup is distinct from the Coscheduling plugin resource that uses minMember, as well as the PodGroup used by Volcano. They share a name but use different schemas and status fields.
Experimental extensions like CompositePodGroup extend this model toward nested hierarchies, but remain alpha. For daily operations, the field governing post-admission disruption is disruptionMode. It decides whether evictions remove the entire gang together (all) or chip away at individual pods (single). As specified in the upstream PodGroup definition, it defaults to single. An evicted pod leaves surviving peers idling on unallocated hardware, a default that only matters when capacity is contested.
What does workload-aware preemption actually decide, and why is it cluster-scoped rather than node-local?
Workload-aware preemption determines which group of lower-priority pods, when evicted across the entire cluster, creates enough room to admit a pending pod group. It operates at cluster scope because freeing space on a single node cannot satisfy a multi-node gang whose members must be placed simultaneously — group-level reasoning that runs through the 1.37 workload management retrofit.
Traditional preemption reasons node by node. When an individual pod cannot schedule, the engine evicts lower-priority pods on a single machine. To admit a distributed gang, however, the scheduler must evaluate capacity across a cluster snapshot to identify victim groups spanning multiple machines at once.
Evicting part of a running victim group reproduces the deadlock in reverse, stranding survivors without freeing enough contiguous capacity for the incoming job. This requires gang-aware eviction. As outlined in workload-aware scheduling updates in 1.37, the engine evicts victim groups completely or leaves them intact. Release trade-offs include larger blast radiuses, longer grace periods, and coarser priority handling. A group configured with disruptionMode: all is removed together; the default chips it away one pod at a time and reproduces the stall in reverse.
The preemption engine has a second blind spot during in-place pod resizing. When expanding container resources in place, scheduler preemption for pod resize evaluates capacity strictly on the pod’s existing node. Analysis of resize preemption behaviour shows that this calculation bypasses node affinity, pod anti-affinity, and topology spread constraints. Which gates are open varies by release, as detailed when you verify what shipped in your release.
Why can’t workload-aware preemption reclaim GPUs that were allocated through DRA?
Every gang decision so far has assumed the scheduler can count what it frees. Workload-aware preemption reasons over node-local extended resources as integer counters; on eviction that count is returned to the node’s allocatable pool. Dynamic Resource Allocation handles devices via ResourceClaims, which follow an asynchronous control loop that the preemption algorithm does not model.
Standard GPU allocations use integer counters like nvidia.com/gpu. When a pod terminates, the scheduler subtracts the count and immediately restores capacity. By contrast, upstream Dynamic Resource Allocation structures hardware through ResourceClaims, DeviceClasses, and driver-managed ResourceSlices, as explored in operational analyses of DRA scheduling. These claims transition through driver preparation and de-allocation stages entirely outside the scheduler’s internal accounting.
The two models do not compose. Upstream scheduling designs track preemption nominations per node, whereas DRA resources frequently span cluster-wide fabrics and require asynchronous driver cleanup. Testing on Kubernetes 1.37 reports that while you can gang-schedule DRA GPUs, preemption cannot reclaim them.
When an incoming gang preempts a DRA-backed workload, the scheduler terminates victim pods and frees CPU and memory, but the GPU claims remain pinned by driver states. Your cluster dashboards show freed compute capacity, yet the incoming gang remains unscheduled because its accelerator claims cannot be satisfied.
Gang scheduling resolves admission atomicity, and disruptionMode provides group-level eviction, but device reclaim across modern accelerator pools remains outstanding as part of the broader Kubernetes workload retrofit. When assessing whether in-tree gang scheduling can retire a second scheduler, treat these capabilities as distinct layers. Use in-tree placement for atomic fit, rely on queue managers for multi-tenant fairness, and track how future releases tackle device lifecycle management.
Frequently Asked Questions
Does gang scheduling mean my pods run at the same time or share GPUs?
No, gang scheduling is placement only. It decides whether a group’s members are bound to nodes in one atomic decision; it does not synchronise execution, time-slice devices, or start pods together at the runtime level. Kubernetes gang scheduling carries no HPC-style time multiplexing, so once bound, each pod runs and can fail independently. What you get is all-or-nothing admission, not lockstep execution.
What happens if one pod in a placed group dies later?
After admission the contract is spent. The replacement pod is scheduled as an ordinary pod, and PodGroupInitiallyScheduled does not revert, so nothing in status tells you the group has dropped below minCount. Your job keeps reporting running while one worker restarts into a queue that may never place it. Watch the placed count against minCount, not the condition.
Does enabling gang scheduling change how my ordinary deployments are scheduled?
Only pods that declare a PodGroup go through atomic admission. A plain Deployment’s pods still queue, filter, score and bind one at a time, exactly as before, and they are not held back by an unrelated gang’s pending state beyond the capacity that gang is squatting on. The feature is opt-in per workload, not a cluster-wide scheduling mode.
What happens if my group can never fit on the cluster?
It waits indefinitely. There is no built-in deadline that fails a gang because it could not be placed, so a group larger than any achievable capacity simply cycles through failed attempts while holding whatever it already bound. Head-of-line blocking follows. The practical answer is a queue layer or quota policy upstream that refuses impossible work before the scheduler sees it.
Do I still need Kueue or Volcano now that gang scheduling is in-tree?
For placement, no. For queueing, yes, if you need it. In-tree gang scheduling decides physical fit for one group against one snapshot; it does not provide queue-level quota reservation, fair sharing between teams, or admission ordering across jobs. Those remain the concern of queue controllers and second-scheduler lineage projects, which is why the two are usually layered rather than substituted.
How do I tell whether my stuck job is a placement failure or something else?
Check whether pods are Pending with no node assigned, and read the PodGroup’s status.conditions for the scheduling verdict. A placement failure shows pods that were never bound; a runtime failure shows bound pods restarting or erroring. Seven workers Running with idle GPUs and an eighth Pending forever is the deadlock signature, not a slow scheduler.
Can I set minCount lower than the number of pods in the job?
Yes, and that is often correct. minCount defines the smallest subset that still does useful work, so a job that degrades gracefully at seven of eight workers should declare seven and let the group admit at that size. Set it too low, though, and you have reintroduced a partially placed job that runs slowly and then stalls.
Does gang scheduling place my pods on the same node or in the same NVLink domain?
No. Atomicity says the group is admitted together or not at all; it says nothing about the shape of the capacity it lands on. Topology constraints are a separate concern and need their own declaration, such as TopologyAwareWorkloadScheduling. Without that, a group can be admitted atomically onto eight nodes that share no fast interconnect.
What happens to a gang when a node is drained for maintenance?
Drain goes through the eviction API rather than preemption, but disruptionMode still governs the damage. On single, the default, the group is chipped at one pod at a time and the survivors keep holding capacity while the job limps. On all, the whole group is removed together, so it can be re-admitted as a unit elsewhere.
Is gang scheduling only relevant for distributed training?
It matters wherever a partial set of pods is worthless rather than merely degraded. Training is the obvious case, but batch pipelines with a barrier, MPI-style simulations, and a sharded inference deployment that cannot serve below a quorum all qualify. A stateless web deployment with independent replicas gains nothing from it beyond a wider blast radius on eviction.
Is coscheduling the same thing as gang scheduling?
Effectively yes, and that is a naming trap. The Coscheduling plugin is the second-scheduler implementation of the same idea, exposed under a different API group with minMember instead of minCount. It predates the in-tree Workload and PodGroup objects. The concept matches, the API does not, so a migration means translating fields rather than flipping a feature gate.
Does gang scheduling reserve capacity for a group that is queued but not yet placed?
No. Nothing is reserved at admission time; the group is evaluated against whatever is free at that instant. Between that decision and the binding of its last member, other work can take capacity, which is why repeated gang attempts under load keep failing. Reservation is a queue-level property, and it lives outside the scheduler.
