Agent sandboxes arrive in bursts, distributed training jobs fail if ranks cannot land together, and the default scheduler binds one pod at a time. Between 21 September and 19 October 2026 Kubernetes was retrofitted: v1.37 “Garhwal” took gang scheduling and workload-aware preemption to beta behind a single GenericWorkload gate; Dynamic Resource Allocation (DRA) had been generally available and on by default since 1.34; Kueue admits jobs atomically; node swap lifted gVisor sandbox density from 80 to 240 pods per node; and NVIDIA open-sourced Run:ai and the KAI Scheduler under Apache 2.0 on 18 October.
Yet two problems remain. You hold a monthly GPU invoice you cannot split across tenants, and container-toolkit escapes showed time-slicing was never a security boundary. The retrofit closed placement; attribution and trust stay open.
In This Series
- Kubernetes 1.37 for AI Workloads and How to Verify What Shipped: on, off, and how to verify
- How Kubernetes Gang Scheduling Works and Where It Breaks: the placement contract and its blind spots
- Kueue, Volcano and KAI Scheduler Compared and How to Judge the Commitment: three tools, three problems
- MIG, Time-Slicing and MPS Compared for Shared Multi-Tenant GPUs: sharing as a trust decision
- The Unsplittable GPU Bill and the Per-Second Sandbox Alternative: why the invoice resists attribution
What did Kubernetes 1.37 actually change for GPU and agent workloads, and how much of it is on by default?
Kubernetes v1.37 “Garhwal” shipped on 21 September 2026 with gang scheduling and workload-aware preemption at beta, merged into a single GenericWorkload gate that stays disabled by default. DRA is the counter-example, generally available and enabled since 1.34. A beta feature behind a disabled gate runs on no default cluster, so check the gate set and the API your cluster serves.
Think of it as a three-column ledger: on like DRA, beta but off such as GenericWorkload, alpha gates like WorkloadWithJob and CompositePodGroup. The Extended Resource Bridge maps legacy nvidia.com/gpu requests, but driver readiness still gates full DRA adoption. Audit in order: release notes, KEP, gate default, observed behaviour. The first guide walks through how to verify what your release actually turns on.
Why does a distributed GPU job stall halfway, and what does gang scheduling actually guarantee?
The default scheduler binds pods one at a time and never evaluates the group’s total footprint. An eight-node training job can hold seven nodes’ worth of capacity, make no progress, and still report every pod as scheduled. Gang scheduling moves the decision from the pod to the group. At beta it fixes placement but closes nothing else, leaving two blind spots in workload-aware preemption.
These partial-placement deadlocks stay invisible: pods scheduled, job running, nothing saying 90% placed while your job stalls. TopologyAwareWorkloadScheduling sharpens this, because strict NVLink or rack boundaries turn a near-miss into a permanent lock. Workload-aware preemption must also reason cluster-wide, and cannot reclaim GPUs allocated through DRA because ResourceClaims live outside the scheduler’s eviction model. Placement got the retrofit first; preemption maturity is catching up. The second guide examines what gang scheduling actually guarantees.
Kueue, Volcano or KAI Scheduler — which problem are you actually solving?
Each answers a different question: Kueue decides whether a job should start at all given quota, admits atomically, and places nothing. Volcano replaces the scheduling path with its own PodGroup lineage. KAI Scheduler, open-sourced with Run:ai under Apache 2.0 on 18 October 2026, productises GPU-dense placement with a hierarchical fair-share model. Run two and you own two sources of truth for pending work.
Separating admission, placement, and productised scheduling clarifies the architectural trade-offs. Kueue’s quota borrowing optimises utilisation and accepts disputes over stranded capacity. KAI’s project-and-department tree optimises accountability and accepts entitlement arguments. Apache 2.0 removes the licence hurdle but grants no support for a run that stalls overnight. Base your selection on commitment: components to maintain, upgrade cadence alongside Kubernetes and CUDA, and who answers pages at 2am. The third guide details which scheduler you are really committing to.
Is a shared GPU a trust boundary or only a billing arrangement?
A billing arrangement, unless the device is partitioned in hardware. Multi-Instance GPU (MIG) splits the card into dedicated silicon slices; time-slicing and MPS interleave execution contexts and leave memory shared, so the fault domain is the whole device rather than the slice. Container-toolkit escapes showed that the mechanism never promised isolation. Decide the trust question first; the utilisation gain follows.
MIG, time-slicing and MPS are device mechanisms; HAMi sits above them, inheriting their isolation. The oversubscription that makes time-slicing cheap is what makes it unsafe across tenants. Isolation spans four layers: device partition, host toolkit where shared toolkit vulnerabilities compromise every co-tenant, runtime, and tenancy agreement. Confidential containers and gVisor-class sandboxing harden the runtime, not the device. For untrusting tenants, separate devices are the honest answer. The fourth guide covers what to check before tenants share a device.
Why can’t the GPU invoice be split across tenants, and what does per-second billing change?
There is no per-pod GPU metric comparable to CPU and memory. DCGM reports at the physical card, metrics.k8s.io has no GPU resource, and nothing plays the Vertical Pod Autoscaler (VPA) role for accelerators, so rightsizing and cost splits fail at the same point. Per-second sandbox billing sidesteps attribution by removing the shared asset, at the price of control over isolation, drivers, and topology.
Buying isolation pushes teams back to sharing, and the accounting wall waits there. The widely quoted 5% average GPU utilisation, against 8% for CPU and 20% for memory, measures provisioned capacity, not demand, and inference workers saturate bandwidth at modest compute. Cast AI cites roughly $8,850 per GPU per month on demand and 30 to 40% savings, against an unstated reservation baseline. Without per-pod accounting, every chargeback figure is a negotiation. The fifth guide explains why the GPU invoice resists attribution.
What is still missing
Even after this retrofit, three platform gaps remain unresolved:
- Attribution: no per-pod GPU metric exists, so every chargeback figure is an estimate; the bill guide sets out what can honestly be claimed.
- Trust boundaries: time-slicing and MPS remain utilisation mechanisms without hardware isolation; confidential containers and attestation are the practical answer, and the sharing guide treats this as policy.
- The load-shape evidence gap: published density metrics, including the 80 to 240 sandbox jump, come from specific machine profiles, and no public arrival data exists for agent workloads.
Checking whether your cluster can use any of this? Start with the release verification.
Stuck at 90% placed? Start with the placement contract.
Invoice is the pressure point? Start with attribution.
Resource Hub: Kubernetes GPU and Agent Workloads Deep Dives
Understand the retrofit
- Kubernetes 1.37 for AI Workloads and How to Verify What Shipped: what is on, what is beta, how to audit.
- How Kubernetes Gang Scheduling Works and Where It Breaks: the placement contract and its blind spots.
Decide the architecture
- Kueue, Volcano and KAI Scheduler Compared and How to Judge the Commitment: admission, placement and scheduling as separate purchases.
- MIG, Time-Slicing and MPS Compared for Shared Multi-Tenant GPUs: what partitions in hardware, what interleaves.
Follow the money
- The Unsplittable GPU Bill and the Per-Second Sandbox Alternative: why attribution is missing, and when per-second billing wins.
Reading order: release verification, placement, queueing, sharing, cost.
Frequently Asked Questions
Do I have to upgrade to Kubernetes 1.37 to get gang scheduling?
No. Gang admission has been available as an external controller for years, and Kueue runs on clusters well behind 1.37. What 1.37 adds is the in-tree GenericWorkload beta, which lets the scheduler itself evaluate a pod group. If you already run Kueue for quota, upgrading is a convenience, not a prerequisite. Audit the gate before you plan the upgrade.
What is a Workload object, and is it the same thing as a Job?
No, they solve different problems. A Job is a run-to-completion controller that owns pods and tracks success. A Workload is a scheduling descriptor that groups pods so the scheduler and preemption logic can treat them as one unit. A Job can own a Workload, and that group is what gang scheduling evaluates. The distinction matters when debugging a stall, because the Job reports completion while the Workload reports placement.
Is Kueue a scheduler?
No. Kueue is an admission controller: it decides whether a job may start given quota, then admits it atomically or leaves it pending. It does not choose nodes. Placement stays with kube-scheduler or whatever replacement you run, which is why Kueue and gang scheduling are complementary purchases. If you expect Kueue to fix fragmentation across racks, you are buying the wrong layer.
Can I run Kueue and Volcano on the same cluster?
You can, but you should not. Both will claim pending work, and each maintains its own queue state, leaving you with two sources of truth and two sets of quota semantics. Choose by the problem you are solving: admission and fairness with Kueue, or a replacement scheduling path with PodGroup lineage in Volcano. Running both is a migration state, not an architecture.
What is the difference between a device plugin and DRA?
A device plugin advertises whole devices as integer counts, so nvidia.com/gpu: 1 means one opaque card. DRA describes devices through ResourceClaims and ResourceSlices, with attributes such as model, memory, and topology, so a claim can request a shape rather than a quantity. The Extended Resource Bridge keeps legacy requests working during migration, but richer placement only arrives once your driver supports DRA.
Why is my pod pending when the cluster shows free GPUs?
Fragmentation is the likely cause, not absence: the free cards are scattered across nodes and none can satisfy the whole claim. Check three things in order. Whether the queue is stopped, whether the ResourceClaim is allocatable on any single node, and whether topology constraints exclude the leftovers. Kueue and Headlamp both surface the admission decision, which is faster than reading scheduler events by hand.
Does node swap really let me quadruple my agent sandbox density?
Not reliably. The jump from 80 to 240 pods per node was measured on a specific machine shape running gVisor Python sandboxes, which are memory-heavy and short-lived. Swap removes the hard memory ceiling, but it converts memory pressure into latency, and a busy node can start swapping under load. Treat 240 as a ceiling to test, not a number to size against.
Does MIG stop a container escape from reaching other tenants?
Not entirely. MIG partitions memory and compute into dedicated slices, so a fault or runaway workload inside one slice cannot consume another’s silicon. But the driver, the container toolkit, and the host kernel remain shared, and a toolkit escape reaches the node and every GPU attached to it. MIG limits the blast radius of device faults, not kernel-level compromise. Confidential containers address the rest.
How can I get per-pod GPU costs today?
You approximate. Join DCGM’s device-level samples with your own record of which pod held which device and for how long, then present the result as an allocation estimate rather than consumption. It supports showback and trend analysis but not contractual chargeback, because idle time and co-tenancy both distort it. If the invoice must be split precisely, per-second sandbox billing is the honest alternative.
Do agent sandboxes need GPUs at all?
Usually not. Most agent workloads are dominated by tool calls, code execution, and network waits, which are CPU and memory problems. GPUs matter when the sandbox runs local inference or embedding models. Sizing a fleet around accelerators because the workload is called AI is a common and expensive mistake. Measure arrival rates and per-sandbox memory first.
