Most teams read the release notes for Kubernetes v1.37 “Garhwal”, noted gang scheduling reached Beta, and changed nothing. Nothing broke. That silence shows the gap between what releases announce and what clusters run. A release claims code exists; whether that code runs is a configuration fact. While headlines promote this milestone for AI workloads, the retrofit this release is part of scatters capabilities across three states: generally available and on, beta but turned off, and disabled alpha.
Separating intent from capability matters when one analysis counts 22 alpha features while another records 27, and testing by ScaleOps showed gates reporting active while their API is unserved. Verifying what runs means checking the ledger, inspecting gate consolidation, testing API behaviour, and evaluating one migration decision. The starting point is sorting claims by state, noticing that ‘Beta’ does two different jobs in one release.
What did Kubernetes 1.37 actually ship for AI and agent workloads, and what is still alpha behind a disabled gate?
Kubernetes v1.37 does not change whether Dynamic Resource Allocation is on; it changes what a claim can ask for. Gang scheduling and workload-aware preemption remain Beta and disabled by default behind GenericWorkload, while experimental work like WorkloadWithJob and CompositePodGroup remains Alpha and off in a default install. Code shipping in a release does not mean your cluster runs it.
The official Kubernetes 1.37 release announcement links the full changelog; one count puts it at 67 enhancements, but readiness splits across three categories. First are capabilities generally available and on by default. Dynamic Resource Allocation refines the resource model in 1.37, giving drivers a structured vocabulary for device properties.
Second are capabilities graduated to Beta but disabled by default. Gang scheduling and workload-aware preemption belong here, both requiring GenericWorkload. If you want the placement guarantees this release turns on, you must enable that gate. Under KEP-5729, DRAWorkloadResourceClaims also reached Beta to let pod groups share claims, but stays switched off.
Third are experimental capabilities that remain Alpha and gated off. WorkloadWithJob (KEP-5547) links workloads directly to Jobs, but sits disabled. Nested groups under CompositePodGroup (KEP-6012) are Alpha and off in a default install. A Beta label does not mean a feature is running. Two of those three states share one gate, and that is where a release decision becomes your adoption decision.
How does the GenericWorkload feature gate work, and why did GangScheduling and WorkloadAwarePreemption stop existing as separate gates?
In Kubernetes v1.37, GenericWorkload is a single Beta gate, disabled by default, enabling the Workload API, gang scheduling, and workload-aware preemption together. Because these mechanisms share one API object and rely on each other’s operational assumptions, testing them independently created a test matrix too large to cover. One flag controls both behaviours together.
According to KEP-4671, separate flags created an artificial split. Gang scheduling provides all-or-nothing admission, while workload-aware preemption provides atomic evictions when clearing capacity. Atomic admission without atomic eviction breaks running groups pod by pod, leaving abandoned tasks on accelerators.
This consolidation hands you an all-or-nothing bargain. You cannot adopt gang placement without accepting workload-aware eviction rules. In the Workload API specified in KEP-4671, disruptionMode defaults to single. That default evicts single pods, quietly half-undoing a group guarantee set at admission and leaving a running gang open to partial eviction unless configured for whole-group eviction from the start.
Consolidation creates operational risk. Turning off GenericWorkload removes both mechanisms, while the shared gate obscures metrics. Control-plane components reject retired gate strings like GangScheduling or WorkloadAwarePreemption at startup rather than ignoring them. If one flag can hide which half is running, the gate set was never the evidence.
How do I tell whether a beta feature is actually usable in a given release: feature gate set, feature metric reporting, or API actually served?
In Kubernetes v1.37, the order that survives a mis-set gate is gate default, then metric, then observed API behaviour, though only the last carries authority. The gate default reveals release intent, the kubernetes_feature_enabled metric confirms startup configuration, and an API that accepts objects and triggers controller action demonstrates working functionality. An API that round-trips and reconciles outranks flags and metrics.
Signals disagree, and a single indicator is not enough for capacity planning. Gates apply per component across kube-apiserver, kube-scheduler, kube-controller-manager, and kubelet. Managed control planes often prevent modifying process flags, meaning upstream defaults cannot be assumed on managed infrastructure.
The metric records what binaries registered at startup, but confirms declared flags rather than runtime readiness. Testing in code-freeze reviews showed PodLevelResourceManagers in kubelet arguments while its metric continued reporting zero.
Whether the API is served and reconciled remains the strongest evidence. In 1.37, ScaleOps found in its release audit that CompositePodGroup reported enabled in metrics, yet kubectl api-resources listed no such object. A gate can read enabled on one component while the API server does not serve the type at all. The gate exists, but the API does not.
A disciplined audit compares release notes to milestones in KEP-4671, KEP-5710, KEP-5729, and KEP-5690 before confirming live behaviour. Tooling like ScaleOps scans gate and version mismatches across clusters. That method matters most where ‘GA’ is easiest to mistake for ‘working’, in how a pod asks for a GPU.
How does DRA change GPU scheduling compared with the nvidia.com/gpu device-plugin counter, and is the migration worth it before the driver ecosystem catches up?
In Kubernetes v1.37, the legacy device plugin exposes nvidia.com/gpu as an opaque integer per node, unable to differentiate memory, compute generation, or interconnect topology. Dynamic Resource Allocation replaces this counter with structured, attribute-based claims across four API objects. The move pays off when attribute matching changes placement; it does not while any GPU will do.
An integer counter is simple, but reporting on accelerator scheduling outlines its limits: it cannot distinguish an 80GB card from a 192GB card, nor describe memory bandwidth or NVLink topology. A job needing substantial memory lands on a smaller card and crashes while larger GPUs sit idle — and that idle capacity is where the GPU economics story goes next.
DRA replaces the integer with four objects:
- DeviceClass: defines a device category advertised by a driver.
- ResourceSlice: advertises available hardware capacity on a node.
- ResourceClaim: binds an allocated device request to a pod.
- ResourceClaimTemplate: generates per-pod claims dynamically.
Yet migration depends on driver maturity rather than API availability. DRA reached GA in 1.34, but drivers must publish ResourceSlices before scheduling works. In tests on early driver behaviour, DeviceClasses registered correctly while zero ResourceSlices appeared because publishing stalled.
KEP-5004 reached Stable in 1.37, adding the Extended Resource Bridge. This shim maps traditional nvidia.com/gpu requests to a DeviceClass, keeping existing manifests running while drivers mature. That trade-off directly shapes the queueing layers that sit above in-tree scheduling and what a reserved but idle GPU costs each month.
A round-tripping API that reconciles remains the trustworthy proof of any release claim. A release whose sources disagree on the Alpha count is normal. A cluster you have not queried is a different problem. Sound verification checks release notes, KEP milestones, and gate defaults before your team commits budget. As you work through placing, queueing and paying for GPUs across the retrofit, the practical question is whether the guarantees you turn on match the workload profiles your applications require.
Frequently Asked Questions
Do I need to enable a feature gate to use DRA for GPUs in 1.37?
No. Dynamic Resource Allocation has been generally available and on by default since 1.34, so 1.37 adds model objects and attributes rather than a flag. The thing that decides whether DRA works on your cluster is the driver: if the NVIDIA DRA driver is not publishing ResourceSlice objects on a node, a perfectly enabled control plane will still allocate nothing.
Is a Beta feature in 1.37 on by default?
Not necessarily. Beta in 1.37 describes API stability, not enablement, and the release splits it two ways: gang scheduling and workload-aware preemption are Beta and off behind GenericWorkload, while HPA scale-to-zero is Beta and on. Read the gate default in your release before you treat a Beta label as a green light.
What happens if I set a feature gate that was removed in 1.37?
The component refuses to start rather than ignoring the flag. GangScheduling and WorkloadAwarePreemption no longer exist as separate gates, so a Helm value or templated manifest still setting them will take kube-scheduler down on its next restart. Search your configuration for retired gate names before the upgrade, not after the first crash loop.
If I enable GenericWorkload on the scheduler, is gang scheduling fully on?
Only if every component that needs it agrees. Gates are set per component, so the API server has to serve the Workload API, the scheduler has to act on it, and the controller manager has to reconcile it. A gate flipped on the scheduler alone gives you a scheduler waiting for an object that nothing serves.
Why does the feature-enabled metric read zero when I have set the gate?
Because kubernetes_feature_enabled only reports for features a component has already enabled, and each component reports separately. It confirms, it does not discover. PodLevelResourceManagers is the documented case: the kubelet accepts the gate and the metric still reads zero. Scrape the component you care about, then check the served API.
Can gang scheduling evict running pods to free GPUs in 1.37?
The scheduler can evict whole groups of pods to make room, but it cannot release a DRA device from a running claim. That gap is tracked under KEP-5690 and is not closed in this release. If your queue design assumes GPU claims get reclaimed under pressure, that is the assumption to test before you commit.
Do I still need the NVIDIA device plugin after moving to DRA?
You need a DRA driver, and you do not need the plugin running alongside it. The Extended Resource Bridge (KEP-5004) went Stable in 1.37, which lets a pod asking for nvidia.com/gpu: 1 be satisfied through a DeviceClass, so existing manifests keep working untouched. The bridge makes a staged migration possible; the driver’s maturity is still the real constraint.
Does upgrading to 1.37 change anything for existing AI pods without me opting in?
Yes, and SELinuxMount is the one to watch. It is Stable and on by default, and it breaks pods with different SELinux labels sharing a volume on the same node. The per-pod opt-out is .spec.seLinuxChangePolicy: Recursive, and clusters without SELinux are unaffected. Audit label conflicts before the upgrade window rather than during it.
Do I still need Kueue or Volcano now that gang scheduling is in-tree?
For placement alone, no. For quotas, fair sharing, queueing across teams and multi-cluster admission, yes, because 1.37 gives you atomic placement of a group, not a queue in front of it. Treat the in-tree API as the floor and judge the out-of-tree layer on the behaviour it adds above that floor.
Does HPA scale-to-zero work for GPU workloads in 1.37?
Only if your scaling signal arrives as an Object or External metric. minReplicas: 0 is Beta and on by default, but Resource metrics such as CPU and memory are excluded because they require running pods, so GPU utilisation has to come through an external metrics pipeline before an idle inference deployment can park at zero. A ScaledToZero condition confirms it happened.
Where do I find the official Kubernetes 1.37 release notes and the KEPs behind these claims?
The release announcement and full changelog on the Kubernetes blog are the primary source, and the scheduling claims here trace to KEP-4671, KEP-5710, KEP-5729 and KEP-5690. Read the KEP milestone and the gate default together, because the changelog on its own will tell you a feature exists and nothing beyond that.
