Lambda‘s cluster analysis describes an illustrative incident: a sixteen-worker training job half-starts, claims four accelerators, and holds them while twelve pods sit pending. Other teams queue behind hardware reporting 8% utilisation. The hardware is fine; something above it failed, depending on whether the job was ever admitted.
Vendors pitch Kueue, Volcano, and KAI Scheduler as competing answers to one problem. On an incident like that, the issue is which layer broke. An open-source licence settles software fees, but never guarantees operational accountability when systems fail.
You need to know which layer is broken, which tool addresses it, and what evidence would make you reverse the decision in two years. Every evaluation here is relative to the full 1.37 workload management picture in Kubernetes v1.37 “Garhwal”, where in-tree gang scheduling ships as beta behind the disabled GenericWorkload gate.
Kueue vs Volcano vs KAI Scheduler — which problem does each actually solve?
Kueue answers admission by holding workloads until quota for every pod is reserved, then handing placement to whatever scheduler is configured. Volcano answers placement with its own scheduler binary. KAI Scheduler provides productised, GPU-dense placement, with entitlement governed separately by KAI Resource Management. Only Volcano and KAI place pods; Kueue does neither.
As Kueue’s admission concepts detail, the controller queues jobs in front of kube-scheduler. ResourceFlavor, ClusterQueue, LocalQueue, and Cohort abstractions turn heterogeneous hardware into schedulable classes and make quota a first-class object. Enforcing the all-or-nothing contract a queue has to satisfy, Kueue admits workloads atomically, preventing partial-allocation deadlocks.
Volcano operates at placement. As an incubating CNCF project, it replaces kube-scheduler for workloads requesting it. PodGroup custom resources and plugins provide gang scheduling, bin-packing, and dominant resource fairness (the share-of-the-bottleneck rule), introducing a second scheduling path.
KAI Scheduler handles placement for GPU-dense clusters, while KAI Resource Management governs projects and departments across an organisational hierarchy. Both are answers to a queueing question that only makes sense inside the wider Kubernetes workload retrofit, where the scheduler is one layer among several the release changed.
The admission and placement split decides what each tool can promise about queue economics. Queueing analyses show that Kueue borrowing lets teams reclaim idle quota with a repayment obligation, optimising cluster utilisation. KAI’s project and department hierarchy distributes share down an org chart, optimising accountability. Choose based on which failure you tolerate: stranded capacity or disputed entitlements. Read Kueue’s ResourceFlavor documentation beside KAI’s project model, because one acts like a market and the other like an org chart.
In-tree gang scheduling vs Volcano — is a beta feature gated off by default enough to replace a second scheduler?
A beta feature gated off by default is not enough to retire Volcano today. Kubernetes v1.37 ships gang scheduling as beta behind the disabled GenericWorkload gate. Kueue hands pods to kube-scheduler; the coscheduling plugin, not Kueue, handles basic all-or-nothing placement. Volcano still justifies operational overhead through plugin breadth, multi-queue semantics, gang-level preemption, and years of production hardening.
As documented in the Volcano v1.15 release, the project evaluates preemption at job granularity, simulates gang placement before committing evictions, and tunes bin-packing for GPU fleets, carrying production scar tissue that in-tree controllers have not accumulated.
Yet a second scheduler incurs operational drag: dual scheduling paths, webhooks, pod scheduling gates, and two places to look when a pod sits pending. Volcano v1.15 documents support for Kubernetes 1.35, sitting two minor versions behind at its last documented release against Kubernetes v1.37. Tracking Dynamic Resource Allocation (DRA) drivers and short-lived cycles such as CUDA 13.4 multiplies maintenance friction.
The beta label in Kubernetes v1.37’s workload scheduling update signals API stability, but a gate that stays disabled gets less production hardening. You should verify what actually shipped in the release on your clusters. If current incidents involve multi-team queue contention, keep Volcano while in-tree APIs mature. If your cluster runs straightforward gang shapes on a single scheduling path, waiting for native stability is defensible. A second consecutive release where GenericWorkload remains disabled while queue contention causes incidents would move you back to Volcano.
Open-source Run:ai vs Kueue vs building it yourself — what does each commit you to?
Open-source Run:ai commits you to NVIDIA’s release lifecycle and requires purchasing support separately. Kueue commits you to upstream Kubernetes release cadences and its specific quota models. Building an in-house scheduler commits your team to owning scheduling mechanics, driver upgrades, and on-call rotas.
Your commitment spans four areas: code footprint, control-plane components, upgrade cadence against Kubernetes and the CUDA toolchain, and the skills you must keep on payroll. Those four areas are exactly the layers that the placement-to-billing stack treats separately, which is why a scheduler choice cannot be judged on its own.
On 18 October 2026, NVIDIA open-sourced the Run:ai platform under Apache 2.0, incorporating KAI Scheduler. Confirm your team tracks the authoritative upstream at kai-scheduler/KAI-Scheduler rather than vendor mirrors. Upstream identity matters more than press releases.
A permissive licence does not provide an operational entitlement. As OpenSSF analyses explain, Apache 2.0 disclaims all warranties and liabilities. Industry reviews of KAI point out that community releases omit indemnification, guaranteed response windows, and validation for 500-GPU runs stalling overnight. NVIDIA acts as both upstream publisher and commercial support vendor. The invoice survives; only its line items changed.
Kueue commits you to upstream kubernetes-sigs cadences. CNCF governance reduces single-vendor lock-in, but your team still manages driver lifecycles and cost allocation. Building an in-house scheduler carries the highest operating cost, forcing you to maintain custom preemption logic. Check who pays when the queue leaves a GPU idle before writing custom scheduling code.
A ledger implies someone on the other side of it, which makes project durability the real test.
How do I assess whether an open-sourced scheduler will still be maintained in two years?
Audit project durability across five public signals: independent foundation governance, contributor diversity beyond one employer, an uninterrupted release cadence, an open enhancement proposal process, and commercial reliance on the open core. Pair these with driver compatibility testing against CUDA 13.4. Do not confuse code availability with community stewardship.
Governance structure provides an initial filter. As open governance guides observe, foundation ownership establishes legal neutrality, but incubation tiers matter. Volcano operates as an incubating CNCF project. Kueue sits under Kubernetes SIG Scheduling within the CNCF umbrella, offering vendor-neutral governance across releases. KAI Scheduler sits in the CNCF Sandbox under Linux Foundation stewardship, which is an early governance stage; Sandbox indicates donated code and IP, not yet shared merge rights.
Examine repository health directly. Research into open-source stability demonstrates that unstable projects concentrate over half their commits among their top three contributors, whereas stable repositories keep that share below 50%. Watch for warning signs: commit share dominated by one employer, advanced features reserved for commercial tiers, documentation referencing private portals, and changelogs that stall after launch.
GPU schedulers add one maintenance challenge: driver compatibility. A project can merge bug fixes regularly while falling behind Dynamic Resource Allocation drivers and the hardware sharing layers below it. If a scheduler lags short-lived driver branches required for GPU fractioning support, it becomes unusable on modern cards.
The deciding question is who fixes the control plane when it deadlocks at two in the morning.
Conclusion
A permissive software licence answers whether you owe a vendor royalty fees. It never answers who is on the rota when it deadlocks at two in the morning. When a multi-worker training job half-starts, resolving the issue depends on having diagnosed your architecture correctly, from admission control through to cost attribution.
Set a calendar reminder to re-evaluate your scheduling stack annually against public commit health, driver compatibility, and production stability. If the GenericWorkload gate stays disabled through another release while your incidents persist, that is the signal to re-run the comparison. Your team’s operational boundaries and the support contracts behind them decide whether a stalled training job is an incident or an outage. The licence header decides neither.
Frequently Asked Questions
Can I run Kueue and Volcano together, or do I have to choose one?
You can, and often should, because they occupy different layers. Kueue admits the job and hands it to whichever scheduler is configured, while Volcano places the pods; set the pod template’s schedulerName to Volcano and let Kueue own the queue. The cost is the one already named: two control planes to upgrade and two places to look when something sits Pending.
Does Kueue allocate GPUs, or do I still need a device plugin or DRA driver?
Kueue never allocates a device. It reserves quota for the pod set and hands the pods to a scheduler, and the actual GPU assignment comes from the device plugin or, increasingly, Dynamic Resource Allocation. That split explains a queue that looks healthy while pods still fail to start: admission succeeded, allocation did not. Check the DRA driver’s CUDA 13.4 support before blaming the queue.
How do I tell whether a Pending GPU pod is an admission problem or a placement problem?
Read the Workload object first. If it is still pending with a quota or borrowing message against the ClusterQueue, nothing has been admitted and the fault sits upstream of scheduling. If the Workload is admitted and the pod is still Pending, read the scheduler’s events for insufficient resources, affinity or taint failures. Two objects, two answers, and the distinction decides who gets paged.
Is gang scheduling the same thing as quota?
No. Gang scheduling is an all-or-nothing placement contract: every worker starts or none does. Quota is an accounting contract covering how much of the cluster a team may claim and when it may borrow. Kueue supplies the second and leaves the first to a scheduler, which is why swapping Volcano for in-tree gang scheduling does not also hand you multi-team quota semantics.
What happens if one pod in a gang cannot be placed: does the job keep the GPUs it already grabbed?
Under a working gang contract, no: the whole group is held back and any placed pods are released, so the cards return to the pool. The half-started 16-worker job is the symptom of a stack where that contract is missing or bypassed, usually because pods were created outside the queue or the gang plugin was never enabled.
Is KAI Scheduler a drop-in replacement for kube-scheduler?
Not quite. It is a GPU-first scheduler that can run alongside or instead of the default path, and it brings its own project and department model for entitlement. That model is the real migration cost, because your quotas move from Kubernetes objects into a hierarchy, making the change organisational as much as technical. Pilot it on one department before rewriting every manifest.
Do I need any of these three if I only run single-pod inference?
Probably not all three, and possibly only one. Single-pod inference needs no gang contract, so Volcano’s main advantage disappears. If several teams share the cluster, Kueue still earns its place as the admission layer; if one team owns every GPU, the default scheduler plus a device plugin is defensible. Add a queue when contention appears, not before.
What is a Cohort in Kueue, and when should I create one?
A Cohort is the group of ClusterQueues that may borrow from each other, which is where the market analogy becomes literal. Create one when teams have uneven, bursty demand and you would rather lend idle quota than strand it. Do not create one between teams whose chargeback disputes you are unwilling to arbitrate, because borrowing creates exactly that argument.
Do I still pay for Run:ai now that it is open source?
Yes, if you want support, validation at your scale, or someone accountable when a 500-GPU run stalls overnight. Apache 2.0 removed the licence fee question, not the invoice: what you buy now is response time and indemnity rather than the right to run the code. Budget for that line explicitly, or budget for the on-call rota instead.
If the vendor walks away, can we just fork it and support ourselves?
Legally yes, practically rarely. A fork inherits the code and none of the hardening, so you would carry Kubernetes release tracking, CUDA 13.4 and DRA driver changes, plus patches for your own scale. That is a platform team commitment, not a weekend. Fork readiness is an argument for contributor diversity, never a substitute for it.
How can I tell whether the open-source core is the real product or a teaser?
Compare the changelog against the paid tier’s release notes across two or three cycles. If features appear commercially first and reach upstream a year later, or never, the open core is a funnel. If the upstream repository is where the work happens and the commercial layer adds support and scale validation, the arrangement is durable enough to build on.
Which of the three is the least bad commitment for a small team without a platform engineer?
Usually Kueue, because it adds a queue without adding a second scheduling path. Volcano and KAI both place pods, which means a control plane you must upgrade and debug yourselves. If you genuinely have no platform engineer, the honest answer may be a managed service, since all three open-source options transfer the 2am pager to whoever signs the architecture decision record.
