The Unsplittable GPU Bill: Why GPU Cost Attribution Breaks and Per-Second Sandboxes Fix It

A cloud invoice arrives with a single line item for accelerator nodes, and nobody in the room can say which team owns which slice of it. The blocker is GPU cost attribution, a primitive the container ecosystem has not shipped.

The confusion sits in the distinction between allocated capacity and used capacity. In a standard cluster, the scheduler records whole-device claims such as nvidia.com/gpu: 1. It assigns a physical card to a single pod, whether that pod runs heavy matrix multiplications or sits idle. The scheduler marks the GPU fully booked, while the hardware sees almost zero kernel executions. That gap between what you reserved and what you executed is your idle tax, and standard cloud billing leaves you to absorb it.

Two engineering models address this waste. On one side stands Kubernetes, using gang scheduling and node elasticity. On the other sits per-second sandbox billing, where you pay only for active execution time. The two models make different bets about where the accounting should live. Both sit downstream of the Kubernetes GPU retrofit in full, which changed how these workloads are placed without settling who pays for them.

Why does average GPU utilisation in Kubernetes sit near 5% when CPU averages 8% and memory 20%?

The percentage measures execution against a reservation you already paid for, so a device kept for a twice-daily job is correctly sized and almost always idle. In its 2026 State of Kubernetes Optimization Report, Cast AI analysed tens of thousands of production clusters and found average GPU utilisation hovering near 5% (Azure Kubernetes Service at 2%, Elastic Kubernetes Service at 5%, and Google Kubernetes Engine at 6%). CPU utilisation averaged 8% and memory 20%.

Three structural causes explain why GPUs sit lower than other resources:

1. No cheap overcommit. Unlike a CPU core, which the Linux kernel can throttle, time-slice and oversubscribe, a physical GPU offers no standard mechanism to share capacity safely. The Kubernetes API treats nvidia.com/gpu: 1 as an integer. An allocation of 0.5 fails API validation.
2. No rightsizing loop. The usual feedback loop, the Vertical Pod Autoscaler (VPA), observes CPU and memory usage over time and adjusts pod requests downward so estimates converge on reality. No equivalent feedback loop exists for GPU compute or video RAM (VRAM), so requests never move off the developer’s first guess.
3. Bursty agent workload load shapes. AI agent loops run in short spikes followed by long stretches of inactivity. A single-tenant deployment can show an active duty cycle under 25%, saturating hardware for a few seconds and idling while awaiting input or API responses.

Inference serving adds another nuance. Modern engines using PagedAttention algorithms (such as vLLM) pre-allocate memory pools for the KV cache at startup. The device appears fully allocated, yet compute occupancy can stay low because memory bandwidth binds before compute does. A device reporting low occupancy can still be correctly sized for throughput.

Treat claims such as Cast AI’s US$8,850 monthly cost for an on-demand H100 or cited 30% to 40% queueing savings with care. Check whether the metric measures active streaming multiprocessor time or broad context reservation, verify the baseline, and confirm whether the reference workload mirrors your traffic patterns. Currently, no public dataset tracks real-world agent arrival distributions, making circulating density benchmarks stack-specific anecdotes rather than universal rules. If whole devices leave capacity idle, the obvious fix is to share the hardware across multiple workloads.

What does a shared GPU still not give you in the way of per-pod metrics, rightsizing and cost splits?

Sharing improves density and removes per-pod attribution at the same time. The underlying driver counters aggregate across all active tenants on the card. When three pods share one physical GPU through time-slicing or NVIDIA Multi-Process Service (MPS), most exporters assign the full card’s activity to each container, so the shares add to 300% of a GPU that does not exist.

The core metric stack reflects this limitation:

• DCGM Exporter: The standard DCGM Exporter monitors device health, power draw, temperature and hardware errors. When run with pod-resource collection flags, it maps hardware metrics to Kubernetes metadata, but this mapping functions accurately only when a single pod occupies the device.
• GPU Fractioning Operator: Custom controllers, such as the NRI-based GPU fractioning operator, attempt to enforce memory boundaries and export normalised per-pod metrics. However, they report an estimated share of a shared truth, and compute attribution collapses when concurrent kernels run simultaneously.
• metrics.k8s.io: The standard resource metrics API that powers kubectl top contains no GPU counters whatsoever. The most expensive hardware in your fleet is invisible to your primary command-line tooling.

This gap leads straight to an accounting problem that the wider 1.37 workload management retrofit left open. Without per-pod metrics, you have no automated rightsizing signal to feed a controller. Without rightsizing, you cannot create a defensible cost split across cost centres. The practical fallback for internal showback is to bill teams based on VRAM residency via NVML process accounting, treating compute occupancy as an advisory signal. Before committing to a shared model, you need to establish whether those shared GPUs are safely shared at all.

Kubernetes with gang scheduling vs a serverless per-second sandbox platform for agent workloads?

When density costs you the measurement, the comparison stops being about price per GPU-hour and becomes about who owns the accounting. Neither architecture wins outright on cost; they solve different problems. Per-second sandbox platforms (such as Modal or Spheron) eliminate the idle tax by denominating cost in active execution seconds. If an agent task takes 40 seconds, you pay for 40 seconds of compute, and the bill attributes cleanly to the initiating user or team. You trade away host-level control and custom drivers in exchange for a granular bill and zero idle overhead.

Running self-managed Kubernetes means assembling these scaling levers yourself:

• Autoscaling Mechanics: Kubernetes 1.37 introduced beta scale-to-zero capabilities under KEP-2021, but the Horizontal Pod Autoscaler cannot evaluate CPU or memory when replica counts hit zero because there are no running pods to measure. Scale-to-zero requires a signal that exists independently of the pods, which is why external event scalers watching queues or request rates carry the workload.
• Node Elasticity: Node autoscalers such as Karpenter can launch targeted instances in 45 to 60 seconds, which improves on older tools but still introduces latency during sudden traffic bursts.
• Gang Scheduling: Multi-node agent or training jobs require atomic placement. Partial allocations waste expensive hardware while pods wait for peer ranks to clear, making the queue that decides who gets the device the main lever for managing admission.

The trade-offs between managing your own clusters and consuming per-second sandboxes break down across six dimensions:

• Cost Attribution: Self-managed Kubernetes relies on device-level estimation and manual showback joins, while sandbox platforms provide native per-second, per-tenant accounting.
• Cold-Start Profile: Kubernetes clusters need minutes for cold nodes and tens of seconds on warm pools, whereas sandboxes start in sub-second to several seconds.
• Isolation Model: Kubernetes shares a Linux kernel across namespaces or sandboxed runtimes, while sandboxes enforce microVM or hardened hypervisor boundaries.
• Elasticity: Kubernetes scales within node provisioning limits and warm pool buffers, while sandboxes scale directly to vendor pool limits.
• Infrastructure Control: Kubernetes gives you CUDA, driver and topology control, whereas sandboxes provide fixed execution runtimes and constrained networks.
• Operational Ownership: You own scheduling, scaling and on-call rotations, while the sandbox vendor manages cluster uptime and capacity.

The platform choice follows your accounting requirements. If your business requires cross-charging infrastructure costs cleanly to separate product lines this quarter, per-second sandboxes solve the problem by eliminating the shared asset. If your workloads demand custom drivers, private networking or gang-scheduled distributed runs, Kubernetes remains the only path, and attribution remains an internal engineering compromise.

The invoice resists splitting because an indivisible physical card is the wrong unit of cost for bursty work, and because nothing in the stack measures execution per pod. Choosing between clusters and sandboxes comes down to deciding where you want the complexity to live, balancing the placement and trust decisions that shape the bill against the cost of engineering your own attribution platform.

Frequently Asked Questions

How do I get per-pod GPU metrics in Kubernetes?

Run DCGM Exporter with the -k flag, or set DCGM_EXPORTER_KUBERNETES=true. That adds pod, namespace and container labels from kubelet pod-resources data, and it works only while one pod owns the device. Under time-slicing or MPS the labels describe who shares the GPU, not who used it. metrics.k8s.io has no GPU resource at all, so kubectl top cannot help. Check DCGM’s metrics documentation, the gpu-fractioning exporter docs and the metrics.k8s.io API reference before trusting any dashboard.

What is the difference between allocated and used GPU capacity?

Allocated is what the scheduler recorded; used is what the hardware actually did. Kubernetes books a whole device through nvidia.com/gpu: 1 whether or not a kernel ever runs, while DCGM measures kernels. The distance between the two is the idle tax, and no native view surfaces it. Ask which side of that gap a dashboard is reporting, because a cluster can show every GPU claimed and the fleet working at a fraction of capacity at the same time.

How do I assess a vendor’s GPU utilisation and savings claim?

Ask four questions before quoting the number. What is the utilisation definition: SM occupancy, memory residency, or any active context? What is the baseline: provisioned capacity or requested capacity? Is the saving from queueing or from discounting? Does the reference customer have your load shape? Cast AI’s roughly US$8,850 per GPU per month and its 30 to 40% queueing saving are claims with an unstated baseline, to be interrogated rather than repeated.

How do I verify a per-GPU-per-month figure before it enters a budget?

Establish five things: what the figure includes (device, node, storage, egress, idle reserve); whether it is reservation or on-demand pricing, and which of the AWS, GCP or Azure reservation models produced it; the committed term and the exit terms; who absorbs idle capacity; and the variance across the last four quarters. Then check the region. A p5.48xlarge runs around US$71.55 an hour on demand in ap-southeast-2 against US$55.04 in us-east-1, so a us-east-1 baseline will not survive a Sydney invoice.

Can I split the GPU bill by team without per-pod compute metrics?

Partly, and only if you are honest about which half is measured. Bill on memory and treat compute as a signal. NVML per-process memory attributes reasonably well, though driver-reserved memory means the numbers never quite sum to the card, and per-process compute share collapses when jobs run concurrently. Combine namespace tags, quota and a written allocation rule, then label the result showback rather than chargeback so nobody mistakes an estimate for an invoice.

Is MIG the answer to GPU cost attribution?

It is the closest thing to a divisible device inside one card, because each pod owns a hard partition and attribution becomes trivial. It does not make a shared device accountable; it removes the sharing. You then pay for stranded capacity, since a 12GB fine-tuning job claiming a 20GB slice on an H100 leaves roughly 8GB unused, and NVLink is disabled between instances, which rules MIG out for NCCL-based distributed training.

Do per-second sandboxes still make sense for always-on inference?

No. Per-second billing wins when capacity sits idle between jobs, because nothing exists to bill. A model serving steady traffic all day consumes the same seconds either way, and committed or reserved capacity is normally cheaper per hour at that duty cycle. The sandbox model fits bursty, session-shaped agent work. It does not repeal the economics of a device that is genuinely busy, and it should not be sold as though it does.

How does HPA scale to zero work, and why can’t it use CPU or memory?

Since Kubernetes 1.37, KEP-2021 is beta and enabled by default, so an HPA can take a workload to zero using an object or external metric. CPU and memory are excluded because they come from running pods: at zero replicas there is nothing to measure and no signal to scale back up. Queue length or request rate exists independently, which is why KEDA-style external metrics carry the load. Services do not buffer requests, so HTTP workloads need a durable queue.

What do memory QoS and node swap change about how many sandboxes fit on a node?

Both attack the memory floor that keeps a node full while its CPU sits idle. Memory QoS uses cgroup v2 controls to protect a pod’s working set while letting burstable memory be reclaimed; node swap lets inactive pages move to disk rather than pinning RAM. Together they lifted gVisor Python sandbox density from 80 to 240 pods on the same hardware. That is a single-stack result with no published arrival data behind it, so treat it as a demonstration only.

Should I move agent workloads onto Spot instances instead?

Only if the workload tolerates interruption and the capacity exists. Cast AI reports that under 2% of GPUs ran on Spot through 2025, with T4-only availability emerging in 2026, and that AWS raised H200 Capacity Block prices 15% in January 2026. Spot is a discount applied to the same reservation model, so it lowers the rate without fixing attribution, and it does nothing about the idle hours you have already paid for.

Which parts of this are actually generally available in Kubernetes 1.37?

HPA scale to zero is beta and on by default in 1.37, which means API support rather than a default policy. Workload-aware gang scheduling matured through 1.36. Node swap and memory QoS depend on your kubelet and runtime configuration, not on one release number. Verify each lever against the release notes for the version you actually run before it appears in an architecture diagram, and treat vendor slides as a starting point only.

What does the GPU bill look like for a session-shaped workload that runs two hours a day?

About 60 GPU-hours a month against 730 hours of reservation, so a reserved device bills roughly twelve times the hours you consume. At the US$2.64 an hour H100 figure Spheron quotes, that is about US$158 of metered work against roughly US$1,927 of reserved capacity. Per-second billing charges the 60 hours. Kubernetes charges the reservation unless you also scale the node away, which is the whole argument in one comparison.