You have probably sat through the pitch where someone promises each team a quarter of an H100. It looks tidy on a spreadsheet. Ten services using 15% of a card ought to fit across two devices. GPU utilisation across managed Kubernetes clusters averages near 5%, which makes sharing commercially tempting.
A fractional GPU describes how capacity is sold and scheduled. It says nothing about what one tenant can do to another.
MIG, time-slicing, and MPS operate on the silicon. HAMi and fractional GPU operators are Kubernetes schedulers sitting above them. Conflating the two layers mistakes accounting for protection. You need to understand the trust-boundary question running through the retrofit across the whole stack.
How do I choose between MIG, time-slicing, MPS, HAMi and a fractional GPU operator for multi-tenant inference and mixed training?
Pick your device mechanism first, because MIG, time-slicing, or MPS determines what the silicon enforces. Use MIG for predictable inference with dedicated memory, time-slicing for interactive notebooks, and MPS for high-throughput cooperative kernels where tenants already share trust. Mixed training requires dedicated devices.
Workload requirements dictate the right device primitive:
- Multi-Instance GPU (MIG) partitions compatible Ampere or newer datacentre GPUs into up to seven hardware instances with dedicated compute and DRAM paths. A profile like 1g.10gb on an A100 or H100 gives an instance its own streaming multiprocessors and memory controllers. Memory is isolated and faults stay inside the slice, though static profile geometry limits dynamic sharing.
- Time-slicing rotates CUDA contexts across one device. It suits bursty development and notebooks where duty cycles are low and latency spikes are acceptable.
- Multi-Process Service (MPS) routes multiple processes through one shared CUDA context, dispatching kernels concurrently. It improves packing for small kernels, but benchmarks show performance degrades by roughly 30% under memory contention.
Once the device mechanism is settled, choose your scheduler for how fractions are represented in Kubernetes. A fractional GPU operator makes fractions schedulable; it does not make them safe. Tools like Run:ai and KAI Scheduler provide cluster scheduling alongside a fractional operator that manages driver memory limits, while HAMi governs resources inside the container via library interception. Both inherit the isolation of whichever device primitive sits underneath.
The sharing mechanisms separate along clear technical lines:
- Multi-Instance GPU (MIG): Partitions silicon at the hardware level. Provides dedicated memory channels and cache paths. Faults stay isolated to the individual slice. Scheduled via the NVIDIA GPU Operator as discrete resource instances.
- Time-slicing: Interleaves CUDA execution over time. Provides no memory isolation across pods. The fault domain is the entire physical GPU. Scheduled via replica counts in the NVIDIA device plugin configuration.
- Multi-Process Service (MPS): Dispatches concurrent kernels into a shared context. Provides software memory limits without hardware isolation. A fatal error can crash the shared server process. Scheduled via experimental device plugin replica settings.
- HAMi: Interleaves execution with in-container memory and core ceilings. Lacks hardware isolation boundaries. Fault isolation is inherited from the underlying device mode. Scheduled via an admission webhook and custom scheduler extender.
- GPU Fractioning Operator: Interleaves execution with driver-enforced memory limits. Lacks hardware-partitioned paths. Fault isolation is inherited from the underlying GPU setup. Scheduled via Kubernetes custom resources alongside KAI Scheduler.
Mixed training is the hardest scenario. Distributed training requires full device memory and predictable bandwidth. Sharing a card with inference causes stalls and out-of-memory crashes. Frameworks like vLLM reserve most VRAM upfront for KV caching, blocking effective sharing. Keep gang-scheduled training jobs on dedicated hardware.
If a scheduler only inherits whatever isolation sits beneath it, you have to ask what that underlying layer actually enforces.
Why is time-slicing a utilisation mechanism rather than a security boundary?
The fault domain under time-slicing is the entire physical GPU. An out-of-memory error, memory leak, or driver crash in one container disrupts your neighbouring tenants on that node. Treat it as an oversubscription technique: it raises reported utilisation and leaves the trust boundary exactly where it was.
Under time-slicing, the NVIDIA device plugin advertises multiple virtual replicas for one GPU. There is no execution quantum or hardware priority scheduler. When one process submits back-to-back kernels, your pods wait.
Because all pods draw from an undivided memory pool, a single workload exhausting VRAM causes the next allocation from a neighbouring pod to fail. Those crashes are easy to misdiagnose as application bugs, leaving your engineers debugging spurious software errors. Increasing your replica count improves packing density while expanding the blast radius.
MPS improves concurrency by pooling processes into one shared context. Trust is unchanged, because the memory space is the same one. As documented by NVIDIA, a fatal client fault can put the MPS server into a FAULT state that terminates every co-located workload on that GPU.
The host stack broadens the risk. As demonstrated in 2024 by CVE-2024-0132 in the NVIDIA Container Toolkit, an attacker exploiting container permissions can escape to the host, accessing every attached GPU and container socket.
MIG restricts the device blast radius to an individual hardware slice, but the host operating system, driver, and container runtime remain shared. Side channels like MIGraine demonstrate that memory contention over Unified Virtual Memory can still leak workload signatures across instances. Because MIG moves the blast radius from the card to the slice, the remaining risk sits above the device.
What should I check before putting shared GPUs in front of tenants who don’t trust each other?
Five layers decide whether shared silicon is acceptable for untrusted tenants. If your contract guarantees only allocation fairness rather than data isolation, treat the infrastructure as shared, not isolated.
Do not let an infrastructure vendor answer a vulnerability at one layer with a promise from another. Work systematically through these checks:
- Device layer: Verify whether hardware isolates memory channels or simply interleaves execution. Ask for test data showing what happens to a neighbour’s allocation when one tenant deliberately exhausts memory. Benchmarks will not show you that.
- Host layer: The host kernel, GPU driver, and container runtime are shared by every tenant on the node. Determine your operational patch window for updates. Check who can deploy privileged containers, because a pod with host privileges sees every raw GPU device node on the machine.
- Confidentiality layer: Trusted execution environments and remote attestation verify GPU firmware and memory state before workloads launch. Confirm whether your partial-allocation setup supports attestation. Without cryptographic attestation, you cannot confirm that residual model data is erased between runs.
- Control-plane layer: Virtual clusters using vCluster isolate API servers, RBAC policies, and custom resource definitions to prevent cluster-level interference. However, vCluster secures the control plane and leaves device memory open, while MIG secures device memory and leaves the control plane open. Buy both, or state plainly which interference you are accepting.
- Contract layer: Read your hosting terms. Check whether your agreement promises isolation or only proportional compute time. Resolving scheduling conflicts and fairness is a queue problem that admission controllers can handle, but administrative fairness does not prevent data leaks. The missing per-pod attribution makes cross-tenant boundaries as difficult to police as shared infrastructure bills.
When your tenants genuinely distrust each other, separate physical GPUs or dedicated nodes remain the honest answer.
Conclusion
The hardware mechanism sets the trust boundary. The tenancy agreement only describes billing and capacity, and only the hardware can enforce isolation.
Use MIG when workloads require dependable latency and hard fault boundaries. Use time-slicing where tenants share trust and jobs are interactive, and reserve MPS for concurrent kernels needing maximum throughput. Keep mixed training out of shared pools.
When workloads belong to separate business units or external customers who require verifiable isolation, separate physical GPUs remain the sensible answer. That distinction is central to the rest of the agent workload retrofit.
Frequently Asked Questions
Does MIG make a GPU safe to share between untrusted tenants?
No, it changes the blast radius rather than removing it. MIG partitions the device into separate GPU instances with their own memory and fault domain, so a crash stays inside its slice. But the driver, container toolkit and container runtime remain shared, and documented cross-instance side channels such as MIGraine still apply. Treat MIG as a large improvement, not a trust boundary.
Is MPS more secure than time-slicing?
No. MPS packs more small kernels through one device, but every client shares the same memory space, so memory isolation is no better than time-slicing. It is worse in one narrow respect: a fatal client fault can put the MPS server into a FAULT state and take every attached client with it. Both are utilisation mechanisms, not boundaries.
Should I use vCluster or namespaces plus MIG to separate tenants on shared GPUs?
Neither replaces the other, because they isolate different layers. vCluster separates the API server, custom resources and controllers, which stops one tenant’s operator from reaching another tenant’s objects. MIG separates device memory and faults, and says nothing about who can patch a CRD. Buy both, or state plainly which interference you are accepting.
Does time-slicing get safer if I limit each tenant to a single replica?
No. Replica count changes how many schedulable units the card advertises, not what one context can reach. Every replica still maps onto the same device, the same memory pool and the same fault domain, so a leaked allocation or a wedged context still surfaces for co-tenants. A larger count improves packing and widens the blast radius at the same time.
What actually happens when one tenant’s pod runs out of GPU memory on a shared card?
Everyone on the node feels it. Under time-slicing or MPS there is no partitioned memory, so the exhausted tenant’s failed allocation and any OOM kill land in the shared pool, and neighbours see latency spikes and crashes they did not cause. Those failures usually get filed as model bugs. MIG contains the OOM inside the instance that caused it.
Can I combine these mechanisms, for example time-slicing inside a MIG slice?
Yes, and it is a reasonable pattern when you want a hard memory floor plus shared access to what remains. Time-slicing inside a MIG instance keeps the partition boundary while letting several pods share that slice. Be clear about what you have bought: the slice is still a device-level boundary, and the tenants sharing it inherit time-slicing’s fault domain.
Which GPUs support MIG at all?
Datacenter parts from Ampere onward, which means compute capability 8.0 or higher, with up to seven instances per card and a fixed set of profiles. Consumer cards and older generations do not offer it, so a node pool built on workstation GPUs cannot be made MIG-isolated by configuration. Check the profile table for the exact part before promising slices to tenants.
What is the difference between a MIG GPU Instance and a Compute Instance?
A GPU Instance is the memory slice; a Compute Instance is the set of streaming multiprocessors inside it. Profiles name both: 1g.10gb means one compute slice and 10 GB of memory. The geometry is fixed at creation, so size for the workload’s peak rather than its average.
Why does the GPU show high utilisation while every tenant says their jobs are slow?
Because utilisation counts busy time, not useful work. Time-slicing raises the reported figure by oversubscribing contexts onto one device, so the card looks saturated while each tenant waits its turn. High utilisation on a shared GPU is evidence of packing, not of health. Measure per-tenant latency and queue time before concluding the fleet is efficient.
Does confidential computing close the side-channel gap on shared GPUs?
It is the only mechanism that addresses co-tenant confidentiality rather than co-tenant fairness. TEE-mode GPU memory with attestation lets you verify GPU state before admitting a workload, which counters residual-data and cross-instance probing. It does not repair a shared toolkit or a shared driver. Ask whether it survives the partitioning you chose, and get the answer in writing.
Will Dynamic Resource Allocation make shared GPUs safer?
No. Dynamic Resource Allocation improves how devices and their attributes are requested and allocated, which helps with fractions, topology and accounting. It does not add memory isolation or shrink a fault domain; those are properties of the mechanism underneath. A better scheduler inherits whatever the device enforces, the same rule that governs HAMi and fractional operators.
