Local inference has crossed the “good enough” threshold. In demos, a roughly $5,000 DGX Spark runs an agentic workload entirely on-device, with no cloud round trip.
None of this is an exodus from the cloud. The economics only pay off at a deliberate split, and DRAM prices rose around 500 per cent in twelve months. Moving agents onto endpoints also expands your attack surface. The counterweight is as real as the headline numbers.
So the placement question, where each workload actually runs, is the one to answer. The decision has moved from which model to where each job lives. Three forces decide every answer: cost, trust and regulatory fit. The four sections below map the whole picture, each closing with a link into the deep dive that covers it in full — such as how to decide where AI runs and what local AI actually costs.
Why Has On-Device AI Crossed the “Good Enough” Threshold?
Two changes arrived together. Small models gained enough capability for real task-specific work, such as classification, extraction, summarisation and tool use, while quantisation matured into a dependable way to fit those models onto hardware you already own. The result is local inference that runs production workloads with no cloud round trip. Together those shifts moved on-device AI from demo to a practical choice, though the fit is still decided workload by workload.
The numbers back it up. A 2025 longitudinal study found the share of queries a local model could serve jumped from 23 per cent to over 80 per cent in two years. A 3B vision model released this year scores 80.7 on a screen-understanding benchmark and runs in-browser through WebGPU with no server behind it. Precision reduction is what makes any of this fit at all, though there is an accuracy cliff at the far end of that trade. The question worth asking is which of your workloads now fall on the local side of the line. How far that line extends is a hardware question, taken up further below.
Read the full argument in why on-device AI is finally good enough.
How Do You Decide Where a Workload Runs — and What Leaves the Machine?
Placement is a governance decision first. Start by asking what can leave the machine: prompts, embeddings, telemetry, metadata. Then check whether that data falls under laws like the US CLOUD Act, which can reach a Sydney server when the provider is US-owned. After that, weigh trust, audit evidence and cost per workload, in that order. The honest answer is almost always a split, sequenced by trust and regulatory fit.
Infrastructure can run almost anywhere, so the binding constraints are data residency, auditability and the CLOUD Act’s extraterritorial reach. The endpoint trust boundary marks where an agent and its data sit outside your control, and crossing it costs you the central logging and audit evidence your business depends on. If your data falls under Australian sovereignty rules, that test is the filter for borderline workloads, and a vendor’s actual egress is the question to press before you sign.
The full framework is in deciding where each workload runs.
What Does Local AI Actually Cost?
Local AI swaps per-token billing for up-front hardware, power, maintenance and engineering time, and its economics hinge on utilisation. At a deliberate split, AMD’s 500-machine model projects 40 to 60 per cent three-year savings, but a roughly 500 per cent twelve-month rise in DRAM prices is compressing that window. The useful question is: at what blend, and for which workload, does local stay cheaper?
“Zero token cost” is the marketing version. The true cost includes capex, power, ops, engineering time and, above all, utilisation, and low utilisation is where local economics break. The memory-cost collision compounds it: DRAM’s 2026 rise, plus hyperscalers pre-buying 2027 supply, shortens the window in which capex-for-opex pays off. Behind every placement call sits the latency-privacy-cost triangle, where no single placement maximises all three.
Work through the arithmetic in the true cost of local AI.
What Can Local Hardware Actually Run?
A roughly $5,000 DGX Spark-class box runs small models at useful speed for a single user, but it hits a concurrency wall when many requests queue at once. Memory bandwidth and KV-cache pressure make latency degrade non-linearly under load. That is why hybrid cloud-edge inference exists: simple queries stay local, hard or high-volume ones escalate to the cloud. Treat the box as a capability ceiling.
The verdict on any local box is what fits. Model size, context window and tokens per second within VRAM headroom define the practical envelope. Clustering, like NVIDIA’s PAIR pooling idle home machines, narrows that wall without removing it. Hybrid cloud-edge inference is the release valve: a routing layer keeps simple queries local and escalates the rest through complexity-based routing or confidence-based cascading.
Scope the envelope in what local hardware can actually run.
Resource Hub: On-Device AI Deep Dives
Understanding the Shift
- Why On-Device AI Is Finally Good Enough for Production Workloads: what on-device AI and the local-first shift are, why the threshold moved, and how small models compare with cloud models. About 6 minutes.
- What Local AI Hardware Can Run and Why Hybrid Inference Exists: what a roughly $5,000 appliance realistically runs, why concurrency is the hidden limit, and how hybrid cloud-edge inference works. About 6 minutes.
Making the Decision
- How to Decide Where AI Runs and What Data Leaves the Machine: why placement is a governance decision, how to assess the endpoint trust boundary, and what to look for in a vendor’s egress. About 7 minutes.
- The True Cost of Local AI and the Latency Privacy Trade-Off: the capex-for-opex arithmetic, the DRAM price shock’s effect, and how to balance latency, privacy and cost. About 6 minutes.
Suggested reading order: start with “Why On-Device AI Is Finally Good Enough” to validate the shift, then “What Local AI Hardware Can Run” to scope the ceiling, then “How to Decide Where AI Runs” and “The True Cost of Local AI” to make the placement call.
Where to Start
The four dimensions come back to one idea: placement is sequenced. You route each workload to where cost, trust and regulatory fit say it belongs, and you re-run that call as the numbers shift.
If you are still validating whether the shift is real, start with the case for on-device AI. If you are already convinced and weighing placement, where each workload runs and what leaves the machine is the heart of it.
Costing a move? Run the arithmetic in the arithmetic behind a local build. Scoping hardware? Check the hardware envelope.
Frequently Asked Questions
What does “on-device AI” actually mean?
On-device AI means the model runs on hardware you control, such as a laptop, workstation or appliance, instead of a remote data centre. Nothing about the request leaves the machine unless you choose to send it. The local-first shift is the move to place a share of production workloads there while still routing the rest to the cloud. It is a placement choice, not a wholesale migration.
Is on-device AI the same thing as edge computing?
Not quite. Edge computing is a broad category that includes telco nodes, retail kiosks and gateway servers, many of which still send data onward to a provider. On-device AI is stricter: the model and the data stay on the endpoint the user is holding or sitting at. The distinction matters because a local device offers a cleaner trust boundary than an edge node you do not own.
Do I need a GPU to run AI on my own machine?
A GPU helps, but it is not strictly required for smaller jobs. A 3-billion-parameter model can run acceptably on a modern laptop using CPU or integrated graphics, while the 7 to 13 billion range realistically wants a discrete GPU with enough VRAM. The verdict on any box is what fits, so match the model to your silicon before buying.
What is quantisation, and does it lower answer quality?
Quantisation shrinks a model’s numbers from 16-bit precision down to 8-bit or 4-bit so it fits in far less memory. Quality holds up well through moderate reduction, which is why quantised small models now handle classification, extraction and summarisation reliably. Push too far, though, and you hit an accuracy cliff where reasoning and nuance degrade noticeably.
Is local AI automatically private?
No. Running the model locally keeps inference traffic on the device, but telemetry, crash reports, update pings and cloud-connected features can still leave it. Privacy depends on what your stack actually egresses, not on where the model sits. That is why the placement question starts by asking which data, including prompts, embeddings and metadata, can leave the machine at all.
What happens when a local model runs out of memory?
Performance falls off a cliff. When a model and its KV cache exceed available VRAM, the system either spills to slower system memory or refuses the request outright, and latency degrades non-linearly as queues build. This is the concurrency wall: one user is fine, many simultaneous requests are not. The fix is fewer concurrent requests or a route to the cloud.
Is local AI slower than the cloud?
Not necessarily, and often the opposite for a single user. A local model skips the network round trip entirely, so time to first token can be excellent on a capable box. The trade flips under load: once requests queue past the concurrency wall, throughput falls behind a provider that can spread work across many accelerators.
Can AI really run inside a web browser with no server?
Yes. A 3-billion-parameter multimodal model can run entirely in-browser through WebGPU, with no server behind it. The model weights download once and inference happens on the user’s own GPU. It is a genuine zero-egress option for lightweight tasks, though the browser’s memory limits keep it to small models.
Which tasks should stay in the cloud?
Anything high-volume, latency-insensitive, or dependent on very large models belongs in the cloud, along with workloads whose audit trail a regulated buyer expects centrally. Simple, private, low-concurrency jobs are the natural local candidates. The result is almost always a split, sequenced by trust and regulatory fit rather than a full migration either way.
What is the KV cache, and why does it limit local AI?
The KV cache stores the key and value tensors for every token already processed, so the model does not recompute them each step. It grows with context length and concurrent users, and it competes for the same VRAM the model weights occupy. That pressure is what makes latency degrade non-linearly under load and sets the concurrency wall.
Does on-device AI still work without an internet connection?
Yes, that is one of its clearest advantages. Once the model is downloaded and running locally, inference needs no network at all, which suits field work, air-gapped environments and privacy-sensitive tasks. The caveat is that any hybrid routing, cloud escalation or licence check that depends on connectivity will simply stop until the link returns.
