If you’re building AI-enabled products, the AI inference market is where your money goes. Not training — inference. Running models in production. That’s $106 billion in 2025, heading to $255 billion by 2030, and it’s consuming 80–90% of all AI computing power on the planet. Training is a sunk cost you pay once. Inference is the meter running with every user request.
Three things are happening at once: hardware consolidation (NVIDIA just spent $20 billion acquiring Groq), provider economics that vary wildly depending on who you use, and $600 billion in hyperscaler capital expenditure locked in for 2026. The AI inference cost crisis isn’t a blip. It’s built into the economics of running AI in production. Here’s what’s actually going on and what it means for the decisions you need to make.
What is the current size of the AI inference market and where is it headed?
Grand View Research and MarketsandMarkets both put the AI inference market at $106 billion in 2025, growing to $255 billion by 2030. Inference has already overtaken training for the first time, sitting at 55% of cloud AI spend in early 2026. Average enterprise LLM spend hit $7 million per company in 2025 — nearly triple the $2.5 million from 2024. One CIO put it plainly: “What I spent in 2023 I now spend in a week.”
Here’s the dynamic you need to wrap your head around. Per-token inference costs dropped approximately 1,000× in three years — yet total inference spending grew 320% over the same period. Cheaper tokens just create more use cases and higher query volumes. Andreessen Horowitz calls it “LLMflation”: total bills go up because demand grows faster than costs fall. The $106B-to-$255B trajectory is rising spend, not falling costs. Plan for that.
What did NVIDIA’s acquisition of Groq mean for the inference hardware market?
On 24 December 2025, NVIDIA acquired Groq’s assets and licensed its inference technology for $20 billion — NVIDIA’s largest deal ever and the biggest consolidation event in AI inference hardware history.
It’s worth being precise about what this was. A licensing-and-acquihire, not a full corporate acquisition. NVIDIA took Groq’s chip assets and licensed the LPU (Language Processing Unit) designs, bringing on founder Jonathan Ross and President Sunny Madra. This mirrors Microsoft’s 2024 licensing of Inflection AI and is widely read as a deliberate move to sidestep mandatory antitrust review.
Groq’s LPU is purpose-built for inference. Independent benchmarks recorded Groq delivering 877 tokens/sec on Llama 3 8B — roughly 2× the throughput of the fastest alternatives at the time.
Before this deal, NVIDIA already held 90–95% of the AI accelerator market. Now it controls the most credible alternative inference chip architecture as well. The 2.9× premium NVIDIA paid over Groq’s September 2025 valuation tells you everything — LPU-style architectures genuinely outperform GPUs for specific inference workloads.
If you’re currently on GroqCloud: the service is still nominally operational, but long-term pricing and product roadmap under NVIDIA ownership are genuinely uncertain. Any infrastructure planning beyond 12 months needs to account for this.
How do AI inference gross margins compare across major model providers — and what does that tell buyers?
Gross profit per token varies a lot. DeepSeek 85%, Perplexity 60%, Anthropic 55%, Manus 50%, Together AI 45%, Groq 40%. These margins tell you whether current pricing is sustainable or subsidised — which matters quite a bit when you’re building production systems.
Traditional SaaS gross margins run 70–90% because software has near-zero marginal cost of delivery. AI margins average around 52% because inference requires continuous GPU compute with every single query. The marginal cost scales with usage in a way that seat-based SaaS simply doesn’t.
DeepSeek’s 85% gross margin is the most instructive number here. It’s achieved through architectural efficiency — a sparse mixture-of-experts design that activates fewer model parameters per inference pass. That’s a structural advantage, not a subsidised pricing scheme. The implication for the market is real: inference-efficient architectures work at production scale, which puts genuine pressure on providers running less-efficient models.
Anthropic’s Series F fundraise valued the company at $183 billion post-money, with run-rate revenue growing from roughly $1 billion to over $5 billion in under eight months at 55% gross margins. Meanwhile OpenAI’s compute margin jumped from around 35% in early 2024 to roughly 70% by October 2025.
Use margin data as a procurement signal. Providers below 45% have limited room to absorb cost increases — expect price pressure as they scale. Cross-reference with valuation multiples: a low-margin provider at a high valuation is pricing for growth rather than stability.
Why does hyperscaler CapEx keep increasing when token prices are already falling?
Hyperscalers committed $600+ billion in AI infrastructure capital expenditure for 2026 — a 36% increase over 2025. Amazon at $200 billion, Google at $175–185 billion, Microsoft at $145 billion, Meta at $115–135 billion.
Falling API token prices and rising infrastructure investment aren’t in conflict — they’re operating on different cost layers. Hyperscalers have to recoup data centre construction, GPU procurement, and energy costs regardless of what they’re charging per token.
Meta and Microsoft are building nuclear plants to power AI data centres. These are decade-scale commitments that have to be recovered through revenue. Energy is a floor cost — US data centres consumed 200 terawatt-hours in 2024, and AI inference is projected to consume 165–326 terawatt-hours annually by 2028. When energy and GPU memory cost more, cloud inference costs more. Simple as that.
AWS raised GPU Capacity Block prices by 15% in January 2026 with no announcement — on a Saturday. Cloud inference pricing does not fall as fast as per-token API rates suggest it should.
Do open-weight models like Meta Llama change the buyer’s negotiating position?
Open-weight models — AI models whose trained weights are publicly released for self-hosted deployment — function as a cost ceiling on proprietary API providers. If API pricing exceeds the all-in cost of self-hosting an equivalent open-weight model at your token volume, you have a rational exit path.
Meta’s Llama 3 series is the obvious example. Llama 3 provides GPT-4-class capability that you can deploy on your own or leased GPU infrastructure. Once workloads are steady and high-volume, self-hosted smaller models can reach cost parity with API-based large models faster than many teams expect.
Self-hosting isn’t free, though. You need GPU infrastructure, operational overhead, and model maintenance capability. Quantify your self-hosting breakeven token volume before you start using open-weight models as a negotiating lever.
AMD’s Instinct MI300X is the primary hardware alternative for buyers who want to avoid NVIDIA lock-in: 192GB HBM3, 5.3 TB/s memory bandwidth, and a 40% latency advantage over the H100 for large models. The CUDA moat is real though — CUDA has nearly two decades of investment baked into PyTorch, TensorFlow, and nearly every major AI framework. AMD’s ROCm 6.x has reached near CUDA parity but still requires more manual tuning. Model the switching costs honestly.
What do these macro market forces mean for a mid-market software company’s infrastructure decisions?
Hardware consolidation, elevated hyperscaler CapEx, provider margin variance, and open-weight model availability are not a transition phase. They are the permanent operating environment.
Don’t assume token prices will fall fast enough to solve the cost problem for you. The Jevons paradox, hyperscaler CapEx recovery requirements, and NVIDIA’s hardware dominance all work against rapid cost deflation at scale. Production costs scale 717× from proof-of-concept to production. That’s not an outlier — it’s the pattern.
Use provider gross margin data as a procurement signal. Providers below 45% are likely to raise prices as they scale; providers above 55% have more structural stability. Open-weight model availability is a real negotiating lever — but only if you’ve already quantified your self-hosting breakeven.
The teams managing inference costs well are treating it as an architectural concern, not a line item to be surprised by at month-end. For a detailed look at how these market forces affect your P&L, see the breakdown of why AI gross margins are structurally lower than SaaS. And for infrastructure decisions shaped by hardware dynamics — particularly how the NVIDIA/Groq consolidation and AMD’s hardware positioning should inform your deployment choices — the cloud vs. on-premises vs. hybrid decision framework covers the real cost data. Deeper guidance across all of these areas is available in what the AI inference cost crisis means for your business.
Frequently asked questions
What is the AI inference market and why does it matter to software companies?
The AI inference market is the ecosystem of hardware, software, and services that enables AI models to run in production. Inference accounts for 80–90% of lifetime AI product costs — training is a sunk cost paid once; inference scales with every user request. At $106 billion in 2025 growing to $255 billion by 2030, it’s the operational cost structure every AI-enabled product faces.
Does the AI inference market growth trajectory mean costs will eventually fall?
Per-token prices dropped 1,000× in three years, yet total inference spending grew 320% over the same period. The Jevons paradox means cheaper tokens drive more consumption, pushing total spend up. Structural factors — hyperscaler CapEx recovery, NVIDIA dominance, rising energy costs — create a floor that limits how fast cloud inference pricing can actually fall.
What exactly did NVIDIA acquire from Groq, and was it a full acquisition?
NVIDIA executed a licensing-and-acquihire of Groq’s core IP and key personnel — LPU architecture designs, founder Jonathan Ross, and President Sunny Madra — for $20 billion. It was not a full corporate acquisition; Groq as an entity continues nominally and GroqCloud remains operational. The structure was designed to reduce antitrust exposure.
What happened to GroqCloud after the NVIDIA acquisition?
GroqCloud remains nominally operational with Groq’s former CFO stepping into the CEO role. Long-term pricing trajectory and product roadmap under NVIDIA ownership are uncertain. If you’re using GroqCloud for production workloads, evaluate alternative providers and revisit pricing assumptions for any planning beyond a 12-month horizon.
Is DeepSeek’s 85% gross margin a sustainable model or an exception?
It appears sustainable. DeepSeek’s margin is driven by a sparse mixture-of-experts architecture that activates fewer model parameters per inference pass — a structural advantage, not subsidised pricing. The implication is that inference-efficient architectures are viable at production scale, which creates competitive pressure on less-efficient providers.
Why does Anthropic’s $183B valuation matter for companies budgeting AI inference costs?
Anthropic grew run-rate revenue from roughly $1 billion to over $5 billion in under eight months, with 55% gross margins and over 300,000 business customers. A provider with that growth trajectory and margin profile is less likely to make sudden pricing changes than a lower-margin competitor under pressure.
Why are AI gross margins lower than traditional SaaS margins?
Traditional SaaS margins run 70–90% because software has near-zero marginal cost of delivery. AI margins average around 52% because inference requires continuous GPU compute with every query — the marginal cost scales with usage. AI companies that price like SaaS face margin compression at scale.
How much are hyperscalers spending on AI infrastructure and should buyers care?
Hyperscalers committed $600+ billion in AI infrastructure CapEx for 2026, a 36% year-on-year increase. Amazon at $200 billion, Google at $175–185 billion, Microsoft at $145 billion. This capital has to be recovered through inference revenue — which is why cloud pricing doesn’t fall as fast as per-token rates suggest it should.
Is AMD a real alternative to NVIDIA for running AI models in production?
AMD is credible but constrained. The Instinct MI300X delivers 192GB HBM3, 5.3 TB/s bandwidth, and a 40% latency advantage over the H100 for large models. ROCm 6.x has reached near CUDA parity for major frameworks but still requires more manual tuning. Model the CUDA switching costs before you make any hardware decisions.
What does the open-weight model trend mean for AI provider pricing power?
Open-weight models function as a cost ceiling for proprietary API providers. If API pricing exceeds the all-in cost of self-hosting an equivalent open-weight model at your token volume, you have a rational exit path. The leverage is conditional on having the infrastructure capability to self-host.
How do I know if my AI provider’s current pricing is sustainable or likely to increase?
The gross profit per token margin is the most accessible signal: providers below 45% (Groq at 40%, Together AI at 45%) have limited room to absorb cost increases. Providers above 55% (Anthropic, DeepSeek, Perplexity) have structural flexibility. Cross-reference with valuation multiples — a low-margin provider at a high valuation is pricing for growth rather than sustainability.
What is the Jevons paradox and how does it apply to AI inference costs?
The Jevons paradox describes how increased efficiency leads to greater total consumption, not less. In AI inference, per-token prices dropped 1,000× in three years while total spending grew 320% because cheap tokens enable more use cases and higher query volumes. Plan for total inference spend to rise even as unit costs fall.
