Your LLM bill is full of work you’ve already paid for. Every call sends the same system prompt, tool schemas and few-shot examples, and pays full input price for them. Then a near-identical question arrives phrased slightly differently and triggers a full model invocation again.
That waste compounds as traffic grows, which is why LLM bills keep rising even as token prices fall. Caching is the fix, but it’s two layers. Prefix caching reuses the computed key-value state of a shared opening; semantic caching reuses whole answers for similar questions. Run both at the LLM gateway and you cut input cost, latency and redundant calls, while thresholds, namespacing and TTLs protect quality.
How does prefix caching cut inference costs on LLM workloads?
Prefix caching stores the key-value (KV) attention state computed for a prompt’s leading tokens and reuses it across every request that shares a byte-identical opening, such as system prompts, tool schemas and few-shot examples. The shared prefix is paid for once instead of on every call, lowering input-token cost and time-to-first-token without changing a token of output.
KV caching already runs inside every request, building attention state once during prefill and reusing it token by token as the model generates output. Prefix caching extends that reuse across requests. Prefill is the expensive phase: all-pairs attention scales quadratically with input length, and most of a prompt is boilerplate. A support bot’s 3,000-token instruction block can sit above a 50-token question. That repeated work is the redundant prefill tax.
A prefix only reuses if it matches byte for byte; a trailing space forces full recomputation. Tool definitions, instructions and reference documents sit first, user content last, which keeps the shared opening stable from request to request. vLLM’s Automatic Prefix Caching documentation is the standard reference for the mechanism.
The savings are measurable: Anthropic charges $0.30 per million tokens on cache reads versus $3.00 for standard input. Self-hosting has a wrinkle: a round-robin load balancer scatters identical prefixes and destroys the hit rate. Prefix-aware routing keeps same-opening requests on one instance, and that build-vs-buy decision has its own trade-offs.
That is the whole trick, and its ceiling: a prefix pays off only when the opening is byte-identical, which does nothing for the near-identical questions your users actually ask.
What is semantic caching and how is it different from a normal cache?
Semantic caching stores past prompt-response pairs and retrieves them by meaning rather than exact string match. A new query is embedded and matched to the nearest stored entry by cosine similarity over a vector store. A normal cache is exact-match and byte-identical; a semantic cache is similarity-matched, so it skips the model call for paraphrases, but every hit is probabilistic.
A normal cache hashes a normalised request and returns a stored response only on a hash match, safe but low-yield. Semantic caching embeds the query, finds the nearest neighbour in a vector store, and serves it when similarity clears a threshold. “How do I reset SSO?” and “How do I reset single sign-on?” map to one answer.
The catch sits in that word “meaning”. Embeddings capture topical similarity, not logical equivalence, so the capitals of France and Germany land close together in embedding space yet demand different answers. A semantic hit claims two prompts mean the same thing, and when it’s wrong, the answer is confidently wrong with no error signal. That silent quality regression is why semantic caching needs governance beyond a threshold dial.
How do you set a semantic cache similarity threshold without letting bad hits through?
The similarity threshold is a precision and recall dial. Low catches more paraphrases but admits wrong matches; high is safe but rarely fires. Tune it per route based on how dangerous a false hit is, and back every hit with guards beyond the cosine score.
A low threshold maximises hit rate and false hits together; a high one may fire so rarely the layer stops paying for itself. The threshold is an empirical choice: labelled pairs of prompts that should and should not share an answer are swept across the value, and the setting lands where false hits reach an acceptable rate for that route. A support-ticket paraphrase tolerates more risk than a clinical answer.
A false miss costs tokens; a false hit costs credibility, because nothing in the pipeline raises an error when a wrong answer is served. Confidence scoring blends similarity with freshness, so an aged entry falls back to the model. Never cache empty or error responses, and scope lookups by tenant, model and prompt version.
Semantic caching vs exact-match caching for an LLM gateway?
Exact-match caching is safe, cheap and low-yield: it returns a stored response only on a byte-identical hash match. Semantic caching is higher-yield but demands a correctness apparatus. An LLM gateway runs both as a layered pipeline and enforces caching policy above individual model APIs.
The gateway is where caching policy lives: model APIs cannot do semantic caching, and provider prompt caches vary in thresholds, TTLs and pricing, so the gateway owns cache keys, per-route thresholds and per-tenant scoping.
The layout is a layered pipeline: exact match, semantic match, prompt cache, then the model, each with a clear responsibility. Redis for exact-match and a vector database for semantic reuse is a common hybrid. Prefix caching stays in the serving stack, while the gateway layers the other two above it. The KV-cache hit rate captures cost and latency in one metric.
Model routing vs prompt caching, which saves more on inference costs?
Model routing reduces cost per call by sending each request to the cheapest model that can handle it. Prompt and prefix caching eliminate calls, or call portions, by reusing computed state or stored answers. Caching wins when prefixes and paraphrases repeat; routing wins when requests are novel but vary in difficulty. Combined, the savings multiply.
Same bill, different levers. Routing sends each request to the right model and machine, paying off when requests are novel and vary in difficulty. Evaluations across OpenAI, Anthropic and Google show prompt caching cutting API cost by 41 to 80 percent.
Run both. Cache the repeats and route the survivors. Router economics explain the unit-price side, and the paradox of falling token prices explains why the bill keeps climbing anyway. Measure the cacheable fraction on your own traffic; the only hit-rate number that matters to your business is yours.
Wrapping it all up
Prefix caching removes the deterministic waste, the redundant prefill tax on every shared opening. Semantic caching removes the near-duplicate calls. The probabilistic layer stays safe only when governed: per-route thresholds, namespacing, TTLs and shadow testing catch a silent quality regression. Caching is a control-plane concern, enforced at the LLM gateway, and it compounds with routing. Inference cost control means eliminating redundant work and governing the one layer that can go wrong.
Frequently Asked Questions
What is the redundant prefill tax, and why is it so expensive?
The redundant prefill tax is the cost of recomputing the same prompt opening on every request instead of paying for it once. Prefill is the compute-bound phase where the model builds its key-value attention state, and its all-pairs attention scales roughly quadratically with length. Because most of a typical prompt is boilerplate such as system instructions and tool schemas, that repeated work dominates the bill.
Does caching change the model’s answers or hurt output quality?
Prefix caching does not change a single token of output, because it only reuses the key-value state of a byte-identical opening. Semantic caching is different, because a hit returns a stored answer instead of generating a fresh one. That is why thresholds, namespacing and expiry exist: to stop the cheap layer quietly becoming the wrong layer.
Is it true that a semantic cache can serve a confidently wrong answer?
Yes, and it is the most dangerous failure mode in the pipeline. A semantic hit is only a probabilistic claim that two prompts mean the same thing, so a paraphrase that looks close can receive an answer written for a different question. Nothing raises an error, which is why shadow testing and per-route thresholds matter more than raw hit rate.
What happens if I update my system prompt or knowledge base?
Cached answers can go stale the moment the underlying source changes, so invalidation has to be designed in from the start. Version your prompts and documents, then use time-to-live expiry so entries age out automatically. Change a policy or a price, bump the version, and the old entries simply stop matching.
Could two customers ever be served each other’s cached answers?
Not if lookups are namespaced, but it is a real risk in a naive setup. Scope every cache key by tenant, model and prompt version so a match can never cross a boundary. In a multi-tenant system this isolation is not a nice-to-have, because a leaked cached answer is a data exposure event rather than a quality bug.
Do I need both prefix caching and semantic caching?
They solve different problems, so most production gateways run both. Prefix caching removes the redundant recomputation of a shared opening, while semantic caching removes the model call entirely for a close paraphrase. Layering them, exact match then semantic match then prompt cache then the model, keeps each failure attributable and keeps the cheap layers cheap.
What is prefix-aware routing and why does it matter?
Prefix-aware routing sends requests that share the same opening to the same server instance, so the cached key-value state is actually found. Caches live in a specific instance’s memory, so a plain round-robin load balancer scatters identical prefixes across machines and destroys the hit rate. Routing on the prefix restores that locality.
Does caching reduce latency as well as cost?
Yes, and the latency win is often the one users notice first. Reusing a prefix skips the compute-heavy prefill phase, which lowers time-to-first-token, and a semantic hit skips the model call altogether. So the two layers cut the bill and the wait at the same time, rather than trading one against the other.
Is caching pointless if all my queries are unique?
Partly, and it is worth measuring before you invest. Prefix caching still pays off whenever prompts share a common opening, even if the user question at the end changes every time. Semantic caching earns its keep only when paraphrases genuinely repeat, so check the hit rate on real traffic before deciding how much governance to build around it.
Do falling token prices make caching unnecessary?
No. Cheaper tokens lower the unit price but do nothing about redundant work, and waste compounds with volume. If the same prefix is recomputed on every call, a lower price per token simply means you are wasting a smaller amount more often. Caching removes the duplicate work, which is why it keeps mattering as prices fall.
Which cache should I implement first?
Start with prefix caching, because it is deterministic and carries almost no quality risk. Stabilise your prompt ordering so shared context sits at the front, then measure the hit rate. Add semantic caching once you have the governance to run it safely: per-route thresholds, per-tenant namespacing and expiry. In that order, the savings arrive before the risk does.
Where can I find the canonical documentation for prefix caching and KV caching?
The canonical self-hosted reference is vLLM’s Automatic Prefix Caching and KV cache documentation, which explains how key-value state is stored and reused. SGLang and TensorRT-LLM implement the same mechanism if you are evaluating serving stacks. For managed caching, check each provider’s prompt-caching docs, because minimum token thresholds, expiry and pricing differ from vendor to vendor.
