Semantic Caching for LLMs: Thresholds, Hit Rates, Pitfalls
How a semantic cache for LLMs works at the AI gateway, what a similarity threshold really trades away, and the pitfalls we measured on 5,000 query pairs.
Microsoft's reference page for the Azure API Management semantic cache policy carries a warning most teams scroll past. Because the cache matches on similarity, not exact text, it "can surface responses that are incorrect, outdated, or unsafe for the current request". It is the most honest description of semantic caching we know, and it is where the design work should start.
Semantic caching is a real cost lever on the right routes and a liability on the wrong ones. This article is for the people who decide which is which: platform owners, AI architects and CTOs. We walk the request path, then show what a similarity threshold trades away, using an experiment you can rerun on a laptop.
TL;DR / Key takeaways
- A semantic cache replays someone else's answer. It skips the model when a new prompt is close enough to a cached one. That saves tokens and latency, and it can be confidently wrong.
- The threshold is a trade, not a setting. On 5,000 labelled query pairs, a cosine threshold of 0.95 served only 28% of genuine duplicates. At 0.85 it served 80%, but 32% of its answers went to pairs labelled as different questions.
- Near misses are the real danger. Pairs that swap one entity or reverse a role scored 0.95 to 0.997. "Gold medallist" against "silver medallist" scored 0.973.
- Thresholds are not portable. At 0.90, a second embedding model served 16 points fewer duplicates (41% against 57%), and every product measures the threshold in different units.
- Partition first, then cache. Key by tenant, model and prompt version, bypass agents and personal data, and run the cache as a GDPR-relevant data store.
Download: Editable draw.io file · Open in diagrams.net · Chart data (CSV)
Why this matters now
The gateways caught up. Azure API Management ships llm-semantic-cache-lookup and llm-semantic-cache-store on all tiers, covering OpenAI Chat Completions and Responses, the Anthropic Messages API (v2 tiers) and Google Vertex AI. LiteLLM supports Qdrant, Valkey and Redis as semantic backends, and Kong has an ai-semantic-cache plugin. Semantic caching is now a policy you switch on, which makes it easy to switch on in the wrong place.
The evidence also got better on both sides. The MeanCache authors (IPDPS 2025) report that repeated queries make up about 31% of user queries. That is the prize. The vCache paper (ICLR 2026) shows that one static threshold for all prompts gives no correctness guarantee. An ICML 2026 paper reports an 86% hit rate for deliberately crafted prompts that hijack cached responses. That is the risk.
There is also a cheaper, safer lever. Provider prompt caching reuses the processing of an identical prompt prefix. Anthropic's standard rate for cache reads is 0.1× the base input price (lower on some newer models), and OpenAI describes cached input as discounted by up to 95%. The model still writes a fresh answer, so prompt caching cannot be wrong. Exhaust it first; our LLM token cost engineering guide shows how.
The request path, step by step
Numbers match the hero diagram. We use Azure API Management as the example; the logic is the same in LiteLLM, Kong or your own code. For the gateway itself, see our AI gateway pattern on Azure.
① Partition key
Decide who may share answers before any similarity search. In API Management this is the vary-by element, and Microsoft recommends user or user-group identifiers to control cross-user access. We key on tenant, model deployment, prompt-template version and, where answers depend on permissions, user group. A key that is too coarse is the classic failure: one customer's answer served to another. Shared caches also leak through timing: an IEEE TIFS paper showed that response-time differences in semantic caches can reveal other users' prompts.
② Eligibility gate
Most traffic should never reach the cache. Bypass it for longer dialogues (API Management's max-message-count), tool calls and agent loops, prompts carrying personal data, and anything live such as prices or status. LiteLLM's documentation is blunt: consecutive agent turns embed about 0.99 similar, its semantic cache "will replay stale responses", and raising the threshold "does not reliably fix this".
③ Embed the prompt
Every lookup costs an embedding call, hit or miss. Embed only the user's text. API Management's ignore-system-messages defaults to false, and Microsoft recommends true, so a long shared system prompt does not dominate the score. Use one embedding model per index, and treat a model change as a schema change.
④ Vector lookup and threshold
The gateway takes the nearest cached prompt inside the partition and compares its score with the threshold. The units are a trap. API Management's score-threshold runs from 0 to 1, lower is stricter, and Microsoft suggests starting around 0.05, warning that values above 0.2 may cause mismatches. RedisVL's distance_threshold is a cosine distance from 0 to 2, also lower is stricter. LiteLLM's similarity_threshold is a cosine similarity, where higher is stricter. Never copy a value between products.
⑤ Store on miss
On a miss the model answers, and the outbound policy stores the response with a TTL in seconds. API Management caches only 200 OK responses by default. Store only what you would replay to everyone in the partition: nothing that failed a content-safety check, nothing built from a user's private context. One bad entry is served many times, which is exactly what the key-collision attack exploits.
⑥ Return and observe
Every response should say whether it came from the cache, and how closely it matched. Kong sets X-Cache-Status; LiteLLM sets x-litellm-semantic-similarity. Log the score, partition and versions, and give users a way to flag a wrong answer. Otherwise you cannot tune the threshold. Our AI agent observability guide covers the tracing.
⑦ Invalidation
A cache needs four ways to forget: a TTL per route, a new namespace whenever the model, prompt template or safety policy changes, a purge when a source document changes, and a targeted delete for erasure requests and wrong-answer reports. Most teams build only the first.
What the numbers say
We wanted real numbers, not vendor claims. We took the GLUE QQP validation split of crowd-labelled Quora question pairs and drew 5,000 pairs with a fixed seed, 1,859 of them labelled duplicates. For each pair we asked one question: if question A were cached and B arrived, would the cache serve A's answer? We embedded both questions with two small open models on a CPU and swept the cosine threshold from 0.70 to 0.99. The script and results are downloadable. Treat this as a proxy for an FAQ workload, not a production benchmark; QQP's labels are noisy.
At 0.85 the cache serves 80% of true duplicates, but 32% of its answers belong to pairs labelled as different questions. At 0.95 the false-hit share falls to 11%, and only 28% of duplicates are served. On this data, no threshold gives most of the savings with almost none of the errors.
Change the embedding model and the curve moves. At 0.90, bge-small-en-v1.5 served 57% of duplicates and all-MiniLM-L6-v2 served 41%. A threshold tuned on one model is wrong for the next.
Then we read all 65 "not duplicate" pairs that still scored 0.95 or more, out of 590 served. Forty-eight were acceptable paraphrases with an arguably wrong label. Seventeen were genuinely different questions, so about 3% of answers served at 0.95 would have been wrong for the asker. "Extrovert in introverted conditions" against the reverse scored 0.997. "Shaving cream as conditioner" against "conditioner as shaving cream" scored 0.987. "Orange, California" against "Orange Cove, California" scored 0.952. Embeddings are good at topic and poor at the one word that changes the answer, and in enterprises that word is usually a product, version, amount or location.
In a recent engagement…
A European financial-services group asked us to review an internal policy assistant. Semantic caching was on at the gateway, with one global partition and a threshold copied from a blog post. Then, in testing, a question about one country's expense rules came back with the cached answer for a neighbouring country. Both were real policies, and both read plausibly.
The fix was not a cleverer threshold. We partitioned by business unit and prompt version and added country and product to the key where the route depended on them. We bypassed the cache for any prompt containing a personnel number. Then we chose the threshold from a few hundred labelled should-match and must-not-match prompts taken from pilot logs. The hit rate dropped, but every remaining hit was one the team could defend.
Decision checklist: should this route use a semantic cache?
| Question | Cache it | Do not cache it |
|---|---|---|
| Same answer for every caller in the partition? | Public FAQ, documentation Q&A | Answers built from a user's own data or permissions |
| How fast does the truth change? | Policies, product docs (TTL of hours to days) | Prices, stock, status, anything live |
| Single-shot prompt? | One question, one answer | Long dialogues, agent loops, tool calls |
| Cost of a wrong answer? | Mild; user can re-ask | Financial, legal, medical, safety or HR impact |
| Do one-word differences matter? | Generic how-to questions | Product names, versions, amounts, locations, negations |
| Observable? | Cache status and score logged, feedback path | No per-response telemetry |
| Tuned on your data? | Threshold from a labelled set of your prompts | Default or copied threshold |
If a route fails two rows, use exact-match caching or provider prompt caching only.
Governance for European enterprises
GDPR. The cache stores prompts and responses, so it holds personal data whenever users type it in. Storage limitation (Article 5(1)(e)) argues for short TTLs. The right to erasure (Article 17) means you must be able to find one person's entries, which is hard if the key does not identify them. Article 32 requires security appropriate to the risk. Keep the cache in the same region as the model deployment, and add it to your records of processing and erasure runbook.
Gateway access control. Microsoft notes that linked access is not checked when a policy references a backend by embeddings-backend-id. Anyone who can edit policies can call any backend with the gateway's managed identity. Treat policy authoring as a privileged role.
EU AI Act. Rarely the driver here. If an assistant is an Annex III high-risk system, its obligations now apply from 2 December 2027 under Regulation (EU) 2026/1744, and your logs should show which responses came from the cache. Step ⑥ gives you that.
A pragmatic adoption path
- Exhaust prompt caching first. Put stable content at the start of prompts and measure cached-token rates. Our Azure OpenAI cost optimisation guide lists the other levers.
- Pick one route and label data. Choose a single-shot, low-stakes route. Label a few hundred real prompt pairs as should-match or must-not-match, including deliberate near misses.
- Run in shadow mode. Enable lookup and store, log the score and candidate answer, but always return the model's answer. Compare offline for a few weeks.
- Go live strictly. Set the threshold from your labelled set, partition by tenant, model and prompt version, keep TTLs conservative and expose cache status in a header.
- Operate it as a data store. Re-tune on every embedding change, purge on source and model changes, wire in erasure requests and review flagged answers monthly.
The verdict
Semantic caching is worth having on far fewer routes than the cost slide implies. The thresholds that save the most are the ones that replay answers to questions differing by one important word. We switch it on for stable, single-shot, low-stakes routes, tuned on the client's own data and tightly partitioned. Everywhere else, prompt caching and good prompt design deliver most of the savings without confident wrong answers. If someone proposes a global semantic cache with a default threshold, the answer is no.
If you want to know which of your LLM routes can safely use a semantic cache, our AI architecture team runs a short gateway and cost review, including a labelled threshold test on your own traffic.
FAQ
What is a semantic cache for LLMs?
A semantic cache stores earlier LLM responses with an embedding of the prompt that produced them. A new prompt is embedded and compared with the cache; if the nearest stored prompt is similar enough, the stored response is returned and the model is never called. It also matches prompts worded differently, which is where both the savings and the risk come from.
How is semantic caching different from provider prompt caching?
Provider prompt caching from OpenAI or Anthropic reuses the processing of an identical prompt prefix. The model still runs and writes a fresh answer; you just pay less for repeated input tokens. A semantic cache skips the model and replays an old answer for a similar prompt. Prompt caching cannot return a wrong answer. A semantic cache can.
What similarity threshold should we use for a semantic cache?
There is no portable number. On 5,000 Quora question pairs, a cosine threshold of 0.95 with bge-small-en-v1.5 served 28% of true duplicates; 0.85 served 80%, but 32% of its answers went to pairs labelled as different questions. At 0.90, a second model served 16 points fewer. Choose the threshold from a labelled set of your own prompts.
How do I enable semantic caching in Azure API Management?
Expose an embeddings deployment as a backend with managed identity, and configure an Azure Managed Redis instance created with RediSearch as the external cache. Add llm-semantic-cache-lookup (score-threshold, embeddings backend, vary-by) inbound and llm-semantic-cache-store (duration in seconds) outbound. Microsoft recommends a rate-limit policy straight after the lookup.
Is semantic caching safe for multi-turn chat and AI agents?
Usually not. LiteLLM's documentation warns that consecutive agent turns embed almost identically, so its semantic cache replays stale responses, and raising the threshold does not reliably fix it. API Management's max-message-count skips caching on longer dialogues. Our default is to bypass the semantic cache for agent loops and tool calls.
What does GDPR mean for an LLM response cache?
If users type personal data into prompts, the cache stores personal data. Storage limitation (Article 5(1)(e) GDPR) argues for short TTLs, Article 17 requires that you can find and delete one person's entries, and Article 32 requires appropriate security. Keep personal-data routes out of the shared cache and add the cache to your erasure runbook.
Sources
All sources checked on 15 November 2026. The charts are our own analysis; the method, seed and script are published next to the CSV.
Tier A: primary documentation and regulation
- Microsoft Learn,
llm-semantic-cache-lookuppolicy reference (updated August 2026): https://learn.microsoft.com/en-us/azure/api-management/llm-semantic-cache-lookup-policy - Microsoft Learn,
llm-semantic-cache-storepolicy reference: https://learn.microsoft.com/en-us/azure/api-management/llm-semantic-cache-store-policy - Microsoft Learn, Enable semantic caching for LLM APIs in Azure API Management: https://learn.microsoft.com/en-us/azure/api-management/azure-openai-enable-semantic-caching
- Microsoft Learn, Tutorial: Use Azure Managed Redis as a semantic cache: https://learn.microsoft.com/en-us/azure/redis/tutorial-semantic-cache
- LiteLLM documentation, semantic caching on the proxy: https://docs.litellm.ai/docs/proxy/caching_semantic
- Kong, AI Semantic Cache plugin: https://developer.konghq.com/plugins/ai-semantic-cache/
- RedisVL, SemanticCache API: https://redis.io/docs/latest/develop/ai/redisvl/0.27.1/api/cache/
- Anthropic, prompt caching: https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- OpenAI, prompt caching guide: https://developers.openai.com/api/docs/guides/prompt-caching
- Regulation (EU) 2016/679 (GDPR), Articles 5, 17 and 32: https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng
Tier B: research and datasets
- Schroeder et al., "vCache: Verified Semantic Prompt Caching", ICLR 2026, arXiv:2502.03771: https://arxiv.org/abs/2502.03771
- Gill et al., "MeanCache: User-Centric Semantic Caching for LLM Web Services", IPDPS 2025, arXiv:2403.02694: https://arxiv.org/abs/2403.02694
- Zhang et al., "From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching", ICML 2026, arXiv:2601.23088: https://arxiv.org/abs/2601.23088
- Song et al., "The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems", IEEE TIFS, arXiv:2409.20002: https://arxiv.org/abs/2409.20002
- Wang et al., GLUE benchmark (QQP task), arXiv:1804.07461: https://arxiv.org/abs/1804.07461
- GLUE QQP validation split on Hugging Face: https://huggingface.co/datasets/nyu-mll/glue
- BAAI, bge-small-en-v1.5: https://huggingface.co/BAAI/bge-small-en-v1.5
- sentence-transformers, all-MiniLM-L6-v2: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
Tier C: vendor and directional
- Redis, LangCache documentation (public preview): https://redis.io/docs/latest/develop/ai/langcache
- Qdrant, FastEmbed (the embedding runtime used in our experiment): https://github.com/qdrant/fastembed
Topics