The Semantic Cache: Recycling Responses in a System That Never Duplicates Itself

Meta description: Learn where similarity-keyed caching saves latency and cost, where it causes wrong answers, and how to choose an honest similarity threshold for production.

Tags: similarity-keyed caching, semantic caching, threshold tuning, production LLMs, latency optimization


This guide is for engineers, product managers, and support teams that already use exact-key caching in search, support bots, document agents, or large language model apps and are considering similarity-keyed caching to improve hit rates. It explains what the technique does, when it creates real savings, when it quietly produces wrong cached answers, and how to choose a threshold from your own traffic instead of a borrowed number.

A Towards AI piece frames the decision in three parts: when the technique earns its cost, when it quietly degrades results, and how to pick a cutoff with real evidence. That framing matters because similarity-keyed caching is not simply a faster cache. It changes the matching rule from “this request is the same” to “this request is close enough.”

What similarity-keyed caching actually does

Traditional caching uses an exact key. If the query string, filters, user context, document version, and model version match, the system can return a stored response. That is safe, but brittle. One extra word, a synonym, a different punctuation pattern, or a changed document ID can create a cache miss.

Similarity-keyed caching relaxes that rule. The system converts the incoming request into an embedding, which is a numeric vector that represents the meaning of the text. It then compares that vector with stored request vectors, often using cosine similarity, a score that measures the angle between two vectors. If the closest stored request is above a chosen threshold, the system returns the old response instead of running the full pipeline.

The cache is therefore making a prediction: a new query is semantically close enough to an old query that the old answer should still be correct. That prediction can be useful, but it is not guaranteed.

Where it pays

Similarity-keyed caching tends to pay when the workload is repetitive, natural, and forgiving.

Support and FAQ traffic is a common fit. Users ask the same practical question in many ways: “how to reset password,” “forgot my password,” “password reset not working,” and “cannot login after password change.” If the underlying answer is the same, a semantic cache can turn several near-identical requests into one stored response.

Document search with stable sections can also benefit. If a user asks about a clause, pricing term, policy detail, or known error message, the answer may not change between requests. The cache can reduce embedding calls, database lookups, search reranking, and model generation time.

The savings are strongest when three conditions hold:

• The cost of a cache miss is high, such as multiple embedding calls, a database scan, reranking, and a large language model call.
• The cache hit is usually correct, because the domain has many paraphrases of the same task.
• The stored response remains valid for a short window, because the underlying data does not change minute by minute.

In those cases, similarity-keyed caching can improve latency and reduce spend without changing the user’s experience much.

Where it silently poisons

The risk is that the cache returns a response that feels right but is wrong for this user, this version, or this moment.

Ambiguous queries are the first danger. “What does it cost?” could refer to one plan, one seat, one API call, one customer segment, or one region. Exact-key caching might miss because the keys are different. Similarity-keyed caching might hit too aggressively and return a price for the wrong item.

Personalization is another. A cached answer for a free user may not apply to an enterprise user. A cached answer for one workspace, customer account, language, or permission level may leak the wrong context into another request. If the similarity key does not include enough of the user context, the cache can create privacy and correctness problems.

Time-sensitive data is a third. Pricing, inventory, legal status, outage notices, user balance, and compliance policies can change. A high similarity score does not mean the answer is still true. It only means the question looked similar when it was stored.

Short or adversarial prompts are also risky. Two short queries can have similar embeddings because they share a few words, even when the intended answer is different. Some users may craft prompts that resemble a benign cached query to retrieve an unintended answer. This is why the cache should not be trusted as a security boundary.

The poison is silent because the system reports a hit, not an error. The response may be fluent and plausible, while the metric that caused it to be returned is a similarity score that nobody reviewed.

How to set the threshold honestly

There is no universal number. A threshold that works for password-reset support questions may be dangerous for financial answers, code generation, or document citations. The honest way is to choose from measured behavior on your own traffic.

1. Separate cache hits into true hits and false hits. Define a true hit as a cached response that would have been acceptable for the new request. Define a false hit as a response that is wrong, stale, unsafe, or missing required context. Then evaluate candidate thresholds against a sample of real requests.

1. Start with a conservative threshold. Begin high enough that most hits are exact or near-exact paraphrases. If your system reports similarity on a 0 to 1 scale, a very high cutoff is safer initially. Then lower it step by step only when you can show that false hits remain acceptable.

1. Measure latency and cost, not only hit rate. A cache can raise hit rate while adding embedding overhead or serving wrong answers. Track end-to-end time, embedding calls, model calls, retries, support escalations, and any user-visible accuracy loss.

1. Include the right fields in the key. Similarity should compare the user question, but the cache key should also be constrained by stable context: user role, customer account, document version, permission set, language, model version, and data freshness window. If those differ, do not let a bare embedding similarity override them.

1. Use a small validation set that mirrors production. Include paraphrases, edge cases, multi-account questions, time-sensitive queries, and known failure modes. If a threshold performs well on the validation set, test it on a small traffic slice before expanding it.

1. Monitor the gap between confidence and correctness. Similarity scores are not probabilities. A high score does not mean the answer is highly correct. It means the vectors are close under that embedding model. Track score distributions over time, because model upgrades and new traffic patterns can shift them.

Common mistakes

• Treating a similarity score as a guarantee of answer correctness.
• Using a default threshold copied from another product or blog post.
• Letting the cache ignore user context, account isolation, permissions, or document version.
• Optimizing hit rate while ignoring false hits and support escalations.
• Caching answers that depend on live data without a clear freshness window.
• Forgetting that embedding models change, so yesterday’s threshold may no longer mean the same thing today.
• Assuming semantic caching is safe for security-sensitive, legal, medical, or financial outputs without a separate validation process.

The practical rule is simple: use similarity-keyed caching where questions vary but answers stay stable, and keep the threshold strict where answers can be harmful, personal, or stale. If you cannot prove the cache hit is acceptable on a representative sample, the honest threshold is the one that does not serve the response.