Semantic Caching to Reduce LLM Costs

In a software system that serves many users or runs many agentic tasks, a notable fraction of LLM queries are semantically similar to queries that have already been answered. A code-review agent that checks five pull requests for the same anti-pattern is asking the same underlying question five times. A support assistant that fields questions about a common configuration error is generating the same response from a different phrasing of the same question every few hours. Each of these calls is billed at full API cost, even though the answer is already known.

Semantic caching addresses this by checking whether a new query is sufficiently similar to a cached query — measured by the cosine distance between their vector embeddings — and returning the cached result if the similarity exceeds a threshold. The cost of a cache hit is the embedding computation (typically a fraction of a cent per query on a fast embedding model) plus a Redis lookup. The cost of serving a full API call is typically 10× to 100× higher depending on the model tier and token count.

The reported hit rates vary by workload, but structured agentic tasks with bounded query sets — code analysis, configuration validation, documentation lookups — routinely see hit rates of 40–65% under good cache design. On a workload with 40% hit rate and average API call cost of $0.008, a cache that costs $0.0001 per lookup reduces effective per-query cost by roughly 32%. At scale — 10,000 queries per day — that is a $28 reduction per 1,000 queries, or $280/day.

What semantic caching solves — and what exact-match HTTP caching does not

Exact-match HTTP caching stores a response keyed to the exact request bytes. If the query changes by even one character — capitalisation, punctuation, word order — the cache misses and a fresh API call goes out. This works well for deterministic APIs where the same input always produces the same output, but LLM queries are rarely identical character-for-character even when they mean the same thing.

Consider these two questions: "What does the AuthService.authenticate() method return when the user doesn't exist?" and "What is the return value of AuthService.authenticate() for a missing user?" They are different strings. An exact-match cache misses on the second query even though the cached answer from the first query would be perfectly adequate.

Semantic caching uses vector embeddings to measure the meaning similarity of the new query against stored queries. A query is embedded into a high-dimensional vector (typically 384–1536 dimensions depending on the embedding model), and the cache index is searched for the nearest stored vector by cosine similarity. If the best match exceeds the configured threshold, the cached answer is returned. If not, the query proceeds to the API, and both the query and the response are stored in the cache for future matches.

This approach handles paraphrase, synonyms, word order variation, and minor contextual differences — all of which break exact-match caching but preserve the underlying intent that a cached answer already satisfies.

Embedding-similarity vs exact-match — the trade-off

The cost of semantic caching's flexibility is the possibility of false positives: returning a cached answer that is not actually appropriate for the new query. This is the accuracy trade-off that makes semantic caching a design decision rather than a free optimisation.

At high similarity thresholds (0.95+), false positives are rare — only near-identical queries match — but hit rates are low, and the cost savings are modest. At low thresholds (0.80 or below), hit rates are high but false positives increase. A query about the AuthService.authenticate() method might match a cached query about UserService.authenticate() if the threshold is loose enough — and the returned answer is wrong for the new context.

The practical calibration is workload-specific. For structured technical queries with bounded vocabulary (code analysis, schema validation, log parsing), thresholds around 0.88–0.92 tend to achieve hit rates of 35–55% with low false positive rates. For natural-language queries with wider semantic variation (free-text user questions, creative tasks), higher thresholds (0.92–0.96) are appropriate to avoid serving stale or off-topic cached answers.

Measure false positive rate empirically, not by tuning the threshold until it looks right. Use a held-out evaluation set of known-different query pairs and measure how often your cache incorrectly classifies them as similar. Set the threshold at the point where false positive rate drops below your acceptable tolerance for the use case — typically 1–3% for informational queries, closer to 0.1% for code that will be executed or advice that will be actioned.

Architecture: Redis-backed with embedding model + similarity threshold

The standard architecture for a production semantic cache has four components: an embedding model, a vector index, a key-value store for the cached answers, and a cache lookup wrapper in the application layer.

The embedding model converts each query string to a dense vector. For latency-sensitive paths, use a fast embedding model hosted locally or in a private endpoint — model-specific embedding latency matters more than embedding quality at the similarity thresholds most use cases require. Current embedding models (as of mid-2026) from providers including Anthropic, OpenAI, Cohere, and open-source options from Hugging Face vary significantly in latency and accuracy; check each provider's documentation for current specifications. What matters for this use case is p99 embedding latency under load, not benchmark accuracy scores on unrelated tasks.

The vector index stores the embeddings of all previously-cached queries and supports approximate nearest-neighbour (ANN) search. Redis Stack's HNSW index type supports vector similarity search directly in Redis, which makes it a common choice because it collapses the vector index and the key-value store into a single infrastructure component. Alternatives include Pinecone, Weaviate, and Qdrant — each with different operational characteristics and managed-vs-self-hosted trade-offs. For most semantic cache workloads, a Redis Stack deployment is sufficient and operationally simpler than a dedicated vector database.

The cached answer store maps a query vector (or its hash) to the cached API response. In Redis Stack, this is the same hash as the vector payload — the query vector and the response are stored together. On a cache hit, the lookup returns both the stored response and the similarity score, which you can log for cache health monitoring.

The cache lookup wrapper sits between your application code and the LLM SDK:

import hashlib
import numpy as np

class SemanticCache:
    def __init__(self, redis_client, embedding_fn, threshold: float = 0.90):
        self.redis = redis_client
        self.embed = embedding_fn
        self.threshold = threshold

    def get(self, query: str):
        vector = self.embed(query)
        results = self.redis.ft("cache_idx").search(
            Query("*").sort_by("__vector_score").limit(0, 1).dialect(2),
            query_params={"vec": np.array(vector, dtype=np.float32).tobytes()}
        )
        if results.docs and float(results.docs[0].__vector_score) >= self.threshold:
            return results.docs[0].response   # cache hit
        return None                            # cache miss

    def set(self, query: str, response: str) -> None:
        vector = self.embed(query)
        key = "cache:" + hashlib.sha256(query.encode()).hexdigest()[:16]
        self.redis.hset(key, mapping={
            "query":    query,
            "response": response,
            "vector":   np.array(vector, dtype=np.float32).tobytes(),
        })

# Usage:
cache = SemanticCache(redis, embed_fn, threshold=0.90)
cached = cache.get(query)
if cached:
    return cached
response = llm.call(query)
cache.set(query, response.content)
return response.content

Cache invalidation strategy: semantic caches are appropriate for workloads where the underlying information is stable over days or weeks. For each cache entry, store a created_at timestamp and a TTL appropriate for the information type. Documentation queries might have a 7-day TTL; configuration-state queries might be 1 hour; creative or user-specific queries should not be cached at all (see below).

Cache hit rate vs accuracy — the threshold calibration table

Observed behaviour across three threshold settings on a structured code-analysis workload (1,000-query evaluation set, queries about function contracts and type signatures):

Numbers here are illustrative of the calibration pattern, not universally applicable — your embedding model, query distribution, and accuracy requirements will produce different numbers. Measure on your own query set before setting a production threshold. What the table shows is that the relationship between threshold and hit rate is nonlinear: moving from 0.90 to 0.85 doubles the false positive rate while increasing hit rate by only 18 percentage points. Moving from 0.90 to 0.95 cuts false positives by 9× while halving the hit rate. The middle range is usually where cost savings and accuracy tolerance intersect for technical workloads.

When NOT to use semantic caching

Semantic caching assumes that similar queries have the same correct answer. There are workloads where this assumption breaks, and applying a cache there produces incorrect outputs at scale.

Creative or generative content. When the goal is novel output — writing a unique product description, generating a creative brief, drafting a personalised email — returning a previously-generated response is a defect, not a cost optimisation. Users notice when they receive cached text that was written for a different context. Do not cache these workloads.

User-specific personalisation. If the correct answer depends on user state — user profile, account history, settings, permissions — a cached answer from a different user's query can be wrong or, worse, a data exposure risk. Segment the cache key space strictly by user if personalisation is in scope, and even then evaluate whether the per-user query volume is high enough to generate meaningful hit rates before investing in the infrastructure.

Legal, medical, or compliance-critical advice. In domains where a stale or slightly-off-topic cached answer can cause material harm — a legal interpretation that has changed due to a recent ruling, a medication interaction query — the false positive cost is not a percentage reduction in accuracy, it is a liability. These workloads require fresh answers on every call. The cost of correctness is the full API cost per query.

Rapidly-evolving information. If the ground truth changes faster than your cache TTL, cached answers are consistently stale. Queries about live market data, deployment status, or real-time system state should not be cached in a semantic layer. Exact-match caching with short TTL (seconds or minutes) may be appropriate; semantic caching with its higher cache-age tolerance is not.

Semantic caching is a cost optimisation for stable, structured, repeated workloads. Applied correctly to that workload type, it is one of the highest-ROI techniques available for reducing LLM API spend. Applied to creative, personalised, or time-sensitive workloads, it introduces correctness problems that cost more to debug and remediate than the token savings are worth. The broader cost-control framework — including how model routing and agent budget gates interact with caching decisions — is covered at AI engineering cost control.