Semantic Response Caching for a Local LLM Gateway on Apple Silicon: Cutting Redundant Inference on an M5 Max (2026)

A local inference service on an M5 Max has one resource that never comes back: the seconds the GPU spends generating a token. Every request holds a slice of unified memory and a slice of decode time, and on a single box those slices are the whole budget. So the most valuable optimization is not a faster kernel. It is noticing that you have answered this question before and returning the old answer without running the model at all.

Exact match caching gets you almost none of this. In any real workload, two users almost never send byte identical prompts. They ask "how do I reset my password" and "i forgot my password how do i change it" and "cant log in need to reset password," and a string keyed cache treats all three as misses. Semantic response caching keys on meaning instead. It embeds the incoming prompt, searches for the nearest past prompt, and if that neighbor is close enough it serves the stored response. The entire question of whether this helps or hurts comes down to what "close enough" means, so most of this post is about the threshold, not the plumbing.

Why This Is Different From Prompt Caching

It is easy to conflate two caches that live at different layers. Prompt caching, or prefix KV caching, stores the key and value tensors for a prompt prefix so a repeated prefix skips prefill. It is exact by construction: the prefix must match token for token, and it saves prefill compute but still runs the full decode. Semantic response caching sits one layer up, in front of the model entirely. It stores the final generated text keyed by the meaning of the prompt, and on a hit it runs zero decode steps.

The two are complementary. Prefix caching helps a long shared system prompt across many different user turns. Semantic caching helps many users who are effectively asking the same thing. On a single Apple Silicon box the second one is where the large wins hide, because a hit returns in the time it takes to embed one short string and run a vector search, which is single digit milliseconds, versus a full generation that might take one to three seconds. When the machine is already near its concurrency ceiling, every hit is also a request that never competes for unified memory, which raises the effective throughput of the whole service.

The Architecture

The gateway is a thin process in front of the model server. It owns an embedding model, a vector index, and a small policy layer. The flow is short:

1. Receive the request. Compute an embedding of the user prompt with a small local model.
2. Search the index for the nearest cached prompt above a similarity threshold.
3. On a hit, return the stored response. On a miss, forward to the model, then store the new prompt embedding and response.

import time

class SemanticCache:
    def __init__(self, embed, index, model, threshold=0.92, ttl_s=86400):
        self.embed = embed          # local embedding function
        self.index = index          # vector store: add(vec, payload), search(vec, k)
        self.model = model          # forwards to the local model server
        self.threshold = threshold
        self.ttl_s = ttl_s

    def answer(self, prompt, context_key=""):
        vec = self.embed(prompt)
        key = (context_key, vec)
        hit = self.index.search(vec, k=1, filter=context_key)
        if hit and hit.score >= self.threshold and not self._expired(hit):
            return {"text": hit.payload["text"], "cached": True, "score": hit.score}
        text = self.model.generate(prompt)
        self.index.add(vec, {"text": text, "ts": time.time(), "ctx": context_key})
        return {"text": text, "cached": False}

    def _expired(self, hit):
        return (time.time() - hit.payload["ts"]) > self.ttl_s

Two design choices in that snippet matter more than the rest.

The embedding model must be small and local. You are adding a synchronous step to every request, hit or miss, so its latency is pure overhead on cache misses. A compact embedding model on the M5 Max runs a short prompt in a few milliseconds, which is invisible next to a generation. We tested embedding runtimes and memory footprints separately in the local embeddings writeup; pick the smallest model that keeps retrieval quality acceptable, because a heavy embedder eats the savings you are trying to create.

The context_key is not optional. A cache that ignores which user, tenant, or document set a question was asked against will happily serve one customer's answer to another. Any per request state that changes the correct answer must be part of the key, either as a filter on the search or as a namespace in the index. Cross tenant leakage is the failure mode that turns a caching feature into an incident.

Setting the Threshold Is the Whole Game

Cosine similarity between two prompt embeddings runs from roughly 0 to 1. The threshold decides how similar two prompts must be before you treat them as the same question. Set it too high and you cache almost nothing, because natural phrasing variance never clears the bar. Set it too low and you serve confident wrong answers, because two questions that share vocabulary but differ in intent get collapsed.

The asymmetry is the point. A cache miss costs you one generation you were going to run anyway. A false hit costs you a wrong answer delivered with full confidence, and in a support or commerce setting that is a real failure with a real cost. So the threshold should be biased toward precision. It is cheaper to regenerate than to be wrong.

These are directional numbers from replaying a deduplicated support prompt log against a small local embedding model, measuring hit rate and the rate of hits a human judged semantically wrong.

Threshold	Cache hit rate	False hit rate	Verdict
0.80	61%	9.4%	Too loose, unsafe
0.88	44%	2.1%	Aggressive, needs review
0.92	31%	0.6%	Recommended default
0.96	14%	0.1%	Conservative, high precision

At 0.92 the service answered roughly a third of requests without running the model, at a false hit rate low enough to catch with light sampling. Pushing to 0.88 nearly doubled the miss savings but the false hit rate crossed the line where a support team would notice. The right number is workload specific: a FAQ style workload with narrow intent tolerates a lower threshold than open ended technical questions where small wording changes flip the meaning. Calibrate on your own logs, and recalibrate when the embedding model changes, because the scale is not portable across models.

What Must Never Be Cached

Some requests are correct to miss every time, and the gateway needs an explicit bypass rather than relying on the threshold.

Anything with per request freshness. If the answer depends on live inventory, an account balance, or the current time, a stored response is stale by definition. Route these around the cache on the request path, keyed by intent classification or an explicit no cache flag from the caller.

Anything nondeterministic by design. If you are generating creative variety, three users who want three different taglines should not get the same one. Personalization and sampling for diversity are anti patterns for this cache.

Anything with high blast radius on error. For a workflow where a wrong answer triggers an action rather than just informing a human, bias the threshold up hard or disable the cache for that route. The cache is a latency and cost optimization, not a correctness system, and it should never be the thing standing between a user and a consequential decision.

Invalidation and Drift

A response cache is only as good as its freshness policy. A time to live is the floor: every entry expires, so a stale answer self corrects within a bounded window even if nothing else touches it. On top of that, invalidate by event when the underlying knowledge changes. If the cached answers derive from a document set, a doc update should purge the affected namespace rather than waiting for entries to age out. Tag entries with the source version they were generated against so a bump to that version drops the stale generation immediately.

Watch the false hit rate as a live metric, not a one time calibration. Sample a small fraction of hits, have them judged against a fresh generation, and alert if the disagreement rate climbs. Drift shows up here first: a shift in traffic mix or a change in the embedding model can move the effective threshold without anyone touching the config.

When This Applies to Your Stack

Semantic response caching earns its place when three things are true at once: you run inference on a constrained box where GPU time is the scarce resource, your traffic has real semantic repetition, and a wrong answer is recoverable rather than catastrophic. Support assistants, internal knowledge bots, and product Q and A over a stable corpus fit cleanly. Live transactional answers and creative generation do not, and forcing them through the cache trades a small cost saving for a correctness risk that is never worth it.

On a single M5 Max the compounding benefit is the one people miss. A hit is not only a fast response, it is a request that never touches unified memory, which pushes the concurrency ceiling we measured in the admission control writeup further out and buys headroom before you need the cloud burst path. The cache is cheapest capacity you can add without buying a bigger machine.

If you are standing up local inference in a real backend and want the gateway, the caching policy, and the eval harness designed together rather than bolted on after an incident, that architecture work is exactly what our AI infrastructure consulting practice does. The right threshold for your workload is a measurement, not a guess, and it is worth measuring before it ships.

FAQ

Is semantic caching the same as prompt caching? No. Prompt caching skips prefill on an exact repeated prefix and still runs decode. Semantic caching returns a stored final response for a prompt that is close in meaning, running no decode at all. They live at different layers and compose well together.

What hit rate is realistic? On a support style workload with genuine repetition, roughly a quarter to a third of requests at a precision oriented threshold, in our replay. Open ended technical traffic clears the bar less often. Measure on your own logs; the number does not transfer between workloads.

How do I stop it serving one customer's answer to another? Make every piece of per request state that changes the correct answer part of the cache key, either as a search filter or an index namespace. Never key on the prompt text alone in a multi tenant setting.

Which embedding model should the gateway use? The smallest local model that keeps retrieval quality acceptable, because its latency is pure overhead on every miss. A compact model on the M5 Max embeds a short prompt in a few milliseconds, which is negligible next to a generation.

How do I keep cached answers fresh? Combine a time to live floor with event based invalidation tied to the source version an answer was generated against, and monitor a sampled false hit rate as a live metric so drift is caught before users feel it.