Semantic Caching for LLM Calls in ASP.NET Core: When to Use It and How

Every AI feature I have shipped eventually hits the same wall. Users ask the same twenty questions in a hundred different phrasings, and each one costs a full round trip to the model. Semantic caching for LLM calls in ASP.NET Core is the pattern that breaks that loop: instead of matching on the exact prompt string, you match on what the prompt means. In one support-desk API we run, roughly a third of incoming questions were semantic duplicates of something answered minutes earlier, and every one of them was being billed at full price.

This is not a theoretical optimisation. It is the difference between an AI endpoint that costs a predictable amount per month and one that quietly triples its bill the week marketing runs a campaign. If you want the complete pattern with the eviction logic, the threshold tuning harness, and the failure paths wired together, the annotated implementation lives on Patreon as a runnable project rather than a set of disconnected snippets.

Getting caching right for AI endpoints means thinking about token budgets, model tiering, and rate limiting at the same time, because they all pull on the same lever. Chapter 15 of the AI-Powered .NET APIs course covers caching LLM calls and embeddings alongside those concerns, and it is explicit about the cases where caching is simply the wrong answer.

What Semantic Caching Actually Solves

A conventional response cache keys on an exact input. That works beautifully for GET /products/42. It works almost never for POST /chat, because natural language has effectively infinite surface forms for the same intent:

• "How do I reset my password?"

• "i forgot my password, what now"

• "password reset steps please"

Three distinct cache keys. Three distinct billed completions. One actual question.

Semantic caching replaces string equality with vector similarity. You convert the incoming prompt into an embedding, search previously answered prompts for the nearest neighbour, and if that neighbour is close enough, you return its stored answer without ever calling the model.

How Is Semantic Caching Different From Normal Response Caching?

Normal response caching matches inputs byte for byte and returns a hit only on an identical key. Semantic caching matches inputs by meaning, using embedding vectors and a similarity threshold, so paraphrases hit the same entry.

Aspect	Output / Response Caching	Semantic Caching
Key	Exact string or URL	Embedding vector
Match rule	Equality	Cosine similarity above a threshold
Hit rate on chat traffic	Very low	Moderate to high
Failure mode	Cache miss (harmless)	Wrong answer returned (harmful)
Cost per lookup	Near zero	One embedding call

That last row is the honest part most write-ups skip. A semantic cache can be wrong in a way a normal cache cannot, and the lookup itself is not free. If your traffic is genuinely all unique, you will pay for embeddings and get nothing back. Our own breakdown of runaway LLM costs in a .NET API walks through how to measure that before you commit.

The Core Concepts: Embeddings, Similarity, and Thresholds

Three moving parts, and each one has a decision attached.

The embedding. In Microsoft.Extensions.AI 10.x you get this through IEmbeddingGenerator<string, Embedding<float>>, registered exactly like any other service. Embedding models are dramatically cheaper than chat models, which is the whole economic basis of the pattern. Microsoft's own embeddings guidance for .NET is a good primer if the concept is new.

// Microsoft.Extensions.AI 10.x, .NET 10
ReadOnlyMemory<float> vector =
    await embedder.GenerateVectorAsync(prompt, cancellationToken: ct);

The similarity measure. Cosine similarity is the default for text embeddings. .NET gives you a hardware-accelerated implementation in System.Numerics.Tensors, so you do not need a library for the maths:

// System.Numerics.Tensors - returns 1.0 for identical direction
float score = TensorPrimitives.CosineSimilarity(candidate.Span, query.Span);

The threshold. This is the single most consequential number in the whole design, and it is the one nobody can hand you. More on that below.

For anything beyond a few thousand cached entries, stop scanning in memory and put the vectors in a real store. Microsoft.Extensions.VectorData gives you one abstraction over sqlite-vec, SQL Server 2025's native vector type, Qdrant, and pgvector. We compared the practical trade-offs in the vector store decision guide for .NET AI apps.

When Semantic Caching Fits

The pattern earns its keep when all of these hold:

• Prompt diversity is low. FAQ bots, support triage, documentation search, product Q&A. Repetition is the fuel.

• Answers are stable over minutes or hours. If the correct answer changes every request, there is nothing to cache.

• The answer does not depend on per-user private data. Two users asking the same thing must be entitled to the same answer.

• Latency matters. A cache hit resolves in tens of milliseconds against several seconds for a fresh completion.

When Semantic Caching Is the Wrong Choice

I have seen this pattern cause real damage in production, so treat the following as hard stops rather than cautions.

Personalised or tenant-scoped answers. "What is my account balance?" and "what's my balance" are semantically near-identical and belong to different people. If you cache across users, you have built a data-leak machine. Partition the cache by tenant and user, or do not cache at all.

Multi-turn conversations. The meaning of "and the second one?" depends entirely on prior turns. Embedding the last message alone produces confident nonsense. Either embed a summarised conversation state or exclude follow-ups from the cache.

Tool-calling and agent turns. If the model's job is to invoke a function against live data, a cached response returns yesterday's reality. Cache the final natural-language rendering if you must, never the decision to act.

Negation and small semantic deltas. "How do I enable two-factor auth?" and "how do I disable two-factor auth?" sit uncomfortably close in embedding space. This is the single most common source of wrong hits.

Regulated or auditable outputs. If you must be able to explain exactly which model produced which answer at which time, a cache layer complicates your audit story more than it saves.

Implementation Sketch in ASP.NET Core

The shape is a decorator around your chat client, not a change to your endpoints. That matters: it keeps the pattern removable.

Step 1 - embed the incoming prompt. One call to the embedding generator, on the request path, before you touch the chat model.

Step 2 - search for a near match. Query your vector store for the top result within the cache partition, then apply the threshold yourself rather than trusting a store default:

if (best is not null && best.Score >= _options.SimilarityThreshold)
{
    _metrics.CacheHit(best.Score);
    return best.Record.Answer;
}

Step 3 - fall through, then store. On a miss, call the model, and write the answer back keyed by the original prompt's vector. Store the prompt text too. You will need it the first time you debug a bad hit, and you will need it sooner than you expect.

Two implementation details that are easy to get wrong:

• Layer an exact-match cache in front. Identical strings are common enough that skipping the embedding call entirely is free money. HybridCache (.NET 9 and later) handles the L1 plus L2 case and gives you stampede protection at the same time, which matters here because a cold cache and a traffic spike arrive together. We covered that failure mode in detail in the cache stampede production fix.

• Version the cache by model and prompt template. When you change the system prompt or swap the model, every stored answer becomes stale in a way no TTL will catch. Bake a version discriminator into the partition key so a deployment invalidates cleanly.

Choosing a Similarity Threshold That Does Not Embarrass You

Do not pick this number by intuition, and do not copy it from a blog post - including this one. Thresholds are specific to the embedding model, the domain vocabulary, and how tolerant your users are of a slightly-off answer.

The approach that has worked for us:

1. Log every lookup with its top score and whether you served a hit, starting with the cache disabled in shadow mode.

2. Take a few hundred real pairs and label them by hand: would serving A's answer for B have been acceptable?

3. Plot the false-hit rate against the threshold and pick the point where false hits approach zero, then add a safety margin.

Expect to land somewhere in the high 0.8s to low 0.9s for cosine similarity on general text, but treat that as a starting bracket for your experiment, not an answer. Push it up when wrong answers are expensive. And re-run the exercise whenever you change the embedding model, because the score distribution shifts underneath you.

Trade-offs You Need to Accept

• You add a dependency on the embedding provider to the read path. If it is down and you have not written a fallback that skips the cache, you have made your API less available, not more.

• You pay on every miss. Embedding plus completion is more expensive than completion alone. Below a certain hit rate, the pattern loses money. Instrument it and be prepared to turn it off.

• Debugging gets harder. "Why did it say that?" now has two possible answers. Log the cache decision, the matched prompt, and the score on every response, behind a header or a trace attribute.

• Freshness is now your problem. Every cached answer is a snapshot. Short TTLs blunt the savings; long TTLs serve stale content. There is no setting that avoids the choice.

A Short Production Checklist

• Partition keys include tenant, user scope where relevant, model id, and prompt version

• Similarity threshold is measured, documented, and alerted on

• Cache decision, matched prompt, and score are emitted as telemetry on every request

• A kill switch disables the cache without a deployment

• Embedding provider failure degrades to a direct model call, not a 500

• Hit rate and cost-per-request are on the same dashboard, so a falling hit rate is visible before the invoice is

FAQ

Does semantic caching work with streaming responses in ASP.NET Core?

Yes, but you have to buffer. A cache hit has the full answer immediately, so you replay it to the client as a stream to keep the user experience consistent. On a miss you stream from the model and accumulate the chunks, writing the assembled answer to the cache once the stream completes successfully. Never cache a stream that was cancelled part way through.

How much can semantic caching realistically reduce LLM costs in a .NET API?

It is entirely a function of how repetitive your traffic is. Support and FAQ workloads with high repetition can see a substantial share of requests served from cache; genuinely open-ended creative workloads may see almost none. Measure your duplicate rate in shadow mode for a week before you build anything, and be ready to walk away if the number is small.

What similarity threshold should I use for semantic caching?

There is no portable answer, because the score distribution depends on your embedding model and your domain language. Label a few hundred real query pairs, find the threshold where wrong hits disappear, then add margin. Re-measure every time you change the embedding model.

Is semantic caching safe for multi-tenant AI APIs?

Only with strict partitioning. The cache key must include the tenant identifier, and anything scoped to an individual user must be partitioned by user or excluded entirely. A shared semantic cache across tenants is a cross-tenant data leak waiting for the right paraphrase.

Should I use HybridCache or a vector store for semantic caching?

Both, at different layers. HybridCache handles exact-string hits and gives you L1 plus L2 with stampede protection for free. The vector store handles the approximate matching. Trying to force similarity search into a key-value cache means scanning every entry, which stops being viable somewhere in the low thousands of records.

How do I invalidate a semantic cache when the underlying knowledge changes?

TTLs alone will not save you, because the staleness is content-driven rather than time-driven. Use a version discriminator in the partition key tied to your prompt template, model id, and knowledge-base revision. When a document is re-ingested, bump the revision and the whole affected partition falls away without a sweep.

---

About the Author

I'm Celin Daniel, Co-founder of Coding Droplets. I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.

• GitHub: codingdroplets

• YouTube: Coding Droplets

• Website: codingdroplets.com