You built the pipeline. Documents are chunked, embedded, and sitting in a vector store. Retrieval returns results. And your RAG answers still hallucinate in .NET, confidently telling users about a student discount policy that does not exist. The instinct is to blame the model. In production the cause is almost always upstream: you retrieved something, so you passed something to the model, and a model handed context will use it whether or not it is relevant.
I have debugged this on a support-desk API running against a local Ollama model, and the fix that finally held was not a better prompt on its own. It was two independent refusal layers plus citations that carry their own scores. The complete implementation, with the calibration data behind the numbers below, is on Patreon if you would rather read working code than assemble it from fragments.
The trust story here is the whole product. An answer with a citation the user can click is worth more than three answers without one, and a system that says "I don't know" keeps its credibility on the fourth question. Chapter 10 of the AI-Powered .NET APIs course builds this exact /ask endpoint end to end: the five-step retrieve-and-ground flow, numbered context with source labels, [n] citation markers, and both refusal paths verified against real query traces.
The Problem: Retrieval Always Returns Something
A vector search does not return "nothing." Ask a support knowledge base about the capital of France and it will still hand back your three least-irrelevant chunks, because nearest-neighbour search is relative, not absolute. There is no natural zero.
That single property causes every symptom teams report:
Confident off-topic answers. The model was given shipping-policy text and a question about student discounts. It bridged the gap, because that is what language models do.
Citations that point at the wrong document. The citation is real, the claim it supports is not. Chunks got retrieved, the model wrote a plausible paragraph, and the source label rode along.
Answers that are right for the wrong reason. Correct today, wrong the moment a document changes, because the answer came from model memory rather than from your corpus.
No way to reproduce a bad answer. Without the retrieved chunks and their scores in the response, you cannot tell whether retrieval failed or generation failed.
Root Cause: Three Things Missing From the Flow
A grounded answer flow has five steps, and most broken implementations only have three.
Embed the question
Retrieve top-k chunks
Filter by score <- usually missing
Refuse or ground <- usually missing
Answer, with citations <- citations usually missing
Steps 3 and 4 are where honesty comes from. Step 5 is where verifiability comes from. Without them you have a search engine wired to a fabricator.
There is a second, subtler root cause: teams threshold on the wrong number. With Microsoft.Extensions.VectorData and a SQLite vector connector using DistanceFunction.CosineDistance, VectorSearchResult.Score is a cosine distance, so lower is closer. Half the broken threshold logic I have seen was written as Score > 0.6 by someone who assumed it was a similarity. That inverted comparison passes exactly the chunks it was meant to block.
What Score Threshold Should You Use for RAG Retrieval?
There is no universal number, and anyone who gives you one is guessing. You calibrate it against your own corpus, which takes about ten minutes.
Run a batch of questions you know are in scope and a batch you know are not, and record the distances. On a real support knowledge base the separation was clean:
| Query type |
Observed cosine distance |
| In scope (returns policy, shipping) |
0.15 to 0.47 |
| Out of scope (capital of France) |
0.61 to 0.69 |
The gap between 0.47 and 0.61 is where the threshold belongs. Setting MaxDistance = 0.6 put it squarely in that gap, and off-topic questions stopped reaching the model entirely.
Two things follow from this. First, the threshold is data, not code, so it belongs in options you can tune per environment:
public record RagOptions(int TopK, double MaxDistance);
Second, you must re-calibrate when the corpus, the embedding model, or the chunk size changes. A threshold tuned for 400-token chunks is wrong for 900-token chunks. If you have not settled on a chunking strategy yet, that decision comes first and we walked through it in How to Chunk Documents for RAG in .NET.
The Fix
Fix 1: Refuse Before the Model Call
The cheapest refusal is the one that never spends a token. After retrieval, drop everything above your distance threshold. If nothing survives, return a canned refusal and stop.
var hits = results.Where(r => r.Score <= options.MaxDistance).ToList();
if (hits.Count == 0)
{
return new AskResult(
Answer: "I don't have information about that in the knowledge base.",
Citations: [],
Grounded: false);
}
Verified behaviour: asking a product support API for the capital of France returned grounded = false with zero citations and no model call at all, because every raw distance landed between 0.611 and 0.691. That is a latency win and a cost win on top of the correctness win.
Fix 2: Build a Grounded Prompt With Numbered, Labelled Context
Do not paste chunks into the prompt as an undifferentiated blob. Number them and label each with its source, because the numbers are what the model will cite:
Answer ONLY from the Context below. Cite sources as [n].
If the Context does not contain the answer, say you do not know.
Context:
[1] (Returns Policy) ...chunk text...
[2] (Shipping Policy) ...chunk text...
Keep the prompt in a file that ships with the app rather than a string literal in a service class. A prompt is an application asset: it gets versioned, reviewed in a pull request, and diffed when answer quality moves.
Fix 3: Keep the Second Refusal Layer in the Prompt
The threshold catches clearly out-of-scope questions. It does not catch the in-neighbourhood-but-unanswerable ones, and those are the questions that produce the most damaging hallucinations.
A real example: a question about a student discount retrieved three chunks that all passed the 0.6 threshold, because discount and pricing language is genuinely near the question in embedding space. The threshold let them through. The refusing instruction in the prompt caught it, and the model replied that it had no information about a student discount in the context, rather than inventing a percentage.
Both layers are needed. Neither is sufficient alone. This is the single most important design point in the whole article.
Fix 4: Return Citations With Their Scores
Citations are not decoration. Model them as data and return them from the endpoint:
public record Citation(string Source, int ChunkIndex, double Score);
public record AskResult(string Answer, IReadOnlyList<Citation> Citations, bool Grounded);
Shipping the score alongside the source turns your API into its own debugging tool. When a user reports a bad answer, the response tells you immediately whether retrieval pulled the wrong chunk (bad score, bad source) or the model misused a good chunk (good score, good source, wrong claim). Requires .NET 8 or later with Microsoft.Extensions.AI and Microsoft.Extensions.VectorData.
The Grounded flag matters too. Your client can render a refusal differently from an answer, and your dashboards can track the refusal rate as a first-class metric. A refusal rate that suddenly drops to zero usually means someone widened the threshold, not that the knowledge base got better.
Fix 5: Add a Search Endpoint You Can Point At
Expose a GET /search that returns raw retrieval results and scores with no model involved. When an answer looks wrong, hit /search with the same question first. Retrieval is upstream of everything, so if the right chunk is not in the results, no prompt change on earth will fix the answer.
This is the single highest-leverage debugging habit in RAG work: always debug retrieval first. In production I have never once found the generation step to be at fault when retrieval was healthy.
Fix 6: Know Where Vector Search Alone Fails
Dense vector search is semantic, which means it is bad at exact strings. Order codes, SKUs, error codes, and version numbers are the classic misses: ERR-4021 and ERR-4012 sit almost on top of each other in embedding space, and neither reliably beats a paragraph that merely talks about errors.
The fix is hybrid retrieval, combining keyword or full-text search with vector search and merging the result sets. If your corpus is full of identifiers, plan for hybrid from the start rather than tuning a threshold that cannot solve the problem. For the broader question of when RAG is the right architecture at all, we covered that in The RAG Pattern in ASP.NET Core.
How to Prevent It Recurring
Treat the threshold as versioned configuration. Record which corpus, embedding model, and chunk size it was calibrated against, so the next person knows when it expired.
Log the retrieval trace on every request. Question, chunk IDs, scores, grounded flag. Sampling is fine. Zero visibility is not.
Alert on refusal-rate movement in both directions. A spike means retrieval or ingestion broke. A collapse means someone loosened a guard.
Re-run a fixed question set on every prompt, model, or corpus change. Keep in-scope, out-of-scope, and unanswerable-but-nearby questions in the set, because the third category is the one that regresses silently.
Never let an ungrounded answer look like a grounded one. Different shape, different flag, different rendering. The moment the two look the same in the UI, users lose the ability to calibrate their trust and the citations stop being worth anything.
Frequently Asked Questions
Why Does My RAG System Answer Questions That Are Not in the Knowledge Base?
Because vector search returns the nearest chunks regardless of how far away they are, and a model handed context will use it. Add a distance threshold after retrieval so clearly out-of-scope questions never reach the model, and add an explicit refusal instruction to the prompt for the ones that squeak through.
Is VectorSearchResult.Score a Similarity or a Distance?
It depends on the distance function the collection was configured with. With DistanceFunction.CosineDistance it is a distance, so lower means more similar and your filter is Score <= threshold. Getting this backwards is one of the most common RAG bugs in .NET, and it fails in the worst possible way: it silently admits exactly the chunks you meant to reject.
How Many Chunks Should I Retrieve for a RAG Answer?
Start at top-k of 3 to 5 and let the threshold do the trimming rather than the k value. A larger k mostly adds tokens and noise, since anything genuinely relevant tends to rank in the first few results. If your correct answer regularly sits at rank 8, the problem is chunking or embedding quality, not k.
How Do I Make the Model Actually Cite Its Sources?
Number the context entries, label each with its source, and instruct the model to cite as [n]. Then return the mapping from [n] to source, chunk index, and score in your response payload so the citation is verifiable rather than decorative. Do not trust a citation the model emits without checking it maps to a chunk you actually retrieved.
Do I Need Hybrid Search, or Is Vector Search Enough?
Vector search alone is fine for prose-heavy corpora like policies, guides, and documentation. Add keyword or full-text search alongside it when your content contains exact identifiers such as SKUs, order numbers, error codes, or API names, because semantic similarity does not reliably distinguish near-identical strings.
Will a Bigger Model Stop the Hallucinations?
It helps with phrasing and instruction-following, and it will not fix a retrieval problem. If the right chunk is not in the context, a larger model produces a more fluent wrong answer, not a right one. Fix the retrieval layer first, then consider whether the model is the remaining constraint.
About the Author
I'm Celin Daniel, Co-founder of Coding Droplets. I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.
