RAG Retrieves the Wrong Chunks in .NET: Root Cause and Fix

The support bot has been live for three weeks. A customer asks about the refund window on annual plans, and it answers confidently and completely wrong, citing a document about trial cancellations. Nobody gets an exception. Nothing is logged as an error. The endpoint returns 200 in 900 milliseconds. When your RAG retrieves the wrong chunks, the failure is silent by design, because the model does exactly what you asked: it answered from the context you gave it. The context was just wrong.

In production I've debugged this more times than any other AI failure, and the single most expensive mistake teams make is starting at the prompt. The prompt is almost never the problem. This article is the diagnostic order I now follow, the four root causes it converges on, and the fix for each one in a .NET stack. If you want the instrumented retrieval pipeline with score logging and the diagnostic endpoint already wired up, the complete version is on Patreon.

The reason this is hard to reason about from a snippet is that retrieval quality is a property of the whole pipeline, not any single call. Chapter 10 of AI-Powered .NET APIs walks the full retrieval flow inside a running support API - top-k, score thresholds, hybrid keyword plus vector search - and then works through the failure modes and how to debug them, which is exactly the loop below.

Targets .NET 10 with Microsoft.Extensions.AI and Microsoft.Extensions.VectorData.

The Symptom

The signature of this failure is a fluent, confident, wrong answer. Specifically:

• The answer is well-formed and on-topic, which is why reviewers miss it.

• Citations point to real documents that are adjacent to the question but do not contain the answer.

• Rephrasing the question slightly sometimes produces the right answer, which makes it look intermittent.

• Latency and error rates look completely normal on your dashboards.

If instead the model says "I don't know" when the answer clearly exists in the knowledge base, that is the same root cause presenting differently - retrieval failed, and your grounding instruction did its job. Treat both as the same investigation.

Why It Happens

Vector search does not find the correct chunk. It finds the nearest chunk in embedding space, and always returns something. There is no null. A cosine similarity of 0.42 and a similarity of 0.91 both come back as results, ranked, looking identical in shape. If your code takes the top three results unconditionally and stuffs them into the prompt, you have built a system that cannot tell "I found the answer" from "I found the least-unrelated paragraph in the corpus."

Everything downstream then works perfectly on garbage input. The prompt is well-constructed. The model is grounded. The citations are accurate references to the chunks you supplied. The output is wrong.

How to Diagnose It

Work bottom-up. Do not touch the prompt until step three fails.

Step 1: Log the Retrieval, Not Just the Answer

You cannot debug what you cannot see. Before anything else, log the top-k results with their scores for every query. VectorSearchResult<TRecord> exposes both Record and Score, so this is a few lines:

var results = collection
    .SearchAsync(question, top: 5, cancellationToken: ct);

await foreach (var hit in results)
{
    logger.LogInformation(
        "Retrieved {DocId} score={Score:F3} : {Preview}",
        hit.Record.DocumentId, hit.Score, hit.Record.Text[..Math.Min(80, hit.Record.Text.Length)]);
}

SearchAsync on VectorStoreCollection<TKey, TRecord> returns an async stream of results. Log the score, the source document id, and enough of the chunk text to recognise it. Within a day of production traffic the pattern is usually obvious from the logs alone.

Add these scores to your traces too, alongside token counts and latency. Our guide on OpenTelemetry for AI endpoints covers wiring that into an existing pipeline.

Step 2: Ask Whether the Right Chunk Was Retrieved at All

Take a failing question and check the logged results by hand. There are exactly two outcomes, and they lead to completely different fixes:

• The correct chunk is not in the top-k at all. This is a retrieval problem. Continue to the root causes below.

• The correct chunk is in the top-k, but ranked third or fourth. This is a ranking problem, and it is usually easier to fix.

Step 3: Test Generation in Isolation

If the correct chunk was retrieved, hand-build a prompt containing only that chunk and ask the question directly. If the model now answers correctly, generation is fine and the problem is entirely upstream - ranking or context dilution. If the model still answers wrongly with perfect context in front of it, then and only then is your prompt or your grounding instruction the issue.

This one test saves days. It cleanly separates "retrieval is broken" from "prompting is broken" and I have never regretted running it first.

Root Cause 1: No Score Threshold

The most common cause by a wide margin. Code that takes top: 3 with no floor will happily return three barely-related chunks for a question the knowledge base has no answer to.

The fix is a relevance floor, and the important part is calibrating it rather than guessing. Run 50 to 100 representative questions through retrieval, log the score of the correct chunk in each, and look at the distribution. Set the threshold just below the worst score at which retrieval was still correct. Filter anything below it out entirely, and if nothing survives, return an honest "I don't have information about that" rather than calling the model at all.

That last point matters more than the threshold value. A RAG endpoint that refuses cleanly when it has nothing is more trustworthy than one that always answers.

Thresholds are not portable. A cosine similarity threshold tuned for one embedding model will be wrong for another, and switching distance functions changes the scale entirely. Recalibrate whenever either changes.

Root Cause 2: Chunking Destroyed the Answer

If the correct chunk never appears in the top-k regardless of the query, look at the chunks themselves, not the search.

Two failure shapes dominate. Chunks too large produce a single embedding averaging several unrelated topics, so it is weakly similar to everything and strongly similar to nothing. Chunks too small or split badly cut the answer in half, so the chunk containing "annual plans" no longer contains "30 days."

The diagnostic is blunt and effective: pull the chunk that should have matched out of the store and read it. Nine times out of ten you will immediately see that it starts mid-sentence, or that the heading giving it context ended up in the previous chunk.

The fix is structural: split on document structure rather than character count where the source allows it, keep overlap between adjacent chunks so a boundary cannot orphan an answer, and prepend the section heading to each chunk's text before embedding so context travels with the content. We cover the strategies and sizes in detail in chunking documents for RAG in .NET.

Re-chunking means re-embedding and re-indexing everything. Budget for it.

Root Cause 3: The Vocabulary Gap

Users search with words your documents do not contain. Someone types "SKU 4471 keeps failing" and your knowledge base only ever calls it "Model 4471-B." Pure dense vector search is weak on exact identifiers, error codes, product names and acronyms, because embeddings capture meaning and an opaque part number carries very little.

The fix is hybrid search: combine dense vector similarity with keyword matching so exact-token queries have a path to the right document. Microsoft.Extensions.VectorData exposes this through IKeywordHybridSearchable<TRecord> and HybridSearchOptions<TRecord> where the underlying store supports it. Our post on hybrid search for RAG in .NET covers the fusion strategies.

If your vector store has no hybrid support, this is a legitimate reason to switch stores. Check that capability against your query mix before committing, using our vector store decision guide.

Root Cause 4: Retrieved but Ranked Badly

If the correct chunk is consistently retrieved at position three or four, dense retrieval is doing its job as a candidate generator and the ordering is the weak link. Models attend unevenly across a long context, so a correct chunk buried behind two mediocre ones often loses.

Two fixes, in increasing order of cost:

• Retrieve wider, keep fewer. Fetch the top 20, then apply your relevance floor and pass only the survivors. This costs one search, not twenty.

• Add a reranker. A cross-encoder scores each candidate against the actual question rather than comparing precomputed embeddings, and it reorders far more accurately. It costs a second model call per query, so it belongs behind a cache and is worth measuring before adopting.

Also check for context dilution: passing eight chunks when two would do measurably degrades answer quality and costs more tokens. More context is not better context.

Root Cause 5: A Stale Index

Worth ruling out early because it is embarrassing and cheap to check. The document was updated; the embedding was not. Retrieval is working perfectly and returning the old text.

Confirm by fetching the record directly by key and comparing it to the source. If they differ, your ingestion pipeline is not keeping up. The fix is a content hash per source document, checked by a background job, so a changed document is re-chunked and re-embedded automatically rather than on someone remembering to run a script.

How to Stop It Recurring

Fixing today's wrong answer is not the goal. Making the failure visible next time is.

• Log retrieval scores on every request, permanently. This is the single highest-value change in this article. Patterns appear within days.

• Alert on the refusal rate and on the top-1 score distribution. A drifting score distribution is an early warning that content or embeddings changed underneath you.

• Keep a golden set. Thirty question-and-expected-source pairs, run in CI. It catches a chunking change or an embedding model swap before it reaches users. Wire it up with Microsoft.Extensions.AI.Evaluation as covered in evaluating LLM output in .NET.

• Expose a diagnostic endpoint behind auth that returns the retrieved chunks and scores for a question without generating an answer. Debugging a user report drops from an hour to a minute.

• Version the index alongside the embedding model. Changing embedding models invalidates every stored vector. Make that a deliberate, versioned migration rather than a config edit.

Frequently Asked Questions

Why Does My RAG System Return Confident Wrong Answers Instead of Saying It Doesn't Know?

Because vector search always returns its nearest neighbours, however distant, and the model is instructed to answer from the provided context. Without a relevance floor, "no good match" and "great match" are indistinguishable to your code. Add a score threshold and refuse explicitly when nothing clears it.

What Score Threshold Should I Use for Vector Search in .NET?

There is no portable number. It depends on the embedding model, the distance function and your content. Calibrate it: run 50 to 100 representative questions, log the score of the correct chunk in each, and set the floor just below the lowest score that was still correct. Recalibrate whenever the embedding model or distance function changes.

How Do I Know Whether the Problem Is Retrieval or the Prompt?

Log the retrieved chunks, then hand-build a prompt containing only the correct chunk and ask the question. Right answer means retrieval is at fault. Still wrong means the prompt or grounding instruction is. Run this test before changing any prompt text.

Will Better Chunking Fix Retrieval on Its Own?

Sometimes, but not usually on its own. Chunking fixes the case where the answer was split or diluted. It does nothing for the vocabulary gap, for a missing score threshold, or for a stale index. Diagnose first: re-chunking forces a full re-embed and re-index, which is expensive to do speculatively.

Does Hybrid Search Always Beat Pure Vector Search?

No. Hybrid search wins clearly when queries contain exact tokens - error codes, SKUs, product names, acronyms - and adds cost and tuning surface otherwise. If your query logs are dominated by natural-language questions with no identifiers, dense retrieval plus a threshold is usually enough. Look at your actual query mix before adding it.

Where Are the Official .NET Vector Search APIs Documented?

The Microsoft.Extensions.VectorData namespace reference documents VectorStoreCollection<TKey, TRecord>, VectorSearchOptions<TRecord>, VectorSearchResult<TRecord> and the hybrid search interfaces, and is the authority on which capabilities each connector supports.

---

About the Author

I'm Celin Daniel, Co-founder of Coding Droplets. I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.

• GitHub: codingdroplets

• YouTube: Coding Droplets

• Website: codingdroplets.com