Retrieval-Augmented Generation lives or dies on one unglamorous step, and it is not the model or the vector database. It is chunking. When you get chunking documents for RAG in .NET wrong, the model retrieves the wrong context, answers confidently from it, and your users lose trust in the feature within a week. I'm Celin Daniel, Co-founder of Coding Droplets, and across 13+ years of building .NET systems in production I've watched more RAG pipelines fail at the chunking layer than at any other point. The embeddings were fine. The prompt was fine. The chunks were garbage.
This walkthrough is the practical version of what I wish someone had handed me on my first RAG project: how to split source documents into chunks that actually retrieve well, which strategies matter in the .NET ecosystem, and how to wire the result into an ingestion pipeline. If you want the full production ingestion service - background re-indexing, admin endpoints, and edge cases handled end to end - the complete annotated codebase lives on Patreon, ready to run and adapt. Getting chunking right also means thinking about embeddings, storage, and index sync at the same time, and the AI-Powered .NET APIs course builds exactly that ingestion pipeline in Chapter 9 inside one running ASP.NET Core support API, so the moving parts are always connected rather than shown in isolation.
Why Chunking Decides Whether Your RAG Pipeline Works
RAG works by embedding your documents into vectors, storing them, and at query time retrieving the closest vectors to the user's question and stuffing that text into the prompt. The unit you embed and retrieve is the chunk. If a chunk is too large, its embedding becomes an averaged blur of several topics and matches nothing precisely. If it is too small, it retrieves a sentence with no surrounding context and the model has nothing to reason over. If it splits mid-thought, the one sentence that answers the question gets cut in half and neither half ranks.
That is why chunking is a design decision, not a utility function you copy from a gist. The most common anti-pattern I see is blind fixed-length slicing:
// Anti-pattern: slice by character count, ignore all structure
var chunks = Enumerable
.Range(0, text.Length / 1000 + 1)
.Select(i => text.Substring(i * 1000, Math.Min(1000, text.Length - i * 1000)));
// Splits mid-word and mid-sentence, no overlap - retrieval quality collapses.
This runs, ships, and quietly ruins your retrieval. Every boundary lands in the middle of a sentence, and with no overlap the context around each cut disappears. It is the single most common reason a RAG demo works and the production version does not.
What Makes a Good Chunk?
A good chunk is a self-contained unit of meaning that fits comfortably inside your embedding model's context and can answer a question on its own. In practice it satisfies three properties:
Semantic coherence - it covers one topic, not the tail of one section and the start of the next.
Right-sized - large enough to carry context, small enough that its embedding stays focused. For most document RAG that lands around 256 to 512 tokens.
Overlapping at boundaries - a small overlap (roughly 10 to 20 percent) so a sentence that straddles two chunks survives in at least one of them.
Hold those three properties in mind and every chunking strategy below is just a different way of trying to hit them.
The Three Chunking Strategies That Matter in .NET
You will read about a dozen exotic strategies online. In real .NET projects, three cover the vast majority of cases.
Fixed-Size Chunking
Split the text into fixed token windows with a fixed overlap. It is the simplest and cheapest option, and it is a perfectly reasonable baseline for uniform content like transcripts or logs. Its weakness is that it ignores document structure, so it will happily cut through the middle of a table or a code block. Add overlap and it becomes tolerable; without overlap it is the anti-pattern above.
Recursive (Structure-Aware) Chunking
Recursive chunking tries progressively finer boundaries - sections, then paragraphs, then sentences - and only falls back to a hard cut when a unit still exceeds the size limit. This respects the natural shape of the document, so headings, paragraphs, and list items stay intact. For most RAG applications over articles, docs, and knowledge-base content, this is the default I reach for first. It is the sweet spot between quality and effort.
Semantic Chunking
Semantic chunking decides boundaries by meaning: it embeds consecutive sentences and starts a new chunk when the similarity between neighbours drops below a threshold, so each chunk aligns with a genuine topic shift. It produces the cleanest chunks but costs extra embedding calls at ingestion time and adds real complexity. Reach for it only when recursive chunking measurably underperforms on your content - not by default.
How Do You Chunk Text in .NET Without Building It From Scratch?
You do not need to hand-roll a tokenizer-aware splitter. The .NET AI stack already ships one. Microsoft.SemanticKernel.Text.TextChunker (in the Microsoft.SemanticKernel.Core package) gives you token-aware paragraph splitting with overlap, and it works standalone even if the rest of your app uses Microsoft.Extensions.AI rather than Semantic Kernel. The API is a two-step split: turn raw text into lines, then combine lines into paragraph-sized chunks.
#pragma warning disable SKEXP0050 // TextChunker is an experimental API
using Microsoft.SemanticKernel.Text;
// Step 1: break the document into small lines
var lines = TextChunker.SplitPlainTextLines(rawText, maxTokensPerLine: 128);
// Step 2: combine lines into overlapping, size-capped chunks
var chunks = TextChunker.SplitPlainTextParagraphs(
lines,
maxTokensPerParagraph: 512,
overlapTokens: 64);
Version note: TextChunker is marked experimental, so you must suppress SKEXP0050 to use it. For Markdown sources, SplitMarkdownParagraphs does the same thing while keeping headings and list items from being bisected - the structure-aware option that gets you recursive-style behaviour for free.
One production detail that bit us: by default TextChunker counts tokens by splitting on whitespace, which overcounts compared to a real model tokenizer. For anything cost-sensitive or close to the embedding limit, pass a proper TokenCounter backed by the actual tokenizer for your model so maxTokensPerParagraph means what you think it means.
Wiring Chunks Into the Ingestion Pipeline
Chunking is one stage of ingestion. The chunks then get embedded and stored so retrieval can find them later. With Microsoft.Extensions.VectorData you can model a chunk as a record whose vector property sources from its own text, and the configured IEmbeddingGenerator produces the vector on upsert - no manual embedding call in the loop.
public sealed class DocChunk
{
[VectorStoreKey] public Guid Id { get; set; }
[VectorStoreData] public string Text { get; set; } = ""; // kept for grounding + citations
[VectorStoreVector(1536)] public string Embedding { get; set; } = ""; // string source -> vector on upsert
}
foreach (var chunk in chunks)
{
await collection.UpsertAsync(new DocChunk
{
Id = Guid.NewGuid(),
Text = chunk,
Embedding = chunk // same text; the embedding generator vectorises it
});
}
Keep the original chunk text in a data property, not just the vector. You need it to build the grounded prompt and to return citations, and vector properties do not give the source text back. This ties directly into how retrieval and grounding work, which I covered in The RAG Pattern in ASP.NET Core, and the store you upsert into is a decision in itself - see Choosing a Vector Store for .NET AI Apps for that trade-off.
How Big Should a RAG Chunk Be?
For most document RAG, start at 256 to 512 tokens per chunk with 10 to 20 percent overlap, then tune against your own retrieval quality. Smaller chunks (around 128 tokens) favour precise fact lookup like FAQs and product specs; larger chunks (768 tokens and up) favour narrative or reasoning-heavy content where context matters more than pinpoint matching. There is no universal number, which is exactly why you should measure rather than guess - retrieval quality on a handful of real questions tells you more than any blog post's default, including this one.
The Trade-offs That Bit Us in Production
A few hard-won specifics from shipping this:
No overlap is the number one silent killer. The fix costs one parameter and recovers a surprising amount of recall.
Structure-blind chunking destroys tables and code. If your corpus has either, use Markdown-aware or recursive splitting, or pre-clean those sections separately.
Chunking and embedding-model choice are coupled. Switching embedding models can change the effective token budget and the ideal chunk size, so re-tune when you swap models.
Re-indexing is a first-class concern. Documents change. You need a background job that re-chunks and re-embeds updated sources and removes stale chunks, or your index slowly drifts from reality.
What to Do Next
Start with recursive or Markdown-aware chunking at 512 tokens and 64-token overlap, store the chunk text alongside the vector, and evaluate retrieval on ten real user questions before you touch anything exotic. Only move to semantic chunking if the numbers demand it. Get those basics right and the rest of your RAG pipeline - retrieval, grounding, citations - has a solid foundation to stand on.
FAQ
What is the best chunking strategy for RAG in .NET?
For most .NET projects, recursive or structure-aware chunking is the best default because it respects paragraphs and headings while capping chunk size. TextChunker.SplitMarkdownParagraphs gives you this behaviour out of the box. Reserve semantic chunking for cases where recursive chunking measurably underperforms on your specific content.
What chunk size and overlap should I use for RAG?
Start at 256 to 512 tokens per chunk with roughly 10 to 20 percent overlap (for example 512 tokens with 64 tokens of overlap). Use smaller chunks for precise fact retrieval and larger chunks for narrative content, then tune based on measured retrieval quality rather than a fixed rule.
Does Microsoft.Extensions.AI include a document chunker?
No. Microsoft.Extensions.AI provides the IEmbeddingGenerator and chat abstractions, but chunking is left to you. The practical option in the .NET ecosystem is Microsoft.SemanticKernel.Text.TextChunker, which works standalone alongside Microsoft.Extensions.AI even if you are not using the rest of Semantic Kernel.
Why does my RAG return irrelevant results even with good embeddings?
The most common cause is chunking, not embeddings. Chunks that are too large blur multiple topics into one vector, and chunks split mid-sentence with no overlap lose the exact context that answers the question. Fix the chunk boundaries and overlap first before blaming the embedding model or the vector store.
How do I keep my RAG index up to date when documents change?
Run a background job that detects changed source documents, re-chunks and re-embeds them, upserts the new chunks, and deletes the stale ones. Treat re-indexing as a first-class part of the pipeline. Without it, your vector index slowly drifts out of sync with the source of truth and answers degrade over time.
About the Author
I'm Celin Daniel, Co-founder of Coding Droplets. I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.
