Managing LLM Conversation History in .NET: Root Cause and Fix for Context Window Overflow

The first AI chat endpoint you ship almost always works. You append each user message and each model reply to a List<ChatMessage>, send the whole list on every turn, and the assistant remembers the conversation. Then it runs for a few weeks in production and the support tickets start: replies get slower, the token bill creeps up, and eventually a long-running conversation returns a hard error about exceeding the model's maximum context length. Managing LLM conversation history in .NET is the difference between a demo that impresses and an endpoint that survives real usage, and almost every team learns that the expensive way.

I have shipped enough Microsoft.Extensions.AI endpoints to know this failure has a predictable shape. It is not a model bug and it is not a bug in IChatClient. It is unbounded history: the message list grows without limit, and every growth costs you tokens on every single turn. Most walkthroughs stop at the happy-path chat loop, so the fix never gets covered. The complete version - session persistence, token budgeting, and the edge cases that bite under load - is something I keep annotated and ready to run on Patreon, where the source code maps directly to what production teams actually deploy.

Getting conversation state right means handling history trimming and token budgeting at the same time, not one after the other. That combination is exactly what Chapter 4 of the AI-Powered .NET APIs course walks through inside one running support-desk API - DB-backed sessions, trimming strategies, and counting tokens before you send, all wired together so the context is never lost.

What Actually Breaks When Conversation History Grows

The symptoms show up in a specific order, and recognizing them early saves you a production incident.

• Latency climbs turn over turn. Every message you resend has to be processed by the model. A 40-turn conversation sends 40 turns of context on turn 41, so time-to-first-token gets worse the longer someone chats.

• Token cost grows quadratically, not linearly. This is the part that surprises people. If each turn adds roughly the same number of tokens, the cost of turn N includes all N-1 previous turns. Over a full conversation the total tokens billed scale with the square of the turn count.

• Then it fails outright. Once the accumulated history plus the next reply exceeds the model's context window, the provider rejects the request. Depending on the client that surfaces as an exception or an error response, and it always happens to your most engaged users first, because they have the longest conversations.

In production I have seen the cost symptom get blamed on "the model being expensive" for weeks before anyone realized a handful of long-lived sessions were resending tens of thousands of tokens per turn. The runaway spend that comes with it is a related failure worth understanding on its own, which I broke down in Runaway LLM Costs in a .NET API.

Why Does LLM Conversation History Overflow the Context Window?

Because most chat services are stateless, and stateless means you resend everything every time. There is no memory on the other side - the model only knows what is in the message list you hand it on this request.

The Microsoft.Extensions.AI docs are explicit about this: with a stateless service, callers maintain a list of all messages, add in all received response messages, and provide the list back on subsequent interactions. That is the standard loop, and it is correct - right up until the list gets too big:

// The pattern that works in a demo and fails in production.
List<ChatMessage> history = [];
while (true)
{
    history.Add(new(ChatRole.User, ReadUserInput()));

    ChatResponse response = await chatClient.GetResponseAsync(history);
    Console.WriteLine(response);

    history.AddMessages(response); // history only ever grows
}

Nothing here trims history. Every turn it gets longer, and every turn you pay to send all of it. A context window is a hard ceiling measured in tokens (a typical model might allow tens of thousands to a few hundred thousand tokens), and unbounded history marches straight at that ceiling. The root cause is not the ceiling - it is the absence of any policy for what to keep and what to drop.

How Do You Diagnose It Before It Bites?

Measure tokens per turn, not just messages per turn. Microsoft.Extensions.AI gives you real usage numbers on the response so you do not have to guess.

Every ChatResponse exposes a Usage property of type UsageDetails, with InputTokenCount, OutputTokenCount, and TotalTokenCount:

ChatResponse response = await chatClient.GetResponseAsync(history);

// UsageDetails lives in Microsoft.Extensions.AI.Abstractions.
long inputTokens = response.Usage?.InputTokenCount ?? 0;
long totalTokens = response.Usage?.TotalTokenCount ?? 0;

logger.LogInformation(
    "Turn used {Input} input tokens, {Total} total for session {SessionId}",
    inputTokens, totalTokens, sessionId);

Log InputTokenCount per turn keyed by session and the problem becomes obvious in your dashboards: healthy sessions stay flat, broken ones climb a ramp. When you stream responses the usage arrives on the final update, so gather it from the last ChatResponseUpdate rather than expecting it mid-stream - a detail I covered while building the streaming endpoint in Stream LLM Responses in ASP.NET Core with IChatClient. Set an alert when per-turn input tokens cross a threshold and you catch this before a user does.

The Fix: Reduce the History Before You Send It

The fix is a policy that runs on every turn and decides what stays in the message list. There are three approaches worth knowing, and real systems often combine them.

Sliding Window: Keep the Last N Turns

The simplest reliable policy: always keep the system message, then keep only the most recent N messages. Older turns fall off the back.

static List<ChatMessage> TrimToWindow(List<ChatMessage> history, int maxTurns)
{
    var system = history.Where(m => m.Role == ChatRole.System);
    var recent = history
        .Where(m => m.Role != ChatRole.System)
        .TakeLast(maxTurns);

    return system.Concat(recent).ToList();
}

This is cheap, predictable, and has no extra model calls. The trade-off is bluntness: anything older than the window is simply gone, so if a user refers back to something they said 30 messages ago, the model no longer sees it. For support chat, task assistants, and most Q and A flows, a window of the last 10 to 20 messages is plenty.

Summarize Older Turns: SummarizingChatReducer

When you genuinely need long-range memory, summarize instead of dropping. Microsoft.Extensions.AI ships a SummarizingChatReducer that compresses older messages into a running summary while keeping recent turns verbatim.

// Experimental API - you must suppress diagnostic MEAI001 to use it.
#pragma warning disable MEAI001

IChatReducer reducer = new SummarizingChatReducer(
    chatClient,        // client used to produce the summary
    targetCount: 10,   // how many recent messages to keep intact
    thresholdCount: 4); // trigger a summary once this many extra pile up

IEnumerable<ChatMessage> reduced = await reducer.ReduceAsync(history, ct);

#pragma warning restore MEAI001

Two things to know before you reach for it. First, it is genuinely experimental - the type carries the MEAI001 diagnostic, so it will not compile until you suppress that warning, and the surface can change between releases. Package this as of Microsoft.Extensions.AI 10.7.0 on .NET 10. Second, summarizing costs a model call, so it is not free - it trades token volume for an extra request. Usefully, the reducer preserves system messages and leaves function-call and function-result messages out of the summary, which keeps tool interactions intact.

Let a Stateful Service Hold the History

Some providers are stateful: they store the conversation server-side and you only send the new message. With Microsoft.Extensions.AI you opt into that with ChatOptions.ConversationId.

ChatOptions options = new();
while (true)
{
    ChatMessage message = new(ChatRole.User, ReadUserInput());
    ChatResponse response = await chatClient.GetResponseAsync(message, options);

    // Carry the id forward; the service keeps the history for you.
    options.ConversationId = response.ConversationId;
}

When ConversationId comes back set, you clear your local history and stop resending it. This pushes the problem to the provider, which is convenient, but it does not eliminate it - a server-side conversation still has a context limit, you just have less visibility and control over the trimming. Know which mode your provider is in before you design around it.

A Token Budget Is the Real Guardrail

Message-count windows are a proxy. The thing the context window actually measures is tokens, so the durable fix is a token budget per request. Decide the maximum input tokens you are willing to spend on a turn, trim until history fits under it, and reserve headroom for the model's reply.

The practical recipe I use: pick a budget well under the model ceiling (leaving room for the response), keep the system prompt and the newest turns, and drop or summarize from the oldest end until the estimated token count fits. Estimate with a real tokenizer for the model family rather than guessing from character counts, because token-to-character ratios vary by language and content. A budget makes cost predictable per turn, which is the number your finance team actually cares about.

How to Stop It Coming Back

A one-time trim is not a fix - the policy has to live in the request path so it applies to every turn, forever.

1. Persist sessions with the policy attached. Store conversation history in a database keyed by session, and apply the window or budget on load, so restarts and scale-out never resurrect an unbounded list.

2. Put reduction in the pipeline, not in a controller. IChatClient composes as a pipeline of delegating clients. Applying reduction as a pipeline step means every call site gets it automatically and no one can forget it.

3. Alert on per-turn input tokens. You already log InputTokenCount from the diagnosis step - wire an alert to it so a regression pages you instead of surprising your users.

4. Cap conversation length by design. For many products, starting a fresh session after a natural boundary is better UX and cheaper than carrying an endless thread.

Get these in place and long conversations become a non-event. The endpoint that used to degrade over a session now holds steady on latency and cost no matter how long someone chats.

Frequently Asked Questions

What is the maximum length an LLM conversation history can reach in .NET?

There is no fixed message limit - the real ceiling is the model's context window measured in tokens, and it is shared across your system prompt, the full history you resend, and the space reserved for the reply. Two short turns or twenty long ones can hit the same wall. Budget in tokens, not messages.

Does Microsoft.Extensions.AI trim conversation history automatically?

No. By default IChatClient sends exactly the message list you provide, so an unbounded List<ChatMessage> grows forever. Trimming is opt-in: either apply your own sliding window, use the experimental SummarizingChatReducer, or delegate to a stateful service via ChatOptions.ConversationId.

How do I count tokens for an LLM conversation in .NET?

Read the real numbers off the response: ChatResponse.Usage is a UsageDetails with InputTokenCount, OutputTokenCount, and TotalTokenCount. For pre-send estimates, use a tokenizer that matches your model family rather than approximating from character length, since the ratio varies by content and language.

Is SummarizingChatReducer production ready?

It is marked experimental and gated behind the MEAI001 diagnostic, so treat it as subject to change and pin your package version. It works well for long-lived assistants that need long-range memory, but it costs an extra model call per summarization. For most support and Q and A flows, a plain sliding window is simpler and sufficient.

Should I store conversation history in a database or keep it in memory?

Persist it. In-memory history dies on restart and does not survive horizontal scale-out, both of which are normal in production. Store history per session in a database and apply your trimming policy when you load it, so no instance ever rebuilds an unbounded conversation.

Why does my token cost grow so fast in a long chat?

Because a stateless chat resends the entire history every turn, so turn N pays for all the turns before it. Total spend over a conversation scales with roughly the square of the turn count. Bounding history with a window or a token budget is what turns that curve back into a flat line.

---

About the Author

I'm Celin Daniel, Co-founder of Coding Droplets. I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.

• GitHub: codingdroplets

• YouTube: Coding Droplets

• Website: codingdroplets.com