Resilient LLM Calls in .NET: Retries, Timeouts, and Fallbacks Done Right

The first time an AI feature I shipped went down in production, nothing in my code had changed. The model provider had a bad afternoon: a wave of 429 Too Many Requests, then a stretch of 503s, and every request that touched the LLM either hung for thirty seconds or threw. The endpoint that summarized support tickets went from "delightful" to "the reason the on-call phone rang" in about ten minutes. That day taught me that resilient LLM calls in .NET are not an optional polish step. They are the difference between an AI feature that degrades gracefully and one that takes the whole request path down with it.

The uncomfortable truth is that a call to an LLM is the least reliable dependency in your entire system. It is a network hop to a shared, rate-limited, occasionally-overloaded service that charges you per token and sometimes answers a question differently than it did a second ago. If you want the full resilience layer wired into a running support API - retry policies, a fallback model, budget-aware backoff, and the tests that prove it works - the annotated implementation lives on Patreon, assembled end to end rather than as isolated snippets.

Getting this right means treating retries, timeouts, and fallbacks as one connected design rather than three switches you flip independently, because on an AI endpoint they interact with cost and latency in ways a normal HTTP dependency never does. That is exactly what Chapter 15 of the AI-Powered .NET APIs course walks through: token budgets, model tiering, caching, and resilience (retries, timeouts, and fallback models) built into one real ASP.NET Core codebase, so the trade-offs are always concrete.

What the Resilience Pattern Actually Solves

At its core, the resilience pattern wraps an unreliable call in a pipeline of strategies that decide what to do when the call misbehaves: retry the transient failures, give up quickly on the permanent ones, cap how long any single attempt can run, stop hammering a service that is clearly down, and provide an alternative path when the primary one fails. In .NET this is the domain of Polly and Microsoft.Extensions.Http.Resilience, and the strategies themselves are not new.

What is new is applying them to an LLM call, where three assumptions from ordinary HTTP resilience quietly break:

• Every retry costs money. A retried REST call wastes a few milliseconds. A retried LLM call re-sends the entire prompt and pays for the input tokens again. Aggressive retry counts that are harmless on a JSON API can multiply your bill on an AI endpoint.

• The response is non-deterministic. Retrying a failed database write and getting the same row back is normal. Retrying an LLM call can return a subtly different answer, which matters if you already streamed part of the first attempt to the user.

• Latency is measured in seconds, not milliseconds. A model call can legitimately take ten to twenty seconds. A timeout tuned for a database query will sever healthy requests, and a retry with a short delay will pile on before the first attempt was ever going to fail.

The pattern fits when you have accepted those realities and want the LLM to be a well-behaved dependency: predictable under load, bounded in cost, and honest about failure.

When to Apply It, and When Not To

Reach for a full resilience pipeline when the LLM call sits on a user-facing or revenue-relevant path: a chat endpoint, a classification step that gates a workflow, a RAG answer your customers see. These are the places where a transient 429 should never surface as a 500, and where a slow provider should not hold a thread hostage.

Be more restrained in a few cases. For a fire-and-forget background job, a simple retry with a long delay and a dead-letter queue is usually enough - you do not need a circuit breaker guarding a job that runs once an hour. For a streaming response that has already sent tokens to the client, a mid-stream retry is often worse than a clean failure, because the user watches the answer restart. And if you are calling a local model through Ollama on the same machine, most of the network-failure surface simply is not there; a timeout is the only strategy that earns its keep.

The anti-pattern I see most often is the opposite of under-engineering: stacking a five-retry policy with exponential backoff on top of a provider that returns 429 because you are over quota. Every retry makes the quota problem worse and the outage longer. Resilience is about absorbing transient failures, not brute-forcing a structural one.

The Two Layers Where Resilience Lives

A point that trips up a lot of teams: in a Microsoft.Extensions.AI application there are two distinct places you can add resilience, and they do different jobs. Getting them confused leads to either duplicated retries or gaps where you thought you were covered.

The transport layer is the HttpClient underneath your provider client. This is where generic HTTP resilience belongs - retrying transient socket errors and 5xx responses, enforcing a per-attempt timeout, and tripping a circuit breaker when the provider is broadly unavailable. If your provider client is built on IHttpClientFactory, one line gets you the standard pipeline (available via Microsoft.Extensions.Http.Resilience on .NET 8+):

builder.Services
    .AddHttpClient("llm")
    .AddStandardResilienceHandler();

The chat-client layer is the IChatClient pipeline, and this is where LLM-aware resilience belongs - anything that needs to understand what a chat response is. Model fallback, honoring a Retry-After header on a 429, and budget-aware decisions all need semantics the transport layer cannot see. Microsoft.Extensions.AI gives you a builder that composes middleware around the raw client:

IChatClient client = new ChatClientBuilder(innerClient)
    .UseFunctionInvocation()
    .Use(new ModelFallbackChatClient(fallback))  // custom, shown below
    .Build();

The order matters. Function invocation and caching sit closer to the inner client; your resilience middleware wraps the outside so it governs the call as a whole. My rule of thumb: put generic transient-fault handling on the transport, and cost-and-model-aware handling in the chat pipeline. Do not put a blanket retry in both, or a single provider hiccup becomes 3 x 3 = 9 real calls.

How Do You Retry an LLM Call Without Wasting Tokens?

The direct answer: retry only genuinely transient failures, cap attempts low, use exponential backoff with jitter, and honor the provider's Retry-After header instead of guessing. On an AI endpoint, restraint is the whole game because each attempt re-bills the input tokens.

Start by being strict about what you retry. A 429 (rate limit), 408 (request timeout), and 5xx are transient and worth another attempt. A 400 (malformed request), 401 (bad key), or a content-filter rejection are permanent - retrying them just spends money to fail again. A minimal Polly v8 pipeline that respects this looks like:

new ResiliencePipelineBuilder<ChatResponse>()
    .AddRetry(new RetryStrategyOptions<ChatResponse>
    {
        MaxRetryAttempts = 2,               // low on purpose - retries cost tokens
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,
        Delay = TimeSpan.FromSeconds(2)
    })
    .AddTimeout(TimeSpan.FromSeconds(30))   // per-attempt ceiling
    .Build();

Two attempts, not five. The jitter matters more than it looks: when a provider throttles you, every instance of your API tends to fail at the same moment, and synchronized retries create a thundering herd that keeps the 429s coming. Jitter spreads the retries out so the provider can actually recover.

The piece the generic guides miss is the Retry-After header. When a provider sends 429, it frequently tells you exactly how long to wait. Guessing with fixed backoff either wastes time or retries too early; reading the header and delaying by that amount is both faster and kinder to your quota. Because retried prompts re-bill input tokens, an over-eager retry policy is a direct line to the kind of bill I broke down in Runaway LLM Costs in a .NET API - resilience and cost control are the same conversation.

Timeouts and Circuit Breakers, Tuned for Model Latency

A timeout on an LLM call is not there to make slow calls fast - it is there to stop a broken call from holding a connection and a thread until the client gives up. Set a per-attempt timeout generously (a real model call can run twenty seconds or more, especially with a long context), and set a separate total timeout across all retries so a request cannot spiral into a two-minute hang. Polly v8 lets you compose both: an inner timeout inside the retry loop and an outer timeout around the whole pipeline.

The circuit breaker is the strategy teams forget until they need it. When a provider is genuinely down, you do not want every incoming request to wait for its own timeout before failing - that turns a provider outage into a thread-pool starvation incident on your side. A breaker that trips after a run of failures and fails fast for the next thirty seconds contains the blast radius. The same shape carries over from other transports; I use the identical mental model I laid out in gRPC Client Resilience in ASP.NET Core, just with timeouts scaled up for model latency. State it plainly near any snippet: ResiliencePipelineBuilder, AddTimeout, and AddCircuitBreaker are Polly v8 APIs and assume the .NET 8+ resilience packages.

Model Fallback: The Strategy Unique to AI

Here is where LLM resilience diverges from every other kind. When your primary path fails, an ordinary system returns a cached value or a default. An AI system can do something better: call a different model. If the flagship model is throttled or down, a smaller, cheaper, or alternate-provider model can often answer well enough to keep the feature alive. This is model tiering used defensively.

The clean way to express this in Microsoft.Extensions.AI is a custom middleware built on DelegatingChatClient, the base class designed for exactly this - it forwards every call to an inner client and lets you override only what you need:

public sealed class ModelFallbackChatClient(IChatClient primary, IChatClient fallback)
    : DelegatingChatClient(primary)
{
    public override async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        CancellationToken ct = default)
    {
        try { return await base.GetResponseAsync(messages, options, ct); }
        catch (Exception ex) when (IsTransient(ex))
        {
            return await fallback.GetResponseAsync(messages, options, ct);
        }
    }
}

One gotcha that cost me an afternoon: if you only override GetResponseAsync, your streaming callers silently bypass the fallback, because streaming flows through GetStreamingResponseAsync. When a middleware needs to cover both, override both methods - otherwise half your traffic runs code you think is protected but is not.

Fallback earns its complexity when availability matters more than getting the best possible answer every single time. Skip it when a downgraded answer is worse than no answer - for example a legal or medical summarization step where a weaker model's output could be actively misleading. In that case, fail cleanly and tell the user, rather than quietly swapping in a model you would not have chosen.

The Trade-Offs Worth Naming

No resilience choice is free, and pretending otherwise is how teams end up surprised in production:

• Retries trade cost and latency for success rate. Each attempt adds token spend and wall-clock time. Two retries on a slow model can turn a 20-second call into a 60-second one for the user who hits the bad path.

• Timeouts trade completeness for predictability. A tight timeout protects your threads but will cut off a legitimately long generation. Tune it to your real p99, not to a database-era instinct.

• Circuit breakers trade individual requests for system health. While the breaker is open, everyone fails fast - including requests that might have succeeded. That is the correct trade during an outage and the wrong one if the breaker is too sensitive.

• Fallback trades answer quality for availability. A cheaper model keeps the lights on but may lower the bar. Decide per feature whether that is acceptable.

The reason to write these down is that resilience settings are not "set once and forget." They are dials you tune against real traffic, and the right position moves as your load, your provider, and your budget change.

Frequently Asked Questions

What is the best way to make LLM calls resilient in .NET?

Layer the strategies deliberately. Put generic transient-fault handling (retry on 5xx and socket errors, a per-attempt timeout, a circuit breaker) on the HttpClient transport via Microsoft.Extensions.Http.Resilience, and put LLM-aware handling (model fallback, Retry-After awareness, budget-aware backoff) in the IChatClient pipeline using DelegatingChatClient middleware. Keep retry counts low because each attempt re-bills tokens, and only retry failures that are genuinely transient.

Should I retry a 429 Too Many Requests from an LLM provider?

Yes, but carefully. A 429 is transient, so a small number of retries with exponential backoff and jitter is appropriate. The key is to read the Retry-After header when the provider sends it and wait exactly that long, rather than guessing. If you are hitting 429 constantly, that is a quota or capacity problem, not a transient blip - retrying harder will make it worse, and the real fix is rate limiting your own callers or raising your quota.

How long should the timeout be for an LLM call?

Longer than you think. Model calls routinely take ten to twenty seconds, and more with large contexts or reasoning models, so a per-attempt timeout in the 30 to 60 second range is reasonable for a user-facing call. Set it from your measured p99 latency, not from HTTP or database defaults. Pair the per-attempt timeout with a separate total timeout across all retries so a bad request cannot hang for minutes.

Does adding retries to LLM calls increase my token costs?

It can, significantly. A retry re-sends the full prompt, so every retried attempt pays for the input tokens again. This is the biggest way LLM resilience differs from ordinary HTTP resilience. Keep MaxRetryAttempts low (one or two), only retry transient failures, and cache where you can, so resilience does not quietly inflate your bill.

What is model fallback and when should I use it?

Model fallback means routing to a different model - usually smaller, cheaper, or from another provider - when the primary model is unavailable or throttled. It is unique to AI resilience: instead of returning a default value on failure, you get a real, if slightly lower-quality, answer. Use it on availability-critical paths where a good-enough answer beats an error. Avoid it where a weaker model's output could be misleading, and fail cleanly instead.

Where should resilience live: the HttpClient or the IChatClient pipeline?

Both, for different concerns. The HttpClient layer handles transport-level transient faults and does not understand what a chat response is. The IChatClient pipeline handles anything that needs chat semantics, like fallback and cost-aware logic. The mistake to avoid is putting the same retry in both layers, which multiplies your real call count during a single provider hiccup.

---

About the Author

I'm Celin Daniel, Co-founder of Coding Droplets. I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.

• GitHub: codingdroplets

• YouTube: Coding Droplets

• Website: codingdroplets.com