# LLM Model Routing in .NET: Cheap Model First, Expensive Only When Needed

Look at a week of traffic through any AI endpoint and the distribution is always lopsided. A large majority of requests are trivial - classify this ticket, extract these three fields, answer a question the docs already answer. A small minority genuinely need the best model you can buy. If every request goes to the frontier model, you are paying premium rates to do keyword matching. LLM model routing in .NET fixes that by choosing the model per request instead of per deployment, and it is the single highest-leverage cost control I have shipped on an AI feature that was already in production.

The pattern is straightforward. What is not straightforward is deciding *when* to escalate without quietly degrading answer quality, which is where most implementations fall apart. The complete routing client with the classifier, the escalation path, and the evaluation harness that proves quality held is available on [Patreon](https://www.patreon.com/CodingDroplets).

Routing is one lever among several, and pulling it in isolation tends to just move the cost somewhere else. [Chapter 15 of AI-Powered .NET APIs](https://aiapis.codingdroplets.com/) walks through model tiering next to caching, token budgets, and fallback models, so you can see how they interact inside one working support API instead of optimising one number at a time.

[![AI-Powered .NET APIs](https://newsletter.codingdroplets.com/images/ai-api-course-banner-1.jpg align="center")](https://aiapis.codingdroplets.com/)

## What Model Routing Solves

A single model choice forces one compromise across every request your API serves. Pick the cheap model and complex requests come back weak. Pick the frontier model and you overpay for the easy ones, while also inheriting its latency on requests that never needed it.

Routing turns that global decision into a per-request one:

| Request type | Typical share of traffic | Tier |
| --- | --- | --- |
| Classification, extraction, short factual answers | Majority | Small |
| Summarisation, multi-step reasoning over short context | Moderate | Medium |
| Long-context analysis, complex reasoning, code generation | Small | Large |

The economics follow directly from that distribution. Because the small tier absorbs the bulk of the volume at a fraction of the per-token price, the blended cost lands far closer to the cheap model than the expensive one. The exact saving depends entirely on your own traffic mix, which is why the first step is measurement, not implementation. We covered how to instrument that in [Runaway LLM Costs in a .NET API](https://codingdroplets.com/runaway-llm-costs-dotnet-api).

Latency improves for the same reason, and that benefit is often the one users actually notice.

## How Do You Decide Which Model a Request Should Use?

You classify the request before dispatching it, using one of four strategies that trade accuracy against cost and latency.

**Static routing by endpoint.** The simplest and, in my experience, the most underrated. If `/classify` is always a small-model job and `/analyse-contract` is always a large-model job, you do not need a classifier at all - you need two registrations. Start here. A surprising share of the available saving is captured by this alone, with zero risk of misrouting.

**Heuristic rules.** Input token count, presence of an attachment, conversation depth, whether tools are enabled. Deterministic, free, and instantly explainable when someone asks why a request cost what it did.

**Embedding similarity.** Embed the request and compare it against labelled example sets for each tier. Cheap relative to a completion, and it handles phrasing variety that keyword rules miss.

**A small classifier model.** Ask a fast, cheap model to grade complexity, then route. The most flexible option and the one with the worst failure mode: you have added a model call to the critical path of every request, including the ones that were about to be trivially cheap.

Start with the first two. Add the others only when you can measure that routing accuracy, not just cost, is the thing limiting you.

## The Escalation Pattern

Classification-before-dispatch has an obvious weakness: you are guessing at difficulty before seeing the answer. The cascade pattern removes the guess.

Send the request to the cheap model first. Inspect the result. If it meets a confidence bar, return it. If not, re-run against the larger model and return that instead.

This is more robust than pure upfront classification because the signal is the actual output rather than a prediction about it. Practical confidence signals that work without another model call:

*   The model refused, hedged, or returned an explicit "I don't know"
    
*   Structured output failed schema validation
    
*   For RAG, retrieval scores were below your grounding threshold
    
*   A required field came back empty or obviously malformed
    

The cost of the pattern is that escalated requests pay twice and take roughly twice as long. That is fine when escalation is rare, and quietly disastrous when your confidence check is badly calibrated and half of traffic escalates. Track your escalation rate as a first-class metric, and alert on it.

## Implementation Sketch in ASP.NET Core

Model this as a client in front of the tier clients, not as branching inside your handlers. The whole point is that calling code should not know routing exists, and [`Microsoft.Extensions.AI`](https://learn.microsoft.com/en-us/dotnet/ai/microsoft-extensions-ai) is built for exactly this kind of decorator.

Register each tier as a [keyed service](https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection#keyed-services) so they can coexist in DI:

```csharp
// Microsoft.Extensions.AI 10.x, .NET 10
builder.Services.AddKeyedChatClient("small",  sp => BuildClient("gpt-4.1-mini"));
builder.Services.AddKeyedChatClient("large",  sp => BuildClient("gpt-4.1"));
```

Then the router itself resolves the tier and delegates:

```csharp
public sealed class RoutingChatClient(IChatClient small, IChatClient large)
    : IChatClient
{
    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        CancellationToken ct = default)
    {
        var tier = _classifier.Classify(messages, options);
        return (tier is Tier.Large ? large : small)
            .GetResponseAsync(messages, options, ct);
    }
    // streaming + Dispose omitted
}
```

Two things to get right in the surrounding pipeline. Put routing **inside** any caching decorator, so a cache hit costs nothing regardless of tier. And record the selected tier as a telemetry attribute on every request, because without it you cannot answer "why did spend go up?" three weeks later. Our walkthrough of [OpenTelemetry for AI endpoints](https://codingdroplets.com/opentelemetry-ai-endpoints-aspnet-core) covers where those attributes belong.

## When Routing Fits

*   **Heterogeneous traffic.** There must genuinely be easy and hard requests. Uniform difficulty means nothing to route.
    
*   **Measurable quality.** You need an evaluation set that tells you whether the small tier is good enough for the requests you send it. Without that, "cost went down" and "quality went down" are indistinguishable.
    
*   **Meaningful spend.** Routing adds real complexity. Below a certain bill it is not worth the maintenance.
    

## When Routing Is the Wrong Choice

*   **Safety-critical or regulated outputs.** If a wrong answer has legal or clinical consequences, do not let a heuristic decide which model produced it. Pin the model and pay.
    
*   **Strict latency budgets with an LLM classifier.** Adding a model call before every model call can cost more time than the cheap tier saves.
    
*   **You have not measured your traffic yet.** Routing built on assumed distributions optimises for a workload you do not have.
    
*   **Prompt-sensitive behaviour.** Models differ in how they interpret the same system prompt. A prompt tuned on the large model can behave noticeably differently on the small one, so tiers need their own prompt validation rather than a shared template assumed to be portable.
    

## Trade-offs You Are Accepting

*   **Non-deterministic quality across requests.** Two similar questions can land on different tiers and get noticeably different answers. Users perceive this as inconsistency, which is worse than uniformly average.
    
*   **More things to evaluate.** Every tier needs its own regression suite, and every routing rule change needs re-validation. Our walkthrough of [evaluating LLM output in .NET](https://codingdroplets.com/evaluate-llm-output-dotnet) is the mechanism that makes this manageable.
    
*   **Model deprecation multiplies.** You now track lifecycle for several models instead of one.
    
*   **Debugging gets a new question.** "Which tier served this?" has to be answerable from a trace, or support escalations become guesswork.
    
*   **A new failure path.** If the small tier is down, does the request escalate or fail? Decide it explicitly. This is the same reasoning as [resilient LLM calls in .NET](https://codingdroplets.com/resilient-llm-calls-dotnet), applied to model choice.
    

## A Short Rollout Sequence

1.  Instrument tokens, cost, and latency per endpoint, and leave it running for a week
    
2.  Apply static per-endpoint routing where the answer is already obvious
    
3.  Build an evaluation set from real requests before changing anything else
    
4.  Add heuristic routing behind a flag, in shadow mode, logging the tier it would have chosen
    
5.  Compare eval scores per tier, then enable for the endpoints that hold quality
    
6.  Add cascade escalation only where upfront classification proves unreliable
    

## FAQ

### How much can LLM model routing actually save?

It is bounded entirely by your traffic mix. If most requests are genuinely simple, most of your volume moves to a tier costing a fraction of the frontier price and the blended cost drops sharply. If your traffic is uniformly complex, routing saves close to nothing. Measure the distribution first, because that number decides whether the pattern is worth building at all.

### Should I use a small model or heuristics to classify requests for routing?

Start with heuristics. Input length, endpoint, attachment presence, and conversation depth are free, deterministic, and explainable, and they capture most of the available saving. A classifier model adds a call to the hot path of every request, including the trivially cheap ones, so only adopt it once you can show heuristics are the accuracy bottleneck.

### What is the difference between model routing and the cascade pattern?

Routing predicts difficulty before calling any model and dispatches once. Cascade calls the cheap model first and escalates only when the result fails a confidence check. Cascade is more accurate because it judges a real answer instead of a guess, but escalated requests pay for two calls, so it depends on escalation staying rare.

### How do I stop model routing from degrading answer quality?

Build an evaluation set of real requests with expected output characteristics, and score each tier against it before routing anything live. Then run routing in shadow mode, recording which tier would have been chosen without acting on it, and compare quality per tier. Ship only for the request classes where the cheaper tier holds its scores.

### Can I implement model routing with Microsoft.Extensions.AI?

Yes, and it is a natural fit. Register each tier as a keyed `IChatClient`, then implement a routing client that selects a tier and delegates to it. Because everything downstream depends on the `IChatClient` abstraction, your endpoints, tools, and tests are unaffected by routing existing at all.

### Where should routing sit relative to caching and rate limiting?

Caching goes outermost so a hit costs nothing regardless of tier. Routing sits inside it. Token-based rate limiting needs care, because a request's cost now depends on which tier serves it, so reserve against the worst-case tier and settle against the actual usage the response reports.

* * *

## About the Author

I'm Celin Daniel, Co-founder of [Coding Droplets](https://codingdroplets.com/). I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.

*   GitHub: [codingdroplets](http://github.com/codingdroplets/)
    
*   YouTube: [Coding Droplets](https://www.youtube.com/@CodingDroplets)
    
*   Website: [codingdroplets.com](https://codingdroplets.com/)