Token-Based Rate Limiting for AI Endpoints in ASP.NET Core: A Real-World Walkthrough

The first AI endpoint I put behind a rate limiter was protected by a policy that allowed 60 requests per minute per user. It felt responsible. Two weeks later one customer generated a bill larger than the rest of the tenant base combined, without ever exceeding 60 requests a minute. They were pasting entire PDFs into the prompt. That is the moment token-based rate limiting for AI endpoints in ASP.NET Core stopped being an optimisation for me and became a correctness requirement: on an LLM endpoint, a request is not a unit of cost.

Two calls to the same endpoint can differ by three orders of magnitude in tokens consumed. Counting requests bounds your traffic; it does not bound your spend, your latency, or your provider quota. What follows is the design we settled on after getting this wrong once. The complete implementation, including the distributed store and the reconciliation background worker, is available on Patreon if you would rather read working code than assemble it from fragments.

Token budgets only work when they sit alongside model tiering, caching, and provider-side resilience, because each one changes the shape of the others. Chapter 15 of AI-Powered .NET APIs works through token budgets per user and per endpoint together with those concerns, against one running support API rather than isolated samples.

The Business Problem: Requests Are the Wrong Unit

Classic API rate limiting assumes requests are roughly interchangeable in cost. For GET /orders, that holds. For POST /chat, it collapses:

Request	Input tokens	Output tokens	Relative cost
"What are your hours?"	~10	~30	1x
Summarise a support thread	~2,000	~400	~50x
Summarise an uploaded contract	~40,000	~1,500	~700x

All three are one request. If your limiter counts requests, your worst-case monthly cost is unbounded by anything except how large a payload your API will accept. You will also hit your provider's own tokens-per-minute quota long before you hit your requests-per-minute policy, which surfaces as sporadic 429s from upstream that look like a provider outage rather than your own capacity planning failure.

We covered the broader financial blast radius in Runaway LLM Costs in a .NET API. This article is about the specific control that puts a ceiling on it.

Why Can't You Just Use the Built-In Rate Limiter?

You can, and you should - but you have to feed it token counts instead of request counts, and you have to handle the fact that the true cost is only known after the call completes.

ASP.NET Core's rate limiting middleware is built on System.Threading.RateLimiting, and its limiters already support acquiring more than one permit at a time. That single capability is what makes token budgets possible: one permit becomes one token of model usage rather than one HTTP request.

The awkward part is that you cannot know the output token count before the model produces it. So a naive "acquire exactly what you will use" approach is impossible. The pattern that works is borrowed from payments: reserve, then settle.

The Design: Reserve, Call, Settle

1. Estimate the cost of the request before dispatching it. Input tokens you can count exactly. Output tokens you cap by setting MaxOutputTokens on the request, so the worst case is knowable.

2. Reserve estimatedInput + maxOutput permits from the caller's bucket. If the reservation fails, reject with 429 before spending a cent.

3. Call the model.

4. Settle the difference using the actual usage the provider reports, returning the unused reservation to the bucket.

Without step 4, every user is charged their worst case, and a tenant asking short questions burns their budget at the rate of their longest possible answer. With it, the budget tracks reality closely.

Microsoft.Extensions.AI surfaces the real numbers on the response, which is what makes settlement straightforward:

// Microsoft.Extensions.AI 10.x, .NET 10
ChatResponse response = await chatClient.GetResponseAsync(messages, options, ct);
long actual = response.Usage?.TotalTokenCount ?? reserved;

Partitioning: The Decision That Actually Matters

Choosing the partition key is a product decision disguised as a technical one. Get it wrong and you either fail to stop abuse or you punish legitimate heavy users.

• Per tenant is the correct default for B2B SaaS. It maps to the entity that pays, and it stops one tenant starving another.

• Per user within a tenant is a second, tighter bucket. A single compromised account should not consume the whole organisation's budget.

• Per endpoint matters when a cheap classification endpoint and an expensive summarisation endpoint share a deployment. One shared bucket lets the expensive path starve the cheap one.

• Per API key is what you want for machine-to-machine traffic, since there is no user identity to key on.

In practice we run nested buckets: reject if either the tenant budget or the user budget is exhausted. PartitionedRateLimiter.CreateChained composes these cleanly.

// Token bucket sized in model tokens, partitioned by tenant
options.AddPolicy("ai-tokens", context =>
    RateLimitPartition.GetTokenBucketLimiter(
        partitionKey: context.User.FindFirst("tenant_id")?.Value ?? "anonymous",
        factory: _ => new TokenBucketRateLimiterOptions
        {
            TokenLimit = 200_000,                          // burst ceiling
            TokensPerPeriod = 50_000,                      // sustained refill
            ReplenishmentPeriod = TimeSpan.FromMinutes(1),
            QueueLimit = 0                                 // fail fast, do not queue
        }));

Two deliberate choices in that snippet. QueueLimit = 0 because queuing an LLM request that will take eight seconds behind others that will take eight seconds each is a timeout generator, not backpressure. And TokenLimit above TokensPerPeriod so a legitimate large document is possible occasionally but not continuously.

If the difference between fixed window, sliding window, and token bucket is not yet second nature, our breakdown of rate limiting algorithms in ASP.NET Core covers when each one fits. For token budgets the token bucket is the natural match, since it already models a replenishing resource.

Rejecting Properly

A 429 without guidance is a support ticket. Return Retry-After and a Problem Details body that says which budget was exhausted and when it recovers. Clients can then back off intelligently instead of hammering you.

options.OnRejected = async (context, ct) =>
{
    context.HttpContext.Response.StatusCode = StatusCodes.Status429TooManyRequests;
    if (context.Lease.TryGetMetadata(MetadataName.RetryAfter, out var retryAfter))
        context.HttpContext.Response.Headers.RetryAfter =
            ((int)retryAfter.TotalSeconds).ToString();
    // then write a ProblemDetails payload naming the exhausted budget
};

Distinguish "you are out of budget for the next 40 seconds" from "you are out of budget for this billing period". They look identical to a client and require completely different reactions.

The Distributed Problem You Cannot Ignore

System.Threading.RateLimiting keeps its counters in process memory. Behind a load balancer with four replicas, each replica enforces the full budget independently, so your effective limit is four times what you configured. For request limiting that is often tolerable. For token budgets tied to real money it is not.

Options, in the order we would reach for them:

• Redis-backed counters with an atomic reserve-and-settle script. This is the standard answer and the one we run.

• Provider-side quotas where your model provider supports per-key limits. Use these as a hard backstop even if you also limit locally, because they are the only limit that cannot be bypassed by a bug in your own code.

• Sticky partitioning by tenant hash so a tenant always lands on one replica. Simple, but it degrades badly during deployments and rebalancing.

Trade-offs You Are Accepting

• Estimation error. Token counts are model-specific, and counting input tokens exactly means running the right tokenizer. A conservative overestimate plus settlement is usually good enough and far simpler.

• Streaming complicates settlement. With streamed responses, usage arrives at the end of the stream. If the client disconnects halfway, you must still settle for what was generated, or you leak budget.

• A new failure mode. If the Redis holding your counters is unreachable, decide in advance whether you fail open (serve, risk cost) or fail closed (reject, risk outage). Write it down; do not let it be decided by an unhandled exception.

• Budgets need a product story. Once you enforce token limits, someone has to decide what each plan tier gets and what happens at the ceiling. That conversation is not optional.

What to Do Next

Start by measuring. Emit input tokens, output tokens, and cost per request as telemetry before you enforce anything, and look at the distribution per tenant for a week. The limits will pick themselves once you can see the shape of your own traffic. Then enforce in shadow mode, logging what would have been rejected, before you return a single 429. From there, wire the budget into the rest of your production checklist - our AI-powered .NET API production readiness checklist covers the surrounding controls.

FAQ

How do I count input tokens before calling the model in .NET?

Use the tokenizer that matches your model. Microsoft.ML.Tokenizers provides tokenizers for common model families and gives you an exact count for the input. For output, you cannot count ahead, so set MaxOutputTokens on the chat options and reserve that ceiling, then settle against the actual usage reported on the response.

Can the built-in ASP.NET Core rate limiter handle token-based limits?

Yes. The limiters in System.Threading.RateLimiting accept a permit count on acquisition, so you can treat one permit as one model token instead of one request. What the built-in middleware does not give you is post-hoc settlement or a distributed store, so those parts you build around it.

What is the difference between request-based and token-based rate limiting for LLM APIs?

Request-based limiting caps how many calls a caller makes, which bounds traffic but not cost, because payload sizes vary enormously. Token-based limiting caps how much model capacity a caller consumes, which is what actually maps to your bill and to your provider's quota. Most production AI APIs need both: requests to stop hammering, tokens to stop overspending.

How do I rate limit streaming AI endpoints by tokens?

Reserve the full worst-case budget before the stream starts, then settle once the stream completes and usage is reported. Handle client disconnection explicitly: settle for the tokens generated up to that point rather than releasing the whole reservation, otherwise a client that disconnects repeatedly consumes capacity for free.

Should token limits be enforced per user or per tenant?

Both, as nested buckets. The tenant bucket protects your margin and stops one customer starving the others; the user bucket contains the damage from a single compromised or misbehaving account inside a tenant. Enforce the tighter of the two and tell the caller in the 429 which one they hit.

What happens if my distributed rate limit store goes down?

That is a decision you have to make deliberately. Failing open keeps the API available but removes your cost ceiling during the incident. Failing closed protects spend but turns a cache outage into an API outage. For paid AI features we fail closed on the tenant budget and fail open on the finer-grained per-user bucket, which keeps the money ceiling intact while limiting the blast radius.

---

About the Author

I'm Celin Daniel, Co-founder of Coding Droplets. I've been building .NET and ASP.NET Core systems in production for 13+ years - APIs, distributed backends, enterprise platforms. Everything I write here comes from real shipping experience: patterns that held up, trade-offs that bit us, and lessons learned the hard way.

• GitHub: codingdroplets

• YouTube: Coding Droplets

• Website: codingdroplets.com