The Competing Consumers Pattern in ASP.NET Core: When to Use It and How

When a single message consumer can no longer keep pace with the volume of incoming work, the natural response is to run more of them — and let them race. That is the competing consumers pattern in a sentence: multiple independent consumers reading from the same queue, each claiming and processing messages without central coordination. For teams building scalable ASP.NET Core services today, understanding when this pattern genuinely helps — and when it quietly introduces more problems than it solves — is one of the more consequential architectural decisions you will face.

If you want to see the pattern operating inside a real production-grade ASP.NET Core background service — wired up with BackgroundService, channel-based queuing, and proper cancellation handling — the full working source code is available on Patreon, where members get annotated, production-ready code that maps directly to what enterprise teams actually ship.

Understanding this pattern in isolation is useful — seeing it work as part of a complete production API, alongside rate limiting, resilience, and structured error handling, is what makes it click. That is exactly what Chapter 12 of the Zero to Production course covers, including BackgroundService, System.Threading.Channels, and Hangfire for durable persistent jobs — with source code you can run immediately.

What Problem Does This Pattern Actually Solve?

The competing consumers pattern addresses a specific class of problem: a message producer generates work faster than a single consumer can drain it, and the backlog is growing.

In ASP.NET Core, this commonly surfaces when:

• An API endpoint enqueues tasks — image processing, report generation, email dispatch — that a single BackgroundService cannot handle at the required throughput.

• A message broker topic or queue (Azure Service Bus, RabbitMQ, AWS SQS) is accumulating messages because one processing instance has hit its CPU or I/O ceiling.

• Scaling the whole application vertically is disproportionately expensive relative to the isolated processing work.

The pattern's answer is to run multiple consumer instances — either as separate BackgroundService registrations within the same process, as parallel workers on System.Threading.Channels, or as multiple replicas of a hosted service across pods — and let the broker or channel handle the coordination. Each consumer independently claims a message, processes it, and acknowledges completion. No consumer sees a message another has already locked.

The Core Mechanics

Message Ownership and Lease-Based Processing

Most production message brokers implement a lease model: when a consumer picks up a message, the broker marks it as "in flight" and hides it from other consumers for a configurable visibility timeout. If the consumer crashes or fails to acknowledge within that window, the broker makes the message visible again and another consumer can claim it.

This is why idempotency is not optional in this pattern — it is a hard requirement. A message may be delivered more than once if a consumer acknowledges too slowly, crashes after processing but before acknowledging, or if network partitions cause the broker to assume failure. Every handler must be designed to produce the same outcome whether it processes a given message once or five times.

In-Process vs. Distributed Competing Consumers

There are two distinct flavours worth distinguishing:

In-process competing consumers use System.Threading.Channels with multiple concurrent readers — an UnboundedChannel<T> or a BoundedChannel<T> with multiple IHostedService registrations all reading from the same channel writer. This is appropriate for parallelising CPU-bound or I/O-bound work within a single process, with no external broker involved.

Distributed competing consumers involve multiple process instances — separate Kubernetes pods, container replicas, or Azure Container App revisions — all connected to a shared external broker. This is the form most teams picture when the pattern is mentioned, and it is what delivers true horizontal scale.

The decision between these is not just technical. In-process scaling is simpler to operate, but it ties throughput to the capacity of a single machine. Distributed scaling unlocks independent elasticity at the cost of broker management, at-least-once delivery semantics, and the operational overhead of running and monitoring multiple instances.

When the Competing Consumers Pattern Fits

High-Throughput, Order-Independent Work

The pattern is most naturally suited to workloads where individual messages can be processed independently and the relative order of completion does not matter. Email sending, thumbnail generation, data export pipelines, and webhook fanout are canonical examples — any consumer can pick up any message, and there is no dependency between them.

Asymmetric Producer-Consumer Rates

When your API can receive bursts of work — say, a batch upload triggering thousands of notifications — but processing each item takes non-trivial time, the competing consumers pattern absorbs the burst behind a queue and lets you tune concurrency separately from ingest rate. The queue becomes the buffer; the consumer count becomes the throughput dial.

Stateless Processing Logic

Consumers must not share mutable state with each other. If your processing logic reads from a shared in-memory structure that is mutated during processing, competing consumers introduce race conditions that are difficult to reproduce and dangerous in production. Clean competing consumers each have their own isolated execution context — dependencies are injected per-scope, database connections are obtained per-message, and no cross-consumer state is assumed.

Workloads That Need Independent Scaling

When the processing logic is the bottleneck — not the API layer, not the database, not the broker — horizontal scale through consumer replicas is the right lever. If you are on Kubernetes, this translates naturally to KEDA (Kubernetes Event-Driven Autoscaler) watching queue depth and scaling your consumer Deployment dynamically, without any change to the producer.

When This Pattern Does Not Fit

When Message Order Must Be Preserved End-to-End

Competing consumers process messages concurrently. If consumer A takes longer than consumer B on adjacent messages, the completion order diverges from the arrival order. This is fine for most workloads, but fatal for scenarios like event sourcing replay, financial ledger updates where sequence integrity is mandatory, or any workflow where a downstream system assumes ordered delivery.

For ordered processing, the right patterns are either a single consumer with sequential processing, or partitioned consumers where each partition is processed sequentially — the model Kafka uses with consumer groups and partition assignment.

When Processing Logic Is Stateful or Correlates Across Messages

If a message at position N must know about the result of message at position N-1, competing consumers break the assumption. This pattern does not give you a coordination mechanism between consumers — they are intentionally isolated. Stateful workflows belong to an orchestrator, not a pool of competing readers. If you find yourself reaching for shared cache or in-process signals between consumers, that is the pattern telling you it does not fit.

When the Volume Does Not Justify the Complexity

Running a pool of competing consumers against an external broker is meaningful operational overhead: the broker must be provisioned, monitored, and operated; dead-letter queues must be handled; consumer health must be tracked independently from the API process; and message visibility timeouts must be tuned against processing latency.

For low-volume, low-criticality workloads, a single BackgroundService with a BoundedChannel<T> is almost always sufficient. Do not reach for distributed competing consumers because the pattern is interesting. Reach for it because the simpler solution has demonstrably failed.

When the Processing Side Effect Is Not Idempotent

If your consumer modifies an external system — charges a payment, sends an SMS, triggers a third-party workflow — and that system does not support idempotency keys, a duplicate delivery will cause a duplicate action. Before applying this pattern, either make the side effect idempotent (by keying on a message ID) or use a broker that guarantees exactly-once delivery semantics (which most do not, and which comes with its own trade-offs).

Decision Matrix

Criterion	Use Competing Consumers	Do Not Use
Message order required	✗	✓
Idempotent handlers	✓	✗
High throughput needed	✓	Not justified at low volume
Stateless processing	✓	Stateful / correlating logic
Horizontal scaling needed	✓	Single-process is sufficient
External broker available	Ideal	Use in-process channels

How It Fits in ASP.NET Core

In-Process: BackgroundService with System.Threading.Channels

For in-process competing consumers, System.Threading.Channels is the right primitive. A BoundedChannel<T> with a fixed capacity acts as the queue; multiple BackgroundService registrations each hold a reference to the same ChannelReader<T> and loop independently until cancellation.

The key here is registering multiple instances of the same BackgroundService type, or registering a single worker that internally spawns multiple processing tasks with Task.WhenAll. Both work. The choice depends on whether you want DI scope isolation per consumer (separate registrations) or a single coordinating worker that manages its own parallelism.

A production detail that matters: when using BoundedChannel<T>, set BoundedChannelFullMode.Wait rather than DropOldest or DropNewest. Silently dropping messages is almost never the right production behaviour.

Distributed: Multiple Hosted Service Replicas with Azure Service Bus or RabbitMQ

For distributed competing consumers, each pod or container replica runs the same BackgroundService that connects to the shared broker. Azure Service Bus and RabbitMQ both implement the lease model natively — competing consumers need only connect and start consuming; the broker handles message assignment, visibility timeouts, and dead-lettering.

The operational concern here is graceful shutdown. When Kubernetes sends SIGTERM, ASP.NET Core's IHostedService.StopAsync is called. Consumer implementations must honour the CancellationToken propagated through the processing loop, complete any in-flight message within the deadline, and not start new message picks after cancellation is signalled. Failing to do this leaves messages locked until the visibility timeout expires, causing unnecessary re-delivery to other consumers.

For working reference code showing BackgroundService wired with proper cancellation and scope management, the dotnet-background-services-hostedservice repository on GitHub covers these details in a runnable project.

Anti-Patterns to Avoid

Shared mutable state between consumers. Any data structure written by one consumer and read by another without synchronisation is a data race. Use message-local scope for everything.

Missing dead-letter handling. Competing consumers fail silently when there is no dead-letter queue and no alerting on redelivery count. Poison messages — those that fail every consumer that processes them — will loop indefinitely. Configure dead-letter thresholds and monitor the dead-letter queue.

Consumer instance count as a configuration constant. The right consumer count depends on message volume and processing latency, both of which change over time. Design for runtime configurability; do not hardcode services.AddHostedService<OrderConsumer>() three times in Program.cs with no way to adjust it without a redeploy.

Ignoring broker-side scaling signals. Queue depth, message age (time since enqueue), and consumer idle rate are the signals that drive autoscaling decisions. If you are not emitting or monitoring these, you are flying blind on capacity.

Treating StopAsync as optional. Hosted services that ignore cancellation during shutdown leave in-flight messages locked. Implement graceful shutdown properly — it is not a nice-to-have.

The Relationship with the Outbox Pattern

Competing consumers answer the question of how to process messages at scale. The Outbox Pattern answers the complementary question of how to reliably get messages onto the queue in the first place — without dual-write inconsistency between your database and your broker.

In most reliable message-driven architectures, these two patterns work together: the Outbox ensures messages are written to the queue exactly once, atomically with the domain state change; the competing consumers ensure those messages are processed at the throughput the business requires. Neither is sufficient alone for a production-grade, reliable messaging system.

Similarly, Domain Events vs Integration Events is a worthwhile read before deciding what kind of work ends up on the queue in the first place — the distinction shapes how consumers are designed and what idempotency guarantees are needed.

☕ Prefer a one-time tip? Buy us a coffee — every bit helps keep the content coming!

FAQ

What is the Competing Consumers pattern in ASP.NET Core? The Competing Consumers pattern runs multiple independent message processors against the same queue, each claiming and handling a message without coordination. In ASP.NET Core, this is implemented using multiple BackgroundService instances reading from a shared channel, or multiple deployed replicas all connected to the same message broker queue.

When should I use Competing Consumers instead of a single BackgroundService? Use competing consumers when a single consumer cannot drain the queue fast enough — when you observe growing queue depth, increasing message latency, or dropped messages under load. If a single BackgroundService handles your current and projected volume comfortably, the additional complexity of multiple consumers is not justified.

Does the Competing Consumers pattern require an external message broker? No. In-process competing consumers can be implemented entirely with System.Threading.Channels — a BoundedChannel<T> with multiple concurrent readers running as IHostedService registrations. An external broker like Azure Service Bus or RabbitMQ is only necessary for distributed consumers across multiple process instances.

How do I handle message ordering with the Competing Consumers pattern? If strict ordering is required, the Competing Consumers pattern in its pure form is not appropriate. Consider partitioned consumers (one consumer per logical partition, sequential within each), or a single ordered consumer if throughput allows it. Azure Service Bus sessions and Kafka partitions both provide ordered processing within a partition while allowing parallelism across partitions.

What happens when a consumer crashes mid-processing? Most message brokers implement a visibility timeout. If the consumer does not acknowledge the message within the timeout, the broker makes it visible again and another consumer picks it up. This is why handlers must be idempotent — the message will be redelivered. Configure the visibility timeout to be comfortably longer than the maximum expected processing time for a single message.

How many consumers should I run? There is no universal answer. Start with (target_throughput / single_consumer_throughput) + a headroom buffer, and measure. For Kubernetes deployments, use KEDA to autoscale based on queue depth rather than fixing a static count. For in-process channels, measure throughput with different concurrency levels before committing to a number.

Is System.Threading.Channels suitable for production competing consumers? Yes, for in-process workloads. System.Threading.Channels is a high-performance, thread-safe primitive designed for exactly this use case. For distributed workloads spanning multiple processes or pods, you will need an external broker — channels are in-memory and do not survive process restarts.