Webhook Delivery in ASP.NET Core: Synchronous vs Queue-Based vs Event-Driven — Enterprise Decision Guide
Webhooks are now table stakes for SaaS platforms, payment processors, and any API that needs to push state changes to external systems. The pattern looks deceptively simple — send an HTTP POST when something happens — but the moment reliability, ordering, fan-out to hundreds of tenants, and failed delivery retries enter the picture, the architecture decision becomes genuinely hard. Teams that pick the wrong delivery strategy early pay for it with on-call incidents, missed events, and painful refactors under load. If you want to go deeper with annotated, production-ready implementations of the patterns covered here, they are all available on Patreon — ready to wire into a real ASP.NET Core codebase.
Understanding where webhooks sit in your system — and why the delivery strategy you choose now shapes every SLA conversation you will have later — is one of those architectural decisions that looks obvious in hindsight but almost never is. The tricky part is not writing the HTTP client that fires the POST. It is knowing how that call fits into the request lifecycle, what happens when the subscriber is down, and how your approach scales from five tenants to five thousand. Building this correctly alongside background processing, resilience policies, and concurrency controls is exactly what Chapter 12 of the Zero to Production course covers — with a full production codebase you can run and adapt immediately.
This guide compares the three main delivery architectures your .NET team will evaluate — synchronous in-request dispatch, internal background queue, and external event-driven broker — and gives you a clear framework for choosing the right one based on your reliability requirements, team capacity, and operational maturity.
What Is Webhook Delivery Architecture?
Webhook delivery architecture refers to the design of the pathway that an event takes from the moment it is generated inside your system to the moment it is confirmed received by an external subscriber endpoint. It includes how the dispatch is triggered, how failures are handled, how retries are scheduled, and how delivery state is persisted.
The three primary strategies are:
- Synchronous dispatch — fire the HTTP POST inside the same request or immediately after the database write, before returning the API response
- Background queue dispatch — write the event to an in-process channel or durable database queue, then let a hosted worker dispatch it asynchronously
- Event-driven broker dispatch — publish the event to an external message broker (RabbitMQ, Azure Service Bus, Kafka) and let a separate consumer service dispatch it
Each strategy sits at a different point on the trade-off curve between simplicity, reliability, and operational cost.
Strategy 1: Synchronous In-Request Dispatch
How It Works
The API endpoint that triggers the event immediately calls the subscriber URL via HttpClient before returning its own response. The business logic, database write, and outbound HTTP call happen inside the same request scope.
When to Use It
- Internal integrations where you control both sides and latency is acceptable
- Low-volume, non-critical notifications (< 10 deliveries per second)
- Prototyping and early-stage products where operational overhead matters more than reliability
- Situations where the subscriber's success must gate the primary operation — for example, a payment hook that requires acknowledgement before writing the order as confirmed
When Not to Use It
Synchronous dispatch should be off the table for any consumer-facing API in production at scale. The problems compound quickly:
- Latency coupling — your API response time is now bounded by the subscriber's response time, which you do not control
- Failure leakage — a slow or unavailable subscriber causes your API to time out or return 500s to users who have nothing to do with the webhook
- No retry path — if the dispatch fails, the event is gone unless you build explicit retry logic inline, which defeats the purpose
- Resource exhaustion — thread pool pressure and connection pool saturation under concurrent load
Anti-Patterns
- Calling multiple subscriber URLs sequentially in a loop inside the request pipeline
- Swallowing
HttpRequestExceptionsilently and logging "webhook sent" regardless of outcome - Setting no timeout on the outbound
HttpClientcall (the defaultHttpClienttimeout is 100 seconds, which is catastrophic inside a request handler)
Strategy 2: Background Queue Dispatch (In-Process)
How It Works
The API endpoint writes the event to a durable queue — either an in-memory System.Threading.Channels channel or a database-backed queue table — and returns its response immediately. A hosted BackgroundService dequeues events and dispatches them asynchronously.
This is the most common production pattern for teams that do not yet want to operate an external broker. The Outbox Pattern is the gold standard variation: the event is written to a database table transactionally alongside the business entity write, guaranteeing that an event is never generated without being persistable, and never lost due to an application crash between the database write and the queue write.
When to Use It
- Medium-scale SaaS products (up to thousands of webhook deliveries per minute) where broker infrastructure is not yet justified
- Teams that need durability guarantees but want to avoid external dependencies in development
- Applications already using a relational database where adding a queue table is trivial
- Scenarios where event ordering within a tenant matters and a single-consumer pattern is acceptable
Trade-Offs
| Property | System.Threading.Channels | Database-Backed Queue (Outbox) |
|---|---|---|
| Durability | None (in-memory, lost on restart) | Full (survives restarts, crashes) |
| Transactional write | No | Yes (same DbContext.SaveChangesAsync) |
| Throughput ceiling | Very high | Bounded by DB write throughput |
| Operational complexity | Minimal | Low (one extra table) |
| Retry scheduling | Manual | Built-in with ProcessedAt and retry count |
For production webhook delivery, the Outbox Pattern is almost always the right choice within this strategy. Writing a WebhookOutboxEntry row inside the same transaction that writes the business entity eliminates the entire class of "event lost between write and dispatch" bugs.
What Determines a Retry?
A delivery attempt should be marked as failed — and scheduled for retry — when any of the following occur:
- The subscriber returns a non-2xx HTTP status code
- The HTTP request times out (set an aggressive timeout: 5-10 seconds)
- The subscriber connection is refused or DNS resolution fails
Retry schedules should use exponential backoff with jitter to avoid retry storms. A common production default: 30 seconds, 5 minutes, 30 minutes, 2 hours, 24 hours, then dead-letter after five attempts.
Is Ordering Guaranteed?
Not by default. Background queue workers may dispatch events concurrently. If your subscriber cares about ordering — for example, order.created must arrive before order.updated — you need to partition processing by tenant or entity ID and ensure single-threaded consumption per partition. This is where in-process solutions hit their ceiling before a broker becomes the natural answer.
Strategy 3: Event-Driven External Broker Dispatch
How It Works
The application publishes an event message to an external broker — RabbitMQ, Azure Service Bus, or Kafka. A dedicated consumer service (often a separate Worker Service or ASP.NET Core background service) reads from the broker, resolves the active subscriber endpoints for the event type, and dispatches the HTTP POST to each one.
This is the enterprise-grade architecture for webhook delivery at scale. The broker absorbs spikes, provides durable storage, enables at-least-once delivery guarantees, and decouples the publishing service from the dispatch worker entirely.
When to Use It
- Large-scale platforms with hundreds to thousands of active webhook subscribers
- Multi-tenant SaaS products where fan-out — one event to N tenant subscribers — is the dominant pattern
- Teams that already operate a message broker for other inter-service communication
- Scenarios requiring strict ordering guarantees (Kafka consumer groups per partition), dead-letter queue visibility, or cross-region delivery
Broker Choice for Webhook Dispatch
| Broker | Sweet Spot | Tradeoff |
|---|---|---|
| Azure Service Bus | Azure-hosted .NET teams; managed topics + subscriptions | Locked to Azure; cost at high message volume |
| RabbitMQ | On-premises or multi-cloud; flexible routing via exchanges | Requires cluster operations expertise |
| Kafka | Extremely high throughput; event replay needed | Heavyweight for pure webhook dispatch; steep learning curve |
For most SaaS teams on Azure using .NET, Azure Service Bus with topics and subscriptions maps almost perfectly to the webhook delivery mental model: one topic per event type, one subscription per tenant group.
What to Watch For
- Fan-out amplification — a single business event dispatched to 500 subscribers generates 500 outbound HTTP calls. Quota and thread pool limits apply. Use bounded parallelism (e.g.,
SemaphoreSlim) to cap concurrent outbound connections - Poison messages — a permanently failing subscriber endpoint will exhaust retries and land in the dead-letter queue. Implement a subscriber health check to auto-disable endpoints that fail consistently
- Ordering vs. throughput — Kafka and Service Bus handle ordering differently. Know your broker's guarantees before promising your subscribers anything
HMAC Signature Verification: Non-Negotiable for Production
Regardless of which delivery strategy you choose, every webhook payload your system sends must be signed with an HMAC-SHA256 signature over the raw request body, using a per-subscriber secret key. The subscriber validates this signature before processing the payload.
This is how Stripe, GitHub, and every other production webhook platform works. Without it, a malicious actor can forge webhook calls to your subscribers and trigger arbitrary processing on their end.
What the HMAC Header Looks Like
The signature is sent as a custom HTTP header — commonly X-Signature-SHA256 or X-Hub-Signature-256 — containing the hex-encoded HMAC digest of the raw body. The subscriber recomputes the digest using the shared secret and compares it to the header value using a constant-time comparison to prevent timing attacks.
For generating and verifying HMAC signatures in .NET, System.Security.Cryptography.HMACSHA256 is the right tool. The key must be stored per subscriber — rotating it on demand is a common requirement for enterprise customers. The Request Correlation middleware in the Coding Droplets GitHub repo demonstrates the pattern of attaching per-request context headers at the middleware layer, which complements the signature verification approach well.
How Should Your Team Make the Decision?
Decision Matrix
| Criterion | Synchronous | Background Queue | External Broker |
|---|---|---|---|
| Delivery reliability | Low | High (with Outbox) | Very High |
| Operational complexity | None | Low | Medium–High |
| Subscriber count | < 10 | 10–10,000 | 10,000+ |
| Event ordering required | Not applicable | Requires partitioning | Native (Kafka/FIFO queues) |
| Fan-out per event | 1–3 | Up to ~500 | Unlimited |
| Retry observability | None | DB table | Broker dead-letter + dashboards |
| Fits existing stack | Always | Yes (EF Core + BGS) | Only if broker already exists |
| Time to production | Minutes | Hours | Days |
The Right Default for Most Teams
Start with the Background Queue + Outbox Pattern unless you have a specific reason not to. It handles the overwhelming majority of real-world SaaS webhook workloads, integrates cleanly with your existing EF Core and BackgroundService setup, requires no new infrastructure, and gives you a clear upgrade path to an external broker when the time comes.
Move to an external broker when fan-out to thousands of subscribers per event is your primary use case, when you need cross-service delivery guarantees, or when your team is already operating broker infrastructure for other purposes.
Synchronous dispatch is acceptable only in controlled, low-stakes internal scenarios — never as the primary delivery mechanism for an externally-facing API.
Anti-Patterns to Avoid
Delivering webhooks without a dead-letter strategy. Events that exhaust retries must go somewhere observable. A queue table column, a broker dead-letter queue, or an alerting hook are all valid. Silently discarding failed deliveries is not.
Storing raw webhook secrets in your database unencrypted. Per-subscriber signing secrets should be encrypted at rest using ASP.NET Core Data Protection or Azure Key Vault. A leaked database backup should not compromise your subscriber security model.
Not idempotency-proofing your dispatch worker. If your background worker crashes mid-dispatch, the same event may be dispatched twice. Subscribers must be designed for at-least-once delivery. Your delivery records must prevent duplicate HTTP calls for the same event ID.
Dispatching directly from domain event handlers. Domain events are synchronous, in-memory, and within the unit of work. Treating them as the dispatch trigger — rather than writing to the outbox and letting the background worker dispatch — breaks transactional integrity and leaks side effects outside the aggregate boundary.
Logging delivery success before confirming the subscriber's HTTP response. The subscriber returned 200 is the delivery confirmation. Anything before that is an attempt, not a delivery.
☕ Prefer a one-time tip? Buy us a coffee — every bit helps keep the content coming!
FAQ
What is the difference between a webhook and a callback URL?
A webhook is an event-driven outbound HTTP POST sent proactively by a producer when a business event occurs, without the consumer needing to poll. A callback URL is typically a one-time URL provided by a client to receive the result of an async operation — for example, after a long-running job completes. Webhooks imply a subscription model with repeating delivery; callbacks imply a single response to a specific request.
Should webhook delivery use at-least-once or exactly-once semantics?
At-least-once is the practical standard. Exactly-once delivery requires distributed transaction guarantees that are expensive and complex to implement across an HTTP boundary. The industry convention — used by Stripe, GitHub, and most major platforms — is at-least-once delivery with the expectation that subscribers implement idempotency using the event ID in the payload.
How many retry attempts should a webhook delivery system support?
Five to seven attempts with exponential backoff is the common production default, covering a window of roughly 24 hours. After that, the event should be dead-lettered for manual review or auto-escalation. More retries reduce lost events but increase long-term queue depth and can mask permanently broken subscriber endpoints that should be disabled.
What HTTP status codes should a webhook subscriber return?
The subscriber should return a 2xx status code — typically 200 OK or 204 No Content — to confirm receipt. Any non-2xx response (including 3xx redirects) should be treated as a delivery failure and queued for retry. Subscribers should not return 2xx codes conditionally based on processing outcome; the webhook protocol confirms receipt, not processing success.
Is it safe to log webhook payloads for debugging?
Webhook payloads frequently contain PII, payment data, or business-sensitive information. Logging them in full to a general-purpose log sink violates data minimisation principles and may breach GDPR or PCI-DSS depending on your domain. Use structured log scrubbing to redact sensitive fields, or log only the event ID and event type at the delivery layer, with full payload available only in secured audit storage.
How should fan-out be handled for multi-tenant platforms?
Fan-out should be decoupled from the triggering event write. Write one outbox entry per event, then let the dispatch worker resolve all active subscriber endpoints for that event type and tenant context. Dispatching in parallel with bounded concurrency (e.g., SemaphoreSlim(20)) prevents thread pool exhaustion. For very high subscriber counts, partition dispatch by tenant group and consider moving to an external broker topic-per-event model.
What is a dead-letter queue in the context of webhook delivery?
A dead-letter queue (DLQ) is a holding store for events that have exhausted all retry attempts without a successful delivery. It prevents permanently failing events from blocking the main queue, provides visibility into subscriber health problems, and enables manual inspection and replay once the subscriber issue is resolved. In a database-backed queue, this is a status value on the outbox row; in a broker, it is a dedicated topic or queue.
