Webhooks are a distributed systems problem

Learn the sender's contract first

Different webhook providers do not offer one shared delivery guarantee. Stripe documents retries, possible duplicate events and a lack of delivery ordering. GitHub explicitly says that failed deliveries are not automatically redelivered, while providing mechanisms for manual or programmatic redelivery. Those differences change the receiver's recovery plan. A queue behind the endpoint cannot rescue an event that never reached it. Before implementing business logic, I would record the provider's acknowledgement deadline, retry policy, event identity, retention window and supported method for discovering missed delivery attempts.

The receiver's own acknowledgement should have an equally precise meaning. A successful response can mean that the event was authenticated and durably accepted for later processing. It need not mean that every downstream side effect has finished. If the endpoint returns success while the only copy sits in process memory, that meaning is false. Conversely, holding the response open while several remote services complete increases the chance that an otherwise successful operation is reported as failed and retried. The HTTP response is a protocol decision, not a progress indicator for the entire workflow.

References: [1] Stripe: Webhooks[2] GitHub: Handling failed webhook deliveries

Authenticate the bytes before trusting the fields

Signature verification belongs at the ingestion boundary, before the event type selects privileged work. Stripe's webhook guidance requires the raw request body for its verification process. A receiver that parses and reserialises JSON first can change the signed bytes. I would bound the request size, preserve the original body for verification, use the provider's maintained verification library and make secret rotation explicit. The event payload should not choose which tenant's secret is trusted without an independently validated relationship between the endpoint, account and delivery.

Authentication and deduplication solve separate problems. A valid signature shows that a request satisfies the sender's authentication protocol; it does not make replaying a business action safe. A duplicate can be legitimate redelivery, an old delivery replay or another message representing the same underlying change. Store enough provenance to distinguish those cases, but avoid treating the raw payload as permanent free storage. Sensitive fields need access controls and a retention policy. A useful diagnostic record can preserve identifiers, verification outcome and a protected payload reference without copying full customer data into every application log.

References: [1] Stripe: Webhooks

Make the inbox the acknowledgement boundary

A durable inbox is a simple receiver design. After verification, insert a record keyed by provider, provider account and event identifier. Include the body digest, event type, received time and processing state. Return success only after that transaction commits. A worker then processes the inbox independently. If the response is lost, the sender can repeat the request and encounter the same durable identity. If the database is unavailable, the endpoint must not claim durable acceptance. The correct failure response and the sender's retry contract then determine how the event can recover.

Consider an illustrative inbox that already contains event E with digest H1. A repeated delivery of E with H1 can be acknowledged without scheduling a second independent effect. A delivery of E with H2 needs investigation or a documented provider-specific rule; silently replacing the payload would change the meaning of an already accepted event. The code sketch shows this decision but deliberately omits a queue publish. If queue notification is useful for latency, treat it as a hint or use an outbox. The inbox remains the recoverable source when that notification is lost.

verify_signature(raw_body, trusted_endpoint_secret)
event = decode(raw_body)
begin transaction
  insert inbox(provider, account, event.id, digest(raw_body))
  if identity exists with a different digest: reject and inspect
commit transaction
return HTTP 200

worker: claim durable inbox rows, apply effect, record outcome

Ordering belongs to the resource, not arrival time

Suppose a hypothetical subscription produces created, upgraded and cancelled events. Network delay causes cancelled to arrive first. Applying later arrivals as unconditional assignments can resurrect a subscription that is already cancelled. Sorting by receipt time only makes that bug deterministic. A provider timestamp may not help if its resolution is coarse or if the field describes a different lifecycle moment. I would first ask whether the provider supplies a monotonic resource version or an event sequence with an explicit ordering contract. An opaque event identifier is not such a contract.

When no usable sequence exists, an event can act as a prompt to read the authoritative resource and reconcile local state. That approach trades historical transition fidelity for current-state convergence. It is unsuitable when every transition triggers an irreversible action, and it can still race with another update while the read is in flight. Conditional writes using a supported source version help when available. Otherwise, define a domain-specific merge rule and a periodic reconciliation process. A cancellation might be terminal in one product and reversible in another, so the correct rule cannot come from HTTP alone.

Deduplicate the effect, not just the receipt

An inbox unique constraint prevents duplicate acceptance from creating multiple inbox rows. It does not by itself make downstream work once-only. A worker can send an email, crash before marking the inbox complete, and send the email again after restart. The effect needs its own identity and recovery boundary. For a database update, the effect and processed marker can commit together. For an external operation, use an idempotency contract if one exists, or record an unresolved outcome and reconcile it rather than assuming that a timeout means nothing happened.

The business identity may also span multiple source events. Imagine an invoice-paid event and a separate payment-succeeded event that both suggest granting the same entitlement. Deduplicating on event identifiers still permits two grants because the identifiers are different. I would key the grant by the entitlement operation, such as account plus billing period plus entitlement type, and validate the source conditions inside its transaction. This requires domain modelling, not just transport plumbing. The inbox tells us which messages arrived; the business ledger tells us which intended actions have already been carried out.

A backlog needs a capacity model

Assume a hypothetical burst of 120,000 events arrives over ten minutes, while workers can sustainably process 150 events per second. Arrival averages 200 per second, so the backlog grows by 50 per second and reaches 30,000 events. If arrival then falls to 50 per second, spare capacity is 100 per second and recovery takes another five minutes. This simple arithmetic establishes whether the proposed durable buffer and freshness target are plausible. It says nothing about tail behaviour when a small fraction of events require slow external calls.

I would isolate poison messages and slow event classes so one repeated failure cannot monopolise the whole queue. Use bounded attempts, backoff, a recorded failure reason and an explicit replay path. Age of the oldest actionable event is often more useful than queue length because a quiet but stuck customer can disappear inside a low average backlog. Measure acceptance failures separately from processing failures. Those signals describe different loss boundaries and require different action: restoring the endpoint, resuming a worker, fixing a schema incompatibility or asking the provider to redeliver missing events.

Reconciliation is the counterweight to delivery assumptions

The strongest objection is that a durable inbox, deduplication ledger and reconciliation job seem excessive for a small integration. They can be. A low-value notification might tolerate occasional loss or duplication, and a simple handler can be an honest choice. The key is to state that choice in terms of consequences. A webhook that refreshes a cache is different from one that changes an account's access. Complexity should follow the required recovery guarantee, not the visual simplicity of the provider's setup screen.

For important state, I would periodically compare the local projection with the provider's authoritative records within a bounded interval. Record which range was checked, page through results with stable cursors where supported, and leave exceptions visible. Then test the full integration by dropping responses, repeating deliveries, reversing their order and interrupting processing between durable steps. A webhook system is dependable when it can explain and repair divergence. Receiving a well-formed POST request is only the beginning of that responsibility, and a green endpoint health check is not evidence that the two systems agree.