Webhooks: Duplicate Events and Retry Delivery

Webhooks And Duplicate Events

Webhooks are HTTP callbacks where a sender posts an event payload to your endpoint. Duplicate events show up when the sender retries after timeouts, network errors, or ambiguous delivery outcomes. A common example: a payment or appointment system posts “payment_succeeded” to your server, your server processes it, but the sender never receives a 2xx response and retries a minute later.

Retry delivery is not a bug in the sender; it is a safety mechanism. If the sender cannot confirm that your endpoint accepted the event, it schedules another attempt. This behavior interacts with load balancers, slow downstream services, and request body parsing time, so duplicates can appear even when your code works correctly.

To reason about duplicates, treat every webhook request as potentially repeated. You then design your endpoint so that repeating the same event does not create repeated side effects like double-charging, double-booking, or duplicate database rows. That design usually hinges on an event identifier and a clear idempotency strategy, not on hoping that retries never happen.

Main Problems And Pain Points

Teams often assume that “one event equals one request,” then build side effects directly into the request handler. That assumption fails when the sender retries due to timeouts, 5xx responses, or connection resets. Even a transient database lock can delay your response long enough for the sender to retry.

Another frequent mistake is relying on HTTP status codes without controlling response timing. If your endpoint performs heavy work before returning 2xx, the sender may time out and retry. Some senders also treat certain response bodies as non-acknowledgements, even when the status code looks successful, and the exact behavior depends on the sender’s webhook implementation.

Duplicate handling depends on supporting technologies: request signing (to prevent spoofing), idempotency keys or event IDs (to detect repeats), and durable storage (to record processed events). If you store “processed” markers in memory, duplicates still happen after restarts, autoscaling events, or container redeployments. I have seen teams run webhook workers on version 1.9.3 and later discover that a redeploy cleared in-memory dedupe state, which turned retries into duplicate rows.

Finally, payload parsing can create hidden retries. If your server reads the request body twice, or a middleware throws after partially processing, you may return a 500 even though the side effect already occurred. That pattern produces duplicates that look like “random” failures, but they follow a deterministic code path.

Solutions And Advice

Use Idempotency With Event IDs

Design your endpoint so that each event ID maps to exactly one side effect. Many webhook providers include an event identifier such as event_id or id in the payload, and some also include a header like X-Event-Id. Store a record keyed by that identifier in durable storage, then check it before applying side effects.

In practice, you can create a table like webhook_events with a unique constraint on event_id. When a duplicate arrives, the unique constraint rejects the insert, and your code returns 2xx without repeating the side effect. This pattern works even if the sender retries multiple times, and it survives restarts.

When the payload lacks an event ID, you can derive a dedupe key from stable fields such as event type plus a provider-specific reference number. Derived keys are less reliable than provider IDs, so you should document the derivation and revisit it if the sender changes payload structure.

Acknowledge Fast, Process Later

Return a 2xx response quickly after validating the request, then process the side effects asynchronously. This reduces the chance that the sender times out and retries. A common approach is: verify signature, parse JSON, write an idempotency record, enqueue a job, then respond.

Set a response-time target that matches your sender’s timeout window. If your sender times out at 10 seconds, keep your synchronous work under 1–2 seconds. I once saw a webhook handler that took 6–8 seconds due to synchronous calls to a third-party API; after switching to a queue, duplicate retries dropped sharply because the sender started receiving 2xx responses consistently.

Queue-based processing must still be idempotent. A job retry inside your worker can create duplicates too, so the same event ID dedupe record should guard the worker’s side effects.

Verify Signatures And Replay Safety

Use the sender’s signature scheme to confirm that the request is authentic. Many providers include a signature header and a timestamp, and they expect you to compute an HMAC over the raw request body. If you verify after parsing JSON, you can accidentally change whitespace or encoding; signature verification should use the raw body bytes.

Replay safety depends on the sender’s headers and your policy. If the sender includes a timestamp, reject requests older than a small window such as 5 minutes. Some teams skip this check, then accept replayed payloads that look valid because the signature still verifies.

Be careful with clock skew. If your servers run with incorrect time, you will reject legitimate events. A mild operational aside: I have seen webhook failures correlate with a container image that lacked NTP sync, and the signature timestamp checks started failing after a host reboot.

Handle Partial Failures Cleanly

Define what “accepted” means. If you write the idempotency record but fail before enqueuing the job, you need a recovery path. One approach is to store a processing state such as received, enqueued, and completed, then have a periodic job reconcile stuck states.

If your side effect is a database transaction, keep the idempotency insert and the side effect in the same transaction when possible. If you cannot, you can still maintain correctness by using the idempotency record as the source of truth and making the worker idempotent.

When you return non-2xx, the sender retries. When you return 2xx but your worker later fails, the sender will not retry, so your internal retry policy must cover that path. Use exponential backoff for worker retries and cap the number of attempts to avoid infinite loops.

Case Examples

Appointment Booking Duplicate

An anonymized scheduling service posts appointment.created to a clinic’s webhook endpoint. The clinic server validates the signature and writes a row keyed by event_id with a unique constraint. The handler returns 2xx within 500 ms and enqueues a job to create the calendar entry.

When the clinic’s calendar provider times out, the worker retries the job. The worker checks the same event_id record and sees that the calendar entry is not marked completed, so it retries safely. If the sender retries the webhook because it did not receive the 2xx response earlier, the unique constraint prevents a second calendar entry.

Outcome: duplicates are absorbed at two layers—webhook ingestion and worker processing—so the clinic’s calendar stays consistent.

Payment Success Double Write

A billing system sends payment.succeeded events to an accounting service. The accounting endpoint initially wrote a ledger row before returning 2xx, then called a tax calculation API that occasionally exceeded the sender’s timeout. The sender retried, and the ledger row was duplicated.

After adding an idempotency table with a unique constraint on event_id, the endpoint records the event first and returns 2xx quickly. The worker performs the ledger write only once per event ID. On a later audit, the team noticed that the original bug correlated with a specific deployment date—2026-03-14—when a new tax API version increased latency.

Outcome: retries no longer create duplicate ledger entries, and the team can trace each event through the idempotency record.

Duplicate Handling Checklist

Step-by-step checklist you can apply to an existing endpoint:

1. Log the provider’s event ID and request ID for every webhook request, including duplicates.
2. Add a durable dedupe store with a unique constraint on the event ID.
3. Change the handler to return 2xx after validation and dedupe record creation, not after downstream side effects.
4. Move side effects to a worker and guard them with the same event ID record.
5. Add a reconciliation job for events stuck in a non-completed state.
6. Measure handler latency and correlate it with sender retry counts from your logs.

If you use a framework like Express, check middleware order; a body parser configuration change can alter raw-body access, and signature verification may fail in a way that looks like “random 401s” (I saw this after upgrading a dependency around 2024-11).

Common Mistakes

One mistake is treating 409 Conflict as a “failure” and returning it to the sender. Some senders interpret non-2xx responses as a signal to retry, which turns dedupe into a retry loop. If your dedupe store detects a duplicate, return 2xx with a clear response body that does not trigger sender-specific error handling.

Another mistake is using a dedupe key that changes across retries. For example, using the entire JSON payload as a hash fails when the sender adds fields, changes ordering, or includes a non-stable timestamp. Prefer stable identifiers provided by the sender, then fall back to derived keys only when stability is documented.

Teams also forget that signature verification depends on the exact raw request body. If a middleware reads and modifies the body, the computed signature may not match. When signature verification fails, some teams return 400, which triggers retries and floods logs, even though the requests are authentic but mis-verified.

Finally, many implementations store “processed” markers without retention policy. Over time, the dedupe table grows and slows queries. Use a retention window aligned with the sender’s retry horizon and your audit needs, and index the event ID for fast lookups.

FAQ

Why do webhooks arrive twice?

Duplicate deliveries happen when the sender cannot confirm your endpoint accepted the event, such as timeouts, network errors, or non-2xx responses. Retries can also occur after ambiguous failures where the sender does not know whether your side effects completed.

How do I stop duplicate side effects?

Use idempotency keyed by a stable event identifier from the payload or headers. Store a durable record with a unique constraint on that identifier, then return 2xx for duplicates without repeating the side effect.

Should my webhook handler do work before returning 2xx?

Keep synchronous work minimal: verify signature, parse payload, record idempotency, enqueue processing, then respond. Heavy downstream calls before 2xx increase timeout risk and trigger sender retries.

What if the webhook payload lacks an event ID?

Derive a dedupe key from stable fields that the sender documents as consistent across retries, such as an external reference number plus event type. Derived keys can break if the sender changes payload structure, so add monitoring and revisit the key when payloads evolve.

How long should I keep dedupe records?

Keep records long enough to cover the sender’s maximum retry window and any internal replay needs. If the sender’s retry horizon is unknown, start with a conservative window and adjust based on observed retry patterns in your logs.

Author's Insight

Duplicate webhook events are a predictable outcome of at-least-once delivery and failure ambiguity, not a rare edge case. The most reliable pattern uses durable idempotency keyed by a stable event identifier, combined with fast 2xx responses and asynchronous processing. Signature verification should use the raw request body bytes, since body parsing can change what gets signed. When teams treat retries as exceptional and side effects as non-repeatable, duplicate rows and double actions appear. When teams treat every request as repeatable and track processing state, duplicates become a logging and reconciliation problem rather than a correctness problem.

Key Takeaways

• Assume webhook deliveries repeat; design for idempotency from day one.
• Use a stable event ID (or a carefully derived key) with a unique constraint in durable storage.
• Return 2xx quickly after validation and dedupe recording, then process side effects asynchronously.
• Verify signatures against the raw body and reject stale requests when timestamps are provided.
• Track processing state and reconcile stuck events so “accepted” does not mean “completed.”