Idempotency Keys For Runs
An idempotency key is a client-supplied identifier that tells a server, “Treat this request as the same logical action as any earlier request with the same key.” When the server receives a duplicate request, it returns the same result instead of running the action again. This matters when networks drop packets, clients retry after timeouts, or a job runner restarts mid-flight.
In practice, you see idempotency keys in payment APIs, booking systems, and order placement endpoints. A typical flow looks like this: the client generates a key per intended action, sends it with the request, and expects the server to store enough metadata to map that key to the outcome. If the client does not receive a response and retries, the server recognizes the key and avoids a second run.
Idempotency keys do not magically fix every duplicate. They prevent duplicates for the specific action and scope the server defines. If the server uses a different scope than the client expects, or if the server forgets the mapping too early, duplicates still slip through, and the logs become a mess.
Main Problems And Pain Points
Duplicate runs usually come from retries and partial failures, not from “bad clients.” A request can time out after the server already committed the change, then the client retries and triggers the action again. Background workers can also reprocess messages after a crash, and the same message payload can be delivered twice.
People often get the scope wrong. They generate a key per HTTP request instead of per logical action, so a retry uses a new key and the server treats it as a new action. Others reuse the same key across different users or different target resources, which can cause cross-account collisions. A third group uses idempotency keys only for the first step, then triggers downstream side effects twice because the downstream calls do not share the same idempotency strategy.
Supporting technologies determine whether idempotency keys work in real systems. The server needs persistent storage for key-to-result mappings, plus a way to decide what “same result” means. Many teams use a database transaction to write both the business change and the idempotency record, then return the stored outcome on duplicates. If the mapping is stored in a cache without persistence, a restart can erase the mapping and duplicates reappear.
Solutions And Advice
Choose A Key Scope That Matches The Action
Generate one idempotency key per intended logical action, not per network attempt. For example, if a user clicks “Schedule Appointment” once, the key should represent that appointment request. If the user edits the time and submits again, the key should change because the logical action changed.
Keep the scope explicit. A common pattern is to include the user identifier and the target resource identifier in the key material, then hash it into a fixed-length string. If you do not include scope, you may accidentally treat two different actions as duplicates. I have seen teams use a plain UUID per click, which works until they add a “resend confirmation email” button that reuses the same UUID across different operations, which is where things get weird.
Use a stable key format that survives retries. If you use a random key, store it client-side until you get a final response. If you regenerate the key on each retry, you defeat the purpose. In one internal test I ran with a simple Node.js client, retrying with a new key produced two rows in the database, even though the server had idempotency logic.
Store Key Mappings With A Clear Retention Policy
The server must store the mapping from idempotency key to outcome. “Outcome” can be a status code, a created resource ID, or a serialized response body. The server also needs to store enough information to return the same result on duplicates without re-running side effects.
Set an expiry that matches the longest realistic retry window. Many systems use retention windows measured in hours or days, depending on client behavior and network conditions. If you expire too quickly, a late retry triggers a second run. If you retain too long, the mapping table grows and query performance can degrade.
In a PostgreSQL setup, a practical approach is a table with a unique constraint on (idempotency_key, scope columns) and columns for outcome data. A uniqueness constraint prevents the “two requests create two mappings” race. I once reviewed a schema where the uniqueness constraint was missing, and the idempotency code relied on an application-level check that failed under load.
Make The Server Return The Same Result On Duplicates
Idempotency is not only about preventing duplicate side effects. It also needs consistent responses. If the first request returns a created resource ID, the duplicate request should return that same ID. If the first request failed after partially processing, the server must decide whether to treat the duplicate as a retryable failure or a final failure.
Define the behavior for “in progress” states. One common pattern is to store a record with a status like “processing,” then update it to “succeeded” or “failed.” If a duplicate arrives while processing, the server can either wait briefly, return a “still processing” response, or return the last known status. Which choice you make affects client retry logic and user experience.
Be careful with timeouts. If the server times out while processing but still commits the business change, the client may retry. Idempotency keys help, but only if the server writes the idempotency record in the same transaction as the business change or in a way that preserves ordering guarantees.
Test With Retries, Crashes, And Concurrent Requests
Test idempotency behavior using controlled failure modes. Simulate a client timeout after the server commits, then retry with the same key. Simulate a worker crash after it reads a message but before it marks it complete. Simulate two identical requests arriving concurrently.
Measure outcomes with concrete checks. For example, verify that the database has exactly one row for the target resource and that the idempotency mapping has exactly one final outcome record. If you use a message queue, verify that the consumer does not produce duplicate side effects when the message is delivered twice.
Tools help you reproduce timing issues. In one test harness, I used a simple curl script with a fixed idempotency key and a forced client-side timeout, then inspected the database after retries. The key detail was keeping the key constant across attempts and verifying the uniqueness constraint actually blocked duplicates.
Case Examples
Appointment Scheduling With Retries
A clinic system exposes an endpoint to schedule an appointment. A patient’s browser sends a request with an idempotency key stored in session storage. The server creates the appointment row and writes an idempotency record in the same database transaction, then returns an appointment ID.
If the browser times out before receiving the response, it retries with the same key. The server detects the existing idempotency record and returns the same appointment ID without creating a second appointment. If the patient submits a new time slot, the client generates a new key, so the server treats it as a new logical action.
Background Job For Lab Results
A lab integration service receives a “new sample” event and enqueues a background job to fetch results and update a patient record. The queue can deliver the same event more than once after a consumer restart. The job runner includes an idempotency key derived from the sample ID and the integration run date.
The worker writes an idempotency record before updating the patient record, then updates both the patient record and the idempotency outcome. If the event is delivered again, the worker sees the idempotency record and skips the update. A mild frustration shows up here: if the key derivation changes between versions, duplicates can return, so teams often pin the key derivation logic to a specific version string, like “v1.3,” and migrate carefully.
Comparison Table And Checklist
| Approach | Prevents Duplicate Side Effects | Returns Same Result | Common Failure Mode |
|---|---|---|---|
| No Idempotency Key | No | No | Timeout retry triggers a second run |
| Client Key Per Attempt | Sometimes | Inconsistent | Retry uses a new key, so duplicates slip through |
| Server Key With Unique Constraint | Yes | Yes | Mapping expires before late retries |
| Partial Idempotency Only On First Step | No | Mixed | Downstream side effects run twice |
Checklist you can apply to an API or job runner:
- Define the logical action boundary. Decide what counts as “the same run.”
- Generate one key per logical action and keep it stable across retries.
- Use a server-side unique constraint so concurrent requests cannot both create a mapping.
- Store the final outcome data needed to return the same result.
- Choose a retention window that covers expected retry delays.
- Test with timeouts, duplicate deliveries, and concurrent requests.
Common Mistakes
One frequent mistake is treating idempotency keys as a logging feature. Logging duplicates does not prevent them, and it rarely helps when side effects already happened. Another mistake is returning a generic “success” on duplicates without returning the original resource ID, which forces clients to guess state and can trigger follow-up duplicates.
Teams also mis-handle failure states. If the first request fails after writing some data, the server must decide whether a duplicate should retry the remaining steps or return the same failure. If the server retries automatically on duplicates without careful design, it can turn a partial failure into repeated partial writes.
Key generation mistakes show up as collisions. If keys are derived from low-entropy inputs like timestamps without user scope, two different users can share a key. That can cause one user’s request to “claim” another user’s mapping. A mild annoyance in audits is when teams store keys in plaintext logs, which creates a privacy risk because keys can correlate actions.
FAQ
What Should An Idempotency Key Represent?
It should represent a single logical action, such as “create this appointment” or “charge this invoice,” not each HTTP attempt or each internal retry.
How Long Should Idempotency Records Be Kept?
Keep them long enough to cover expected client retry delays and network timeouts. Many systems use hours to days, but the exact window depends on client behavior and operational requirements.
What Happens If Two Requests Use The Same Key?
The server should treat the second request as a duplicate and return the same outcome as the first. This requires a unique constraint and stored result data.
Do Idempotency Keys Work For Background Jobs?
Yes, when the worker uses a stable key derived from the job’s logical identity and stores the outcome mapping. Queue redelivery alone does not prevent duplicates without that mapping.
Can Idempotency Keys Prevent All Duplicates?
No. They prevent duplicates for the specific action and scope the server defines. Duplicates can still occur if the key changes across retries, expires too early, or downstream side effects lack idempotency.
Author's Insight
Idempotency keys are a contract between client and server, and the contract only holds when the server stores a durable mapping from key to outcome. The most common real-world failures come from scope mismatches, missing uniqueness constraints, and unclear handling of in-progress or partially failed states.
In systems design reviews, I look for evidence that the idempotency record is written with the business change under the same transactional guarantees, or with a proven ordering strategy. I also check whether the key is propagated across internal calls, because side effects often live outside the first endpoint.
One practical detail: teams sometimes version their key derivation logic, like “v2,” and migrate carefully so old clients do not generate keys that no longer match the server’s expectations. That small operational step prevents a class of duplicates that otherwise appear only after deployments.
Key Takeaways
- Use one idempotency key per logical action and keep it stable across retries.
- Store key-to-outcome mappings on the server with a uniqueness constraint to handle concurrency.
- Return the same result on duplicates, including resource identifiers when clients need them.
- Choose retention long enough for expected retry windows, then test with timeouts and concurrent requests.
