You Added a Validation Check. The Race Condition Is Still There.

What idempotency actually means

Idempotency does not mean "calling something twice gives the same result." That definition sounds right, but it is missing the important part. The real definition: an operation is idempotent if running it many times leaves the system in the same state as running it once. It does not matter how many times someone retries it, in what order the repeated tries arrive, or how much time passes between them.

This matters because the response can look the same while the system underneath does not. A GET request returns the same data every time you call it. Nothing changes, so this case is easy: it is idempotent by nature. A "reserve seat 14B" request is a different story. If it succeeds twice, both responses might say "seat reserved" — but if the system actually reserved that seat twice, or sold it to two people, the state is no longer correct, even though nothing in the response showed a problem. Idempotency is about what happens to the state, not about what the response says.

This is why you cannot add idempotency at the end, as an extra fix. It has to be part of how the operation is built from the start.

The practical case: seat reservation at scale

Picture an event ticketing platform, the kind that opens ticket sales for a stadium concert and gets hit by 40,000 people trying to buy at the same second.

The operation we will use: ReserveSeat(eventId, seatId, userId).

Here are the situations that make this operation dangerous:

• Network retries. A user's app sends the reservation request. The connection breaks, the app does not get a response, and it does not know if the request went through. So it tries again. If the first request already worked, the retry must not create a second reservation.
• Real concurrency. Two different users click "buy" on the same seat within milliseconds of each other. Only one of them can win, but the system has to make sure of that, not just hope it works out.
• Partial failure across services. Reserving a seat is not one single write to a database. It involves inventory (mark the seat as held), payment (charge the card), and notifications (send a confirmation). Any of these can fail on its own, and the system still has to stay correct when that happens.

The rest of this article follows how this same operation gets harder to solve correctly, one step at a time, starting with the version a Junior engineer would ship under deadline pressure, up to the version a Staff engineer would design knowing the system has to survive Black Friday-level traffic across several services.

Junior: no protection

The solution a Junior engineer implements under deadline pressure is the most direct one possible: read the seat's status, and if it looks available, write the reservation. No lock, no atomic check, nothing that remembers whether this exact operation already ran.

This isn't carelessness. The happy path works fine in every manual test. One user, one seat, one click, and the reservation gets created without a hitch. The problem doesn't show up until real, concurrent traffic hits the system.

This approach breaks in two distinct ways, and it's worth keeping them separate:

• Overselling from real concurrency. Two different users read the same seat's status within milliseconds of each other. Both see "available," because neither has written anything yet. Both go ahead and reserve. The system ends up with two active reservations for one physical seat. Neither request did anything wrong on its own. The system's state is just broken.
• Duplication from a retry. One user, one real purchase intent, but the network drops the response before it reaches the client. The client doesn't know if the request went through, so it tries again. The server processes the second request exactly like the first. It has no memory of already handling this, so it creates a second reservation for the same user and the same seat.

Both failures come from the same root cause: the operation treats "read, then decide, then write" as three separate steps, with nothing stopping something else from happening in between.

Intermediate: validation without protection

The Intermediate engineer's fix looks reasonable on paper: before writing the reservation, check whether the seat is actually available. This catches the obvious mistake from the Junior version, where the code wrote blindly without checking anything first.

Under a single request, it works. Someone tries to book a seat that's already taken, the validation catches it, and the request gets rejected with a clean error message. It feels like the bug is fixed.

It isn't. The validation only checks the state at one instant, then a separate step writes the reservation a moment later. Nothing locks the seat between those two steps. Two requests can both pass the check, one right after the other, before either one has written anything. Both see "available." Both proceed. The write step has no idea the other request even happened.

This is a well-known trap called check-then-act, or time-of-check to time-of-use if you want the formal name. It shows up anywhere a system reads a value, decides something based on it, and then acts on that decision as a separate operation. Adding a check here doesn't remove the race condition. It just moves it one step later and makes it easier to miss, because now there's a validation step that looks like it should have caught the problem.

The real issue was never "we forgot to check." It's that checking and writing are two separate operations, and nothing guarantees they happen together, without anything else slipping in between.

Senior: closing the race at the source, then naming the states

A Senior engineer stops treating the race condition as something to catch after the fact. The Intermediate version tried to detect the problem with a check. The real fix is to make it impossible for the problem to happen at all, by relying on the one thing that can actually guarantee atomicity: the database.

Atomic constraint closes the race

Here's the move. Instead of "check, then insert" as two steps, the insert itself carries a uniqueness guarantee, usually a unique constraint on the seat, or on the idempotency key tied to that specific reservation attempt. Now two competing requests race to insert, but only one of them can win. The database enforces that, not application code. The second insert doesn't silently corrupt anything. It gets rejected with a constraint violation, and the server turns that into a clean "already reserved" response instead of a duplicate reservation.

There's a design choice buried in this, and it's worth naming explicitly: who generates the idempotency key, and when? A client-generated key seems convenient, since the client already knows when it's retrying. But a client can forget to reuse the same key on retry, or reuse the same key across two genuinely different requests by mistake, and either way the safety guarantee depends on the client behaving correctly. The more solid version has the server generate the key first, the moment it accepts the reservation attempt, before doing anything else. That key becomes the one true reference for "has this exact operation already happened." Every retry, from any client, points back to the same record.

Naming the states

Once the write itself is safe, a new problem surfaces: what states can a seat legally be in, and what's allowed to move it from one to the next? Without an explicit answer, teams end up with a scatter of boolean flags, isReserved, isHeld, isSold, that can combine into states nobody actually designed for, like a seat marked both held and sold at once.

The Senior move here is to name the states directly and treat the seat as a finite state machine: available, held, reserved, sold, and released or expired. Every transition is a deliberate, guarded step, not a side effect of updating a flag. And each transition has to be idempotent on its own terms: confirming a sale on a seat that's already sold shouldn't throw an error or charge the card twice, it should just recognize the seat is already where it's supposed to be and return the same result as the first time.

Staff: crossing service boundaries

The atomic write and the state machine solve the problem inside one service, backed by one database. A Staff engineer has to answer a harder question: what happens once "reserve the seat" also means charging a card and sending a confirmation email, and those live in different services with their own databases?

The tempting shortcut is to update the seat, then call the payment service, then call the notification service, all in the same request handler. This looks fine until the payment call times out after the charge actually went through on the provider's side, or the notification service is down for thirty seconds. Now the seat, the payment, and the confirmation can disagree with each other, and there's no single transaction to roll them back into agreement.

The standard fix is the outbox pattern: write the seat state change and a record of "this event needs to be published" in the same database transaction. Nothing gets sent to anyone yet. A separate relay process reads unpublished outbox rows and pushes them onto a message broker, retrying safely if that delivery fails, since the row stays in the outbox until it's confirmed sent.

This is where a phrase Staff engineers throw around gets tested for real: exactly-once delivery. Message brokers don't actually give you that. What they give you is at-least-once delivery, meaning a message might arrive twice, and your consumer has to behave as if that's normal, not exceptional. The payment service consuming this event needs its own idempotency check, keyed on the event id, so processing the same event twice charges the card once. "Exactly-once" isn't a broker feature. It's at-least-once delivery plus an idempotent consumer, and pretending otherwise is how systems end up double-charging customers during a broker retry storm.

The other piece a single-service design doesn't have to think about: what happens when payment fails after the seat is already held? You can't roll back a charge that another service owns. The answer is a compensating action, the core idea behind the Saga pattern: instead of one transaction spanning services, each step can be undone by a follow-up step. If the payment fails, the payment service publishes its own event, and the inventory service reacts by releasing the hold.

One last piece a Staff engineer has to plan for, since it doesn't show up until the system has been running for a while: idempotency keys and stored event ids can't live forever. Every one of them needs a retention window, long enough to cover realistic retry delays (minutes to a few hours is typical, not days), after which it expires and gets cleaned up. Skip this, and the idempotency table grows without bound, or worse, a key from a completely unrelated request months later collides with one that should have expired.

And not every side effect is naturally undoable. Charging a card can be reversed with a refund, but refunds aren't instant and aren't free of their own edge cases. Sending a confirmation email can't be unsent. For these, the compensating action isn't "undo it," it's "acknowledge it happened and handle the consequence," like issuing a refund instead of pretending the charge never occurred, or sending a follow-up "your reservation was cancelled" email instead of trying to claw back the first one.

Same state, no matter how many times

The opening of this article defined idempotency as running an operation many times leaving the system in the same state as running it once. Everything since then has been the same idea, applied at a different scale.

The Junior version failed because there was no protection at all between reading the seat's state and writing to it. The Intermediate version added a check, and it still failed, because a check that isn't tied to the write doesn't protect anything, it just adds a step that looks safe. The Senior version closed that gap by making the write itself atomic, and then went further, naming the seat's actual states instead of letting boolean flags drift into combinations nobody designed for. The Staff version took the same guarantee and had to rebuild it across service boundaries, where there's no single database transaction to lean on, only events that might arrive twice and compensating actions when something can't simply be undone.

None of these are different problems. They're the same problem, "what happens when this runs more than once," showing up at a different point in the system: inside one request, inside one service's database, and across a distributed architecture where failure is the default assumption, not the edge case.

That's also why idempotency can't be bolted on with a single trick, like "just add a key." A key without atomicity doesn't help. Atomicity without state modeling turns into unmanageable flags. State modeling without compensating actions falls apart the moment a side effect can't be reversed. Each level in this progression exists because the one before it was solving a smaller version of the same question, and got a smaller answer than the system actually needed.