An idempotency key should remember successes, not failures

#The question

A crew member on an apron with no signal records a departure. The app queues it. Later, back in range, the queue drains and the server refuses the departure, because a gate condition was not met. Fine so far: that is the system working.

An engineer clears the blocker. The crew member taps Retry.

It fails, identically, with the same refusal. It will keep failing that way for ever, no matter what anyone fixes, because the answer is no longer being computed. It is being replayed.

#What I thought was happening

My model of an idempotency key was: a client attaches a key to a request, the server does the work once, stores the response, and replays it for any later request with the same key. Retries are then safe, because the work cannot happen twice.

The rule I had internalised for what to store was about transience. Do not store a 5xx, because the server broke and the request might well succeed if tried again. Store everything else, because everything else is an answer the server meant.

That rule is wrong, and the reason it is wrong is more interesting than the bug.

#What I found

The store was caching every non-5xx response. A queued departure that the takeoff gate refused returned a 409, and the 409 was written against the operation's key.

The client's retry affordance then had nothing to do. It sent the same key, the store recognised it, and the stored refusal came back without the handler running at all. The gate was never consulted again. The blocker being cleared changed nothing, because nothing was asking.

What made this properly bad is the combination with offline. The whole point of the queue is that work recorded without signal reaches the server eventually. A refusal that permanently pins itself to the operation means the one path back to a correct state, try again now that the world has changed, is closed. The crew member is looking at a button labelled Retry that cannot retry.

#The mechanism

Go back to what idempotency is actually protecting.

It exists to stop a side effect happening twice. Two identical charge requests must not charge twice; two identical departure records must not create two flights. The key identifies a unit of work so the effect can be made to happen at most once.

Now ask what a 4xx means in that frame. A 4xx means the handler refused before doing anything. In this codebase that is concrete: the handler threw before its transaction committed, so there is no side effect, no row, no partial state. Nothing happened.

If nothing happened, there is nothing for idempotency to protect. Re-running the request is safe by definition, and it is frequently the only way forward, because a 4xx often depends on state outside the request that somebody else can change.

The distinction is not transient versus permanent. It is did this produce an effect. A 5xx is excluded from the cache because the effect is unknown. A 4xx should be excluded because the effect is known to be absent.

#The decision

The claim on the key is released on any 4xx as well as any 5xx. Only a 2xx is stored and replayed.

That is the whole change, and it collapses the rule to something I can say in one line: an idempotency key remembers that the work succeeded. It has no business remembering that it was refused.

#What it costs

Two concurrent copies of a request that will both fail now both execute, and both fail, where before one would have been served from the store.

That costs a round trip and some handler work. It changes nothing about the data, because neither one produces an effect. I am comfortable with it: the saving was on the path where the client is being told no, which is not a path worth optimising, and it was buying that saving by making the no permanent.

#The model that made it click

"The same request" and "the same answer" are different claims.

An idempotency key asserts the first: this is the same unit of work, do not do it twice. Replaying a stored response asserts the second: the answer has not changed. For a success those coincide, because the work is done and the result is a fact about something that happened. For a refusal they come apart, because a refusal is a statement about the world at the time of asking, and the world moves.

#What I would remember in six months

Any cache keyed on a request rather than on a resource is asserting that the answer is a property of the request. Check whether it is. If the response depends on state the request does not contain, which is what a validation gate is, caching it freezes a moment and then serves that moment for ever.

And the smaller, more general one: a retry affordance that cannot change the outcome is a lie, and users will believe it. If a button says Retry, something on the path has to be genuinely re-evaluated. Worth checking, because it is very easy to build the honest-looking button on top of a layer that has quietly decided the question is settled.