Seven predicates for one question

#The question

Ground an aircraft, then look at it on four different screens. One of them said it was grounded. The other three said Serviceable.

The data was not corrupt. Every screen was reading real values out of the real database and rendering them correctly. They simply did not agree about what the question meant.

#What I thought was happening

I assumed this was a caching problem, or a refresh problem. Something had written the status and three readers had not noticed yet. That is the ordinary shape of "the screens disagree", and it has an ordinary fix.

Underneath that was a more basic assumption: that "is this aircraft available" had an answer, stored somewhere, and the job was making sure everyone read it.

#What I found

There was no single answer to read. The audit counted seven mutually incompatible predicates for "is this aircraft available", scattered across the surfaces that needed to ask.

They were not copies of one rule that had drifted apart. They were seven different rules, each written at the moment some screen needed to know, each correct for the case its author had in front of them, and each subtly different about the cases its author was not thinking about.

Exactly one surface computed it correctly, a deriveDisplayStatus function behind the aircraft summary endpoint. And here is the detail that made the whole thing legible: that surface was also the only one displaying the status correctly. The correct derivation and the correct display were the same place, because the derivation had been written for that display and had never left it.

Two more findings came with it. Six write paths bypassed the transition() function that was supposed to be the choke point for status changes. And the aircraft state machine, along with the guard that was meant to stop an aircraft with an open grounding order being dispatched, was dead code. It existed, it was correct, and nothing called it.

That last one is worth sitting with. We had a state machine enforcing the rule. It had simply been routed around, one write path at a time, by people who needed to set a status and found a more direct way to do it.

#The wrong fix I nearly made

The obvious repair is to add the missing states to the stored enum. If the enum had reserved, in_flight, pre_flight and the rest, every screen could read one field and agree.

I got some way down that road before noticing it makes the disease worse. Every value added to a stored enum is another thing that has to be written at the right moment by every code path that could cause it. Adding in_flight means every departure path must set it and every arrival path must clear it, and a missed clear leaves an aircraft permanently airborne. The seven predicates existed because writing state correctly from many places is hard; the answer to that is not more state to write from more places.

The disease is drift, not vocabulary.

#The decision

Split the question in two, along the line of what is durable.

The stored aircraft_status enum stays, with its five values, and its meaning is narrowed to the durable reason the aircraft cannot fly: serviceable, aog, maintenance, parked, leased_out. That is a fact about the aircraft that outlives any particular flight, and it is genuinely written by a human decision.

The transient operational state, Reserved, In Flight, Pre-flight, Awaiting Post-flight, Awaiting Approval, is not stored at all. It is derived, by one server-side function, from the records that already exist. Every surface calls that function. And all status writes route through transition(), which is now the only way in.

The property that matters: one derivation cannot disagree with itself. Seven predicates can drift because they are seven things. A function is one thing, and if it is wrong it is wrong everywhere at once, which is a much easier failure to notice and a much easier one to fix.

#What it cost

Every surface now has to call the shared function, and a list query needs a join it did not need before. That is a real cost on the list endpoints and it is the reason the derivation is cached.

Caching then brings back a smaller version of the original problem, staleness, which has to be invalidated on the aircraft tag. I would rather have one cache with one invalidation rule than seven predicates, but it is not nothing, and it is the part most likely to bite later.

And one thing the derivation deliberately does not solve. in_flight still needs a hard uniqueness constraint in the database, independent of any derivation, because the audit proved live that two flights could be airborne on the same tail. A derived view of the world tells you what the records say. It cannot stop the records saying something impossible. That needs a constraint, at the level where constraints are enforced.

#The model that made it click

If a fact can be computed from records you already have, computing it is safer than storing it, because stored state has to be written correctly by every path that could change it, and derived state has to be written correctly by nobody.

The corollary is the useful part. When you find several incompatible definitions of one concept, the fix is not to choose the best one and propagate it. Propagating it creates the eighth. The fix is to make it impossible for there to be more than one, which usually means making it a function rather than a field.

#What I would remember in six months

Counting is diagnostic. "How many places in this codebase answer this question?" is a cheap query and the answer is often startling. Seven was not a number anyone would have guessed, including the people who wrote all seven.

And when you find a state machine or a guard that is correct but dead, do not just wire it back up. Ask how it came to be bypassed, because six independent write paths did not appear by accident. They appeared because the choke point was harder to use than going around it, and if that is still true after you reconnect it, it will happen again.