A fact that had no owner

An order was paid. Not in one movement: an advance first, then the balance some weeks later. Every rupee had gone. The people who paid it knew it was paid.

And a queue inside our own software kept asking for it to be paid again.

Why it happened#

The record of this is paid was a single date stamp on the task. Paying in stages does not write that stamp, and that is not a bug: when a payment has a schedule, the stages are the payment record. There is nothing missing.

So any screen that asked "is the paid date set?" was blind to every rupee that had ever moved in instalments. One screen asking the wrong question is an ordinary defect. What we actually found was the same question being asked all over the application, in place after place, each in its own words, each written by somebody who reasonably assumed the stamp meant what it looked like it meant.

Four of them were queues and counters: the list of payments cleared to pay, the list waiting for a signature, the number in the sidebar, and the sweep that puts a decision back in front of somebody who has already made it.

Five more were found afterwards, and one of them was expensive.

The one that cost people time#

A material line on an order paid in stages sat at "waiting on payment" forever. That stage of the journey has a chaser attached to it, and the chaser rings Finance every three days.

So Finance was being reminded, every three days, about money that had already left the account.

Nobody reported it. That is the part worth sitting with. It is not that somebody noticed and was ignored. It is that a recurring reminder which is occasionally wrong teaches you to skim the whole class of reminder, and once you skim it, the one that matters costs the same as the ones that do not. The chaser was not merely useless here. It was quietly spending the credibility of every other chaser in the building.

The one that could have stopped a lorry#

The same stale question was being asked at the gate. The guard's list of loads to expect refuses any line whose money has not settled, which is sensible. A load paid to the last rupee in stages carried no stamp, so it never reached the list, and a lorry that had been fully paid for arrived at the barrier unannounced.

That is the exact event the screen exists to prevent. It had been prevented in every case except the ones paid in the way the biggest orders are paid.

The fix is not one fix per screen#

Writing the correct condition into each of those places would have produced a copy of a rule per screen, which is the same problem again with a longer fuse on it. There is now one function that answers "has the money for this actually settled?", living in the file that owns payment schedules, with a mirror of it in the database query language beside it so the two cannot drift apart. Every surface asks it. None of them owns a second opinion.

One detail from the repair is worth stating plainly, because it is the least comfortable thing we found.

One of our own regression tests was holding the bug in place. The test that guarded the "this is paid, close it?" prompt was anchored on the stale date stamp, because at the time it was written that was what the code said. It passed happily while the prompt failed to appear on exactly the orders it was built for. A test that describes the code rather than the rule will defend a defect as loyally as it defends a feature, and it will do it while reporting green.

The shape#

Any fact your work depends on that is computed in more than one place will eventually disagree with itself. It will not disagree loudly. Nothing crashes. Two screens simply say different things about the same order, on the same morning, and each person takes the answer from whichever screen they happen to be standing in front of.

It survives because everybody's own view of it stays consistent. The person in Finance sees an unpaid order and chases. The person who paid it sees a paid order and does not understand the fuss. Neither of them is looking at a contradiction; they are looking at their own screen, which agrees with itself.

If you want to find these in your own tools, do not look for error messages. Look for a fact that several teams each know how to check, and ask each of them to describe the check out loud. The gap is the difference between their answers. It is usually the case nobody enumerated: the part payment, the cancelled order, the one that was settled a different way.

A fact worth trusting has one owner. Everything else asks it.

The same structure, described for anybody running a purchase ledger rather than a codebase, is in why staged payments break accounting systems. The test that was holding this particular bug in place has its own piece: a test that describes the code will defend a bug.