A test that describes the code will defend a bug

The purpose of a test is to state a rule and check that the system obeys it.

A very large number of tests do something subtly different: they state what the code currently does and check that it still does it. Those two things look identical while the code is correct, and they diverge sharply the moment it is not.

The test that held a defect in place#

A concrete case from our own system, and it is the least comfortable thing we found that week.

A payment made in stages was not being recognised as paid, because the instalment path never wrote the field that means this is paid. There is a prompt that should appear on an order once its money has settled — this is paid, close it? — and on staged orders it never appeared.

The prompt had a test. The test passed.

It passed because it was anchored on the same stale field the code was reading. At the time it was written, that field was what the code checked, so the test faithfully described the code. It went on passing happily while the prompt failed to appear on exactly the orders it was built for.

A test that describes the code rather than the rule will defend a defect as loyally as it defends a feature, and it will do it while reporting green.

Why this happens so easily#

Because the code is right there.

When you write a test after the implementation, the implementation is the clearest available description of what the system does, and the path of least resistance is to assert against it. The test then has exactly the same blind spots as the code, because it was derived from it.

The rule, meanwhile, exists in somebody's head, in a requirement written months ago, or in a conversation. Getting it into the test requires going back to it deliberately.

This is not a discipline problem that can be solved by trying harder. It is a consequence of the order in which things are written, and the only reliable counter is to state the rule in the test's own language — in domain terms, with values a person would recognise — rather than in the code's terms.

The check that was wrong about itself#

A second case, and the mechanism is different enough to be worth telling.

We had a defect where an invisible overlay was swallowing clicks across a whole page. The fix was general: an overlay that cannot be seen may not take a click.

To stop it coming back, we wrote an automated check that looked at the relevant code and confirmed the two necessary things sat together.

The check passed. It passed with the bug sitting directly in front of it.

The long comment explaining the fix had been placed between the two things the check needed to compare, which pushed one of them outside the window it was looking at. So the check saw one and not the other, concluded nothing was wrong, and would have gone on concluding that for as long as anybody cared to trust it.

We only found out because we put the bug back on purpose to watch the check go red. It did not. The check now strips comments before it looks, and it was re-tested the same way. The whole episode is in the complaint names the wrong thing.

Watch it fail#

The single most valuable habit in this area takes about a minute.

Break the thing on purpose and confirm that something notices.

A test you have never seen fail is a test you are trusting on its own word. It might be asserting nothing. It might be looking at the wrong object. It might have been silently skipped for a year because a filter changed. All of these are common and none of them is visible in a green result.

Almost nobody does this, which is why so many green suites are green for reasons unrelated to the code being correct.

Anchor on the rule, not the mechanism#

Practically, the difference shows up in what the test asserts against.

Describes the code: asserts that a particular field is empty, that a particular function was called, that a query has a particular shape, that a value is stored in a particular place.

Describes the rule: asserts that after an advance and a balance are recorded, the order is treated as settled — by whatever means the system uses to decide that.

The second survives a refactor. More importantly, the second fails when the refactor is wrong, which is the entire reason the test exists.

The best available anchor is a real case with real values that a person from the business would recognise. Our own money rules are held to the figures from an actual settled order, which means the assertions are checkable by somebody who is not a programmer.

One rule, one implementation, one test#

There is a structural point underneath all of this.

If a rule is implemented in two places, testing one of them tells you nothing about the other. And rules do get implemented twice — we found the same question about whether money had settled being asked in nine separate places, each written by somebody who reasonably assumed the obvious field meant what it looked like it meant. The account is a fact that had no owner.

The remedy is the same one that fixes the underlying defect: one function owns the rule, everything asks it, and the test tests that function. A test suite cannot compensate for a rule that has been copied. It can only ever check the copy it was pointed at.

What a suite should be for#

Not proof of correctness, which it cannot provide.

A suite is a set of statements about what must remain true, written so that a change which breaks one of them is caught by somebody other than a customer. Its value is entirely a function of whether those statements are about the domain or about the implementation.

Which suggests a review question worth asking of any existing suite: pick five tests at random and ask, for each, what rule would a business person recognise here? If the answer is a description of code structure, that test will defend whatever the code does, correct or not.

The short version#

A test written from the implementation inherits the implementation's blind spots and will guard a bug while reporting green.

State the rule in domain terms, with values somebody outside engineering would recognise. Break the thing on purpose and watch the test go red before you trust it. And remember that a test can only check the copy of a rule it was pointed at — which is one more reason a rule should exist only once. The principle behind that is in append-only records and, more directly, in every gap we have written about here.