One outage should be one ticket

A message fails to reach eight people because a service was unavailable for five minutes.

Eight tickets open. Each has to be read, understood and closed by hand. All eight describe the same five minutes.

This is not a rare pathology. It is the default behaviour of almost every system that raises alerts, because alerts are naturally written at the point of failure, and the point of failure is per-attempt.

Why per-symptom alerting is the default#

Nobody designs it. It emerges.

Somewhere in the code, a send fails. The person writing that code does the responsible thing and raises a ticket. It is one line, it is correct, and it is tested with a single failure.

In production, failures are not single. They are correlated — one dead service, one expired credential, one network partition — and every correlated failure takes the same path through that line.

So the number of alerts is proportional to the number of attempts during an outage, which is a function of your traffic rather than of the problem's size. A busy hour produces a hundred tickets for the same cause.

What the flood costs#

Manual closing. Somebody must close each one, and closing tickets teaches people that tickets are noise.

The real one is buried. During a flood, an unrelated genuine problem arrives and lands in the middle of ninety copies of something else.

The pattern is destroyed. Ninety tickets from one cause and one ticket from another cause look like a ratio, and any counting you do afterwards is wrong.

Attention is spent at the worst moment. The flood arrives exactly when somebody needs to think clearly about one thing.

Group by cause, carry the details inside#

The fix is to make the unit of alerting the cause, not the attempt.

One outage, one ticket. The affected recipients, records or attempts travel inside it as a list. If more failures occur while the ticket is open, they join it rather than creating siblings. The same discipline on a building is one snag, one row, rather than four reports of the same cracked tile — a snag list that actually closes.

Three implementation questions decide whether this works.

What is the key? What makes two failures the same event. Usually the service or resource that failed, plus a time window. Getting this too narrow reproduces the flood; too wide and genuinely different problems merge.

When does the ticket close? Ideally on recovery, automatically, with the duration recorded. A ticket that must be closed by hand after the problem has gone will be closed late or not at all.

What if the cause changes? A ticket that stays open for a week accumulating unrelated failures is worse than a flood, because now nothing is separable. A time bound on grouping is necessary.

The ticket has to describe the system that exists#

A related failure, and in some ways a worse one, because the ticket count is fine and the content is wrong.

Our own message-delivery tickets used to tell people to create an approved template in a service we had retired months earlier. That service was gone. The bridge that replaced it has no templates and no approvals. So the ticket had been sending people to fix a problem that could not exist.

Its sibling was worse. When a handset stopped answering, the ticket said messages were being routed via the other provider. There is no other provider. They were not being sent at all — and the ticket's wording actively reassured whoever read it.

Both now describe the system that is actually running, and each names the real cause with the one action that fixes it: the handset needs re-pairing, the secret no longer matches, nothing is answering at the address, or the service was too busy to reply.

The general lesson is that alert text ages faster than alert logic. The condition that fires the alert gets maintained because it breaks visibly. The sentence describing it does not, because a wrong sentence produces no error. A message describing a system that no longer exists is a silent, permanent misdirection, and it is worth explicitly re-reading alert text whenever architecture changes. The wider argument is in the app should tell you why.

Tell the person who can fix it#

Grouping correctly and describing correctly still fails if the ticket goes to the wrong desk.

Six mailboxes in our estate had lost their authorisation to send, one of them for over a week, and they stayed broken because of who was being told. The failure raised a ticket for the IT team — but IT cannot reconnect somebody else's mail account. Only its owner can, from their own browser, with their own password.

So the ticket sat open while the person whose mail had quietly stopped flowing never heard a word.

Now the owner is told directly, with a one-tap link that starts the reconnect and an explanation of why it happened — usually a password change, which revokes the permission by design. IT still gets its ticket for the record.

That is the right shape: the actionable notification goes to whoever can act, and the record goes to whoever needs the record. They are different messages with different purposes and they should not be the same message sent to a list.

Silent failure is the worse cousin#

Everything above is about too many alerts. The opposite failure is quieter and more expensive.

A classifier in our system asked a small local model, in words, for a JSON array, and hoped. It got prose wrapped around the answer, or ran out of room mid-array and stopped before the closing bracket. It failed twenty-nine times over twenty-four days, quarantined real mail, and opened a ticket each time.

Twenty-nine tickets is a flood by the standards of this article, and it was still the good outcome, because the alternative — discarding the mail quietly — would have been invisible.

The eventual fix removed the failure rather than the alert: constrained decoding, where anything but valid output is impossible to produce. That is always the better order. Suppressing an alert because it is noisy, without fixing what it is reporting, converts a loud problem into a silent one.

The short version#

Alert on causes, not on attempts. One outage, one ticket, with the affected items carried inside it and an automatic close on recovery.

Re-read the words in your alerts whenever the architecture changes, because alert text ages faster than alert logic and a wrong sentence produces no error.

Send the actionable message to whoever can act, and the record to whoever needs a record.

And when an alert is noisy, fix the thing it is reporting before you quieten it. The related discipline of routing by what a message asks somebody to do is in an alert that rings for everything.