A pilot that tells you something

The usual pilot is run by the person who wanted the software, on the newest project, with the vendor's engineer sitting in the site office for a fortnight.

It measures nothing. It measures whether an enthusiastic person, given help, can operate a system. That was never in doubt.

A pilot is an experiment, and an experiment that cannot come out badly is not an experiment. It is the last stage of choosing construction software, and the design decisions below are all aimed at one thing: making failure visible while it is still cheap.

Design rules#

One complete process, not one department. Take a record and let it travel the whole way — order raised, lorry received at the gate, bill checked, payment released. A pilot confined to the stores tells you the stores screen works and nothing about the handoffs, which is where systems actually fail.

An ordinary site, not the best one. The best site has your best people, your newest cabin and, usually, your best connectivity. If it succeeds there you have learnt that it works under favourable conditions. Choose the site with average staff, average signal and an average storekeeper.

Long enough to include a month-end. Month-end is when the awkward cases surface all at once: bills that must be booked in the right period, deliveries made on the last evening, reconciliations, the accountant's queries. A trial that ends on the twenty-fifth has skipped the hardest week of the cycle.

The vendor does not do the data entry. This is the rule most often broken and the one that ruins the most trials. If the vendor's engineer keys in the receipts, or sits beside the storekeeper prompting, you are testing the engineer. Let him train, then let him leave the room.

Pass criteria in writing, before it starts. Two paragraphs are enough, but they must exist on paper before anybody is emotionally committed. Without them, the trial ends with a discussion in which whoever argues best wins, and the person who argues best is usually the person who chose the software.

What to measure#

Five things, all of them observable without anybody's opinion.

• Did the parallel register stop? The notebook at the gate, the spreadsheet kept "for our own hisaab". If it is still being written, the tool is not the record. Everything else you measure is secondary to this.
• How long does the slowest real user take on the commonest transaction? Not the average, and not the champion. Stand at the gate with a watch during a normal delivery. The commonest transaction is done hundreds of times a month, and a slow screen is a tax collected every time.
• How many records needed a correction? And of those, how many were corrected because the person made a mistake, and how many because the system could not hold what actually happened.
• What did people do when they got stuck? This is the most informative question in the whole exercise. Did they read the message on screen and recover, ask a colleague, ring the head office, or write it on paper and carry on? The last answer is the one to worry about.
• How many questions had to go to the vendor? Count them, and keep them. After go-live those questions become a support queue with a response time, and the volume you saw in the pilot is the volume you will pay for.

How pilots lie#

The champion who compensates for the tool. One person quietly fixes the entries every evening, chases the site for what was missed, and re-keys what came over WhatsApp. The dashboard looks correct. Roll it out to twelve sites and there is one of him, not twelve, and the numbers go wrong everywhere at once. Ask the champion, privately, what he has been doing after hours.

The sample too small to hit the awkward cases. Partial deliveries, credit notes, cancelled orders, a rate revised mid-order, material returned to the supplier — these are a minority of transactions and a majority of the pain. If none occurred during the trial, the trial did not test the system. Force them: run a real short delivery and a real cancellation on purpose.

Scope creep, so nothing is concluded. Someone adds payroll in week three, then asks whether it can do drawings. Now the trial is a general enquiry, ends without a verdict, and the decision falls back to whoever is keenest. Freeze the scope on day one and write down what is out.

The pilot that succeeds and is then rolled out to strangers. The trial site was trained face to face over a fortnight. The other sites get a video call and a PDF. The pilot proved the software works when people are taught; it proved nothing about the training you are actually willing to pay for. Budget the real training before you count the pilot a success — the shape of that cost is in what a price list does not tell you.

Two conditions deserve deliberate stress while you have the vendor's attention. Take a phone to the basement or the far corner of the site and enter a receipt with no signal, which is the real form of the claim examined in evaluating offline claims. And ask for an export in the first week rather than the last.

Ending it cleanly#

Pilots that are not ended stay alive for years as a half-used login and a small monthly charge.

Decide against the written criteria, in one meeting, with the people who did the entry present rather than represented. If it passed, say what the rollout depends on. If it failed, say which of the criteria it failed, because that sentence is worth more to the next evaluation than the whole trial.

Then deal with the data. Ask for a full export of everything entered during the trial, in a format you can open, and check it before the account is closed — the tests for that are in getting your data back out. If you are not continuing, ask in writing for the trial data to be deleted, and for confirmation once it is done. Your vendor rates, your site addresses and your staff's phone numbers were all in there.

Finally, take the failures back into the next evaluation. Every awkward case the pilot found is a case to put in front of the next vendor at demo stage, which is the point of the demo is not the product. A failed pilot that produced a good list is not a wasted quarter.

The short version#

Pick one whole process on an ordinary site, run it past a month-end, keep the vendor's hands off the keyboard, and write the pass criteria before you start.

Measure whether the parallel register stopped, how slow the slowest user is, and what people did when they got stuck. Then end it in one meeting, get the data out, and carry the awkward cases forward.