Skip to content
All insights
Guide · Strategy

How to Write a Pilot Plan That Ends in a Decision

One page written before a pilot starts is what lets it end. Including the pilot I am running now, and the measure we added to it together.

· 5 min read

Patrick Laughlin, Founder, Chief AI Officer

Naval Academy, class of 2011. Decorated naval aviator, 10+ years. President, 2023 Inc. 5000 company.

I am in a pilot right now, and it taught me to write the ending before anything else.

The client is a national wealth management firm with several hundred advisors. We are building an advisor portal and its agents with them, on the data lakehouse their own data team built, inside their own cloud environment. Ten advisors and five advisory leaders get access. Each spends roughly an hour with it over a week. A decision about wider rollout follows a few weeks later.

The test protocol was good. Testers work through the portal against the views they use today and classify what they find as fixed, broken, missing, or not working as expected. It is a proper defect hunt run by a competent product organization, and it will produce a proper defect list.

A defect list cannot say whether the thing is worth rolling out to several hundred people. That takes an agreed measure, a baseline, and a date on which a named person reads a figure. Raising that early was our job, and we raised it later than I would like.

So we sat down with the firm and added three things together.

Two questions added to the tester pass. For one real client meeting, how many minutes did preparation take in the new portal against your normal route. And how many separate systems did you avoid opening.

The baseline captured before access. Once an advisor has used the tool, their sense of what the old way cost is gone, and it does not come back. That measurement has an expiry date and the date is the release.

Product analytics approved before launch. Ours was still in the client's security review. If it does not clear before the release, the pilot produces defect tickets and opinions and no usage data at all, and a rollout decision for several hundred people gets made on impressions.

None of that is exotic, and a well-run program with capable people on both sides can still reach launch week without it. That is the point, and it is why I now write it into every pilot before it starts. Pilot purgatory is almost never a technology failure. It is a pilot with no defined ending, which means the only available outcome is another pilot.

The one page

Write this before anyone touches the software.

The decision. Not "learn about AI." The sentence reads: on this date we decide whether to roll this out to everyone, and here is who signs. If nobody will sign a sentence like that, you are not running a pilot, you are running an evaluation, and you should call it one and spend less.

The number. One primary measure, in a unit your company already tracks. Minutes per case. Days sales outstanding. Errors per hundred records. Tickets resolved without escalation. If the measure requires you to invent a new metric, you will spend the pilot arguing about the metric.

The baseline. Captured before anyone gets access, timed rather than remembered, from the people who will be in the pilot. This is the item that expires.

The population. Who exactly, by name, and what happens if three of them never log in. Decide in advance whether that is a data problem or a finding, because it is usually the finding.

The floor. The result below which you stop. Write it down before you see any data, because afterwards every number is arguable and the sunk cost is doing the arguing.

The production owner. Name the person who will own the system after launch, during the pilot, and put them in the pilot. It changes how the pilot is run, and it removes the handover that otherwise never happens.

The sentence that creates a second pilot

"We will decide when we see it."

It sounds open-minded and reasonable and it guarantees another six months, because there is no version of "it" that settles anything. Seeing something is not a decision procedure. A threshold agreed in advance is.

Two more things learned expensively

Instrument before you launch, not after. If telemetry arrives in week three, the first cohort is unmeasured, and the first cohort is the one whose behavior tells you the most. Retrofitting usage data also means you are reconstructing what happened from memory, which brings you back to the same problem.

Give people a channel for saying it is worse. If the only feedback route is a defect form, you will hear about bugs forever and never hear that the new way takes eleven minutes where the old one took six. That is the finding that should stop a rollout, and a defect form cannot express it.

What I will tell a client

If the pilot fails against the floor we agreed, I will say so and recommend stopping. That is the whole reason to write the floor down while everyone is still calm.

A pilot is either an experiment with a stated ending or it is a demo on a longer schedule. The difference is one page, written first, with a date on it.

Tell us what you want to get done.

A first conversation with a Chief AI Officer about your goals and what you need, whether that is a build, training, change management or expert advice. We shape the engagement around your answer.

A developer at work on a build