How to Validate a New Dashboard Before You Trust It

A sales lead at a licensed apparel company builds the weekly sales numbers by hand. Pull the orders, add them up, format the page, send it out.

We built a dashboard that produces the same figures from the source systems. Then we took one four-week window and compared our totals against that hand-built report, and the two came within half a percent. Our arithmetic, not a measurement anybody at the company made, and still the only reason to read the dashboard instead of the spreadsheet.

To validate a new dashboard, run it beside the report it replaces for one full cycle, compare the totals, and explain every difference rather than averaging it away. It is ready when the person who owns the number can say why the two disagree, and will put their name on the new one.

What a parallel run is

A parallel run means both versions of the number get produced for the same period, under the same rules, with nobody switching over yet. It is old accounting practice and it transfers directly to anything a dashboard or an agent now produces.

1. Name the artifact you are replacing. The actual file, as it is built today, not a description of it. If someone keeps that file on their own machine, it is the standard your build will be judged against whether you agreed to that or not.
2. Run both for one full cycle. A week for a weekly report, a month for a month-end number, one event for anything that happens per event. Anything shorter and you have not yet seen what only happens at the end.
3. Compare the totals, then go one layer down. Two matching totals can hide two errors that cancel. Compare the parts that make the total up: by customer, by account, by location.
4. Write down every difference and what caused it. One line each. That list is more of a deliverable than the dashboard is.

Why the small difference is the interesting part

A small gap between the two versions is a finding, not a pass mark. Half a percent still has a cause, and the cause is worth an hour of somebody's time.

Gaps like that almost always come from definitions rather than from arithmetic. A cutoff that falls on a different day. A credit memo one side counts and the other does not. An account somebody excluded by hand for a good reason two years ago and never wrote down.

A difference you can explain is a definition the two of you now share. A difference you cannot explain is the one that surfaces in a board meeting, when the number you published does not match the number somebody else brought.

So the tolerance belongs in writing before the build starts, in the same place as the date and the acceptance criteria: the largest gap you will accept, and the rule that every difference inside it still gets a cause written next to it. Reconciling one number across your systems is the same work done earlier, and earlier is cheaper.

Who signs it off, and who does not

The builder does not sign. That includes us.

Our own statement that a build is accurate is not evidence. It is the start of the conversation, and working software shown on real data is what settles it, read by the person whose name is on the number.

So we hand over the login and ask that person to check the new thing against their own method before anything goes live. On the collections build at the same apparel company, that was the person who works the past-due list, checking our statements against the weekly sweep they do by hand. On the royalty accrual it was the accountant, asked to run a full month through both his own worksheet and our tool and tell us whether the totals tied. The honest version of that story has its own post.

That ask is the part that slips, and it slipped for us. A parallel run finishes only when somebody sends you two numbers, and a person with a day job will not send them while nothing is forcing the question. Give it a date, give it to one named person, and go back for it.

What to do with the old spreadsheet

Do not delete the old spreadsheet on launch day.

Keep it running one cycle past sign-off, then set a date to stop and say that date out loud to the team. People keep private copies of numbers because nobody ever showed them the two versions agreeing. Once they have seen it happen twice, most of them put the copy down on their own. The ones who do not are usually telling you about a case the new report does not handle yet, and that is worth more to you than their compliance.

So before the next build gets scoped, take the number you would most like to stop producing by hand and ask the person who produces it one question: what would you have to see to stop keeping your own copy of this. The answer is your acceptance test, and it is better than the one you were going to write for them.

Nobody puts a spreadsheet down because the new system is impressive. They put it down the week the two versions agree and somebody sits with them and shows them why.