How to Decide How Often Your Data Should Sync

A sales lead at a licensed apparel company sent us a short email in August. Artwork for a new product had landed four hours earlier, the proposal tool we built still could not see it, and they wanted to know how often it was syncing.

They asked again the next day.

The answer was every six hours. Nobody at the company had picked that number. We had.

How often your data should sync is set by the person waiting on it, not by the job that produces it. Find the shortest gap between a change arriving upstream and somebody needing to act on that change, then refresh inside the gap. A six-hour job sitting in a four-hour workflow is late, however cheaply it runs.

Why we had set it to six hours

We chose that interval from the data's side of the problem, and the data's side is the wrong side.

Everything you look at while building a sync points the same way. How long a full pull takes. What the source system will tolerate being asked, and how often. How much the upstream records actually change, which for new product artwork is a few times a week rather than a few times an hour. Against those facts, six hours looks generous. It is defensible on every axis except the one that decides whether the tool gets used.

The event that matters is not the artwork arriving. It is a rep on a call with a buyer who wants a proposal today. The window between those two moments is the specification, and the window was one afternoon.

A build that misses the window fails quietly. The person opens the tool, does not find the thing they know exists, and goes back to the method they used before. They do not file a bug. They build the deck by hand, get on with the day, and your usage report shows a tool nobody wanted. It is the same reflex that keeps a private spreadsheet alive next to a new report, and it takes the same work to undo as a parallel run and a signed-off number.

That proposal tool is in daily sales use, and the only quantitative fact anybody has produced about its speed is that complaint. So I will not publish a turnaround figure for it until the refresh is right and somebody times a real proposal end to end. The delay got raised twice in two days, which is roughly what adoption sounds like when a build is nearly there.

How to pick the interval

Work backwards from the moment a person acts, in four steps.

1. Name the upstream event. The thing that changes: an order lands, artwork gets approved, a meeting ends, an invoice ages past thirty days.
2. Name the moment somebody acts on it. Who opens what, and what they are trying to finish. If nothing happens in response, you are refreshing a screen for its own sake and nightly is plenty.
3. Measure the gap between the two at its shortest, not its average. The rep who needs a deck this afternoon. The controller on the last working day of the month. An average will talk you into an interval that is fine most weeks and wrong in every week that counts.
4. Set the refresh inside that gap, then give the user a button that pulls now. A manual refresh is worth more than halving the interval. It covers the case you did not predict, and it tells the person their screen has a clock on it, which is the more useful thing for them to know.

What each cadence is for, and what faster costs

Almost everything a mid-market company builds lands in one of four cadences, and the choice is about the work rather than the technology.

Event-driven, inside a few seconds, is for the cases where the action follows the event immediately and somebody is standing there waiting: a check-in desk where the record has to exist before the next person steps up, a form submission that has to create a task before anyone makes the next call. A refresh every few minutes suits a person working a live queue. Hourly is the right answer for data behind a screen people open through the day, which is why the sales tracker portal we built at a retirement planning firm runs on an hourly sync: advisors look at it between meetings. Nightly is for reporting, reconciliation, and anything read once a day with coffee.

Faster is not free, and the cost is not mainly the bill. Every step up asks more of the source system, and that system belongs to somebody whose job is keeping it running. Event-driven pipelines also fail differently from scheduled ones. A missed event leaves no trace, where a job that did not run is a question you can ask directly. That is most of the reason every build needs a way to say it ran and a person who notices when it did not, and the reason stale numbers do more damage than an empty page.

So pick the slowest cadence that fits inside the window, and spend what you saved on knowing when it breaks.

The line to write for every job you run

This week, open the scheduler and find every job that moves data for somebody. Against each one, write the window it is supposed to serve: what changes upstream, who acts on the change, and how long they have to act. Three facts, one line, no architecture.

Then read the interval and the window next to each other. In most companies at least one job is slower than the window it serves, and nobody has escalated it, because the people it fails have already found a way around it.

Those workarounds are the cheapest product research you will ever be handed. They are sitting in somebody's downloads folder, named final_v3.