Guide
Projected, observed and validated results in store operations
A staffing change can come with three different numbers: one from a model, one from a before-and-after, one from a test against comparable stores. They are not interchangeable, and the gap between them is where rollout money gets lost.
In store operations, a projected result is an estimate of what a change should do, built from a store's own history before the change is made. An observed result is how a measure moved after the change, compared with before, usually at one store. A validated result is the difference between test stores and comparable control stores over the same weeks. Projected results help choose what to test, observed results show that a measure moved, and only validated results support the claim that the change caused it.
Three numbers, one change, three different meanings
A regional operations team is asked whether to open a second register at the Saturday lunch peak in all 25 stores. Within a week three numbers are in circulation. A model says line time should fall by 18 minutes. The store that tried it reports 14 fewer minutes than last month. A test at three stores against three comparable stores shows 7. All three are real numbers, from real work. They are not the same kind of evidence, and the rollout decision should not treat them as if they were.
In store operations it helps to give them three names and to use each name only for what it can support.
The definitions
A projected result is an estimate of what a change should do, made before the change is made, from your own history and stated assumptions. It answers "which change is worth testing, where, and how big an effect should we look for?" It is a plan, not a measurement.
An observed result is how a measure moved after a change compared with before it, for the same hours, adjusted for traffic and day of week. At one store it answers "did the measure move?" It does not answer "why?", because nothing in the comparison separates the change from everything else that changed in the same weeks.
A validated result is the difference between how test stores moved and how comparable control stores moved over the same weeks, measured the same way, against a decision rule written before the test. It answers "did the change most likely cause the difference, and by how much?" It is the strongest basis for a fleet-wide decision, and it still has limits (see below).
Three neighbors are worth separating from these:
- A pilot is a trial run of a change at one or a few stores to find out whether it can be executed. Its numbers are usually observed.
- An experiment (a controlled store test) is a change at selected test stores with comparable control stores running as usual. Its numbers can be validated.
- A rollout is the change at scale. Its first months are a check on the result, not the end of measuring.
What each result can and cannot support
| Projected | Observed | Validated | |
|---|---|---|---|
| Comes from | A model of the change against your own history, with stated assumptions | A before-and-after at one or more stores, adjusted for traffic and day of week | Test stores vs comparable control stores, same weeks, same measure |
| Answers | What should we test, and how much might it move? | Did the measure move? | Did the change most likely cause the move, and by how much? |
| Good enough to | Choose the test, pick the stores, size the effect to look for | Decide to test at more stores | Decide to scale, refine or stop |
| Not good enough to | Report a result or commit fleet-wide labor | Claim the change caused the improvement | Promise the same result in every store, or in every season |
| Main weakness | Only as good as its assumptions; forecasts tend to run optimistic | The season, a promotion, the weather or a new manager moves the same measure | Matched stores are never identical; test teams run the change at their best |
| Safe wording | "Projected impact", "estimated", "if the assumptions hold" | "Observed change", "moved after the change" | "Validated against comparable stores", "the test stores moved X more than control" |
Why a projection is a reason to test and not a result
Forecasters tend to lean optimistic. Flyvbjerg (2006) describes optimism bias as a tendency to judge future events more favorably than past experience justifies, which leads to underestimated costs and overstated benefits, and recommends checking a forecast against the actual outcomes of comparable past projects. His examples are large public projects, not store operations, but the habit transfers: a projected lift from a staffing change deserves the same check against what similar changes actually delivered in your stores. A projection built from your own history is a better starting point than a rule of thumb. It is still a starting point.
Why an observed change cannot carry the claim of cause
Goodacre (2015), writing about health-service evaluations, lists what makes a comparison with no control group hard to trust: changes in the mix of cases, other changes made at the same time, extra resources, and selection into the intervention. Store changes have the same problems under different names: promotions, weather, school calendars, a new assistant manager, and the fact that stores are usually picked for a fix because they had a bad stretch. A bad stretch tends to be followed by a more ordinary one on its own. Barnett, van der Pols and Dobson (2005) call this regression to the mean and say it should always be considered as a possible cause of an observed change. An adjustment for traffic and day of week makes an observed change fairer. It does not turn it into a cause.
Why a control group is what makes a result validated
Every causal claim is about something that did not happen: what the test stores would have done without the change. Control stores stand in for that. Mendoza (2026) explains that a control group estimates what the treated group's outcome would have been without the treatment, and that an observational comparison mixes the effect of the change with systematic differences between the groups. In a store test, matching control stores on format, size, register count, traffic pattern and baseline trend before the test starts is how you keep that mixing small. Assigning test and control at random from a matched list, as the researchers did with 28 Gap stores in the Stable Scheduling Study (19 treatment, 9 control, 2015 to 2016), is stronger still.
A worked example: one change, three numbers
This is a hypothetical chain, built to show the method. It is not customer data.
A 25-store chain considers opening a second register from 12:00 to 14:00 on Saturdays. The inputs are assumed: each Saturday window adds 2 labor hours at $22 an hour, which is $44 per store per Saturday; each additional verified purchase carries a $16 contribution; there are 4.3 Saturdays a month. At those inputs the change breaks even at about 2.75 additional purchases per store per Saturday ($44 / $16).
| Stage | What was done | Minutes with 5+ in line, Saturday 12:00 to 14:00 | Additional purchases per store per Saturday | Potential monthly contribution, 25 stores, net of the added labor |
|---|---|---|---|---|
| Projected | Simulated against each store's own history | About 18 fewer | About +20 (assumed) | About $29,670 |
| Observed | One pilot store, 4 weeks before vs 4 weeks after, adjusted for traffic and day of week | 14 fewer | +14 | About $19,350 if every store matched the pilot store |
| Validated | 3 test stores vs 3 comparable control stores, 4 weeks before and 6 weeks during | Test 13 fewer, control 6 fewer: 7 fewer | +8 | About $9,030 if the stores rolled out to match the test stores |
The monthly figures are the additional purchases times $16 times 4.3 Saturdays times 25 stores, less $4,730 a month of added labor ($44 x 4.3 x 25). They exclude rollout and training time, any software fees, any effect on other hours of the day, and every sale that was not verified in the POS. They hold only if the lift is verified, attributable to the change and sustained after rollout.
Read the table the way the decision would. The projection was right to say the change was worth testing. The one-store result was right to say the line time moved, and it overstated the effect because part of that move was the calendar: the control stores improved by 6 minutes without the change. The validated number is the smallest and the one to plan on. It still clears break-even, so the team rolls out in stages to stores that share the Saturday pattern, and keeps measuring.
If the validated effect had been +2 purchases, which is below break-even, the same pilot number would have justified a rollout that lost money. That is the practical case for keeping the labels apart: the three numbers lead to different decisions at the margin.
This example is built so the numbers shrink from stage to stage. Real changes do not always behave that way, and a validated result can come in above a projection. The point is that each number should be used for the decision it can support, whichever way it comes out.
How to word each result in a report
Wording does most of the work of keeping the three apart. Patterns that hold up:
- Projected: "Our estimate, built from the last 12 weeks at these stores, projects 14 to 22 additional purchases per store per Saturday if a second register opens from 12:00 to 14:00. This is a projection from assumptions. We recommend testing it."
- Observed: "At the pilot store, minutes with five or more in line on Saturday lunch fell by 14 compared with the four weeks before, adjusted for traffic and day of week. There was no control store, so we cannot separate the change from the calendar."
- Validated: "Over six Saturdays, the three test stores improved by 7 more minutes than the three matched control stores, with transactions up 8 per store per Saturday against a decision rule of 5 minutes and no guardrail worse."
- Financial: "Potential incremental contribution of about $9,030 a month across 25 stores, at $16 per verified purchase and $22 an hour of added labor, before rollout cost and fees, and only if the lift holds after rollout."
Words to keep out of a report on any of the three: "will deliver", and anything that implies certainty. Even a validated result describes what happened at the test stores, not what will happen in every store.
What has to be true before you call a result validated
| Check | If it is missing |
|---|---|
| Control stores were chosen before the test, from stores that looked like the test stores at baseline | Call it observed. Controls picked afterwards tend to flatter the change |
| The same measure, the same way, the same hours, in both groups | The groups are not comparable |
| One change, and it was actually carried out at every test store | You learned about a bundle, or about a half-executed change |
| The test ran long enough to see the changed hours several times (four weeks or more for a weekly peak) | Early weeks swing the most |
| The decision rule was written before the test | A convenient number can be chosen after the fact |
| No holiday, promotion or remodel hit only one group | The calendar swamps the change |
| Guardrails (transactions, labor hours, coverage) held | A gain in the primary measure may have been bought somewhere else |
| For financial ROI: verified purchases from the POS and labor cost from scheduling data, or labeled assumptions | You have an operational result, not a financial one |
How to test this change before rolling it out
Moving a result from projected to validated takes one page and a few weeks. Write the change, the question, the one primary measure, two or three guardrails, the test and control stores, the baseline and test dates, and the decision rule before anything changes. Start at one store to find out whether the change can be run, and report that as observed. Then run it at a few test stores against comparable controls for four to eight weeks, with a baseline of the same length, and report the difference between the groups as validated. If you can only test at one store, say so plainly: it is an observed change, and that is still useful as the first step. The full method, with the arithmetic and a control-store checklist, is in How to test a retail store change before rolling it out.
The limits of the three labels
The labels help you say what a number can support. They do not make the number certain. A projection can be wrong in ways its inputs did not show. An observed change can be real and still not be caused by the change. A validated result describes the test stores in the test weeks, and nearby stores can influence each other, test teams know they are being measured, and a result from autumn may not hold in December. Fisher, Gallino and Netessine (2021), who validated a staffing approach in a few stores before implementing it in 168, found that the effect on revenue varied substantially by store, which is why scaling to stores that share the pattern beats scaling to every store.
A validated result is also not always achievable. A very small fleet may not have enough comparable stores, and some changes cannot be held back from a control group. In those cases an observed change with an honest caveat is a reasonable basis for a small, reversible decision, and the wrong basis for a large one.
How Super Vision approaches this
Super Vision is AI-powered operational intelligence and optimization for multi-location retail. Monitoring generates the evidence that makes optimization possible: the cameras your stores already have become a timestamped history of how each location operates, including visitor traffic and peak periods, and staff presence and service coverage where cameras cover the zone.
The three labels map onto three parts of the product. Simulations model a staffing or workflow change against your own history before you commit resources (projected impact). Benchmarks compare performance before and after a change, adjusted for traffic and day of week (observed change). Experiments test a change at selected locations against comparable control locations (validated impact). Super Vision suggests tests to run, based on history across your locations, and your team decides what to change. Add POS context for verified revenue or transaction metrics. A single location has no control group, so a one-location test reports an observed change, and that is often the right place to start: test at one location before you spend fleet-wide, then scale what works across locations.
Frequently asked questions
What is the difference between projected, observed and validated results?
Projected results are estimates made before the change from your own history. Observed results are the measured change after it, compared with before. Validated results compare test stores with comparable control stores over the same weeks, which is what separates the effect of the change from the calendar. Each answers a different question, so each supports a different decision.
What is projected ROI?
Projected ROI is the return a change is expected to deliver, estimated before it is made, from stated assumptions such as the expected lift, the contribution per sale and the added labor. It is a reason to run a test, not a result. A projection is only as good as its inputs, so every projected ROI figure should show them.
Is a before-and-after comparison good enough to roll out a change?
It is good enough to decide to test at more stores, and not good enough to decide on a fleet-wide rollout. A before-and-after at one store cannot separate the change from the season, a promotion, the weather or a new manager. Adjusting for traffic and day of week narrows the gap; a comparable control store closes most of it.
What makes a result validated?
The comparison groups were chosen before the test from stores that looked alike at baseline, the same metric was measured the same way in both groups for the same hours, one thing changed, the test ran long enough to see the changed hours several times, and the decision rule was written first. If any of those is missing, call the result observed.
Can a validated result be wrong?
Yes. Matched stores are never identical, a test team may run a change better than the whole fleet will, and a result from one season may not hold in another. Validated means the evidence for cause is strong enough to act on, not that the outcome is certain. Keep measuring after the rollout.
Do I need POS data to report ROI?
To report financial ROI, yes. A change in line time or coverage is an operational result. Turning it into sales or contribution needs verified purchases from your POS and the labor cost from your scheduling data, or assumptions that you label as assumptions. Camera data on its own does not confirm sales.
Where does a pilot fit?
A pilot checks whether a change can be run: whether the team can follow it, what breaks and what it costs. Its numbers are usually observed, because one store has no control group. An experiment checks whether the change caused a result. Most rollouts use a pilot first, then an experiment, then a decision.
Sources
- Uncontrolled before-after studies: Discouraged by Cochrane and the EMJ (Goodacre). Emergency Medicine Journal 32(7), 2015. Editorial on why a comparison with no control group makes causation hard to establish, and why controlled designs are preferred.
- From Nobel Prize to Project Management: Getting Risks Right (Flyvbjerg). Project Management Journal 37(3), 2006. Describes optimism bias in forecasts of project costs and benefits, and the remedy of comparing with the actual outcomes of similar past projects.
- Experiments, A/B Testing, and Causal Evidence (Mendoza), in Applied Business Analytics for Marketing Decision-Making. NYU Manifold, CC BY 4.0, 2026. Explains why a control group estimates what would have happened without the change, and why observational comparisons mix the effect of a change with differences between groups.
- Stable Scheduling Study report (Williams, Lambert, Kesavan et al.). Center for WorkLife Law, with the University of Chicago and UNC Kenan-Flagler, in partnership with Gap Inc., 2018. Randomized experiment in 28 Gap stores, November 2015 to August 2016: 19 treatment and 9 control stores. Reported median sales up 7% and labor productivity up 5% in treatment stores.
- Regression to the mean: what it is and how to deal with it (Barnett, van der Pols, Dobson). International Journal of Epidemiology 34(1), 2005.
- Setting retail staffing levels: a methodology validated with implementation (Fisher, Gallino, Netessine). Manufacturing & Service Operations Management 23(6), 2021. Staffing approach tested in a few stores, then implemented in 168 stores; reported 4.5% revenue increase, with effects that varied substantially by store.