Pre/post difference-in-differences · quasi-experimental test planning
Designing a reliable test
Why should you choose a difference-in-differences test?
Sometimes you can't randomize your test sites. Maybe you're locked to a specific geography, or your sites spill over into one another. A difference-in-differences test (DiD) can still prove impact, and a triple difference test (DDD) goes further, subtracting out a recurring gap between your groups, like different seasonal rhythms, or different market patterns, which would otherwise bias your results. This is a robust test for any business decision where you can control the intervention, but can't fully randomize your test sites.
Difference-in-differences (DiD) takes two differences and subtracts them. Difference one: how your test group moved from before to after. Difference two: how the control group moved over the same window. Subtract the second from the first and you're left with the part that's yours - the control group absorbs seasons, weather, the economy, and anything else that hit both groups. A triple difference adds one more subtraction to strip out a recurring gap between the groups, like different seasonal rhythms. This is quite useful because it reduces the need for your groups to trend perfectly in parallel.
Here's an overview of how we'll setup the test:
Determine number of test sites and test duration. The size and length of your test will depend on how precise of a measurement you need and the variability of your data, among other factors. The calculator can help here.
Select test and control locations. The test group gets the intervention, and the control group is left alone. You'll aim for two groups that, as a whole, are as similar to one another as possible in terms of key metrics related to your test. If they're not a perfect match, the math further down can adjust for the difference. This will be the most time consuming step. The relevant metrics will be unique to your business, so it's important that the person setting up the test has a good understanding of what matters, and how to compare the groups.
Decide when to run the test. You'll need a clean stretch of "normal" as a baseline pre-period for comparison, and a testing period for the intervention. Use whole weeks, because if you have too many weekends relative to weekdays (or the reverse), it can bias results.
Apply the intervention to the test group only. Then wait, monitor for bias (bad weather impacting only one group, data interruptions, etc.), and get ready to measure results.
Compare how much each group moved. The amount the test group moved beyond what was expected, as calculated from the control group, is your effect.
Start with your test size
If you haven't yet, work out how many locations you need and for how long - that's what the calculator is for.
Before we jump into the actual formulas, let's play with some example test results to understand how this works. Type in your own pre-period and test-period averages for each group and watch the impact and the chart update.
Average weekly revenue per store
Test group (received the intervention)
$
$
Control group (left alone)
$
$
Test groupControl groupExpected (test grows at control's rate)
Each dot is one week; the intervention lands between weeks 4 and 5. The dashed line is the test group's expected level - where it would have landed if it had grown at the same rate as the control group. The gap above it in the test period is your effect.
Your intervention's real effect
-
-
The actual calculation
The math, broken into steps
These steps get you the percentage impact. These update live with the example above.
Step 1 — what the test group would have done anyway-
Step 2 — compare actual to expected-
Now that you've seen it in two steps, here's the condensed one-liner. This is the same thing, but as a single formula to get the percent impact.
Textbook difference-in-differences subtracts absolute changes: (test after − test before) − (control after − control before). That's fine when the groups sit at similar levels, but for many business metrics like revenue, where a bigger store moves more dollars, and last year's dollars aren't this year's, it's cleaner to compare growth rates (difference-in-differences on percentages). That's what this page does, which is also why a seasonal adjustment is a ratio, never a fixed dollar amount.
The one-liner (same result)-
A couple of things worth knowing so your result is trustworthy:
Work at the location level. Give each location one pre number and one test number, then average. Don't pour in every single day as if the days were independent - that makes your result look far more certain than it is.
Each location's days are correlated - a busy store tends to be busy all week - so treating 28 days as 28 independent readings is like counting the same opinion 28 times. Collapse each location to a single pre and a single test number (or use clustered standard errors by location). This is also why the sizing calculator counts locations, not location-days.
To get significance, run it as a regression. The same impact pops out as one coefficient, with a p-value and error bars.
In Excel you can run a regression via Data ▸ Data Analysis ▸ Regression (or the LINEST function). List each store in column A, its average pre-period revenue in B, and its test-period revenue in C. In column D work out each store's growth (=C/B), and in column E mark 1 for the stores that got the intervention and 0 for the rest. Run the regression with column D as the Y values and column E as the X values - the coefficient on E is your effect, and its p-value tells you whether it's real (below 0.05 = significant at 95% confidence).
Making it robust
How to account for seasonality
The calculation above leans on one assumption: that your two groups would have moved together if you'd done nothing. This is often fine, but it breaks when your test and control locations have different seasonal rhythms - say your test stores are in beach towns that spike every summer and your controls aren't. If you run the test over summer with a plain DiD, you may just get the impact of vacationers on your business.
This is the parallel-trends assumption. DiD already forgives the two groups sitting at different levels (one always outsells the other) - that cancels out. What it can't forgive on its own is the two groups being on different trajectories, which is exactly what mismatched seasonality creates.
The fix: measure the same weeks last year, when nothing was being tested. The seasonal gap that showed up between the groups last year is the gap that you'll fold into your expected value before comparing. This turns your test into a triple difference. Here's the same two-step math, with the seasonal correction added - using this year's numbers from the example above, plus last year's growth over the same weeks:
A triple difference is just DiD with one more subtraction: (this year's test-vs-control gap) minus (last year's test-vs-control gap over the same weeks). The first holds your effect plus any recurring seasonal skew between the groups; the second is the recurring skew alone; subtract and you're left with your effect.
Last year, same weeks, no intervention — growth over the period
Last year's growth
%
%
Step 1 — last year's recurring seasonal gap-
Step 2 — fold it into the expected value-
Step 3 — recompute the impact-
-
Two cautions. This only helps if last year was a normal year for those locations, so lean on two or three prior years if you can. And note that this is an add-on to your live control group, not a replacement for a concurrent control group.
Before you trust the result
Validate the test
Getting the math right doesn't make the test fair or acurate. Before you believe any number, sanity-check parallel trends with your own eyes:
Chart the run-up, not just the result
Plot both groups' numbers for the several months before your test starts. The two lines should rise and dip together - roughly parallel. If the test group was already pulling ahead before you intervened, DiD will attribute this lift to your intervention.
If the lines aren't parallel in the run-up, fix it before you even start the test, or at least before you measure results. Swap some locations between groups until they match, pick a better-matched control, or use the seasonality adjustment above to absorb a recurring difference.
Ready to size it?
Work out how many locations and how many weeks you need for a result you can trust.
One final thought: This page assumes you're getting your groups as close to random as you can, aggregating to one number per location, and running a two-sided test decided in advance. It's a practical guide, not a substitute for a pro on a high-stakes rollout. If you need a pro, I'm happy to help with this sort of thing. :) Learn more here.
«