Live demo
Three numbers describe the same four weeks. One of them is right.
A campaign ran in 4 markets and not in 20 others, starting Jul 31. This data has a known incremental effect injected into it, so for once the true answer is available to check against. Everything below is fitted in your browser from the daily revenue series.
The generator added $18K a day to the treated markets after launch and nothing to the others, so the correct answer is $490K. The synthetic control estimate lands 8.7% away from it. The platform number is off by $1.66M.
The counterfactual
What the treated markets would have done
The dotted line is a weighted blend of the 20 held-out markets, with weights fitted only on the 140 days before launch. After launch it keeps running as the counterfactual, and the shaded wedge between the two lines is the effect.
The credibility check
Whether you should believe any of that
A synthetic control that does not track the treated markets before the campaign is not a counterfactual, and the estimate built on it is meaningless. This is the part a measurement vendor has no incentive to show you, which is exactly why it goes first.
The blend tracks the treated markets to within 1.0% of daily revenue across 140 pre-campaign days, and refitting it with a fake launch date four weeks early finds $661 a day, which is nothing. Both had to hold before the estimate below was worth printing. If the first number were 4% instead of 1.0%, the honest readout is “inconclusive”, and that is what this would say.
Non-negative and summing to one, which is the constraint that stops the fit from extrapolating. The counterfactual is a blend of real markets, so you can name every market that went into it and argue about whether it belongs.
The decision
What the campaign returned
$640K of media in the treated markets over 28 days. The platform’s number and the measured number disagree about whether that was a good trade, and they disagree by a factor of 4.8.
The campaign did cause revenue. It caused $448K of it, against $640K of spend, so every dollar of media returned 0.70 dollars of revenue before any cost of goods. The platform reported 3.36×. Those two numbers lead to opposite budget decisions, and only one of them was measured against markets where the campaign never ran.