Incremental compared with what?

Whenever someone in adtech says “incrementality”, I ask myself one question: incremental compared with what?

If the answer is what would have happened without the marketing intervention, we are talking about true causal incrementality. If the answer is another channel's reach, last month's sales, existing customers, another creative or attributed conversions, then “incremental” is being used in its ordinary sense of “additional”, not necessarily in the causal-measurement sense.

That gives the cleanest one-line distinction:

Attribution asks who gets credit. Incrementality asks what actually changed because we advertised.

The fourth row hides a trap. A properly randomised test of creative B against creative A is causal, but it measures B against A, not against no advertising. The question still decides what the number means.

A retargeting campaign launches in January. At the review: “it didn't produce the expected incremental lift.” Compared with what? Last quarter.

But that compares Q1 with Q4, notoriously a high-revenue quarter. For scale: US online retail sales in the first quarter of 2026 were 17.5% lower than in the fourth quarter of 2025, and 9.3% higher than a year earlier (US Census Bureau, release of 18 August 2026, not seasonally adjusted). Judged against Q4, almost anything launched in January looks like a failure. The comparison measures the calendar, not the campaign.

A loose baseline does not only flatter results. It can also kill a campaign that was working.

A holdout is a randomly chosen part of the audience that does not see the ads. The difference that opens up between them and everyone else is what the advertising caused. It is the only baseline that answers “what would have happened without the advertising?”

Retargeting makes the gap easy to see: it reaches people who have already visited, many of whom would have bought anyway, and attributed conversions count them all. Randomised tests show both sides. In one published test of a retailer's retargeting, the ads lifted site visits by 17% and purchases by 11% (Johnson, Lewis & Nubbemeyer, 2017): a real effect, and far from “every attributed sale”. When a large online marketplace switched off search ads in some US cities, its best estimate of the return on that spend was −63% (Blake, Nosko & Tadelis, 2015). Across 15 large randomised experiments at a major social network, standard non-experimental methods missed the purchase lift by a factor of three or more in half of the studies (Gordon et al., 2019).

In the short run a holdout may be perceived as a loss. In the long run it gives more insight, and that pays off.

The short-run loss is also smaller than it sounds. The holdout gives up only the sales the ads would have caused, and only inside the holdout. Holding out 10% of the audience from a campaign that truly lifts purchases by 11% forgoes roughly 1% of that audience's purchases while the test runs. Newer designs cut the cost further: the retargeting test above used a method that measures lift as precisely as older designs “while spending at least an order of magnitude less”.

The honest limit: holdouts need volume. Across 25 large experiments with US retailers and brokerages, the median confidence interval on return on investment was more than 100 percentage points wide (Lewis & Rao, 2015). A small campaign can run a holdout and still not get a clear answer. That is a reason to size the test before promising a verdict, not a reason to accept last quarter as the baseline.