Incrementality Testing Explained: How to Prove Your Ads Are Actually Causing Conversions
Your tracker says a campaign produced 400 conversions last week. The question most media buyers never ask is how many of those 400 would have happened if the campaign had never run. Incrementality testing answers that question, and the answer is often uncomfortable. In this post, we explain what incrementality is, why attribution reports cannot measure it, and how to run a simple holdout test on RTB traffic without a data science team.
What Incrementality Means
Incrementality is the number of conversions, sales, or sign-ups that happened because of your ads, not merely after your ads. A conversion is incremental if it would not have occurred without the ad exposure.
That distinction matters because every tracking system you use is built on correlation. A user sees or clicks an ad, then converts, and the tracker connects the two. It cannot see the alternate universe where the user never saw the ad. If that user was already heading to your offer, the ad gets credit for a sale it did not cause.
Attribution answers "which touchpoint gets the credit?" Incrementality answers "did the spend change anything?" Those are different questions, and only the second one tells you whether your budget is working. If you want a refresher on how credit gets assigned in the first place, see our guide to ad attribution models.
Where Non-Incremental Conversions Come From
On ad network traffic, several patterns produce conversions that look great in reports but add little real value:
- Retargeting existing intent. Users who already visited your site are likely to return anyway. Retargeting campaigns often show excellent CPAs for this reason. Some of that performance is real, and some is credit for conversions that were going to happen regardless. Our post on retargeting strategies covers how to structure these campaigns well.
- Brand or returning-customer traffic. If your landing page attracts repeat buyers, an ad that happens to reach them gets a conversion it did not create.
- Overlapping campaigns. When you run the same offer across several traffic sources, each tracker claims the conversion. Add up the reported totals and you may be counting one sale three times.
- Long lookback windows. A click thirty days ago gets credit for a purchase today. The longer the window, the more coincidental conversions get attributed.
None of these are fraud. They are simply the limits of click-based measurement. An incrementality test is how you check what attribution cannot.
The Core Method: The Holdout Test
The cleanest way to measure incrementality is a holdout test, sometimes called a ghost ads or control group test. The logic is straightforward:
- Split your target audience into two random groups.
- Show ads to the test group. Withhold ads from the control group.
- Measure conversions in both groups over the same period.
- The difference between the two conversion rates is your incremental lift.
If the test group converts at 2.0% and the control group converts at 1.5%, your ads produced 0.5 percentage points of lift. The other 1.5% would have happened anyway. Divide the incremental conversions by your spend and you get your true incremental cost per acquisition, which is almost always higher than the CPA your tracker reports.
The hard part is making the split truly random and making sure the control group is genuinely unexposed. Any leakage between groups shrinks the measured difference and understates your real impact.
How to Run a Holdout Test on RTB Traffic
You do not need a lab environment to run a useful incrementality test on popunder, redirect, or banner traffic. Here is a practical approach.
Option 1: Geographic holdout
Pick a set of regions or cities that behave similarly and have comparable baseline conversion volume. Run your campaign in half of them and pause it in the other half for the test period. Compare total conversions per region, including conversions from organic, direct, and other channels, before and during the test.
Geographic holdouts work well on ad network traffic because geo targeting is precise and easy to control. The weakness is that regions are never perfectly matched, so run the test long enough to smooth out day-to-day noise.
Option 2: Time-based holdout
Run the campaign for a period, pause it completely for a matched period, then run it again. Compare total conversions during the "on" and "off" windows. This is the simplest test to set up, but it is vulnerable to seasonality, day-of-week effects, and anything else that changes over time. Alternate on and off periods in short cycles rather than one long block of each to reduce that risk.
Option 3: Audience holdout
If your tracker or ad platform supports it, split users randomly at the impression level and suppress delivery for a percentage of them. This produces the cleanest results but requires more technical setup. Postback tracking that records every exposure, not only clicks, is a prerequisite. Our guide to postback conversion tracking explains how to get that data flowing.
Making the Results Trustworthy
A holdout test is only as good as its design. Keep these principles in mind:
- Measure total conversions, not attributed ones. The whole point is to capture conversions your tracker would not credit to the campaign. Pull numbers from your offer's backend or your CRM, not just the tracker.
- Hold everything else steady. Do not launch a new creative, change bids, or start another traffic source mid-test. Any change contaminates the comparison.
- Size the test properly. Small differences between groups need large sample sizes to be meaningful. Our post on statistical significance for media buyers walks through how much data you need before trusting a result.
- Run it long enough. Cover at least one full weekly cycle, and ideally two, so weekday and weekend behavior are both represented.
- Repeat before you reorganize your budget. One test is a signal. Two consistent tests are a conclusion.
What to Do With the Answer
Incrementality results change decisions in ways attribution reports never can.
If a campaign shows strong lift, you have evidence to scale it with confidence, and you can defend that spend to anyone who questions it. If lift is weak or absent, you have found budget that can be moved to campaigns that actually generate new customers. Media buyers often discover that their lowest-CPA campaign is their least incremental, while a prospecting campaign with a mediocre reported CPA is driving most of the real growth.
Incrementality also sharpens your ROAS calculations. Reported ROAS based on attributed revenue flatters every campaign that touches warm traffic. Incremental ROAS tells you what your dollars actually produced.
Conclusion
Attribution tells you where the credit went. Incrementality tells you whether the money mattered. The two are not in competition, and you need both: attribution for day-to-day optimization and incrementality for the periodic reality check that keeps your strategy honest.
You can start with a simple geographic or time-based holdout this week, using nothing more than the targeting controls already in your campaign dashboard and the conversion data in your offer backend. The answer may confirm what you believed, or it may free up budget you did not know you were wasting. Either way, you will be making decisions on cause rather than coincidence.
Ready to put this into practice? Squren's precise geographic targeting, flexible campaign scheduling, and token-based tracking make holdout tests easy to set up on RTB traffic. Sign up as an advertiser at Squren.com to launch your first campaign, or contact our 24/7 support team if you need help designing a test.