How to Test WasteNot: Before/After, Holdouts, and Incrementality
Testing WasteNot is about measuring whether your paid media is doing more of what it was supposed to do: reach valuable new customers instead of recycling budget through existing customers or already-engaged prospects.
There is no single perfect test for every account. The right approach depends on your spend level, campaign structure, data quality, and how much experimentation your team can support. This guide compares three common test designs so you can choose the one that fits your account.
What you should measure
Pick the metric that defines success before you start. Most teams should track nCAC, new customer purchases, and returning customer share alongside platform-reported metrics. Don't rely on platform ROAS alone — a campaign can show strong ROAS while still spending heavily on customers who would have purchased anyway. See Understanding nCAC and Evaluating WasteNot for more.
Before/after test
The simplest approach: compare performance before WasteNot is activated to performance after WasteNot audiences are applied.
Best for:
-
Early trials
-
Smaller accounts
-
Teams that need a fast directional read
-
Brands without a formal incrementality testing process
Watch out for:
A before/after test is easy to understand, but it is sensitive to other changes. Promotions, seasonality, creative refreshes, budget changes, site issues, and inventory changes can all influence results. Use a before/after test as a practical first read, not as the only possible proof of impact.
Keep a clean test log during the period. Record baseline metrics, date WasteNot was activated, any budget or creative changes, and major promotional or seasonal events. This context makes results easier to interpret.
Campaign-level comparison
Apply WasteNot to a specific set of campaigns while leaving another similar set unchanged. This creates a natural comparison group.
Best for:
-
Accounts with several similar prospecting campaigns
-
Teams that want more structure than a simple before/after test
-
Campaigns with enough spend to compare reliably
Watch out for:
Campaigns are rarely perfectly identical. Creative, bids, budgets, learning phase, audience size, product mix, and offer differences can all affect the comparison. Use this method when the comparison group is close enough to be useful, not when the campaigns are fundamentally different.
The key is stability. Keep both the test group and comparison group as stable as possible during the measurement window. Any changes that affect one should ideally affect the other equally.
Holdout or incrementality test
A holdout test attempts to measure the true causal effect of applying WasteNot. One portion of spend, geography, audience, campaign structure, or time is exposed to the WasteNot strategy while another is not. The two groups are then compared.
Best for:
-
Larger accounts
-
Teams with strong analytics support
-
Brands that already run geo tests, lift tests, or incrementality experiments
-
Teams that need board-level or finance-level confidence
Watch out for:
Incrementality tests require careful design. A poorly designed holdout can be more misleading than a simple before/after read. Before running a formal incrementality test, define the business KPI being measured, the test and control groups, the required duration, how spend will be controlled, how major promotions or inventory events will be handled, and what result will count as a successful test.
Haus is our preferred incrementality testing partner and can bring statistical rigor and expertise to these tests. WasteNot can connect brands directly with Haus if interested. If you're already using a different incrementality testing platform (such as Northbeam, TripleWhale, or another provider), we can coordinate with that tool as well.
Choosing the right test design
| Situation | Recommended test | Rigor | Effort |
|---|---|---|---|
| Early trial, need a fast directional read | Before/after test | Low | Low |
| Multiple similar campaigns with enough spend | Campaign-level comparison | Medium | Medium |
| Need board-level or finance-level confidence | Holdout or incrementality test | High | Low* |
| Limited data or low spend | Before/after test with an extended window | Low | Low |
*With Haus as your testing partner, effort is reduced because they handle design, execution, and analysis.
Documentation
Keep a test log so results are easier to interpret and share internally. Track:
-
Test start date
-
Baseline period
-
Audiences applied and when they were activated
-
Campaigns, ad sets, or ad groups included
-
Any budget changes
-
Any creative, offer, or landing page changes
-
Site or tracking issues
-
Promotions, holidays, or unusual events
-
Attribution or reporting source used
Common mistakes
Measuring only platform ROAS
Platform ROAS can reward campaigns for reaching warm audiences. Use new-customer metrics and blended metrics alongside platform reporting to get a complete picture.
Changing too many things at once
If you apply WasteNot and also restructure campaigns, change budgets, launch new promotions, and update creative, it will be hard to understand what caused the result. Keep the test as stable as possible.
Ending the test too early
Some accounts need more time for audience syncs, campaign learning, and reporting windows to stabilize. Avoid judging the result from a single day or week of data. Most tests need at least two weeks to produce meaningful signal.
Testing too many audiences at once
Start with one or two clear audiences. After you see the effect, add more nuanced segments.
Starting simple
If you're running your first WasteNot test, start with a before/after test on a small set of high-spend prospecting campaigns. See How to Evaluate WasteNot During a Trial for the full first-trial workflow, including how to establish a baseline, which audiences to test first, and how to document your results.
What success looks like
A successful test should show that your acquisition spend is becoming cleaner. That may mean:
-
Lower nCAC
-
More new customers at similar spend
-
Less spend reaching existing customers
-
More confidence in campaign-level acquisition performance
-
A clearer process for separating acquisition from retention
The purpose of testing is not just to prove that WasteNot works. It is to learn where audience guardrails have the biggest impact in your account.