When You Cannot Easily Randomize People: Experimentation in Marketing

In 2017 Uber sued one of its advertising agencies, Fetch Media. The complaint alleges that Uber paid Fetch more than $82.5 million between 2016 and early 2017, and that the agency “allowed networks and publishers to take credit for Uber App installs that would have happened regardless of advertising.” One chart in the filing is titled “Effect of Fetch Campaign Pause: Fetch Signups Replaced by Organic Signups”.

These are allegations, and Fetch sued Uber in turn. But the shape of the claim is familiar to anyone who has measured paid media. Attribution, the reporting that assigns each sale or install to the ads a customer saw, said the ads were working. A pause suggested much of the credit belonged to demand that already existed. The difference is incrementality: the sales that happened only because of the ads. The most reliable way to measure it is an experiment, which shows the ads to one group, withholds them from a comparable group, and compares the two.

Part 1 argued that as AI makes building faster, most tested ideas still lose, so the bottleneck moves from making things to learning which of them work. It looked at how Amazon and Netflix made product experiments cheap, trustworthy and routine. The same logic applies to marketing: more campaigns, audiences and ad variants only help if you can tell which ones add sales that would not have happened anyway.

Marketing experiments are harder than product tests on all three counts. The companies that run them well rely on designs built for those constraints, and they treat testing as a standing program rather than a one-off project.

Why marketing experiments are harder

The treatment costs money. In a product test, the control group sees the existing product. In an ad test, every treated user is paid for. Netflix’s data scientists describe the consequence directly: “the treatments (the ads that we purchase) have a direct monetary cost to Netflix. As a result, we are risk averse”.

The signal is tiny against the noise. Purchases, in the words of the Yahoo researchers below, “are occasional, unpredictable and highly variable in amount”, so sales vary enormously from person to person and an ad’s effect is a small shift buried in that variation. In a Yahoo experiment on 3 million past customers of a retailer, sales per person had a standard deviation (the typical distance from the average) of “$125 for the two-week campaign”. The researchers asked whether the test could detect an effect of $0.51 per exposed customer, the lift that would give the retailer a 50% return on its ads.

That gap shows up in statistical power: the chance that a test detects an effect that is really there. With 570,000 users in each of the main treatment and control groups, and the researchers’ methods for cutting noise, the test had 79% power to detect that effect, so about one time in five it would have missed a profitable campaign. An earlier Yahoo study “had only 47% power”. Netflix says its advertising experiments “generally have low power”.

That is why the Yahoo test matters. It had more than a million people, a platform that controlled delivery and researchers who built methods to cut the noise, and it still had only a four-in-five chance of detecting a profitable campaign; without those methods, the chance was 49%, close to a coin flip. Most advertisers’ tests are far smaller, so “no lift detected” rarely means the ads did nothing. A test is only worth running if it is designed and powered to detect the effect that matters, which is what the rest of this piece is about.

Often you cannot randomize people at all. A fair test assigns people to see or not see the ads at random, so the two groups differ only in the ads. A TV spot or a billboard reaches everyone in a region. Some platforms offer advertisers no testing framework, and user-level identifiers can be blocked.

The old control design broke. Advertisers used to show a control group a public-service announcement (PSA), an unrelated charity or public-interest ad, in the same slot, so they could see which control users would have been reached. Under algorithmic delivery, that fails: the optimizer shows the placeholder and the real ad to different kinds of people. The paper that introduced ghost ads (below) states it plainly: “PSA experiments are rendered invalid when marketers use performance-optimizing computer algorithms to deliver ads.” Citing a 2014 figure from the IAB, an advertising industry group, it notes that two-thirds of online display ad spending used such optimization.

Designs that make it cheap

The last decade produced a set of designs that make marketing experiments affordable enough to repeat. Several were built inside the ad platforms.

Ghost ads. Instead of buying placeholder ads, the system records when a control user would have been shown the ad. That lets analysts compare exposed users with would-have-been-exposed users, without paying for the control. The method’s authors, Garrett Johnson (then at the University of Rochester), Randall Lewis (then at Netflix) and Elmar Nubbemeyer of Google, built it into Google’s display network. In their 2016 paper they report the same precision as older designs for “at least an order of magnitude less” spend, and an implementation that “records more than 100 million predicted ghost ads per day.”

Design can beat more data. In the Yahoo experiment, removing “the half of in-campaign sales data that is unaffected by the ads” gave “31% more precision”, equivalent to growing the sample from 3 million to 5.3 million users. Adding two years of purchase history as covariates (extra variables used to explain away some of the noise) improved precision by only 5%. The often quoted 50% variance reduction from CUPED, a method that adjusts for each user’s past behavior, comes from Bing product experiments. It is worth trying in ad tests, but it should not be assumed.

Geo experiments. When people cannot be randomized, regions can: ads run in some cities or regions and not in comparable others, and the difference in sales is the lift. Two Google researchers describe geo experiments as “a standard tool for the causal measurement of online advertising at Google”. Google’s published methods include time-based regression, built for cases with few regions, and Trimmed Match, which “trims the poorly matched pairs” of markets before estimating incremental return on ad spend (iROAS: the extra revenue the ads caused per dollar spent). Google’s geo Conversion Lift names both as its “underlying methodologies”. Meta’s open-source GeoLift uses synthetic control, which builds a counterfactual market from a weighted combination of untreated ones. GeoLift’s documentation makes a point worth learning before the first test, not after it: “running a geo-test without a robust prior power analysis leads to a high chance of failing to find lift, even if it actually happened.”

Switchbacks. A switchback randomizes time instead of people: the ads are switched on and off in alternating periods, and the periods are compared. When DoorDash wanted to measure app-store search ads, the platform offered no A/B framework, no precise geo targeting, and no user-level randomization. The team turned campaigns on or off by day in week one, the sequence reversed in week two. They had to account for ads taking “a couple of hours” to disappear after a pause.

Budget splits. LinkedIn’s design for its jobs marketplace “creates two copies of every ad, with the total budget split asymmetrically”, and compares how the higher- and lower-budget copies perform. That measures what an extra dollar returns without a holdout group, the set of people deliberately kept from seeing the ads.

Programs, not projects

The difference between a company that has run an incrementality test and one that runs incrementality testing is whether the results feed back into everything else.

Always-on holdouts. Instacart measures ad incrementality “on an ongoing basis” at the advertiser, system and experiment-variant level. Each advertiser’s holdout is randomized by a hash of user and advertiser, the holdouts are independent of each other, and product experiments run on top of them. (The ongoing measurement covers Instacart’s Sponsored Products ads.) LinkedIn’s budget split “is deployed continuously, not as a one-off measurement exercise.”

Experiments calibrate the models. Airbnb uses “ghost ads, geography based experimentation, and longitudinal tests” to measure channel incrementality, and the outputs “are used to calibrate our multi-touch attribution model”, the model that splits credit for a new sign-up across the ads a person saw. DoorDash says its switchback results will inform calibration of its marketing mix model (a statistical model of how spending in each channel relates to sales) and future budget allocation, and can yield an “incrementality scalar,” the ratio between experimental incrementality and attribution. The ratio is the point. Attribution keeps running; the experiment tells you how far to discount it.

Past tests become priors. A prior is what you already believe before a new test, based on earlier evidence; a new result is weighed against it rather than read in isolation. This is the compounding benefit of a program. Netflix takes “a Bayesian approach to experimentation in growth advertising”, using “the distribution of metric movements from past tests as an additional input”. It is explicit about why: with low power, exaggerated effect estimates “could lead to overestimating our optimal budget level”, because in an underpowered test the results that clear the bar for significance tend to be the lucky, inflated ones. A question Netflix’s data scientists ask is a good template for any team: “What information can I synthesize from the last year of tests, to inform opportunity sizing for next year’s learning roadmap?”

The platforms are moving the same way. Google announced that “in 2025 there will be a gradual transition to Bayesian methodology for all Google Ads accounts” for Conversion Lift, using historical campaign data as priors so that studies can report with lower budgets and fewer conversions.

Netflix’s marketing engineering team summed up the philosophy in one line: “every dollar we spend is a dollar we can learn from”.

What programs find

Some channels, once tested, turn out not to pay back. Finding them is the point of testing.

At eBay, brand-keyword search ads (ads bought on searches for the company’s own name) had “no measurable short-term benefits”; when they were switched off, almost all of the paid clicks came back as free ones through the ordinary search results. Across 15 field experiments with 13 US retailers, broad category search ads did lift brick-and-mortar sales, but the average return on ad spend “does not breakeven”, although several retailers reached or passed break-even on store sales alone. A positive, measurable effect and a profitable one are different findings.

Sometimes the evidence arrives without a designed test. In March 2020 Airbnb paused performance marketing. Its IPO prospectus reports paid channels falling from about 23% of traffic in 2019 to about 9% in the first nine months of 2020, and says it believes it can “maintain or increase overall traffic levels with lower performance marketing spend.” Airbnb’s co-founder and CEO, Brian Chesky, later said traffic “came back to 95% of the traffic levels of 2019 without any marketing spend.” That was a pandemic-era pullback with no control group, measured in traffic rather than bookings, and Airbnb still buys performance marketing, which Chesky described as “like a laser.” It is a lesson about what attribution had been claiming, not a measurement of incrementality.

Redrawn from Airbnb’s IPO prospectus (Form 424B4, December 2020), which reports “approximately 23% and approximately 9%” of traffic from paid performance marketing channels for 2019 and the nine months to September 30, 2020. A before-and-after comparison during the pandemic, not a controlled test.

Reading the platform lift tools

Google and Meta both offer lift studies inside their ad platforms, and both are worth using. They also need careful reading.

• Access is gated. Google says Conversion Lift “isn’t available for all Google Ads accounts”, and user-based studies need “at least 1,000 observed conversions” and a “minimum campaign budget of $5,000 USD.” Meta’s developer documentation says access is “currently limited.”
• Power is shown before launch. For user-based studies, Google displays study power from 50% to 95% and recommends aiming for 90%. Meta’s developer documentation says a study that does not meet its power threshold gets a warning. Take the warnings seriously; an underpowered test that finds nothing tells you almost nothing.
• Null results happen, even in the examples. A null result is one where the test cannot tell the effect apart from zero. The sample output in Meta’s own documentation shows negative incremental conversions with an interval spanning zero.
• Beware slicing after a null. When no lift is detected, Google’s geo results page suggests checking “specific conversion slices” and focusing on “those segments with positive lift.” Search enough subsets and one will show lift by chance, so a segment found after a null result is a false positive until a new test confirms it.
• Do not peek. Google’s geo tool shares results while the study runs, though it recommends waiting until the end. Checking repeatedly and stopping as soon as a result looks significant raises the odds of a false win. Classical p-values, the standard test of significance, are “wholly unreliable” when people stop tests by watching them, in the words of Ramesh Johari of Stanford University and colleagues, including Leo Pekelis and David Walsh, who worked at the testing company Optimizely on methods that allow continuous monitoring. Wait for the planned end, or use methods designed for continuous monitoring.

The tools make an experiment easy to launch. They do not make it well designed, adequately powered or honestly read.

The takeaway

• Marketing tests are harder than product tests. Every treated user costs money, the effect is tiny against the noise (in one Yahoo test, a $0.51 effect against a $125 standard deviation in sales), and often individual people cannot be randomized.
• Designs exist for most constraints. Ghost ads avoid paying for the control group; geo experiments and synthetic controls randomize or model regions; switchbacks randomize time; budget splits work inside the platform.
• Run a program, not a project. Standing holdouts, experiment results used to discount attribution and calibrate mix models, and past tests used as priors for new ones.
• Expect some channels to fail. eBay’s brand-keyword ads showed no measurable short-term benefit, and category search ads across 13 retailers lifted store sales without breaking even on average.
• Read platform lift tools with care. Check power before launch, wait for the planned end, and do not hunt through segments after a null result.

The final part of this series looks at the organization around the tests: who runs the platform, how results are made trustworthy, and who gets to decide.

Appendix: technical notes

These notes expand five claims made above: why placeholder (PSA) controls fail and ghost ads work (“The old control design broke” and “Ghost ads”), what the Yahoo power figures assume and why CUPED’s gains may not carry over (“The signal is tiny against the noise” and “Design can beat more data”), how Google’s geo methods work (“Geo experiments”), how synthetic control, switchbacks and budget splits work (the rest of “Designs that make it cheap”), and why peeking is a problem (“Do not peek”).

Ghost ads, PSAs and intent-to-treat

Johnson, Lewis and Nubbemeyer (2016) compare three control designs. Intent-to-treat compares everyone assigned, “including unexposed users who contribute only noise to the estimator”, but needs no special infrastructure and works with geographic randomization. PSA designs tag would-be-exposed control users with a placeholder ad, which fails once an optimizer serves the placeholder to different people. Ghost ads have the platform run “a second, simulated auction that includes the focal ad” for control users and log where it would have won. Predicted ghost ads run that simulated auction for both groups, keeping the comparison symmetric when a publisher can reject an ad.

Why exposure filtering raises precision, restated from first principles: if a fraction e of users is exposed, the intent-to-treat difference is diluted to about e times the effect while its noise comes from everyone. With equal outcome variance, keeping only exposed and would-be-exposed users improves the signal-to-noise ratio by a factor of 1/√e.

What the Yahoo power figures assume

Johnson, Lewis and Reiley (2017) set the alternative at a 50% return on ad spend. With “the $0.17 cost of display ads” and a 50% contribution margin, that implies a $0.51 effect on the treated. Power is 79% after their methods cut the standard deviation from $125 to $111, and 49% without. Covariates, including two years of purchase history, added 5%, because purchases “are occasional, unpredictable and highly variable in amount”.

That is also why CUPED’s 50% may not transfer. Deng et al. (2013) show that adjusting for a pre-period covariate multiplies variance by (1 − ρ²), where ρ is its correlation with the outcome; halving variance needs ρ of about 0.71 (arithmetic added here). For new users, the paper notes, “there are no pre-experiment data to work with.”

Time-based regression and Trimmed Match

Kerman, Wang and Vaver (2017) split a geo test into pretest, intervention and cooldown. On pretest data they fit y_t = α + βx_t + ε_t, with y_t and x_t the treatment and control series, then use the fit to predict the treatment group’s counterfactual through the test period. The cooldown captures lagged effects, but once cumulative effects stop rising it “only adds more uncertainty”.

Chen and Au (2022) randomize within pairs of geos. At the true iROAS θ, each pair’s response difference minus θ times its spend difference is symmetric around zero. Trimmed Match solves for the θ that sets the trimmed mean of those residuals to zero, dropping the worst-matched pairs; the estimate is total incremental response over total incremental spend in the pairs kept. Google’s geo Conversion Lift defines iROAS as “Incremental conversion value/Incremental Cost”.

Synthetic control, switchbacks and budget splits

In GeoLift, “SCMs find the combination of untreated units that most closely replicate the treated” before launch; divergence afterward is the lift. Augmented SCM de-biases the estimate and Generalized SCM supplies inference.

DoorDash’s switchback (Pan and Chen 2022) assumes the effect switches off within the period: ads took “a couple of hours” to disappear, and the design needed conversion to follow the click almost immediately. Simple random days were rejected because of day-of-week effects; reversing week one’s sequence in week two balances them across arms (reasoning added here).

LinkedIn’s asymmetric budget split (Hermle and Martini 2022) randomizes users into equal, disjoint sub-markets, splits each ad into high- and low-budget copies and randomly assigns the copies. Performance difference over budget difference estimates the return to a unit of budget, not to an exposure.

Always-valid inference

A classical p-value is valid at one planned sample size. Johari et al. (2022) construct “always valid p-values and confidence intervals”, which remain valid at whatever point a user stops, so watching the dashboard does not inflate false positives. The price, restated here, is wider intervals at any given sample size than a fixed-horizon test would give.