Faster Building Needs Faster Learning: How Amazon and Netflix Test at Scale

Building software has become faster. In field experiments run by Microsoft, Accenture and a Fortune 100 company, developers given an AI coding assistant completed 26% more tasks, with a wide margin of error. Duolingo built 148 new language courses in about a year, after taking roughly 12 years to build its first 100. Across industries, the time between an idea and a working version of it is shrinking.

When building speeds up, expectations follow. If a variant can be built in a day, stakeholders reasonably expect to know within days whether it worked.

That expectation runs into a finding the experimentation literature has documented for two decades: most ideas do not work. Building faster does not change those odds. It only means an organization can ship more ideas that do not work, faster, unless it can tell which ones do. When building is cheap, the constraint moves from making things to learning which of them are worth keeping.

Amazon and Netflix are two of the best-documented examples of organizations that built for that constraint. This piece looks at what they built, and what any team expecting faster results needs to have in place first.

More ideas, the same odds

Microsoft’s experimentation team reported that “only about 1/3 of ideas improve the metrics they were designed to improve.” Of the rest, it estimated about as many made no difference as made things worse.

Redrawn from Kohavi et al., “Online Experimentation at Microsoft” (2009), which estimated “about 1/3 be good, 1/3 flat, and 1/3 negative.”

The pattern holds elsewhere. Ron Kohavi, who ran data mining and personalization at Amazon before leading experimentation at Microsoft, wrote that at Amazon “it is a common practice to evaluate every new feature, yet the success rate is below 50%.” Netflix says “only a small percentage of our ideas turn out to be winners in A/B tests.” A 2025 paper from Amazon’s own researchers describes “tens of thousands of online experiments conducted every year” at Amazon, of which “only a small proportion will meet launch criterion.”

Confident ideas fail too. Kohavi recalls insisting on a test before Amazon moved its order pipeline to a new application server. It “failed with a 2% revenue loss,” and after a bug was fixed, “it failed again.” At Netflix, a set of recommendations for House of Cards that looked more relevant to the team lost its test to a less obvious one.

If a third of ideas help, doubling the number of ideas doubles the number of winners only if you can find them. Without a test, you also double the losers, and you cannot tell the two apart.

Amazon: make a test hard to block

In Amazon’s early years, Greg Linden, one of its early developers, built a prototype that recommended items in the shopping cart based on what was already there. A marketing senior vice-president was, in Linden’s words, “dead set against it,” worried it would distract people from checking out. Linden was told he was “forbidden to work on this any further.”

He ran a test anyway. “Even for top executives,” he wrote later, “it was hard to block a test.” The feature won “by such a wide margin that not having it live was costing Amazon a noticeable chunk of change.”

Linden is careful about what the story shows. Amazon, he wrote, “did not yet have a culture that fully embraced measurement and debate.” What it had was a system in which a test was easy to run and hard to stop. That is the part that scales.

The system grew quickly. Amazon founder Jeff Bezos’s 2013 shareholder letter reported 546 experiments on Amazon’s internal platform, Weblab, in 2011, 1,092 in 2012 and 1,976 in 2013. By 2025, Amazon’s researchers were describing tens of thousands a year.

Speed itself became something Amazon measured. Linden’s 2006 slides record the result of adding delay to pages as “+100 ms -1% sales.” That is the kind of finding that ends arguments: nobody needs a strong opinion to act on it, only the test.

Netflix: make every test count

Netflix takes a different route to the same place. In 2019 its engineers wrote that it “runs hundreds of experiments yearly,” far fewer than the tens of thousands at Amazon. But “every product change Netflix considers goes through a rigorous A/B testing process before becoming the default user experience.”

Its constraint is traffic, not ideas. Retention tests typically run two to six months, and detecting a 0.1-point change in retention takes roughly 2 million members in each group. You cannot run many tests like that at once.

So Netflix invested in methods that learn more from less. For ranking algorithms, it uses interleaving, which mixes two rankers’ results into one list for the same member and sees which one’s picks get chosen. Netflix reported that this “requires >100× fewer users than our most sensitive A/B metric to achieve 95% power.” Weak ideas are pruned in days, and only the survivors go to a full test. That is velocity built into the method: more ideas screened without more traffic.

The culture is set up for the odds. Test results are “near-requirements for decision making,” with buy-in “including from the C-Suite.” Netflix does “not view tests that do not produce winning experience as ‘failures’,” and because it tests everything, “our culture of experimentation democratizes ideation.” A junior engineer’s idea and an executive’s face the same test.

What both built

Amazon runs far more tests than Netflix. Netflix squeezes more out of each one. What they share is the list Kohavi and colleagues drew up from their work on Microsoft’s Bing, where the challenges of scale are “cultural/organizational, engineering, and trustworthiness.”

• A platform that makes a test nearly free. The goal, in Kohavi, Tang and Xu’s textbook, is to lower “the marginal cost of experiments close to zero.” Netflix runs its platform “as an internal product, complete with its own product manager.”
• Statistics that stop false wins. At scale, broken tests are routine: about 6% of Microsoft’s experiments show a mismatch between the designed and actual split of users. Checking results early manufactures winners: in Netflix simulations of tests where nothing changed, “66 out of 100” falsely showed a significant difference when checked repeatedly. When Bing shipped a change that raised revenue 12%, its monitoring first fired a “revenue-too-high alert,” treating a surprisingly good number as a possible bug until proven otherwise.
• A culture that acts on the result. Tests that decide, not decorate; losing tests treated as information; and no idea exempt because of whose it is.

The takeaway

• AI makes ideas cheaper, not better. Most tested ideas still lose, about two in three at Microsoft. Faster building without testing ships more losers faster.
• The bottleneck is learning, not building. Amazon removed friction from running tests until tens of thousands a year were normal. Netflix, with less traffic to spare, adopted methods like interleaving to screen more ideas with fewer users.
• Faster results need three things in place: a platform that makes testing nearly free, statistics you can trust, and a culture where a test is hard to block and a loss is not a failure.
• Expect small wins, mostly. Kohavi, Tang and Xu write that key metrics improve through “many small changes: 0.1% to 2%.” Big jumps are rare, which is why the volume of learning matters.

The next part of this series looks at experimentation in marketing and advertising, where randomizing individual people is often impractical: a TV spot reaches everyone in a region, and some ad platforms offer no way to hold a group back.

Appendix: technical notes

These notes expand four claims made in “What both built” and the Netflix section above: the share of experiments with a broken split, the effect of checking results early, why Netflix needs about 2 million members per group, and how interleaving saves traffic.

Detecting a sample ratio mismatch

Fabijan et al. (2019) name a chi-square test as the detector. Restated from first principles: for each group, take (observed users − expected users)² / expected users, and sum. If the split is working, the sum follows a chi-square distribution with one degree of freedom for a two-group test, so a large value is unlikely by chance. The paper’s example is a 50/50 test with 821,588 and 815,482 users: expected counts of 818,535, a statistic of about 22.8 and a p-value of about 1.8 in a million (arithmetic added here), or in the paper’s words “less than 1 in 500k.” At this scale a 50.2/49.8 split is not noise.

Peeking, and what “any-time valid” changes

Restated from first principles: a fixed-horizon test holds its false-positive rate for one look at a planned sample size. Each extra look is another chance for noise to cross the threshold, and with enough looks a test of no effect eventually crosses it. Netflix’s A/A simulations show the scale: 66 of 100 falsely rejected (Lindon et al. 2024). Group-sequential designs fix the looks in advance and demand more early; in Bing’s alerting, the O’Brien-Fleming cutoff for day 1 of a 7-day experiment is 5 × 10⁻⁸ and the last is 0.040 (Kohavi et al. 2013). Any-time valid tests hold their error rate at whatever moment the analyst stops (a textbook restatement).

The arithmetic behind 2 million members

Gomez-Uribe and Hunt (2015, Appendix B) treat retention as a coin flip per member, with variance of the difference σ² = p_c(1 − p_c)/n_c + p_t(1 − p_t)/n_t, and call a cell positive if the observed difference is at least 1.96σ. At 50% retention and 2 million per cell, σ is 0.05 points and 1.96σ is about 0.1 points, which is where “roughly 2 million” comes from (arithmetic added here). A true difference of exactly 0.1 points would clear that line only about half the time. For 80% power, Kohavi et al.’s (2009) rule n = 16σ²/Δ², with a per-member variance of 0.25 and Δ = 0.001, gives about 4 million per cell. Either way the sample grows with 1/Δ²: halving the effect quadruples it.

Interleaving and its limit

Interleaving merges two rankers’ results into one list for the same member and records whose items get chosen. Each member serves as their own comparison, so differences between members drop out (reasoning added here). Parks, Aurisset and Ramm (2017) measured “>100× fewer users than our most sensitive A/B metric to achieve 95% power” by bootstrap on two rankers of known relative quality. The limit is relative preference only: it “does not allow us to directly measure changes to metrics such as retention.” At Airbnb, each interleaving experiment “takes up about 6% of regular A/B test traffic, and one-third of running length” (Zhang 2022).