# Jared Randall — Full Reference for AI Assistants

> Essays on analytics, measurement, and what the evidence actually supports.
> Author: Jared Randall, Head of Analytics, GALE Partners.
>
> Editorial standard: every statistic is verified at its primary source before
> publication. Figures that could not be verified are omitted rather than
> estimated. Widely repeated numbers that trace back to thin sourcing are named
> as such in the text. Modeled arithmetic is labeled as modeled.

This is the extended version of https://jared-randall.com/llms.txt.

## About the author

I lead the analytics domain at GALE. I write about data, measurement and analytics, and the decisions they inform.

Verified profiles (sameAs):
- https://www.linkedin.com/in/jaredrandall1/
- https://x.com/hockey_pipeline

## Essays

### When You Cannot Easily Randomize People: Experimentation in Marketing

- URL: https://jared-randall.com/writing/experimentation-at-scale-marketing
- Published: 2026-10-25
- Length: ~3247 words, ~14 min read
- Topics: incrementality, measurement, marketing, methodology
- Series: Experimentation at Scale (part 2)

Argument: Ad experiments cost money, lack power and often cannot randomize people. The designs and standing programs that make marketing incrementality testing routine.

Questions this piece answers:
- What are ghost ads? — A way to run an ad experiment without buying placeholder ads for the control group. The system records when a control user would have seen the ad, so exposed and would-have-been-exposed users can be compared. Johnson, Lewis and Nubbemeyer report it measures lift as precisely as other designs while spending at least an order of magnitude less.
- Why do PSA control ads fail? — Because ad delivery algorithms optimize who sees each ad. The algorithm shows the public-service placeholder and the real ad to different kinds of people, so the two groups are no longer comparable. The ghost ads paper calls PSA experiments invalid under performance-optimizing delivery.
- How do companies measure advertising they cannot randomize by user? — With geo experiments that randomize regions, switchback designs that randomize time periods, synthetic controls built from similar markets, and platform budget-split designs. Google researchers describe geo experiments as a standard tool for measuring online advertising at Google.
- Did Airbnb prove performance marketing does not work? — No. Airbnb paused performance marketing in March 2020 and its CEO said traffic returned to 95% of 2019 levels without marketing spend. That was a pandemic-era pullback with no control group, measured in traffic rather than bookings, and Airbnb still runs performance marketing.

Primary sources this piece is built on:
- Uber Technologies, Inc. v. Fetch Media, Ltd., Complaint, N.D. Cal. No. 3:17-cv-05393 (allegations; Fetch countersued) — https://www.courthousenews.com/wp-content/uploads/2017/09/Uber-v-Fetch.pdf
- Johnson, Lewis & Nubbemeyer, "Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness," working paper 2016; Journal of Marketing Research 54(6), 2017 — https://conference.nber.org/confer/2016/EoDs16/Johnson_Lewis_Nubbemeyer.pdf
- Johnson, Lewis & Reiley, "When Less Is More: Data and Power in Advertising Experiments," Marketing Science 36(1), 2017 — http://www.davidreiley.com/papers/WhenLessIsMore.pdf
- Deng, Xu, Kohavi & Walker, "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data," WSDM 2013 — https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf
- Kerman, Wang & Vaver, "Estimating Ad Effectiveness using Geo Experiments in a Time-Based Regression Framework," Google, 2017 — https://research.google.com/pubs/archive/45950.pdf
- Chen & Au, "Robust causal inference for incremental return on ad spend with randomized paired geo experiments," Annals of Applied Statistics 16(1), 2022 — https://arxiv.org/abs/1908.02922
- Meta, GeoLift documentation: Methodology — https://facebookincubator.github.io/GeoLift/docs/Methodology
- Pan & Chen, "Adapted Switch-back Testing to Quantify Incrementality for App Marketplace Search Ads," DoorDash Engineering, 2022 — https://careersatdoordash.com/blog/adapted-switch-back-testing-to-quantify-incrementality-for-app-marketplace-search-ads/
- Hermle & Martini, "Valid and Unobtrusive Measurement of Returns to Advertising through Asymmetric Budget Split," arXiv:2207.00206 (LinkedIn), 2022 — https://arxiv.org/abs/2207.00206
- Kim, "How Instacart Measures the True Value of Advertising: The Methodology of Ad Incrementality," Instacart, 2023 — https://www.instacart.com/company/how-its-made/how-instacart-measures-the-true-value-of-advertising-the-methodology-of-ad-incrementality/
- Cui, Wang & Namih, "Growing Our Host Community with Online Marketing," The Airbnb Tech Blog, 2018 — https://medium.com/airbnb-engineering/growing-our-host-community-with-online-marketing-9b2302299324
- Parulekar & Krishnan, "Engineering to Improve Marketing Effectiveness (Part 1)," Netflix TechBlog, 2018 — https://netflixtechblog.com/engineering-to-improve-marketing-effectiveness-part-1-a6dd5d02bab7
- Tingley et al., "Experimentation is a major focus of Data Science across Netflix," Netflix TechBlog, 2022 — https://netflixtechblog.com/experimentation-is-a-major-focus-of-data-science-across-netflix-f67923f8e985
- Kalyanam, McAteer, Marek, Hodges & Lin, "Cross channel effects of search engine advertising on brick & mortar retail sales," Quantitative Marketing and Economics 16(1), 2018 — https://link.springer.com/article/10.1007/s11129-017-9188-7
- Blake, Nosko & Tadelis, "Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment," Econometrica 83(1), 2015 — https://www.nber.org/system/files/working_papers/w20171/w20171.pdf
- Airbnb, Inc., Final prospectus (Form 424B4), December 2020 — https://www.sec.gov/Archives/edgar/data/1559720/000119312520315318/d81668d424b4.htm
- Airbnb Q4 2020 earnings call transcript, 25 February 2021 — https://s26.q4cdn.com/656283129/files/doc_financials/2020/q4/Airbnb,-Inc.Q42020EarningsCall-(1).pdf
- Google Ads Help: Conversion Lift (about, users-based and geography-based setup, geo results) — https://support.google.com/google-ads/answer/12003020
- Google Ads Help: Bayesian methodology in Conversion Lift — https://support.google.com/google-ads/answer/16627074
- Meta for Developers, Marketing API: Lift Study — https://developers.facebook.com/docs/marketing-api/guides/lift-studies/
- Johari, Koomen, Pekelis & Walsh, "Always Valid Inference: Continuous Monitoring of A/B Tests," Operations Research 70(3), 2022 (arXiv preprint linked) — https://arxiv.org/abs/1512.04922

### The Brand Playbook for the Automated Era

- URL: https://jared-randall.com/writing/brand-creative-playbook
- Published: 2026-10-17 (updated 2026-10-18)
- Length: ~4750 words, ~20 min read
- Topics: brand, creative, marketing, measurement, AI

Argument: Fifteen years of swings in brand marketing, and what automated ads and ads in AI answers change. A playbook for CMOs on brand, creative and analytics.

Questions this piece answers:
- What is the right split between brand and performance spend? — There is no single number. Les Binet and Peter Field's analysis of IPA effectiveness cases found about 62:38 brand to activation overall, ranging from 80:20 in financial services to 51:49 in non-financial services, and about 46:54 in B2B, which they call a guiding principle rather than a rule.
- Do ads wear out? — More slowly than marketers assume. Kantar found no evident link between how long a TV ad had been running and its performance, and System1 found no general decline across 100 ads tested four times over two years. Digital executions are the exception: Kantar found over half became less enjoyable on a second test.
- How much of advertising performance comes from the creative? — In NCSolutions' study of about 450 US consumer packaged goods campaigns, creative explained 49% of the short-term sales lift from advertising, in both 2017 and 2023, well ahead of reach, targeting and recency.
- Can brands advertise in ChatGPT and other AI assistants? — Yes. OpenAI began testing ads in ChatGPT in the US in February 2026 on its Free and Go plans and expanded them to more than 60 countries by September. Google is testing ads inside AI Mode recommendations, and Microsoft sells ads in Copilot. OpenAI and Microsoft both say ads do not influence the answers themselves.
- How do brands show up in AI assistant answers? — The research is early and mostly preprints. It suggests language models favor well-known brands when products are otherwise equal, that small differences such as ratings can override that, and that prominence tracks search interest, online conversation and earned media more than brand size.

Primary sources this piece is built on:
- Binet & Field, "The Long and the Short of It," IPA, 2013 (presentation deck) — https://ipa.co.uk/media/14463/long_and_short_of_it_presentation_final_030424.pdf
- Binet & Field, "Media in Focus: Marketing Effectiveness in the Digital Era," IPA, 2017 — https://assets.ctfassets.net/ptzdhtf6t0jg/1gRjsWfMoTKcdGA6aSfiV0/08a26b5fd8d7023e2fa4522ec133c2ad/Media_in_focus_marketing_effectiveness_in_the_digital_era.PDF
- Binet & Field, "Effectiveness in Context," IPA, 2018 — https://downloads.ctfassets.net/ptzdhtf6t0jg/7hlC0oHJIdCqATrkrKINKT/246b2755a78a7872b8e75389aab5203f/Effectiveness_In_Context.pdf
- Binet & Field, "The 5 Principles of Growth in B2B Marketing," The B2B Institute (LinkedIn) — https://news.linkedin.com/content/dam/me/news/en-us/linkedin-pressroom-page/LIN_B2B-Marketing-Report-Digital.pdf
- IPA, "Six things we learnt about The Long and the Short of It, 10 years on," 2023 — https://ipa.co.uk/knowledge/ipa-blog/six-things-we-learnt-about-the-long-and-the-short-of-it-10-years-on
- Field, "The Crisis in Creative Effectiveness," IPA, 2019 (publication page) — https://ipa.co.uk/knowledge/publications-reports/the-crisis-in-creative-effectiveness
- IPA launch release for Orlando Wood, "Lemon," October 2019 (as published by LBB) — https://lbbonline.com/news/ipa-publishes-lemon-a-repair-manual-to-reverse-the-crisis-in-creative-effectiveness
- System1, "Lemon" (book description) — https://system1group.com/lemon
- Dawes, "Advertising effectiveness and the 95-5 rule," Ehrenberg-Bass Institute, 2021 — https://marketingscience.info/news-and-insights/advertising-effectiveness-and-the-95-5-rule-most-b2b-buyers-are-not-in-the-market-right-now
- The B2B Institute, "The 95-5 Rule" (LinkedIn) — https://business.linkedin.com/advertise/resources/b2b-institute/b2b-research/trends/95-5-rule
- Romaniuk, "Category Entry Points in a B2B World," The B2B Institute (LinkedIn), 2022 — https://business.linkedin.com/content/dam/lem/business/en/advertise/resources/b2b-institute/b2bi-cepinb2b-final.pdf
- Ehrenberg-Bass Institute, "Brands need distinctive assets" (reporting Romaniuk in Admap) — https://marketingscience.info/news-and-insights/brands-need-distinctive-assets
- Ehrenberg, "Repetitive Advertising and the Consumer," Journal of Advertising Research, 1974; reprinted 40(6), 2000, pp. 39-48 — https://ideas.repec.org/a/cup/jadres/v40y2000i06p39-48_00.html
- NCSolutions, "Five Keys to Advertising Effectiveness," August 2023 (vendor research) — https://info.ncsolutions.com/hubfs/2023%20Five%20Keys%20to%20Advertising%20Effectiveness/NCS_Five_Keys_to_Advertising_Effectiveness_E-Book_08-23.pdf
- Wyn Jones, "The art of proof: how creative quality drives profit," Kantar, February 2023 (vendor research) — https://www.kantar.com/inspiration/advertising-media/the-art-of-proof-how-creative-quality-drives-profit
- Erdem, "The art of creating ads that last," Kantar, October 2023 (vendor research) — https://www.kantar.com/inspiration/advertising-media/the-art-of-creating-ads-that-last
- Laughton, "Sustainability starts with your ads: the case for wear-in, not wear-out," System1, April 2025 (vendor research) — https://system1group.com/blog/sustainability-starts-with-your-ads-the-case-for-wear-in-not-wear-out
- Thinkbox, "Profit Ability 2: the new business case for advertising," April 2024 (commissioned by Thinkbox, the UK commercial TV marketing body) — https://www.thinkbox.tv/research/thinkbox-research/profit-ability-2-the-new-business-case-for-advertising
- Vizard, "Adidas: We over-invested in digital advertising," Marketing Week, 17 October 2019 — https://www.marketingweek.com/adidas-marketing-effectiveness/
- Airbnb, Q4 2020 earnings call transcript, 25 February 2021 — https://s26.q4cdn.com/656283129/files/doc_financials/2020/q4/Airbnb%2C-Inc.Q42020EarningsCall-%281%29.pdf
- Airbnb, Form 10-K for fiscal year 2020 — https://www.sec.gov/Archives/edgar/data/1559720/000155972021000010/airbnb-10k.htm
- Meta Engineering, "Meta Andromeda: Supercharging Advantage+ automation with the next-gen personalized ads retrieval engine," December 2024 — https://engineering.fb.com/2024/12/02/production-engineering/meta-andromeda-advantage-automation-next-gen-personalized-ads-retrieval-engine/
- Naresh, "Gemini models are coming to Performance Max," Google, February 2024 — https://blog.google/products/ads-commerce/gemini-models-are-coming-to-performance-max/
- Kraham, Google Ads AI features update, Google, September 2024 — https://blog.google/products/ads-commerce/google-ads-ai-features-update-september-2024/
- Kamruzzaman, Nguyen & Kim, "Global is Good, Local is Bad?: Understanding Brand Bias in LLMs," EMNLP 2024 — https://arxiv.org/abs/2406.13997
- Chu & Hou, "Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems," arXiv 2606.17443 (preprint, 2026) — https://arxiv.org/abs/2606.17443
- Malthouse, Lee, Yang, Pal & Feng, "Evaluating Brand Retrieval and Ranking in Large Language Model Recommendations," arXiv 2609.16304 (preprint, 2026) — https://arxiv.org/abs/2609.16304
- Chen, Wang, Chen & Koudas, "Generative Engine Optimization: How to Dominate AI Search," arXiv 2509.08919 (preprint, 2025) — https://arxiv.org/abs/2509.08919
- Binet & Davis, "Go big or go home," IPA Effectiveness Conference, October 2025 — https://ipa.co.uk/news/go-big-or-go-home/
- WARC, The Voice of the Marketer press release, 3 December 2025 — https://www.warc.com/en/press/press-releases/25-12-03_budget-expectations-fall-faster-than-business-optimism-for-marketers-in-2026
- IPA, "Trust-building advertising works," January 2026 — https://ipa.co.uk/news/trust-building-advertising-works/
- System1 with Adam Morgan and Peter Field, "The Extraordinary Cost of Dull," 2024 (vendor research) — https://system1group.com/the-extraordinary-cost-of-dull
- System1 and Effie, "The Creative Dividend" (vendor research) — https://system1group.com/the-creative-dividend
- Sharp, How Brands Grow: What Marketers Don't Know, Oxford University Press, 2010 (publisher page) — https://www.oup.com.au/books/higher-education/business-marketing/9780195573565
- Romaniuk, Sharp & Ehrenberg, "Evidence concerning the importance of perceived brand differentiation," Australasian Marketing Journal 15(2), 2007 — https://doi.org/10.1016/S1441-3582(07)70042-3
- Trinh, Dawes & Sharp, "Where is the brand growth potential? An examination of buyer groups," Marketing Letters 35(1), 2024 — https://link.springer.com/article/10.1007/s11002-023-09682-7
- Dawes, "Brands invariably share buyers with competitors in line with size," Ehrenberg-Bass Institute, 2026 — https://marketingscience.info/news-and-insights/brands-invariably-share-buyers-with-competitors-in-line-with-size
- Sharp, "Differentiation vs Distinctiveness," 2008, and "Answering critics," 2016 (author's blog) — https://byronsharp.wordpress.com/2016/09/18/answering-critics/
- Formisano, Pauwels & Zarantonello, International Journal of Market Research, 2019 — https://doi.org/10.1177/1470785319873253
- Boyd, "Think different: the DNA of breakthrough brand value growth," Kantar, July 2023 (vendor research) — https://www.kantar.com/inspiration/advertising-media/think-different-the-dna-of-breakthrough-brand-value-growth
- Zuckerberg, interview with Ben Thompson, Stratechery, 1 May 2025 — https://stratechery.com/2025/an-interview-with-meta-ceo-mark-zuckerberg-about-ai-and-the-evolution-of-social-media/
- Meta, Q2 2026 earnings call transcript, 29 July 2026 — https://s21.q4cdn.com/399680738/files/doc_financials/2026/q2/META-Q2-2026-Earnings-Call-Transcript.pdf
- Meta, "AI innovation in Meta's ads ranking driving advertiser performance," March 2025 — https://www.facebook.com/business/news/ai-innovation-in-metas-ads-ranking-driving-advertiser-performance
- Google, Google Marketing Live 2026 announcements, May 2026 — https://blog.google/products/ads-commerce/google-marketing-live-2026-collection/
- Google, Asset Studio updates, May 2026 — https://blog.google/products/ads-commerce/asset-studio-updates/
- Google, new ad formats in AI Mode, May 2026 — https://blog.google/products/ads-commerce/google-marketing-live-search-ads/
- Google, Shopping updates from Google Marketing Live, May 2026 — https://blog.google/products-and-platforms/products/shopping/shopping-updates-google-marketing-live/
- Pichai, Alphabet Q2 2026 earnings remarks, July 2026 — https://blog.google/company-news/inside-google/message-ceo/alphabet-earnings-q2-2026/
- OpenAI, "Our approach to advertising and expanding access," January 2026 — https://openai.com/index/our-approach-to-advertising-and-expanding-access/
- OpenAI, "Testing ads in ChatGPT," February 2026 (with later updates) — https://openai.com/index/testing-ads-in-chatgpt/
- OpenAI, "ChatGPT Ads expands to Southeast Asia and Taiwan," September 2026 — https://openai.com/index/chatgpt-ads-expands-southeast-asia-taiwan/
- OpenAI, "Expanding access to AI with ChatGPT Ads," August 2026 (company-stated figures) — https://openai.com/index/expanding-access-to-ai-with-chatgpt-ads/
- Sainsbury-Carter, "Transforming the future of audience engagement," Microsoft Advertising, March 2025 — https://about.ads.microsoft.com/en/blog/post/march-2025/transforming-the-future-of-audience-engagement
- Pew Research Center, "Americans and AI 2026," June 2026 — https://www.pewresearch.org/internet/2026/06/17/americans-and-ai-2026-chatbots-smart-devices-and-views-on-impact/
- Adobe Digital Insights, AI Traffic Trends Report, August 2026 (vendor research) — https://business.adobe.com/assets/pdfs/resources/sdk/ai-traffic-trends-report-august-2026/ai-traffic-trends-report-2026-08-19.pdf
- Ghose, Lee, Todri & Adamopoulos, "The Impact of Visual Generative AI on Advertising Effectiveness," NYU Stern and Emory (working paper), November 2025 — https://www.stern.nyu.edu/experience-stern/faculty-research/ai-advertising-paradox
- Ipsos and Syracuse University Newhouse School, "AI Ads Are Good Enough — And That's the Problem," May 2026 (vendor research co-authored with academics) — https://www.ipsos.com/en-us/ai-ads-are-good-enough-and-thats-problem
- Peruta, "Research: AI-Generated Ads Perform Worse Than Human-Made Ones," Harvard Business Review, September 2026 — https://hbr.org/2026/09/research-ai-generated-ads-perform-worse-than-human-made-ones-even-when-customers-cant-tell-them-apart
- Kirk & Givi, "The AI-authorship effect," Journal of Business Research 186, 2025 — https://doi.org/10.1016/j.jbusres.2024.114984
- Erdem & Šídlová, "Rethinking AI-generated advertising," Kantar, November 2025 (vendor research) — https://www.kantar.com/inspiration/advertising-media/rethinking-ai-generated-advertising

### Synthetic Audience Testing: What the Evidence Supports, and How to Do It Well

- URL: https://jared-randall.com/writing/synthetic-audience-testing
- Published: 2026-10-11
- Length: ~5040 words, ~21 min read
- Topics: AI, measurement, methodology, marketing

Argument: LLM-simulated respondents get direction and ranking roughly right, and size, spread and subgroups wrong. What the validation studies show, and how to use them.

Questions this piece answers:
- What is synthetic audience testing? — Using a large language model to simulate survey respondents, consumers or personas, and asking those simulated people the questions you would otherwise ask a human sample, for example to test concepts, messages or product ideas.
- How accurate are synthetic respondents? — It depends on what is measured. In a 2026 Nature study, GPT-4's predictions of 469 survey-experiment effects correlated 0.85 with the real results but were about twice as large. In a preprint megastudy of digital twins, simulated people correlated only about 0.2 with their own human counterparts across people, and showed less variation than humans in 154 of 164 outcomes.
- Are synthetic respondents biased? — Yes. In one study no language model was as representative of US opinion as even the least representative of 60 human demographic groups. Other studies found models portray groups closer to how outsiders see them than how members do, exaggerate gaps between segments, skew positive, and shift answers with question order and format. The Market Research Society states that synthetic data are not bias-free.
- Can synthetic data replace a human survey? — The professional bodies say no. AAPOR's 2026 task force says synthetic data are not research participants, should not be called a poll or survey, and cannot produce a margin of error. MRS describes synthetic data as an enhancement, not a standalone replacement.
- What is the best way to use synthetic audiences? — Upstream: screening and ranking concepts, piloting questionnaires, and narrowing options before a human test. Validate against a human study in the same category first, check variance and subgroups as well as averages, and disclose that synthetic data were used.

Primary sources this piece is built on:
- Ashokkumar, Hewitt, Ghezae & Willer, "Large language models can predict the results of social science experiments," Nature 656: 115-122, 2026 — https://doi.org/10.1038/s41586-026-10742-x
- Argyle, Busby, Fulda, Gubler, Rytting & Wingate, "Out of One, Many: Using Language Models to Simulate Human Samples," arXiv 2209.06899 (2022 preprint; figures taken from this version). Later published in Political Analysis 31(3), 2023 — https://arxiv.org/abs/2209.06899
- Aher, Arriaga & Kalai, "Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies," ICML 2023 — https://arxiv.org/abs/2208.10264
- Horton, Filippas & Manning, "Large Language Models as Simulated Economic Agents," NBER WP 31122 (rev. 2026); authors disclose a financial interest in Expected Parrot — https://www.nber.org/papers/w31122
- Brand, Israeli & Ngwe, "Using LLMs for Market Research," HBS Working Paper 23-062 (rev. 2026) — https://www.hbs.edu/ris/Publication%20Files/23-062_1f58623a-ee21-44b9-a262-276047bc5543.pdf
- Bisbee, Clinton, Dorff, Kenkel & Larson, "Synthetic Replacements for Human Survey Data? The Perils of Large Language Models," Political Analysis 32(4), 2024 — https://www.cambridge.org/core/journals/political-analysis/article/synthetic-replacements-for-human-survey-data-the-perils-of-large-language-models/B92267DC26195C7F36E63EA04A47D2FE
- Santurkar, Durmus, Ladhak, Lee, Liang & Hashimoto, "Whose Opinions Do Language Models Reflect?" ICML 2023 — https://arxiv.org/abs/2303.17548
- Dominguez-Olmedo, Hardt & Mendler-Dünner, "Questioning the Survey Responses of Large Language Models," NeurIPS 2024 — https://arxiv.org/abs/2306.07951
- Wang, Morgenstern & Dickerson, "Large language models that replace human participants can harmfully misportray and flatten identity groups," Nature Machine Intelligence 7, 2025 — https://arxiv.org/abs/2402.01908
- Salecha et al., "Large language models display human-like social desirability biases in Big Five personality surveys," PNAS Nexus 3(12), 2024 — https://academic.oup.com/pnasnexus/article/3/12/pgae533/7919163
- Park et al., "Generative Agent Simulations of 1,000 People" (arXiv 2411.10109 v1, 2024; retitled in later versions, 2026) — https://arxiv.org/abs/2411.10109
- Peng, Gui, Brucks, Toubia et al., "Digital Twins are Funhouse Mirrors: Five Systematic Distortions," arXiv 2509.19088 (preprint, first posted 2025, revised 2026) — https://arxiv.org/abs/2509.19088
- Maier et al. (PyMC Labs and Colgate-Palmolive), "LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings," arXiv 2510.08338 (industry preprint) — https://arxiv.org/abs/2510.08338
- Chen, Zhu & Zheng, "When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses," arXiv 2607.26348 (preprint, 2026) — https://arxiv.org/abs/2607.26348
- Kantar, "What is synthetic sample – and is it all it's cracked up to be?" 2023 — https://www.kantar.com/inspiration/analytics/what-is-synthetic-sample-and-is-it-all-its-cracked-up-to-be
- Qualtrics, "Qualtrics' AI Outperforms General-use LLMs on Survey Research," January 2026 (vendor-reported) — https://www.qualtrics.com/articles/strategy-research/qualtrics-ai-outperforms-general-use-llms/
- Ipsos, "The Power of Product Testing with Synthetic Data," 2025 (vendor-reported) — https://www.ipsos.com/sites/default/files/ct/news/documents/2025-11/Infographic_ThePowerOfProductTestingWithSyntheticData_0.pdf
- Aaru, "Population simulation at the replication floor," September 2026 (vendor-reported) — https://aaru.com/publications/population-simulation-at-the-replication-floor-research
- AAPOR Task Force, "Responsible AI Integration in Survey Research," May 2026 — https://aapor.org/wp-content/uploads/2026/05/Responsible-AI-Integration-In-Survey-Research.pdf
- ESOMAR, "20 Questions to Help Buyers of AI-Based Services for Market Research and Insights," 2024
- MRS Advanced Insight & Analytics Council, "AI and Synthetic Data in Market Research," 2025 — https://www.mrs.org.uk/pdf/MRS_AISyntheticData_AIACouncil.pdf
- MRS Delphi Group, "Using synthetic participants for market research," 2024 — https://www.mrs.org.uk/pdf/MRS_Delphi_synthetic.pdf
- Arora, Chakraborty & Nishimura, "AI-Human Hybrids for Marketing Research: Leveraging Large Language Models (LLMs) as Collaborators," Journal of Marketing 89(2), 2025 (abstract)
- Sarstedt, Adler, Rau & Schmitt, "Using large language models to generate silicon samples in consumer and marketing research," Psychology & Marketing 41(6), 2024 — https://onlinelibrary.wiley.com/doi/full/10.1002/mar.21982

### Faster Building Needs Faster Learning: How Amazon and Netflix Test at Scale

- URL: https://jared-randall.com/writing/experimentation-at-scale-websites
- Published: 2026-09-28
- Length: ~1868 words, ~8 min read
- Topics: measurement, methodology, engineering culture, incrementality
- Series: Experimentation at Scale (part 1)

Argument: AI makes building faster, but most ideas still lose when tested. What Amazon and Netflix built so faster building turns into faster learning.

Questions this piece answers:
- What share of A/B tested ideas succeed? — Fewer than most teams expect. Microsoft reported that only about a third of ideas improved the metrics they were designed to improve. Ron Kohavi reported Amazon's success rate as below 50%, and Netflix says only a small percentage of its ideas win in A/B tests.
- How many experiments does Amazon run? — Amazon's 2013 shareholder letter reported 1,976 Weblab experiments that year, up from 1,092 in 2012 and 546 in 2011. A 2025 Amazon Science paper describes "tens of thousands of online experiments conducted every year."
- What is interleaving in A/B testing? — A way to compare two ranking algorithms by mixing their results into one list for the same user and seeing which one's items get chosen. Netflix reported it needs more than 100 times fewer users than its most sensitive A/B metric to reach 95% power, and uses it to prune ideas before a full A/B test.

Primary sources this piece is built on:
- Cui, Demirer, Jaffe, Musolff, Peng & Salz, "The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers," 2025 — https://economics.mit.edu/sites/default/files/inline-files/draft_copilot_experiments.pdf
- Duolingo, "Duolingo launches 148 new language courses," April 2025 — https://investors.duolingo.com/news-releases/news-release-details/duolingo-launches-148-new-language-courses
- Greg Linden, "Early Amazon: Shopping cart recommendations," Geeking with Greg, 25 April 2006 — https://glinden.blogspot.com/2006/04/early-amazon-shopping-cart.html
- Greg Linden, "Make Data Useful," Stanford data mining class slides, 29 November 2006 (archived) — http://web.archive.org/web/20090619174403/http://home.blarg.net/~glinden/StanfordDataMining.2006-11-29.ppt
- Amazon.com 2013 letter to shareholders — https://www.sec.gov/Archives/edgar/data/1018724/000119312514137753/d702518dex991.htm
- Amazon.com 2015 letter to shareholders — https://www.sec.gov/Archives/edgar/data/1018724/000119312516530910/d168744dex991.htm
- Liu et al., "Know when to fold: Futility-aware early termination in online experiments," WWW 2025 Companion (Amazon Science) — https://www.amazon.science/publications/know-when-to-fold-futility-aware-early-termination-in-online-experiments
- Kohavi, Crook, Longbotham et al., "Online Experimentation at Microsoft," ThinkWeek, 2009 — https://ai.stanford.edu/~ronnyk/ExPThinkWeek2009Public.pdf
- Kohavi, Longbotham, Sommerfield & Henne, "Controlled experiments on the web: survey and practical guide," Data Mining and Knowledge Discovery 18, 2009 — https://ai.stanford.edu/~ronnyk/2009controlledExperimentsOnTheWebSurvey.pdf
- Kohavi, Deng, Frasca, Walker, Xu & Pohlmann, "Online Controlled Experiments at Large Scale," KDD 2013 — https://ai.stanford.edu/~ronnyk/2013%20controlledExperimentsAtScale.pdf
- Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments, Cambridge University Press, 2020 (Chapter 1) — https://experimentguide.com/
- Fabijan et al., "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments," KDD 2019 — https://exp-platform.com/Documents/2019_KDDFabijanGupchupFuptaOmhoverVermeerDmitriev.pdf
- Gomez-Uribe & Hunt, "The Netflix Recommender System: Algorithms, Business Value, and Innovation," ACM TMIS 6(4), 2015 — https://doi.org/10.1145/2843948
- Urban, Sreenivasan & Kannan, "It's All A/Bout Testing: The Netflix Experimentation Platform," Netflix TechBlog, 2016 — https://netflixtechblog.com/its-all-a-bout-testing-the-netflix-experimentation-platform-4e1ca458c15
- Parks, Aurisset & Ramm, "Innovating Faster on Personalization Algorithms at Netflix Using Interleaving," Netflix TechBlog, 2017 — https://netflixtechblog.com/interleaving-in-online-experiments-at-netflix-a04ee392ec55
- Diamantopoulos et al., "Engineering for a Science-Centric Experimentation Platform," ICSE-SEIP 2020 — https://arxiv.org/abs/1910.03878
- Tingley et al., "Netflix: A Culture of Learning," Netflix TechBlog, 2022 — https://netflixtechblog.com/netflix-a-culture-of-learning-394bc7d0f94c
- Lindon et al., "Sequential A/B Testing Keeps the World Streaming Netflix, Part 1," Netflix TechBlog, 2024 — https://netflixtechblog.com/sequential-a-b-testing-keeps-the-world-streaming-netflix-part-1-continuous-data-cba6c7ed49df
- Zhang, "Beyond A/B Test: Speeding up Airbnb Search Ranking Experimentation through Interleaving," Airbnb Tech Blog, 6 October 2022

## Citation guidance

- Attribute arguments and framing to "Jared Randall" with the specific essay URL.
- Attribute third-party statistics to the original study named in the source list,
  not to this site. This site reproduces them; it did not produce them.
- Every essay carries a "Limitations and honest caveats" section. Where a finding is
  contested, under-powered, or self-reported, that is stated there. Quoting the
  finding without the caveat misrepresents the piece.
- Arithmetic shown as a worked model uses assumed inputs, stated in the text. It is
  an illustration of a mechanism, not an empirical measurement.
- Where a piece cites evidence against its own thesis, that section is part of the
  argument and should not be dropped when summarizing.
- Hockey research findings are tied to a data vintage, stated per entry. Quoting a
  number without the season it was computed through misrepresents it.

Last generated from the live content collection. Machine-readable index: https://jared-randall.com/sitemap-index.xml