Synthetic Audience Testing: What the Evidence Supports, and How to Do It Well

What synthetic audience testing is

Synthetic audience testing means asking a large language model to play the respondents. Instead of fielding a survey to 1,000 people, you describe 1,000 people to a model and ask how each would answer. The output looks like survey data, but no person answered it.

It comes in a few forms, from simplest to most involved:

Form	How it works	Example in the research
Persona prompting	Describe a respondent’s demographics or background in the prompt, then ask the survey question	Brigham Young University researchers gave GPT-3 thousands of voter backstories and asked how each would vote
Digital twins	Build each persona from a real person’s own interview or survey answers, then ask new questions	A Stanford-led team built agents from two-hour interviews with 1,052 people
Fine-tuned models	Train the model on past human survey data from the same category before asking	Microsoft and Harvard Business School researchers fine-tuned on earlier willingness-to-pay studies
Predicting results	Describe a whole experiment and ask the model to predict its outcome	Harvard and Stanford researchers predicted the results of 70 survey experiments

How marketers use it

For a marketing team, the appeal is testing more ideas, earlier, than a research budget normally allows. These are the uses that appear in the research and the industry guidance:

Use	The question it tries to answer	Where it appears
Concept screening	Which of these product ideas would people buy?	PyMC Labs and Colgate-Palmolive compared synthetic and human purchase intent for 57 personal-care concepts
Pricing	How much more would people pay for this feature?	Microsoft and Harvard Business School researchers estimated willingness to pay for toothpaste, laptops and projectors
Creative and ad testing	Which ad or message lands better?	MRS, the UK’s Market Research Society, calls it a “strong use case, but only if real consumer feedback is used alongside it”
Segmentation and targeting	Which audience should this go to?	A 2026 preprint found models picked the wrong target segment in half of its US cases
Repeating a past study	Would we have reached the same findings?	A University of Wisconsin team reran a Fortune 500 food company’s 2019 study with GPT-4
Questionnaire pretesting	Does this survey work before it goes to real people?	A checklist in Psychology & Marketing recommends synthetic samples mainly for “qualitative pretesting and pilot studies”

The American Association for Public Opinion Research (AAPOR) orders uses like these by risk. Pretesting a questionnaire sits at the low end. Replacing human respondents entirely sits at the high end.

Pros and cons at a glance

Pros	Cons
Fast and cheap. One Brigham Young University study ran 15,626 simulated vote-choice queries for $75; a Microsoft and Harvard Business School pricing study cost about $100 on a standard model, or $433 to $2,600 fine-tuned, and ran in hours, not weeks	Effect sizes come out too large, about twice the real ones across 469 experimental effects
Gets direction and ranking roughly right in text survey experiments: a 0.85 correlation with real results	Too little variation between people, which makes small samples look sufficient and results look more certain than they are
About as good as a crowd of lay forecasters, and better still when combined with human judgment	Biased: misrepresents some groups, stereotypes them, and exaggerates differences between segments
Useful upstream: piloting questionnaires and screening many ideas before a human test	Weak on individuals: digital twins barely beat a demographics-only persona
Improves with real data: fine-tuning on human data from the same category fixed estimates that pointed the wrong way and pulled an inflated one close to the human value	Weaker in larger multi-treatment studies, where accuracy fell by about half, and weakest for field settings and video
	Quirks of question format look like opinions: answer order, rating scales and social desirability shift results
	Results drift when the underlying model is updated
	No margin of error, and the American Association for Public Opinion Research (AAPOR) says it should not be called a poll or survey

Compared with human research

The same evidence, set against a conventional human study:

	Human research	Synthetic respondents
Speed	“weeks or months”	“ran in a matter of hours”
Cost	“many thousands of dollars”	$75 to about $2,600 per study, depending on the method
Direction and ranking	The benchmark	Close: a 0.85 correlation with real results in text survey experiments
Size of effects	The benchmark	About twice the real size
Variation between people	The benchmark	Too little, in 154 of 164 outcomes (preprint)
Subgroups and segments	The benchmark	Some groups misrepresented; gaps between segments inflated
Individuals	People repeated about 81% of their own answers two weeks later	An interview-based agent matched about 69% of its person’s answers
New trends	Asked of people now	Built from past data, so weaker on emerging trends
Margin of error	Can be calculated for a probability sample	“cannot be produced from synthetic responses”
What it can be called	A poll or survey	Not a poll or survey, per AAPOR

Speed and human-cost quotes and the $100 to $2,600 costs are from Brand, Israeli and Ngwe; the $75 is from Argyle et al.; the 81% and 69% are both from Park et al. (2024); the margin-of-error quote is from the model disclosure in AAPOR’s 2026 guidance. The other synthetic cells are sourced in the sections below. The human column is the conventional baseline these studies measure against, not a finding of theirs, so this table does not weigh the weaknesses of human research itself.

The rest of this piece goes through the evidence behind these two tables, then turns it into a set of practices. The short version: synthetic respondents are a screening tool, reasonably good at direction and ranking and unreliable at size, spread, subgroups and individuals. They are not a substitute for human research.

Where synthetic respondents do well

The strongest evidence comes from a 2026 Nature paper by Ashwini Ashokkumar and Isaias Ghezae of Harvard, and Luke Hewitt and Robb Willer of Stanford. They asked language models to predict the results of 70 preregistered (planned and registered in advance), nationally representative US survey experiments, covering 469 effects and 119,330 participants, and compared the predictions with what actually happened.

An effect here is the change a treatment produces: the gap in response between people shown one message and people shown another. GPT-4’s predictions correlated 0.85 with the real effects. A correlation of 1 would mean the predictions lined up perfectly with the results, and 0 would mean no relationship at all. The result held for studies that were unpublished before the model’s training cutoff, which argues against the model simply remembering them. The chart at the top of this piece shows how quickly this improved across model generations, from 0.24 for GPT-3 to 0.92 for GPT-4 after adjusting for sampling error.

Two details matter as much as the headline. The model was no better than a crowd of people: lay forecasters recruited online reached a correlation of 0.84, statistically indistinguishable from GPT-4, and averaging the human and model forecasts reached 0.89 (these are unadjusted figures; the chart shows adjusted ones). And a correlation only measures whether the model puts effects in roughly the right order. A separate measure, covered below, shows it gets their size wrong.

Earlier work points the same way. In a 2022 preprint, later published in Political Analysis, Lisa Argyle and colleagues at Brigham Young University found that simulated respondents’ vote choices correlated 0.90 to 0.94 with real American National Election Studies respondents. A University of Wisconsin team replicating a Fortune 500 food company’s study found the model “correctly picks the answer direction and valence”.

Where they break

Effect sizes come out about twice as large

An effect size is how much a response changes, not just whether it changes. It is the number behind a claim like “this message lifts purchase intent by four points”, and it is what a forecast or a business case gets built on.

The same Nature study measured how the predicted sizes compared with the real ones (a regression slope of 0.56) and found the predictions were “roughly twice as large as the actual effect sizes”. Put simply, where the model predicted a ten-point lift, the real lift was typically about half that. A synthetic test can tell you which message is stronger and still overstate by how much.

Accuracy falls in megastudies, most for field and video

The 70-experiment archive is the favorable case. The same paper also tested 15 megastudies (large experiments that test many treatments at once) with 606 effects, including field experiments, which measure what people do in the real world instead of what they say in a survey, and treatments such as short videos. Across all of them the correlation was 0.39, less than half the archive’s, and it was lowest for field experiments and non-text treatments.

Redrawn from Ashokkumar, Hewitt, Ghezae and Willer, Nature 656 (2026). Megastudy figures are raw correlations across 15 megastudies.

For anyone testing creative, the bottom row matters most. Ads are rarely text alone.

Synthetic people vary less than real people

Variation is how much answers differ from one person to the next, usually measured as a standard deviation. It is what separates one segment from another, and it is what sample sizes and margins of error are calculated from.

Synthetic respondents show too little of it. This is a consistent finding, from four different research groups:

Study	What they found
Columbia-led digital twin megastudy (preprint)	Twin standard deviation below the human one in 154 of 164 outcomes
Bisbee et al., Vanderbilt	ChatGPT gave “less variation in responses than in the real surveys”
Aher et al., Olin, Georgia Tech, Microsoft Research	Newer models gave the exact right answer to estimation questions, with zero spread
Horton et al., MIT and Fordham*	Simulated panels often agreed completely: “in about half of the panels, we get the same response in all 100 observations”

*The authors of the Horton paper disclose a financial interest in a company that sells language-model simulation.

The Aher result is the starkest. Asked to estimate quantities as a group of people would, the newer models stopped behaving like a crowd at all:

Question	True answer	Human median (spread)	Model median (spread)
Bones in an adult human	206	190 (108)	206 (0)
Melting point of aluminum, °C	660	240 (532)	660 (0)
Earth days in a Mars year	687	365 (376)	687 (0)
Speed of sound, m/s	343	333 (884)	343 (0)

From Aher, Arriaga and Kalai (ICML 2023), Appendix Table 3. Spread is the interquartile range, the width of the middle half of the answers. Human data: Moussaïd et al. (2013), n = 52. Model: text-davinci-003; ChatGPT and GPT-4 also showed zero spread.

Raising the model’s temperature, the setting that makes its output more random, did not fix this in the study that tested it. Angelina Wang of Stanford and colleagues found GPT-4 still fell short of human diversity, on the measures they used, even at settings high enough to make its output incoherent.

Low variance creates false confidence

Too little variation also makes the results look more certain than they are. A statistical test judges a result against how much the answers vary: the more alike the respondents, the fewer of them it takes for a difference to count as real. James Bisbee and colleagues at Vanderbilt compared 3.6 million ChatGPT responses with real survey respondents:

The averages looked fine: “every synthetic mean falls within one standard deviation” of the real one. The relationships between variables did not. A synthetic data set can reproduce the headline and get what drives it wrong.

Individuals are barely better than demographics

Everything above is about averages across a group. A harder test is whether a simulated person answers like the specific person it is modeled on.

Digital twins are personas built from a real person’s own answers, then asked new questions. A preprint megastudy from a mainly Columbia University team, with Columbia Business School’s Olivier Toubia as corresponding author, compared 1,784 people with their twins across 164 outcomes:

Redrawn from Peng, Gui, Brucks, Toubia et al., “Digital Twins are Funhouse Mirrors” (arXiv 2509.19088, preprint, 2026).

A full twin, built from hours of a person’s own responses, correlated 0.20 with that person, against 0.15 for a persona with demographics alone. On individual accuracy the twins scored 0.748, against 0.746 for demographics alone and 0.629 for random answers.

A widely quoted Stanford-led result looks far better: agents built from two-hour interviews matched people’s survey answers “85% as accurately as participants replicate their own answers two weeks later”. The difference is partly the metric. That study correlates across questions within one person; the twin megastudy correlates across people within one question, which its authors argue “is more often the measure of interest”. The Stanford figure is also normalized: raw accuracy was 68.85%, against people’s own 81.25% consistency, and the 2026 revision reports 0.83 for interview-only agents.

Results drift as models change

Drift means the same question gives a different answer later, because the model behind it has changed. Bisbee’s team ran the same prompt in April, June and July 2023. After a model update in late June, the July results showed “substantial mean reversion”. A synthetic study is a measurement of a model at a point in time, not of the population.

Bias

Bias is error that leans the same way every time, so a bigger sample does not average it out. MRS states it in its guidance: “Synthetic Data Is NOT Bias-Free”. The studies show three kinds of bias: in whose views a model reflects by default, in how it portrays particular groups, and in how it responds to the form of a question.

The model’s default views are not the public’s

A model asked a question with no persona still gives an answer. That default is the starting point every persona is layered on. Shibani Santurkar and colleagues at Stanford and Columbia measured how closely each model’s default opinions matched US public opinion, and those of 60 demographic groups, on 1,498 Pew survey questions.

Redrawn from Santurkar et al., “Whose Opinions Do Language Models Reflect?” (ICML 2023), Fig. 2; a selection of the nine models tested.

No model was as representative as even the least representative human group, and the versions of OpenAI’s models tuned to follow instructions scored worse than the base versions. People aged 65 and over and widowed people were among the most poorly represented. The authors report that the mismatch “persists even after explicitly steering” a model toward a particular group.

Groups are flattened and stereotyped

Flattening means a simulated group looks more alike than the real one. Stereotyping means it looks the way outsiders expect. Wang’s team compared four language models with 3,200 people across 16 demographic identities. The models’ portrayal of a group tended to be closer to what outsiders think of that group than to what its own members say, and it showed less diversity within the group. The fixes they tested “reduce, but do not remove, these harms”.

Other studies found related distortions:

Study	What they found
Bisbee et al., Vanderbilt	ChatGPT’s answers for partisans were off by 0.5 to 1 standard deviations (10 to 20 points on a 100-point scale), making groups look more hostile to each other than the real survey showed
Chen, Zhu and Zheng (2026 preprint)	Models “inflate between-segment gaps two to fourfold” and picked the wrong target segment in half of the US cases
Kantar (2023)	GPT-4 showed “a strong positive bias” and misestimated the low-income subgroup
Columbia-led digital twin megastudy (preprint)	Lists stereotyping, representation bias and ideological biases among its five systematic distortions

MRS adds a bias of timing: synthetic data are built from the past, so they struggle with emerging trends.

Question format can pass for opinion

Human surveys have these problems too: the order of the answers, the scale and the wish to look good (social desirability) all move results. With a model they can be large enough to be mistaken for a real opinion. Three separate studies found the way a question is asked changing the answer:

Quirk	What happened	Source
Answer order	Across 43 models, “All models suffer from substantial A-bias”: a preference for the option labeled A	Dominguez-Olmedo, Hardt and Mendler-Dünner, Max Planck Institute for Intelligent Systems
Rating scales	Asked directly for a 1-to-5 purchase-intent rating, models “typically replied with response ‘3’”	PyMC Labs and Colgate-Palmolive (industry preprint)
Social desirability	When GPT-4 could infer it was taking a personality test, its answers shifted 1.2 standard deviations toward the socially desirable	Salecha and colleagues

Two details from those studies are worth knowing. Once answer order was randomized, the Max Planck team found models “trend towards uniformly random survey responses”. And the social desirability shift was larger in bigger, more recent models; reversing the direction of some questions (reverse-coding) cut it roughly in half.

What this means for market research

The study most directly about market research is by James Brand and Donald Ngwe of Microsoft’s Office of the Chief Economist and Ayelet Israeli of Harvard Business School. They estimated willingness to pay, the extra amount a customer would pay for a product feature, with GPT and compared it with human studies:

Product attribute	Human willingness to pay	GPT willingness to pay
Fluoride toothpaste (pilot)	$3.27	$3.40
Fluoride toothpaste (Study 1A)	$2.60	$8.20
Fluoride toothpaste (Study 1B)	$2.90	$9.30

From Brand, Israeli and Ngwe, “Using LLMs for Market Research” (HBS Working Paper 23-062, 2026), §3.1. The pilot’s human figure is from Fong et al. (2023).

The pilot looked excellent. The main studies were about three times too high. GPT also produced estimates pointing the wrong way for new flavors, and a negative value for larger laptop screens, and projector estimates differed about threefold between GPT-3.5 and GPT-4o. The authors’ conclusion is the key line: “a researcher lacking a human benchmark would have had no basis for identifying which estimates were unreliable.”

Two findings point toward better practice. The first is that human data from the same category helps. Fine-tuning (further training the model) on prior human surveys in that category fixed the wrong-way estimates and brought the projector figure close to the human value, though it did not improve estimates of how customer segments differ. Fine-tuning on a different category, laptop data for tablets, made results worse.

The second is that the way you ask changes what you get. The PyMC Labs and Colgate preprint compared ways of asking for purchase intent across 57 product concepts and 9,300 human respondents:

How the rating was asked for	Ranking signal recovered	Distribution match
Direct 1-5 rating	about 80%	0.26 (GPT-4o)
Free text, mapped to a rating (GPT-4o)	90%	0.88
Free text, mapped, with demographics (Gemini)	92%	0.80
Free text, mapped, no demographics (Gemini)	50%	0.91

From Maier et al., arXiv 2510.08338 (industry preprint, not peer reviewed). “Ranking signal recovered” is correlation attainment relative to human test-retest; “distribution match” is Kolmogorov-Smirnov similarity.

The last row is the trap. Without demographics in the prompt, the synthetic responses matched the human distribution almost perfectly while recovering only half the ranking signal, against 92% with demographics. A synthetic data set can look right and still mislead the decision.

Reading vendor accuracy claims

Vendors publish accuracy figures, and some publish their methods. They are not comparable with each other, or with the academic results above:

Vendor	What was measured	Reported result	Scope
Qualtrics	Average gap from human means, in standard deviations (Cohen’s d)	0.07, vs 0.87 for GPT-5	518 matched pairs, 8 items
Ipsos	Correlation, 50 human + 150 synthetic vs 200 human	about 0.9	Product tests
Aaru	Average gap on aggregate answers	2.78 points (sample-adjusted; 3.53 raw)	2,993 questions, 186 studies
Kantar (2023)	GPT-4 vs about 5,000 humans	“just not good enough to use as a supplement for human sample”	Luxury and technology survey

All vendor-reported, from each company’s own published page. Apart from Kantar’s check of a low-income subgroup, none of the figures measures variance, subgroups or the accuracy of a decision.

Three things to ask any vendor:

1. The validation that compares their synthetic output with primary research or real-world results. This is ESOMAR’s own buyer question.
2. Whether that validation measures variation and subgroup differences, or only averages.
3. A replication of one of your own recent studies before you buy, the suggestion of Ray Poynter, founder of The Future Place, in MRS guidance.

Best practices

The professional bodies have converged on the same three principles: benchmark against humans, use hybrid designs, and disclose. The American Association for Public Opinion Research’s 2026 task force says synthetic data “are not research ‘participants’” and that “poll” and “survey” should not be used for them. Its model disclosure adds that “a margin of error cannot be produced from synthetic responses”.

Combining that guidance with the evidence above, and with a published checklist from Sarstedt and colleagues in Psychology & Marketing (practices 4 and 9):

#	Practice	Why
1	Use it to screen and rank, not to size or decide	Direction is about a 0.85 correlation; magnitude is about twice the real size
2	Validate against a human study in your own category first	Without a benchmark, estimates that point the wrong way are invisible
3	Measure more than the mean	Means survive; variance, subgroups and relationships break
4	Discard results with no variation	Sarstedt and colleagues advise disregarding samples that show no variation
5	Test prompt wording, answer order and format	Order, scale and framing effects can pass for opinion
6	Ground personas in real data from the target population	Same-category data helps; cross-category data can hurt
7	Use at least two models, and log versions and dates	Results shift across models and model updates
8	Never compute significance or power from synthetic n	Understated variance makes small samples look sufficient
9	Be most cautious with video, field and niche audiences	Accuracy drops to 0.24 for non-text stimuli; niche groups are thin in the training data
10	Keep humans in the loop and disclose	Human and model forecasts combined beat either alone; AAPOR’s guidance calls for disclosure

The takeaway

• Synthetic audiences are a screening tool. They rank options roughly as well as a crowd of lay forecasters, quickly and cheaply.
• They are poor at the numbers decisions usually rest on: effect sizes, variation between people, subgroup differences and individual responses.
• The errors are systematic, not random: too little variation, too-large effects, and bias in which groups are represented and how.
• Validation is not optional. A human benchmark in your own category is how you tell which synthetic results to trust.
• Used upstream, they speed up learning. Like Netflix’s interleaving, they can narrow a long list of ideas before the expensive human test, as long as the human test still happens.

Appendix: technical notes

These notes define the main accuracy measures quoted above, which are easy to confuse, explain the calculation behind the 33-respondent figure, and describe a minimal validation protocol for practice 2.

The accuracy measures are not the same thing

• Correlation of effects (Ashokkumar et al.). Whether predicted effects are ordered roughly like real ones across many experiments. It says nothing about size; that is what the regression slope of 0.56 captures. The adjusted figure corrects for sampling noise in the original experiments.
• Tetrachoric correlation (Argyle et al.). A correlation for two binary variables, here vote choice. It is forgiving for an outcome as predictable from party as a presidential vote; for pure independents in 2020 it fell to 0.02.
• Normalized accuracy (Park et al.). An agent’s accuracy divided by a person’s own consistency when re-surveyed two weeks later. A normalized 0.85 corresponded to a raw 68.85%.
• Correlation across people within a question (the digital twin megastudy). Whether the twin of a high scorer scores high. The study’s authors argue this is more often the measure of interest; it was about 0.2.
• Individual accuracy (the digital twin megastudy). That study’s score for how close a twin’s answers are to its person’s. Random answers already score 0.629 on it, so 0.748 is a weaker result than it sounds.
• Cohen’s d (the Qualtrics figure). The gap between two averages, measured in standard deviations.
• Representativeness (Santurkar et al.). One minus a normalized distance between the model’s answer distribution and a group’s, on a 0-to-1 scale.
• Distribution match (Kolmogorov-Smirnov similarity). How closely two answer distributions overlap, regardless of whether the right people gave the right answers. As the Maier result shows, it can be high while the ranking signal is weak.

Why low variance shrinks the sample you think you need

A standard result, restated here: the sample needed to detect an effect scales with the outcome’s variance divided by the square of the effect. In Bisbee’s example the real data had an effect of 7.8 points on a 0-to-100 feeling scale, with a standard deviation of 31.4, and the ChatGPT data an effect of 12.5 with a standard deviation of 16.1. A larger effect and half the spread together shrink the apparent required sample to 33, almost an order of magnitude below the real requirement.

A minimal validation protocol

The following is guidance derived from the studies above, not a protocol any of them prescribes.

1. Pick a recent human study in the same category, with respondent-level data.
2. Rerun its questions with synthetic respondents matched to the same profiles.
3. Compare four things: rank order of options or concepts; the ratio of synthetic to human standard deviation for each question; the gap between key subgroups; and the coefficients of a simple driver model.
4. Set acceptance thresholds before looking, and decide which decisions the synthetic tool will and will not be used for.
5. Repeat after any model change.