Synthetic User Bias: What 2025–2026 Research Shows

The short answer:

Synthetic users can provide useful signals for exploration, formative research, and narrowly defined tasks.
Their outputs often show less variation than human responses.
Persona prompts can introduce stereotypes rather than improve representativeness.
Small changes in persona or question wording can change results.
Synthetic users can be useful for generating hypotheses, but population estimates and behavioral predictions require stronger validation.
A small amount of real human data can help determine whether a synthetic population is reliable enough for a particular research question.

A September 2026 review in Nature Computational Science organizes LLMs used as human proxies into four roles: believable agents, task agents, experimental subjects, and “silicon samples.” Its central point is important for synthetic-user research: similarity to humans is not a single property. Different uses require different validity criteria. Large language models as human proxies

A June 2026 systematic review in IEEE Access maps LLM-based synthetic audiences across behavioral replication, experimental extension, market research, and large-scale simulation. It also highlights systematic bias, prompt sensitivity, anthropomorphic overgeneralization, and the need for human calibration. Synthetic Audiences as Proxies for Human Users

Why synthetic-user bias matters

Every research method has limitations. The particular challenge with synthetic users is that their outputs are fluent, coherent, and often persuasive even when they are systematically wrong.

A review of 63 peer-reviewed persona studies published between 2023 and 2025 found that most used only limited sociodemographic attributes, while just 35% discussed whether their LLM personas were representative of the population they claimed to model. The authors argue for more explicit grounding in empirical data and stronger transparency around how synthetic personas are constructed. Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency

This creates a basic methodological problem:

A synthetic user can sound like a person without being a valid representation of that person or population.

New to the topic? Start with our guide to synthetic users.

Eight biases that repeatedly appear

1. Synthetic personas can stereotype rather than represent

A large 2026 study generated 40,000 synthetic personas in four languages using two open-weight models. The researchers found systematic skews in age, gender, occupations, names, sentiment, and behavioral archetypes.

The generated characters tended toward a relatively narrow group of middle-aged, aspirational, and positively framed personas. Some occupations and non-binary identities were underrepresented.

The authors describe this as a form of “narrative sanitization”: the models can produce a more standardized and normative version of people than the underlying population. All too perfect: bias and aspiration in persona generation with LLMs

This matters because the persona itself becomes the input to later simulation. If the representation is distorted at the beginning, the downstream research can inherit that distortion.

2. Synthetic users can flatten variation

Real users disagree.

Synthetic users often agree much more.

A 2026 HHAI study compared 62 human crowdworkers with demographic-matched synthetic users in an HCI-Q experiment. The synthetic users showed severe variance collapse and tended toward a relatively homogeneous consensus, alongside normative and coherence biases. In the machine-machine condition, mean inter-participant correlation reached 0.814, compared with 0.189 in the human-human condition. The Synthetic Persona Fallacy in HCI-Q Methodology

A separate October 2026 study matched GPT-4o personas to 177 real participants and compared demographic-only personas with personas that also included Big Five personality scores. Adding personality information improved some measures of mean alignment, reduced under-dispersion, and modestly improved correspondence with individual differences. But the models still reproduced individual differences only modestly. Do participant-matched LLM personas approximate human survey data?

The practical implication is not that adding more attributes solves the problem. It is that grounding can improve fidelity without making synthetic users equivalent to real participants.

3. Demographics can become too predictive

Giving a model demographic information can sometimes make its output less realistic rather than more realistic.

A 2026 benchmark, When Synthetic Users Fail, compared several models with real responses from the General Social Survey and World Values Survey. The researchers found that models could substantially overestimate the relationship between demographic characteristics and attitudes.

In practical targeting scenarios, the models inflated gaps between segments by two to four times and selected the wrong segment in 50–72% of cases. The study also found cross-cultural scenarios in which models introduced differences that did not exist in the real population. Political leaning explained only 1.5% of variation in the real attitudes studied, but up to 67% of variation in model outputs.

This is particularly important for marketing research.

A persona such as “older enterprise buyer from Germany” may cause a model to generate what it has learned to associate with that description rather than what an actual German enterprise buyer in the target population would say or do.

4. Synthetic users can be too agreeable

Language models are designed to be helpful and cooperative. That behavior can become a liability when the model is supposed to simulate a skeptical, frustrated, or indifferent user.

Research on LLM interaction has documented sycophancy and acquiescence effects, including increased agreement when models are given user context. A CHI 2026 study found that user-memory context increased agreement sycophancy by 45% for Gemini 2.5 Pro in the tested setting, while some models also became more sycophantic with synthetic, non-user context. Interaction Context Often Increases Sycophancy in LLMs

A 2026 psychometric audit of LLMs as synthetic survey respondents found strong acquiescence bias across the models tested: +0.84 standard deviations, meaning a tendency to agree with statements regardless of their content. Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

This matters in usability research because real participants do not always cooperate with the researcher's assumptions. They misunderstand instructions, reject premises, become frustrated, abandon tasks, or simply do something unexpected.

A synthetic user that keeps trying politely can produce an overly optimistic picture of an experience.

5. Small wording changes can change the result

Synthetic users are sensitive to how their persona and task are described.

An ACL 2026 study compared six ways of conveying the same sociodemographic persona across seven models. Different persona cues produced substantial differences in the resulting responses. One Persona, Many Cues, Different Results

Another 2026 study ran more than 334,800 simulated survey interviews across 18 models and tested prompt perturbations. Almost all models showed sensitivity to changes in question wording, including recency effects. Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responses

This means a result that appears to describe a user population may partly describe the prompt used to construct that population.

For important research questions, running one persona formulation once is therefore a weak basis for a strong conclusion.

6. Persona drift can weaken long simulations

A persona assigned at the start of a session does not reliably hold over long conversations.

An EACL 2026 study tested seven models in dialogues of more than 100 turns and found that persona fidelity degraded over time, especially in goal-oriented conversations. As fidelity faded, persona responses became increasingly similar to the models' baseline behavior.

For multi-step simulations, this means a synthetic participant can gradually stop behaving like the participant you originally defined.

7. Plausible behavior is not necessarily accurate behavior

Synthetic users can produce journeys that look human without matching what humans actually do.

A 2026 ACL study used 31,865 real online shopping sessions containing 230,965 user actions to evaluate LLMs' ability to predict the next human action. Prompt-based models achieved only 11.86% accuracy. Fine-tuning a smaller model on real behavioral data improved this to 17.26%, but a substantial gap remained.

The result is an important warning for behavioral simulation:

A believable user journey is not the same thing as a validated prediction of human behavior.

This distinction becomes especially important when synthetic users are used to predict conversion, task completion, clicks, or other quantitative outcomes.

8. Accuracy varies across populations

Synthetic users are not equally reliable for every population.

A 2026 study, Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations, compared LLM-simulated users with participants from the United States, India, Kenya, and Nigeria in agentic retail tasks. Agent success rates varied by up to 9 percentage points depending on which user-simulation model was used.

The researchers also found systematic disparities: simulated users performed worst as proxies for African American Vernacular English (AAVE) and Indian English speakers. The paper reports that AAVE speakers experienced worse success rates and calibration errors than Standard American English speakers, with the disparity increasing with age.

The implication: a synthetic population should not be assumed to be equally representative across languages, dialects, cultures, or demographic groups.

Where synthetic users work

The evidence is not uniformly negative. Several recent studies suggest that synthetic users can reproduce useful signals when the research question is narrow, observable, and evaluated against a human benchmark.

What the evidence shows

Observable task behavior. A 2026 paired usability study compared 10 students testing an educational web application with matched AI “twins” created using Looma.ai. The study reported 75.3% correlation in attitudinal change patterns, 92% overlap in qualitative themes, and moderate to high similarity in navigation behaviors. However, the AI agents consistently underestimated cognitive load and emotional frustration. Because the study used a vendor's tool, it is best treated as one empirical case rather than general validation of AI-persona systems. Evaluating the Trustworthiness of AI Personas in Paired Usability Testing

Structured design screening. Research on AI-driven UI transition evaluation provides another example.

Hu et al.'s 2025 paper, Toward AI-driven UI transition intuitiveness inspection for smartphone apps, evaluated its models against a benchmark of 140 UI screen pairs, each annotated by 20 participants. The models were developed using 1,636 other training samples. In a controlled experiment with 12 UI/UX practitioners, the screening mode helped designers identify unintuitive transitions with 75% accuracy and reduced inspection time by 65.7% compared with manual inspection.

This is a specific prediction task rather than a claim that synthetic users generally reproduce human navigation. The distinction matters: the models were evaluated against human behavioral data for a defined task.

Open-ended exploration. A 2026 Design Society case study investigated LLM-driven synthetic participants in research around augmented reality for education. Synthetic participants generated plausible and comprehensive open-ended responses, although their answers to closed-ended questions diverged from those of human participants. The authors conclude that synthetic participants can augment early qualitative research but cannot replace human studies. Can LLM-driven synthetic participants help user research?

The common feature across these examples is important: the synthetic user is not being treated as a statistically interchangeable replacement for a human participant.

Practical use cases

The strongest practical use cases are situations where researchers need to explore a large hypothesis space quickly.

Exploring possible reactions

Synthetic users can help surface possible objections, interpretations, concerns, or areas of friction before committing to a larger research study.

The output should be treated as a set of hypotheses, not as evidence that a certain percentage of users will react that way.

Comparing design alternatives

When several design variants need an initial screen, synthetic users can provide a repeatable way to compare them under the same assumptions.

This can be useful for formative design work, especially when the alternative is not testing anything until a formal participant study is available.

Testing narrowly defined behavioral hypotheses

A question such as:

“Which of these two navigation elements is more likely to be selected to reach this destination?”

is easier to evaluate than:

“What would our customers think about this website?”

The narrower the behavioral question and the stronger the human benchmark, the easier it is to determine whether the simulation is useful.

Testing research materials

Synthetic participants can also be used before real research begins.

Researchers can test:

survey wording
screeners
interview questions
task instructions
response options
potential ambiguities

The goal is not to predict what real participants will answer. It is to find problems in the research instrument before it is used with people.

Exploring edge cases and scenarios

Synthetic populations can make it inexpensive to explore combinations of goals, experience levels, contexts, and interface variants that would be impractical to test individually with recruited participants.

Again, the value is in exploring the hypothesis space. Generating hundreds of scenarios does not make those scenarios representative simply because the number is large.

Generating hypotheses for human research

This may be the safest general use.

A synthetic study can produce:

“Users may interpret this control differently from what the design team expects.”

That is a useful research hypothesis.

It is not yet:

“Users interpret this control incorrectly.”

The second statement requires evidence from the population being studied.

The question matters more than the persona

One of the biggest mistakes in synthetic-user research is assuming that a more detailed persona automatically produces a more realistic simulation.

Recent research challenges that assumption.

A 2026 sim-to-real study used the Upworthy Research Archive, which contains thousands of real-world headline A/B tests, as held-out ground truth. The main validity analysis was restricted to 399 tests with statistically reliable winners. A no-persona LLM baseline predicted headline click-through better than a panel of demographic personas: Kendall τ was 0.361 for the no-persona baseline versus 0.084 for the persona panel; top-1 accuracy was 49.2% versus 34.6%.

The study is specifically about predicting aggregate headline click-through, not general user behavior. Its result is nevertheless notable: adding demographic personas did not improve predictive validity and actually made it worse in this setting. Do Synthetic Personas Predict Real Audience Response?

It reverses a common intuition:

More persona detail does not necessarily mean more predictive validity.

A useful synthetic-user question is therefore usually narrower than “pretend to be this person.”

For example:

Broad:

What would a 45-year-old SaaS buyer think about this pricing page?

More specific:

Which pricing option is this persona most likely to consider first, and what information would make the choice easier?

A question requiring human validation:

Which pricing design produces fewer decision errors among our target customers?

The first asks a model to construct a person.

The second asks it to simulate a constrained behavior.

The third makes a claim about real users and therefore requires real evidence.

This is the principle we build Synthers around: synthetic users run defined tasks on a real interface, rather than answering open-ended questions about what they might think. The distinction does not remove the need for human validation, but it makes the simulation more concrete and the claim easier to test.

When synthetic users should not be treated as evidence

The current research suggests increasing caution as a use moves from exploration toward measurement and prediction.

Use case	Role of synthetic users	Main limitation
Explore possible reactions	Useful	Results are hypotheses
Generate research questions	Useful	Requires human validation
Compare design variants	Useful for screening	Relative performance may not transfer to real users
Test research materials	Useful	Prompt sensitivity still applies
Explore edge cases	Useful	Scenarios are not automatically representative
Estimate population percentages	High risk	Synthetic bias can distort estimates
Size or prioritize market segments	High risk	Demographic conditioning can exaggerate differences
Predict exact behavior or conversion	High risk	Human behavioral fidelity remains limited
Represent underrepresented populations	High risk	Accuracy varies across populations; simulated users performed worst for AAVE and Indian English speakers in one 2026 benchmark
Validate a major product decision	Not sufficient alone	Requires evidence from real users

A useful rule emerges:

The further a use moves from exploring possibilities toward measuring or predicting real people, the stronger the evidence needed to justify the simulation.

Synthetic users can be calibrated

The choice does not always have to be synthetic users versus a large human study.

A 2026 methodological paper, When Can You Trust Your Synthetic Users?, draws on earlier studies to note that synthetic data typically compresses variance by about 2× and misses subgroup estimates by 10–30 percentage points. Its own doubly robust calibration, using 50–300 real respondents, reduced naive bias by 92.9–99.6% on ANES survey data and by 83–94% on a consumer-pricing dataset.

Another ACL 2026 study found that synthetic survey data alone introduced 24–86% bias, while combining it with statistical rectification reduced bias to below 5%. Valid Survey Simulations with Limited Human Data

This suggests a more useful methodological question than “Are synthetic users accurate?”

It is:

Are they accurate enough for this particular task, population, and claim?

That is something researchers can test.

A practical framework

A simple way to think about synthetic-user evidence is:

Synthetic exploration

→ identify possible reactions, problems, and hypotheses

Human calibration

→ compare selected findings against real participants

Real-user research

→ establish what actually happens in the target population

Combined evidence

→ understand where the simulation agrees with humans and where it systematically diverges

This does not make synthetic users equivalent to real participants.

It makes their limitations measurable.

The bigger picture

The current research does not support either extreme.

Synthetic users are not reliable replacements for human participants. But neither are they simply useless simulations.

Their value depends on the question.

They can be useful for exploring possibilities, screening alternatives, testing research materials, generating hypotheses, and investigating narrowly defined behavioral questions.

They become much harder to justify when the goal is to estimate how many people will behave a certain way, predict real-world behavior, represent a population the model knows poorly, or make a consequential product decision without human validation.

The most important distinction is therefore not synthetic users versus real users.

It is:

What claim are you trying to make, and what evidence is strong enough to support it?

Synthetic users can help explore what might happen.

Real users are still needed to establish what actually happens.

And the most informative research may come from measuring the gap between the two.

Disclosure: Synthers AI builds synthetic-user testing tools. The studies cited here are independent unless noted.

Frequently asked questions

Are synthetic users biased?

Yes. Recent research documents several recurring patterns, including stereotyped personas, reduced variation, over-weighting of demographic cues, agreeableness, prompt sensitivity, persona drift, gaps between plausible and observed human behavior, and uneven accuracy across populations.

Can synthetic users replace real participants?

Not generally. The evidence supports them for exploration, formative work, hypothesis generation, research-material testing, and some narrowly defined tasks. Strong claims about populations, behavior, or consequential product decisions still require human evidence or task-specific calibration.

What are synthetic users good for?

They are most useful when the goal is to explore possibilities quickly: surfacing potential reactions, comparing early design alternatives, testing research instruments, exploring edge cases, or generating hypotheses for subsequent human research.

How do you reduce synthetic user bias?

Ground personas in empirical data, vary persona and question wording, use goal-based tasks rather than prompts that invite agreement, check for persona drift, and calibrate important findings against real participants.

Does running more synthetic users make the results more reliable?

Not automatically. If the underlying bias is systematic, more simulations can reproduce the same error at greater scale. More samples help only when the simulation method itself has been shown to produce useful variation for the question being studied.

Synthetic User Bias: Where AI Personas Work and Where They Fail

In this article