THE EFFECT
Quality can drop each round
A 2024 Nature study found that indiscriminate training on model-made content causes irreversible defects, which the authors called model collapse.
See the numbersWHAT THE RESEARCH FOUND
A copy of a copy
loses the detail.
AI models learn from text and images on the web, and more of the web is now written by AI.
Check a training mix
KEEP IT REALfresh human data
- Rare details vanish first
- Errors build up
- Real data keeps it steady
- Why the training mix matters
Researchers have tested what happens when each new model learns mainly from the last one's output. The rare and unusual drop out first; keeping real human data in the mix is what holds quality up.
Updated
THE SHORT ANSWER
Learning from echoes
drifts from reality.
Three findings from the model collapse studies,
in plain English.
WHAT GOES FIRST
The rare things vanish
The first sign is that the tails of the distribution disappear: unusual, low-probability examples stop being produced. Later, output narrows towards a few common patterns.
Why tails matter
THE FIX
Keep real data in
A Stanford-led study found that adding synthetic data alongside the original real data, instead of replacing it, avoided collapse in its tests.
Compare training mixesA QUICK MIX CHECK
What is the next model
learning from?
Pick a training set-up to see what the studies found. The bar shows how strongly the research links that set-up to collapse.
Summarised from peer-reviewed and preprint studies (2023-2024). Results come from controlled experiments, not from any particular product.
TWO NUMBERS WORTH KNOWING
Human text is finite,
and it matters.
One figure on how long the supply of public human text may last,
one on how far a looped model drifted.
2026–2032
When training sets may match all public human text
Researchers at Epoch estimated that if trends continue, models will be trained on datasets roughly equal to the whole stock of public human text between 2026 and 2032. They put that effective stock at around 4×10¹⁴ tokens, about 400 trillion.
HUMAN TEXT · EPOCH STUDY
9 generations
Looped generation shown drifting off topic
Given a passage about medieval church towers, the ninth generation of a looped model wrote about black-tailed, white-tailed, blue-tailed and red-tailed jackrabbits.
DRIFT · NATURE 2024
The Nature authors say access to the original, human-made data source must be preserved, and that it is unclear how AI-generated content can be tracked at scale on the web.
FOUR THINGS TO TAKE AWAY
Collapse is not fate,
it is a data choice.
The studies point to the same habits for anyone building or judging AI systems: know where the training data came from, and keep real data in the loop.
Ask where the data came from
The Nature authors raise the question of provenance: whether content crawled from the web was written by people or by models.
Keep the original data
Every study points the same way: retaining real, human-made data protects the next model.
Add, do not replace
Accumulating synthetic data alongside real data avoided collapse in the Stanford-led tests; swapping it in did not.
Watch the rare cases
Collapse shows first in unusual examples, so check how a model handles the uncommon, not just the typical.
WHY THE TAILS MATTER
The unusual is
the first to go.
Each model sees only a finite sample of what came before. A rare word, fact or picture may simply not be sampled, and once it is missing the next model never learns it. The Nature paper shows that, step by step, output converges on a narrow set of common patterns.
The authors note that low-probability events are often relevant to marginalised groups and are vital to understanding complex systems, which is why losing them matters beyond quality scores.
Questions
Questions people ask
What is model collapse?
The Nature study defines it as a degenerative process in which data generated by models end up polluting the training set of the next generation, so later models mis-perceive reality.
Does any AI-made training data cause collapse?
Not necessarily. The Nature paper warns about indiscriminate use. A Stanford-led study found that adding synthetic data alongside the original real data, rather than replacing it, avoided collapse in its experiments.
Does it only affect chatbots?
No. The Nature study showed it in language models, variational autoencoders and Gaussian mixture models, and a Rice University study saw quality or diversity fall in image models trained in a self-consuming loop.
Is human-written text running out?
An Epoch study estimated that, if current trends continue, training datasets will be roughly the size of the stock of public human text between 2026 and 2032, and discussed synthetic data as one way forward.
Why not just filter out AI-written text?
The Nature authors say it is unclear how AI-generated content can be tracked at scale, and suggest community-wide coordination on sharing information about where content came from.
Where the facts come from
- Shumailov et al. (2024), AI models collapse when trained on recursively generated data, Nature (open access, PMC)
- Shumailov et al. (2023), The Curse of Recursion: Training on Generated Data Makes Models Forget, arXiv
- Alemohammad et al. (2023), Self-Consuming Generative Models Go MAD, arXiv
- Gerstgrasser et al. (2024), Is Model Collapse Inevitable? Accumulating Real and Synthetic Data, arXiv
- Villalobos et al. (2024), Will we run out of data? Limits of LLM scaling based on human-generated data, arXiv