You know what happens when you photocopy a photocopy of a photocopy. Each generation loses fidelity — not randomly, but systematically. Fine details blur first, then outlines, then structure. By generation 10 you have a grey smear.
That is exactly what AI training on AI-generated output does. The mechanism is called model collapse. If you have been using AI tools for a year or two and something feels off — responses more generic, more hedge-y, less precise about edge cases — you are experiencing what Charlie Guo at Ignorance AI calls “intelligence drift.” It is a structural cause, not a perceptual one. This article is part of our series on what the AI slop epidemic means broadly — covering how machine-generated content is degrading the systems that produce it.
In 2024, Shumailov et al. documented model collapse empirically in Nature (vol. 631, pp. 755–759). What almost no practitioner-facing writing has addressed since is what those findings mean for companies fine-tuning foundation models on their own internal data. That is what this article is about.
What is model collapse — and why does the 2024 Nature paper matter?
Model collapse is a degenerative process where a model trained on AI-generated outputs progressively loses the ability to represent rare or edge-case knowledge. Output distribution narrows. Outputs become homogenised and low-variance. It is not a software bug — it is a structural training phenomenon.
Shumailov et al.’s key finding: “indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear.” The tail is where rare, specialised, and minority knowledge lives. When the tails disappear, the model loses the ability to reason about uncommon cases — exactly the cases where expert knowledge matters most.
Two stages. Early: the model begins losing information about the tails. Late: the model converges to a distribution that carries little resemblance to the original — the “irreversible defects.”
Shumailov et al. characterise it as a structural property of the training process: “this process is inevitable, even for cases with almost ideal conditions for long-term learning.” Given sufficient recursion, collapse is not a risk. It is an outcome.
Why does this matter right now? The internet is no longer predominantly human-authored. AI slop floods the web-crawl corpora used to train the next generation of foundation models. According to Ignorance AI’s analysis, over 74% of newly created webpages contained AI-generated text as of April 2025. The recursive loop is already running at internet scale.
How does the entropy spiral work, step by step?
The recursive training mechanism — which Shumailov et al. term the “self-consuming loop” — works like this: Model N generates outputs. Those outputs enter a training corpus. Model N+1 is trained on that corpus. Model N+1 generates outputs. Those outputs enter the next corpus. Model N+2 is trained on that. Each generation’s output becomes the next generation’s training input.
The entropy framing is not a metaphor. The arxiv analysis (2509.16499, 2025) measured a strong linear correlation between model generalisability and training dataset entropy. As recursive iterations progress, entropy sharply decreases. Generalisability declines with it.
Three sources of error compound across generations:
- Statistical approximation error: any finite training set truncates the full distribution. Tail knowledge gets cut off each generation.
- Functional expressivity error: neural networks have structural limits on what distributions they can represent. Each generation’s learned distribution is a compressed approximation of what it was trained on.
- Functional approximation error: stochastic gradient descent amplifies the most common patterns at the expense of rare ones.
These errors compound. The Gaussian model collapse theorem establishes that the nth generation approximation “collapses to be zero variance as the number of generations increases, with probability 1.”
The observable surface symptom before collapse becomes obvious is semantic drift: outputs converge on common phrasing, vocabulary diversity decreases, answers to unusual queries become vague or confidently wrong.
Semantic drift is distinct from catastrophic forgetting. Catastrophic forgetting happens during fine-tuning when the task is too narrow. Model collapse happens from recursive data contamination — the data is degraded, not the training objective. Different causes, different remediation.
For a richer treatment of hallucination mechanisms that manifest at the output level as model collapse progresses, see vibe citing as a professional-domain symptom of hallucination at scale.
What are data distribution drift and semantic drift — and how do you measure them?
Data distribution drift is the statistical shift in training data characteristics across successive iterations — the precursor condition for model collapse. Semantic drift is the observable symptom in model outputs — the signal that damage has already occurred.
By the time user-reported quality degradation surfaces — developers complaining that the model “feels dumber” — the degradation has typically been underway for weeks or months.
Vocabulary diversity / type-token ratio: measure the ratio of unique tokens to total tokens in model outputs, run against held-out human-authored test queries. A declining ratio over successive training runs signals semantic drift. Direction of travel matters more than absolute value.
Perplexity on held-out human-authored data: a model whose learned distribution has shifted away from human language patterns will show rising perplexity on normal human writing. The Nature paper’s OPT-125m experiments documented exactly this across recursive generations.
Repetitive phrasing audit: manually review a sample of 50–100 production outputs for formulaic sentence openings, stock hedge phrases, or repeated structural patterns. Low-cost, and easy to spot once you know what to look for.
Establish a baseline at initial fine-tuning. Re-evaluate after each subsequent training run. Quarterly monitoring is a reasonable minimum.
Early-stage semantic drift can potentially be arrested by cleaning the corpus and retraining from an earlier checkpoint. Late-stage collapse is not recoverable without clean human-authored training data at scale. Prevention is more tractable than reversal.
Full remediation — data quality gates, provenance tracking tooling, entropy-based data selection — is treated in depth in the companion piece on how to defend against model collapse in fine-tuning pipelines.
Are you already inside the recursive training loop? The SMB fine-tuning blind spot
Most content on model collapse focuses on AI labs training frontier models from scratch. The SMB fine-tuning scenario is a structural analogue that almost nobody discusses explicitly.
The starting condition
Foundation models — GPT-4o, Claude, Gemini — are pre-trained on web-crawled corpora that already contain a growing fraction of AI-generated content. The recursive loop is already running at internet scale before your fine-tuning begins. You are not starting from clean human-authored data.
Frontier AI labs know this. Google has licensing deals with Reddit. OpenAI has deals with News Corp. These are active efforts to preserve access to verified human writing because the free supply is running low — “peak data” is a live concern at every major AI lab.
The internal corpus problem
Think about what “our data” actually contains for a team that has been using AI tools for the last 18 months.
Support tickets drafted with ChatGPT or Copilot. Internal documentation and wikis written with AI writing assistance. Code comments and README files generated by coding assistants. Marketing and HR materials produced with AI tools. None of it is tagged as AI-generated at the time of authorship. To any data ingestion pipeline, it looks like internal human knowledge.
When your team fine-tunes a foundation model on this corpus, it is entering the recursive training loop — not at generation 1, but potentially at generation 2 or 3. Invisible, because nothing is labelled “AI-generated.”
Closing the loop
If the fine-tuned model is then deployed to help write more support tickets, update more internal documentation, or assist with code comments — and that content subsequently enters the fine-tuning corpus in the next training cycle — the loop is now closed internally. The organisation has created a private entropy spiral.
Moveo AI‘s research establishes a practical threshold: even 10–25% of incorrect or AI-generated data in a fine-tuning set causes measurable performance degradation, and at that level the base model outperforms the fine-tuned variant. When fine-tuning data volume increases without curation, performance does not just plateau — it often regresses: “the model becomes less coherent, and the subtle understanding it initially displayed vanishes.”
More training on unaudited data is not safer. It is faster degradation.
If your team adopted AI writing and coding tools 12–18 months ago without tagging authorship, assume a portion of your internal corpus is effectively synthetic data. The question is: “how far along are we, and do we have the baseline measurements to know?”
How do you detect early signs of model degradation in fine-tuned models?
Baseline measurement at the first fine-tuning run is the foundation — measurements at key intervals, compared against a baseline. Not onerous.
1. Baseline capture at initial fine-tuning
At the point of completing the first fine-tuning run, record: type-token ratio on a held-out set of 200–500 human-authored test queries; perplexity on the same set; a sample of 50–100 production outputs, saved verbatim. This is the reference point.
2. Dataset audit before each fine-tuning run
Before adding new data to the fine-tuning corpus, review it for AI-generated content. AI content detection tools (imperfect but directionally useful), internal tagging policies for new content, and similarity searches against your own recent model outputs can reduce contamination risk. The goal: understand what fraction you are adding and whether it is approaching the 10–25% threshold.
3. Output sampling programme
Monthly is reasonable, quarterly is a minimum — sample 50–100 production outputs and assess for vocabulary diversity and formulaic phrasing. You are looking for direction of travel, not an absolute score. Is the type-token ratio declining? Are the same sentence openings appearing more frequently?
4. Regression testing
Maintain a set of domain-specific evaluation prompts with human-authored reference answers. After each model update, compare outputs using a semantic similarity score. A declining score — even a gradual one — is a signal. Keep the same evaluation set across all cycles.
5. Catastrophic forgetting check
After each fine-tuning run, evaluate general-capability benchmarks alongside domain-specific tasks. If general performance degrades while domain performance holds, catastrophic forgetting is the likely cause, not model collapse. They require different responses. Conflating the two leads to applying the wrong fix.
The “worse than baseline” signal
If your fine-tuned model consistently underperforms the unmodified base model on domain-specific tasks, contamination is the likely cause. Fine-tuning should improve domain performance. If the base model is beating your fine-tuned version, the corpus is degrading rather than improving capabilities. If vocabulary diversity has declined more than approximately 20% from baseline, or perplexity on human holdout has increased more than approximately 15–20%, treat corpus contamination as urgent before the next training run.
What comes next: the defence stack for model collapse
Model collapse is a documented failure mode, not a hypothetical risk. Shumailov et al.’s Nature 2024 paper established it as an inevitable outcome of sufficient recursive training. Entropy decreases with each recursive generation, compressing output diversity in a measurable, linear relationship.
SMB organisations fine-tuning foundation models on internal corpora are entering the recursive training loop without necessarily realising it. The 10–25% contamination threshold for measurable degradation is reachable without deliberate action. Early detection is possible, but only if baselines are established before complaints surface.
For organisations that have identified contamination risk or are building fine-tuning pipelines, the defence stack — data quality gates, provenance tracking, human-in-the-loop validation, entropy-based data selection — is treated in depth in how to defend against model collapse in fine-tuning pipelines.
The conditions driving this problem are not improving. As Charlie Guo frames it: “The path of least resistance (and lowest costs) leads towards AI models regurgitating AI content, over and over again.” The entropy spiral is the default trajectory for any system that does not actively counter it. The question is whether the measurements are in place to catch it before it compounds.
For the broader consequences of AI-generated content flooding the internet, see the wider consequences of AI-generated content floods — covering all dimensions of the AI slop epidemic in one place.
Frequently asked questions
What is model collapse in AI?
Model collapse is a degenerative process where a model trained on AI-generated outputs progressively loses the ability to represent rare or edge-case knowledge. Output distribution narrows until outputs become homogenised and low-variance. Shumailov et al. documented this in Nature (vol. 631, pp. 755–759, 2024): “indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear.” Structural training phenomenon, not a software bug.
Is model collapse the same as catastrophic forgetting?
No. Catastrophic forgetting occurs during fine-tuning when the task is too narrow — the model overwrites prior general capabilities with the new task distribution. Model collapse occurs from recursive contamination of training data — the data is degraded, not the training objective. If general performance degrades while domain performance holds after fine-tuning, catastrophic forgetting is the likely cause. If vocabulary diversity and perplexity on human holdout are declining over successive training cycles, model collapse is the more likely cause.
Can model collapse be reversed once it starts?
Early-stage semantic drift can potentially be arrested by cleaning the corpus and retraining from an earlier checkpoint. Late-stage collapse — where rare knowledge has been systematically extinguished from the model’s weights — is not recoverable without access to clean human-authored training data at scale. The Nature paper uses language of “irreversible defects” once collapse compounds. Prevention is more tractable than reversal.
Is it safe to fine-tune an LLM on data my team wrote with AI assistance?
Not without auditing the corpus first. AI-assisted content is indistinguishable from human-authored content at ingestion time. The Moveo research establishes that even 10–25% AI-generated or incorrect data in a fine-tuning set causes measurable degradation — at that level the base model can outperform the fine-tuned variant. If your team has been using AI writing tools for 12–18 months without tagging authorship, assume a portion of your corpus is effectively synthetic data before you fine-tune on it.
Does model collapse affect large frontier models like GPT-4o or Claude?
Yes, in principle — frontier models are trained on web-crawled corpora that already contain AI-generated content. In practice, frontier AI labs have data quality teams, filtering pipelines, and curated licensed corpora that reduce contamination. SMBs fine-tuning on internal corpora lack all of these safeguards. The risk profile for enterprise fine-tuned models is higher than for the foundation models they fine-tune on.
What is the “self-consuming loop” in AI training?
The formal term from the Shumailov et al. literature: each generation of model output, when used as training data for the next generation, creates a feedback loop. For enterprise fine-tuning, the loop can be closed within a single organisation if the fine-tuned model assists with content creation that subsequently enters the fine-tuning corpus. The conditions for this are already met at most teams using AI writing tools for a year or more.
What does “peak data” mean and why does it matter for AI training?
Peak data: usable high-quality human-authored web text is approaching exhaustion as a training resource. Frontier AI companies are actively securing licensed human-authored content because the free clean supply is running low. The base models you fine-tune on are being trained on an increasingly synthetic web, meaning the starting point for fine-tuning is already further down the entropy spiral than it was two years ago.
How do I know if my internal documents count as “synthetic data”?
Any content drafted or materially edited by an AI writing tool — ChatGPT, Claude, Copilot, Gemini — functions as synthetic data in a fine-tuning context, even if a human reviewed it. Content types to audit: customer-facing communications, support ticket responses, internal documentation, wiki pages, code comments, README files, HR and policy documents. If your team has been using AI writing tools for 12–18 months without tagging authorship, assume a portion of your corpus is effectively synthetic data.
What is RAG and is it safer than fine-tuning for avoiding model collapse?
Retrieval-Augmented Generation (RAG) injects relevant context from a retrieval index into the model’s prompt at inference time. RAG does not modify model weights, so the base model is not exposed to the recursive training dynamic — model collapse is not a risk of the RAG approach itself. Trade-off: RAG is more collapse-safe but requires up-to-date retrieval indices. Fine-tuning produces more deeply integrated domain knowledge but carries collapse risk when corpus quality is uncontrolled.
Does model collapse affect coding assistants?
Yes. Coding assistants trained or fine-tuned on repositories containing AI-generated code face the same recursive contamination risk. GitHub repositories increasingly contain AI-generated code comments, boilerplate, and documentation. Observable signals: progressively more generic, less idiomatic code suggestions; increasing rates of hallucinated function signatures or API calls that exist in pattern but not in reality. At advanced stages, outputs drift toward plausible-looking but incorrect code — the AI hallucination dynamic, surfacing through the model collapse mechanism.
