The Looming Threat of AI Model Collapse: Are Our LLMs Already Getting Worse?

The landscape of information access is undergoing a seismic shift, largely driven by the rapid advancements in artificial intelligence, particularly large language models (LLMs). Many, including myself, have found AI-powered tools to be revolutionary for tasks like search, often providing more direct and synthesized answers than traditional search engines. However, a disquieting trend is beginning to emerge: signs that the quality, accuracy, and reliability of these powerful AI models may be starting to degrade. This phenomenon, known in AI research circles as 'model collapse,' poses a significant threat to the future utility of generative AI.

My own experience mirrors this concern. While AI search, particularly tools like Perplexity, initially felt like a significant upgrade over traditional methods that have arguably declined in quality, I've noticed a subtle but persistent deterioration in the results. When seeking concrete data, such as market share statistics or financial figures, AI responses are increasingly sourced from questionable aggregators rather than authoritative documents like SEC 10-K reports. Even when attempting to guide the AI towards reliable sources, the tendency to pull from less credible summaries persists, leading to data that is 'never quite right.'

This isn't an isolated issue with one specific AI. Testing across various major AI search platforms reveals a similar pattern of delivering 'questionable' results, particularly when precise, verifiable data is required. It feels like a return to the fundamental principle of computing: Garbage In, Garbage Out (GIGO). But in the context of AI, this isn't just about poor input from the user; it's increasingly about poor input data for the AI's training itself.

Understanding AI Model Collapse

AI model collapse is a technical term describing a critical failure mode in the training of generative models, particularly when these models are trained on data that includes a significant amount of synthetic content – content previously generated by other AI models. The core problem is that training on synthetic data, which is inherently a projection or interpretation of reality rather than raw, original observation, causes successive generations of models to drift away from the true data distribution. This leads to a loss of accuracy, diversity, and ultimately, reliability.

As a Nature paper published in 2024 starkly put it, in model collapse, "The model becomes poisoned with its own projection of reality." Imagine training a model to identify cats and dogs. If you initially train it on real images, it learns the true features of cats and dogs. But if you then use that model to generate synthetic images of cats and dogs, and subsequently train a *new* model on these synthetic images, the new model learns the characteristics of the *generated* images, not the real animals. Errors, biases, and simplifications present in the first generation's outputs become amplified in the second generation, and so on. Over time, the model's understanding of 'cat' or 'dog' could become severely distorted, perhaps converging on a simplified, generic image that resembles neither.

Model collapse is typically attributed to a combination of factors:

• Error Accumulation: Each generation of AI models inherits and often amplifies the inaccuracies, biases, and stylistic quirks of the models it was trained on. These flaws compound over time, causing the model's outputs to diverge increasingly from the original, real-world data patterns it was initially meant to learn from. Think of it like making photocopies of photocopies – each copy degrades the image slightly, and after many generations, the original is unrecognizable.
• Loss of Tail Data: Real-world data distributions often contain 'long tails' – rare events, unique examples, or subtle nuances that occur infrequently but are crucial for a complete understanding of the data space. When AI models generate synthetic data, they tend to produce examples that are typical or average, effectively smoothing out or ignoring these rare instances. Training on this smoothed-out data causes subsequent models to lose the ability to understand or generate these tail events, leading to a reduction in diversity and an inability to handle edge cases. Concepts that were once distinct might become blurred together.
• Feedback Loops and Reinforcement: When AI-generated content is fed back into the training data pool, it reinforces the patterns and biases present in the AI's outputs. This creates a positive feedback loop where the model becomes increasingly confident in its own generated style and content, even if it's drifting from reality. This can lead to repetitive outputs, amplified biases, and a narrowing of the model's effective knowledge base. As IBM explains, this self-consumption of data can cause outputs to "drift further away from original data patterns."

In essence, when AI is trained on its own outputs, as the AI company Aquant succinctly puts it, "the results can drift further away from reality." This is the core mechanism driving model collapse.

RAG: A Partial Solution with New Risks

One technique developed to mitigate some of the issues with LLMs, particularly hallucinations (confidently making up facts), is Retrieval-Augmented Generation (RAG). RAG systems enhance LLMs by allowing them to access, retrieve, and condition their responses on information from external, authoritative knowledge sources – such as databases, internal documents, or curated web content – instead of relying solely on their potentially outdated or incomplete pre-trained knowledge.

Intuitively, one would expect RAG to improve accuracy and reliability by grounding the AI's responses in verifiable external data. And indeed, RAG is effective at reducing simple hallucinations. However, recent research suggests that RAG introduces its own set of risks, which can paradoxically contribute to the broader problem of unreliable AI outputs.

A recent Bloomberg Research study investigated the performance of 11 leading LLMs (including models like GPT-4o, Claude-3.5-Sonnet, and Llama-3-8B) when augmented with RAG. Using over 5,000 prompts designed to elicit harmful or problematic responses, the study found that while RAG reduced hallucinations, it increased the likelihood of other types of harmful outputs. Specifically, RAG-enabled LLMs showed an increased tendency to produce bad results related to leaking private client data, generating misleading market analyses, and providing biased investment advice.

As Amanda Stent, Bloomberg's head of AI strategy & research, noted regarding this counterintuitive finding, it has "far-reaching implications given how ubiquitously RAG is used in gen AI applications such as customer support agents and question-answering systems." She stressed the need for AI practitioners to be thoughtful about responsible RAG usage, especially considering that the average internet user interacts with RAG-based systems daily.

The Bloomberg study highlights a critical point: RAG's effectiveness is dependent on the quality and nature of the external data it accesses. If that external data pool is increasingly polluted by low-quality or biased AI-generated content, RAG may simply become more efficient at retrieving and presenting garbage, albeit garbage that is less likely to be a pure hallucination and more likely to be a plausible-sounding distortion of reality.

The Proliferation of Synthetic Content and Its Consequences

The primary driver accelerating the potential for model collapse is the sheer volume of synthetic content being produced and published online. AI is no longer just a tool for analysis; it's a prolific content generator. From marketing copy and news articles to academic papers and creative writing, AI is flooding the digital ecosystem with text, images, and code.

While proponents argue that AI frees up humans for higher-level tasks, the reality is that AI is often used to generate content faster and cheaper, frequently prioritizing quantity over quality or authenticity. This isn't just happening at a corporate level; individuals are using AI for everything from drafting emails to writing school reports. The motivation is often efficiency or cost-saving, not necessarily a commitment to producing original, high-quality work.

This widespread adoption of AI for content generation has led to numerous instances of fabricated information entering the public domain:

• Fake Academic Papers: AI has been used to generate plausible-sounding but entirely fabricated scientific research papers, complete with fake data and citations. These can potentially slip through review processes and enter academic databases, polluting the scientific record. Research published in the Harvard Kennedy School Misinformation Review has explored how GPT-fabricated papers appear on platforms like Google Scholar, highlighting the challenge of identifying and preempting such evidence manipulation.
• Fictional Works Presented as Real: A notable example involved a major newspaper's summer reading list that included forthcoming novels that simply do not exist. When queried about one of these fictional books, even ChatGPT, the source of much synthetic text, could only state that no public information about the plot was available – a truthful answer, but one that underscores the AI's inability to distinguish between real and fabricated information it might encounter elsewhere.
• Generic or Inaccurate Information: Beyond outright fabrication, AI frequently generates content that is bland, repetitive, or subtly inaccurate because it averages across its training data, losing specific details and nuances. This is particularly problematic for factual queries, as observed with the financial data example.

The core issue is that much of this AI-generated content is finding its way onto the internet, into databases, and onto platforms that serve as the training data for the *next* generation of AI models. The more AI-generated content enters the training pool, the more diluted and distorted the representation of real-world data becomes. This creates a vicious cycle: AI trains on AI-generated data, producing more AI-generated data, which is then used to train even more AI, leading to a progressive decline in quality and connection to reality.

Consider the scale of this production. Sam Altman, CEO of OpenAI, tweeted in February 2024 that OpenAI alone was generating approximately 100 billion words per day. While not all of this output is published online or used for training, a significant portion undoubtedly contributes to the digital information ecosystem. This immense volume of synthetic text, combined with AI-generated images, code, and other data types, is rapidly altering the composition of the internet – the primary training ground for many foundational models.

The Feedback Loop of Degradation

The process of model collapse driven by synthetic data can be visualized as a destructive feedback loop:

1. Initial Training: An LLM is trained on a large dataset, primarily composed of human-generated text and data from the internet.
2. Synthetic Generation: The trained LLM is used to generate new content (text, code, images, etc.). This content contains inherent biases, simplifications, and errors from the model's training and generation process.
3. Data Pollution: The AI-generated content is published online, added to databases, or otherwise enters the digital information ecosystem.
4. Retraining/Fine-tuning: Future versions of LLMs, or entirely new models, are trained or fine-tuned on datasets scraped from the internet and other sources, which now include a growing proportion of the previously generated synthetic content.
5. Degradation: The new models learn from the synthetic content, inheriting and amplifying its flaws (error accumulation, loss of tail data, reinforced biases). Their understanding of the world becomes based on the previous model's interpretation rather than raw reality.
6. Increased Pollution: These degraded models generate even lower-quality, less diverse, and potentially more biased synthetic content.
7. Cycle Repeats: This new, lower-quality synthetic content further pollutes the data pool, accelerating the degradation in subsequent training cycles.

This loop suggests a future where, without intervention, the digital commons become so saturated with derivative, synthetic content that training data for new models becomes increasingly impoverished, leading to models that are less capable, less creative, and less connected to factual reality.

Impact on Search and Information Trust

The most immediate and noticeable impact for many users is the degradation of AI-powered search and question-answering systems. If these systems are increasingly relying on or trained on data polluted by synthetic content, their ability to provide accurate, nuanced, or verifiable information diminishes. The struggle to obtain reliable financial data from AI search is a potential early warning sign.

Beyond search, the proliferation of synthetic content undermines trust in digital information generally. When it becomes difficult to distinguish between human-generated, fact-based content and AI-generated, potentially fabricated or biased content, the value of online information decreases. This has profound implications for journalism, education, research, and public discourse.

The incentive structure exacerbates this problem. For businesses and individuals prioritizing speed and cost, generating content with AI is appealing, even if the quality is lower. The economic pressure to produce more content faster can easily outweigh the concern for data integrity, especially when the negative consequences of data pollution are diffuse and long-term.

Can Model Collapse Be Mitigated?

Some researchers are exploring potential ways to mitigate model collapse. One proposed strategy involves mixing synthetic data with fresh human-generated content during training. The idea is that including a sufficient proportion of original, real-world data can help anchor the model's understanding and counteract the drift caused by synthetic data.

However, the feasibility and scalability of this approach face significant challenges. As the volume of AI-generated content explodes, the relative proportion of truly 'fresh human-generated content' available for training may shrink. Where will this clean, diverse, and comprehensive human data come from? Creating high-quality, curated datasets is expensive and time-consuming. If the economic incentive is to replace human effort with AI, the supply of new, original human-generated data entering the public domain or available for training could dwindle.

Moreover, distinguishing between human-generated and AI-generated content is becoming increasingly difficult. As AI models become more sophisticated, their outputs can closely mimic human writing styles, making it challenging to filter training data effectively. Watermarking or other methods to identify synthetic content are being explored, but their effectiveness and widespread adoption remain uncertain.

Another potential mitigation involves developing new training techniques that are less susceptible to data pollution or that can better identify and prioritize high-quality data sources. However, these are active areas of research, and it's unclear if they can fully counteract the fundamental problem of training on degraded data.

The Path Forward: A Question of Value

The current trajectory suggests a future where the digital information environment becomes increasingly saturated with AI-generated content, leading to a potential decline in the quality and reliability of the AI models themselves due to model collapse. The early signs – degraded search results, fabricated information – are perhaps just the beginning.

The question is not if we will continue to invest in AI; the economic and perceived efficiency benefits are too strong. The question is how long it will take for the degradation in AI quality, driven by model collapse and data pollution, to become so undeniable that it forces a re-evaluation of our approach. Will we continue down the path of prioritizing volume and cost-saving through synthetic content generation until the AI becomes demonstrably 'worthless' for tasks requiring accuracy and reliability?

The warning signs are present. The mechanisms of model collapse are understood. The proliferation of synthetic data is accelerating. While some researchers are sounding the alarm and exploring technical solutions, the broader economic and societal forces pushing for the mass production of AI-generated content remain powerful. The future quality of general-purpose AI, and the digital information ecosystem it interacts with, may depend on whether we can find a way to break the feedback loop before the 'poisoned projection of reality' becomes the dominant view.

It's possible that I, and others observing these early signs, are simply witnessing temporary glitches or growing pains in a rapidly evolving technology. However, the theoretical underpinnings of model collapse, combined with the practical observations of degraded outputs and the increasing volume of synthetic data, paint a concerning picture. The prediction that general-purpose AI could start getting worse isn't just theoretical; there's growing evidence that it's already happening.