This is the second part of “The Myth of Digital Collapse”. There I discussed the story; here I discuss the mathematics.

The model collapse argument has a flawless formal part and an assumption almost nobody looks at. The formal part proves that a model trained only on what the previous one generated loses, generation after generation, the variety of the original data. The assumption is that human data is thrown away after the first step. My thesis is that if that assumption is replaced by a more realistic one (data accumulates instead of being replaced), collapse disappears and what remains is a bounded drift. This is not a proof about real language models, but about a toy model that is enough to show where the hole is.

A Photocopy of a Photocopy

The image going around is that of making a photocopy of a photocopy: each copy loses a little, and after enough copies nothing legible remains. The academic version is the paper by Shumailov and colleagues in Nature (2024), which showed that indiscriminate use of model-generated content makes the tails of the original distribution vanish: the rare, the technical, the metaphorical. The phenomenon is real. The relevant question is under what conditions it happens.

The Formulation

Let p0p_0 be the distribution of human data. At each generation t≥1t \ge 1 a model is trained on nn samples, and that model is used to produce the samples for the next one. To be able to do the math, the simplest possible case will do: Gaussian data and a model that only estimates mean and variance.

X0(1),…,X0(n)∼N(μ,σ2),Xt(1),…,Xt(n)∼N(μ^t−1,σ^t−12)X_0^{(1)},\dots,X_0^{(n)} \sim \mathcal{N}(\mu,\sigma^2), \qquad X_t^{(1)},\dots,X_t^{(n)} \sim \mathcal{N}(\hat\mu_{t-1},\hat\sigma^2_{t-1})

What matters is which data model tt sees. There are two regimes. In the replacement regime, model tt is trained only on the nn samples generated by model t−1t-1. In the accumulation regime, it is trained on everything there is: the original human samples plus all the synthetic ones produced so far. The formal development of collapse, including the formulas circulating in its popular version, works in the first regime.

Why Replacement Collapses

I start with the mean. In the replacement regime, each estimate is the previous one plus fresh noise, and the noise adds up with nothing to correct it. The mean performs a random walk and its error grows without bound:

μ^t=μ^t−1+εt,εt∼N ⁣(0,σ2n)⟹Var⁡(μ^t−μ)=(t+1) σ2n→t→∞∞\hat\mu_t = \hat\mu_{t-1} + \varepsilon_t, \quad \varepsilon_t \sim \mathcal{N}\!\left(0,\tfrac{\sigma^2}{n}\right) \quad\Longrightarrow\quad \operatorname{Var}(\hat\mu_t-\mu) = \frac{(t+1)\,\sigma^2}{n} \xrightarrow[t\to\infty]{} \infty

(To isolate the effect I assumed known variance; the result does not change in any essential way.)

Something subtler happens with the variance. If the mean is known, each generation’s variance is the previous one multiplied by a random factor WtW_t with mean 1:

σ^t2=σ^t−12 Wt,Wt∼χn2n,E[Wt]=1\hat\sigma^2_t = \hat\sigma^2_{t-1}\,W_t, \qquad W_t \sim \frac{\chi^2_n}{n}, \qquad \mathbb{E}[W_t]=1

Since each factor has expectation 1, the expected variance never changes. And yet it collapses. The reason is that the logarithm of WtW_t has negative expectation (by Jensen’s inequality), so the logarithm of the variance is a sum of terms that, on average, subtract:

E[ln⁡Wt]=ψ ⁣(n2)+ln⁡2−ln⁡n≈−1n⟹ln⁡σ^t2=ln⁡σ^02+∑j=1tln⁡Wj→t→∞a.s.−∞\mathbb{E}[\ln W_t] = \psi\!\left(\tfrac{n}{2}\right) + \ln 2 - \ln n \approx -\frac{1}{n} \quad\Longrightarrow\quad \ln\hat\sigma^2_t = \ln\hat\sigma^2_0 + \sum_{j=1}^{t}\ln W_j \xrightarrow[t\to\infty]{\text{a.s.}} -\infty

By the law of large numbers, almost every trajectory goes to zero even though the average stays put. It is a textbook case in which the expectation deceives: a few exceptional trajectories hold up the average while almost all of them die out. This corrects two common claims in the popular version. First, that the expected variance decays as ((n−1)/n)t((n-1)/n)^t holds only for the maximum likelihood estimator with unknown mean; with the unbiased estimator, the expectation does not decay. Second, the limit is a Dirac delta, but not at the original mean: it sits at the random point where the mean’s walk ended.

There is one more detail, and it is not a minor one: scale. The logarithm of the variance drops by roughly 1/n1/n per generation. With nn the size of a training corpus, collapse exists on paper and is invisible in practice.

The Assumption Nobody Meets

Up to this point, the argument could be read as a confirmation of collapse. But everything hinges on a decision the model makes without announcing it: throwing away human data after the first step. Nobody trains that way. The internet is not erased when a synthetic text appears; corpora grow. Gerstgrasser and colleagues (2024) said so explicitly: the collapse studied in the literature assumes replacement, and the realistic regime is accumulation. Their accumulation scenario is, in fact, deliberately pessimistic: it imagines a future in which synthetic content is dumped uncontrollably onto the web and everything is vacuumed up for training.

The Proof: With Memory There Is No Collapse

In the accumulation regime, model tt estimates the mean from all batches together: the original human one and the tt synthetic ones, each of size nn. If mjm_j is the mean of batch jj, the estimate is the average of the averages, and the recurrence changes decisively:

μ^t=1t+1∑j=0tmj⟹μ^t=μ^t−1+εtt+1\hat\mu_t = \frac{1}{t+1}\sum_{j=0}^{t} m_j \quad\Longrightarrow\quad \hat\mu_t = \hat\mu_{t-1} + \frac{\varepsilon_t}{t+1}

The new noise comes in divided by t+1t+1. Each generation weighs less than the one before, because it shares the corpus with everything already there. Adding up the variances:

Var⁡(μ^t−μ)=σ2n∑k=1t+11k2  <  π26⋅σ2n\operatorname{Var}(\hat\mu_t-\mu) = \frac{\sigma^2}{n}\sum_{k=1}^{t+1}\frac{1}{k^2} \;<\; \frac{\pi^2}{6}\cdot\frac{\sigma^2}{n}

This is the Basel series. The error does not grow without bound: it stays bounded, forever, by less than twice the first generation’s error (π2/6≈1.64\pi^2/6 \approx 1.64). It is the same factor Gerstgrasser and colleagues found for linear regression.

The argument for the variance runs in parallel. If vjv_j is the variance of batch jj, the accumulated estimate is their average, and the recurrence becomes:

σ^t2=σ^t−12(1+Wt−1t+1),E ⁣[σ^t2∣σ^t−12]=σ^t−12\hat\sigma^2_t = \hat\sigma^2_{t-1}\left(1+\frac{W_t-1}{t+1}\right), \qquad \mathbb{E}\!\left[\hat\sigma^2_t \mid \hat\sigma^2_{t-1}\right] = \hat\sigma^2_{t-1}

It is a martingale, as before, but now its jumps shrink like 1/(t+1)1/(t+1). Its second moment stays bounded:

E ⁣[σ^t4]=E ⁣[σ^04]∏j=1t(1+2n (j+1)2)  ≤  E ⁣[σ^04] eπ2/(3n)\mathbb{E}\!\left[\hat\sigma^4_t\right] = \mathbb{E}\!\left[\hat\sigma^4_0\right]\prod_{j=1}^{t}\left(1+\frac{2}{n\,(j+1)^2}\right) \;\le\; \mathbb{E}\!\left[\hat\sigma^4_0\right]\,e^{\pi^2/(3n)}

A martingale bounded in L2L^2 converges almost surely and in mean square to a limit σ∞2\sigma^2_\infty whose expectation is the original variance σ2\sigma^2. It remains to show that this limit is not zero. For that it is enough to look at the logarithm: it is a sum of independent terms whose means and variances are of order 1/(n(j+1)2)1/(n(j+1)^2), two convergent series. By Kolmogorov’s three-series theorem, the logarithm converges to a finite number, and therefore the limiting variance is strictly positive. I leave this as a sketch: the details are standard, but they deserve a careful check before being considered closed. (If the mean is also estimated, the accumulated variance additionally includes the spread between batches, which can only add variety.)

In other words: with replacement, almost every trajectory dies out. With accumulation, the distribution settles into a slightly shifted version of the original, with positive variance. The tails do not disappear.

Not Collapse, but Drift

The name matters, because “collapse” suggests an ending. What the mathematics describes is much closer to a phenomenon population genetics has known for a century: genetic drift. In the Wright-Fisher model, a finite population with no mutation or migration loses diversity every generation, and heterozygosity decays as (1−1/(2N))t(1 - 1/(2N))^t. It is the same form as ((n−1)/n)t((n-1)/n)^t. The replacement regime is an isolated population that only breeds with itself: it ends up fixed on a single variant.

Genetics also knows what breaks drift: mutation and migration, which reinject variety from outside, and selection, which decides what is kept. The accumulation of human data plays the role of memory and migration; curation and verification, that of selection. That is why I propose calling the phenomenon bounded drift, and the whole process evolution with memory. It is an analogy, not an identity: models are not populations and texts are not genes. But the mathematical structure is the same, and the analogy frames the question better than the photocopy metaphor does.

The Limits of Memory and the Drift of Bias

The proof above rests on an assumption that deserves to be exposed as frankly as I exposed the replacement one: that synthetic error is pure, zero-mean noise. A language model does not err that way. It hallucinates systematically, repeats structural biases and falls into the same loops again and again. To see what changes, it is enough to add to the toy model a bias bb that every synthetic generation pushes in the same direction.

mt=μ^t−1+b+εtm_t = \hat\mu_{t-1} + b + \varepsilon_t

In the replacement regime, the bias is added in full at every step. In the accumulation regime, it comes in divided by t+1t+1, just like the noise:

replacement: E[μ^t−μ]=b taccumulation: E[μ^t−μ]=b∑j=1t1j+1=b (Ht+1−1)≈bln⁡t\text{replacement: } \mathbb{E}[\hat\mu_t-\mu] = b\,t \qquad \text{accumulation: } \mathbb{E}[\hat\mu_t-\mu] = b\sum_{j=1}^{t}\frac{1}{j+1} = b\,(H_{t+1}-1) \approx b\ln t

The difference between the two series is the whole argument. The Basel series, which governs the noise, converges: that is why random error stays bounded. The harmonic series, which governs the bias, does not converge: it grows like the logarithm. Accumulation stops the noise, but only slows down the bias. The drift still has no ceiling; it grows so slowly that it goes unnoticed in one generation or in ten. That does not make it harmless. It makes it silent.

This also settles the dilution objection. It is true that the weight of human data falls as 1/(t+1)1/(t+1), but the proof already accounted for that: dilution alone does not return the system to the replacement regime. What does is a different combination: if synthetic volume grows geometrically and is also biased, the weight of each new batch stops shrinking and the drift becomes linear again. On top of that, the human corpus that predates the explosion of generative models is archived: X0X_0 is not lost even if the web buries it. The danger is bias, not volume.

Accumulation is not free either, though the bill should not be exaggerated. The corpus grows as (t+1)⋅n(t+1)\cdot n, linearly, and training each generation on everything before it makes cumulative compute grow quadratically, not exponentially. Nor is it necessary to accumulate everything forever: keeping a fixed fraction of real data already stabilizes the process. The cost that does weigh is the one pointed out by Dohmatob, Feng and Kempe: even without collapse, scaling laws degrade, because each synthetic token teaches less than a human one. And curation is expensive, which favors those who can pay for it.

That leaves the verifier. The biological analogy calls for selection, and selection needs a criterion. In code and mathematics it exists: a compiler or a theorem prover says yes or no without an opinion. In natural language, philosophy or art there is no automatic, unbiased verifier, and using another model as judge brings the loop back with the evaluator’s own bias. That is exactly where the bb that accumulation cannot correct is born.

None of this stops being a toy model: it shows a mechanism, not the behavior of a real system (although Gerstgrasser and colleagues confirmed the effect of accumulation in transformers trained on text as well). But the mechanism is enough to correct my own conclusion.

My initial suspicion was that the debate had picked the wrong word and that the risk was forgetting. I fell short. The Nature paper exaggerated from an unrealistic assumption: erasing the web at every generation. Evolution with memory, on its own, would err on the side of optimism from another equally unrealistic one: that being wrong amounts to averaging independent Gaussians. What lies between the two is not a collapse ending in a single repeated word, but a slow sedimentation: models that are more expensive to train, marginally poorer in nuance and dominated by a synthetic consensus that homogenizes language without any single generation noticing. The drift is bounded in the noise. In the bias, it is not.

Sources

  • Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. and Gal, Y. (2024). “AI models collapse when trained on recursively generated data.” Nature, 631, 755–759. doi:10.1038/s41586-024-07566-y
  • Gerstgrasser, M., Schaeffer, R., Dey, A. et al. (2024). “Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data.” arXiv:2404.01413. doi:10.48550/arXiv.2404.01413
  • Dohmatob, E., Feng, Y. and Kempe, J. (2024). “Model Collapse Demystified: The Case of Regression.” arXiv.
  • Dohmatob, E., Feng, Y., Yang, P., Charton, F. and Kempe, J. (2024). “A Tale of Tails: Model Collapse as a Change of Scaling Laws.” arXiv.