Synthetic data needs a failure model

Describe the process before defending the category

Synthetic data is a broad label for very different procedures. A program can generate arithmetic questions with exact answers, a simulator can produce observations under a known physical model, or a language model can invent both a task and its proposed solution. Each procedure has a different relationship to ground truth. I would specify the generator, seed material, sampling policy, selection rules, label validation and training mixture before arguing about quality. Without that description, two teams can debate synthetic data while one means verified counterexamples and the other means unreviewed model continuations.

The generator is only the start of the distribution. A quality filter decides which outputs survive, a deduplicator changes relative frequencies, and a curriculum decides how often the retained examples are encountered. Those stages can introduce systematic bias even when each seems reasonable in isolation. An acceptance rate is therefore not enough to judge the process. The key question is which kinds of examples are accepted or rejected and whether the resulting coverage still represents the intended task. A pipeline can produce fluent, internally consistent data while removing precisely the difficult cases the next model needs.

A good-looking filter can shrink the tail

Consider an invented population in which one percent of examples belong to a rare but important class. Suppose a filter accepts half of that class and ninety-nine percent of common examples. Out of an expected hundred thousand candidates, five hundred rare and ninety-eight thousand ten common examples survive. Overall acceptance is an impressive 98.51 percent, yet the rare class is now only about 0.508 percent of the retained set. The filter has almost halved its representation. A high aggregate acceptance rate conceals a substantial change in the tail of the distribution.

This example does not assume the filter is malicious or careless. It might reject unusual wording more often, score unfamiliar structures poorly or remove examples whose answers are harder to verify. The mechanism still matters. Measure acceptance by relevant task slices and inspect whether rejection correlates with difficulty, language, source or output form. If rare examples are intentionally excluded, document that scope change. If they are needed, improve their verification path or deliberately restore coverage. Blindly oversampling the remaining rare examples can amplify errors if the retained subset is itself a biased representation of the class.

rare_prior = 0.01
rare_accept = 0.50
common_accept = 0.99
retained_mass = rare_prior * rare_accept + (1 - rare_prior) * common_accept
rare_after_filter = rare_prior * rare_accept / retained_mass
assert abs(retained_mass - 0.9851) < 1e-12
assert 0.00507 < rare_after_filter < 0.00508

# Independent draws from a source with a one-percent rare class.
probability_of_no_rare_example = (1 - rare_prior) ** 100
assert 0.366 < probability_of_no_rare_example < 0.367

Replacement and accumulation are different experiments

Shumailov and colleagues investigate recursive training on generated data and describe model collapse under the conditions they study, including loss of information about distribution tails. That result deserves attention, but it should not be inflated into a claim that all generated examples inevitably degrade all models. Gerstgrasser and colleagues explicitly examine the difference between replacing data and accumulating it, finding that retaining original real data alongside successive synthetic generations changes outcomes in their studied settings. The training policy is a central experimental variable, not an implementation detail to omit from the conclusion.

My engineering reading is to preserve the distinction in every experiment. A fixed-size replacement pipeline, an expanding archive and a fixed-budget mixture with a retained reference set do not expose the learner to the same distribution. Accumulation can prevent one form of forgetting while increasing storage, duplication or weighting problems. A real-data anchor can also be incomplete or biased. I would test the actual proposed mixture and its sampling weights against a stable reference evaluation rather than borrowing a guarantee from a differently configured experiment. Retention is a design choice with measurable effects, not a universal antidote.

References: [1] Shumailov et al. — The Curse of Recursion: Training on Generated Data Makes Models Forget[2] Gerstgrasser et al. — Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

Finite samples can delete events before training begins

Even without an imperfect neural model, sampling can remove rare events. For one hundred independent draws from a source with a one-percent rare class, the probability of observing no rare example is 0.99 to the hundredth power, approximately 36.6 percent. If an intentionally simplistic learner fits only the observed empirical categories and the next generation replaces the original data completely, a missing class has no representation from which to be sampled. That toy process illustrates one route to irreversible loss. It is not a mathematical description of every language model, which can generalize and encode prior information.

The example suggests a practical control: track required coverage explicitly instead of assuming a larger generation count will preserve every important case. Maintain a list of task families whose absence would invalidate the dataset for its purpose. Count examples, inspect their diversity and verify their labels within each family. If a generator cannot produce a rare failure mode faithfully, use a different construction method or retain suitable reference examples. Frequency alone is not importance. A low-probability case can deserve substantial evaluation and training attention when failure on that case has a large practical cost.

Verification should add independent information

Having a generator grade its own answer can be useful for triage, but agreement between two prompts to the same model does not establish an independent reference. The prompts may share the same blind spot. Several models can also inherit correlated errors from overlapping training sources or similar objectives. I would prefer checks that introduce a different source of information: executing a program against meaningful tests, comparing a derived answer with a known database record, or asking a reviewer to inspect evidence rather than only stylistic plausibility. Independence is a property of the evidence, not the number of model calls.

Executable verification is strongest when the oracle captures the intended task. A generated function that passes three superficial examples may still exploit an incomplete test suite. Vary the tests, include invalid inputs and preserve the specification from which expected results were derived. For open-ended writing, no single oracle will settle quality; use a clearly defined rubric and inspect disagreement. The pipeline should record what was checked and what remains uncertain. Calling a record verified without naming the scope of verification encourages later consumers to assume a stronger guarantee than the process actually supplied.

Lineage is needed for both training and evaluation

A useful synthetic record should identify its parent material, generator revision, prompt or generation procedure, filtering decisions and verification result. This does not require preserving sensitive source text indefinitely; stable identifiers and appropriate retention controls can carry lineage without uncontrolled copying. Parentage matters because apparently different examples may derive from the same underlying task. If one derivative enters training and another enters evaluation, a row-level split can exaggerate generalization. Preserve family boundaries from the beginning, when the relationship is known, instead of trying to reconstruct them with text similarity after the dataset has grown.

I would evaluate candidate mixtures on held-out examples that were not generated by the same pipeline, and report results by the failures the synthetic data was meant to address. If the goal was better handling of malformed inputs, show that slice alongside ordinary valid inputs. If synthetic data improves one slice while degrading another, the tradeoff should be visible. Keep a control run under a comparable compute budget where feasible. More training tokens can improve a model for reasons unrelated to the novelty or quality of the synthetic procedure, and the experiment should avoid confusing those effects.

Spend generation where verification is credible

The strongest counterargument is that human-created data is expensive and fallible too. That is correct, and it argues for comparing complete processes under realistic budgets rather than idealizing either origin. Synthetic generation can be especially useful for systematically varying a known construction, exercising boundary conditions or producing counterexamples with reliable labels. A smaller amount of carefully verified generated data may add more value than a large unfiltered corpus. Conversely, a broad task with weak validation may benefit more from improving reference material than from multiplying confident variations of the same uncertain answer.

My decision rule is to name the failure model before increasing volume. Identify how the pipeline could lose coverage, reinforce an incorrect label, contaminate evaluation or overrepresent an easy pattern. Add measurements and controls for those failures, then examine whether the resulting data improves the intended behaviour. This approach leaves room for synthetic data to be extremely useful without treating it as free information. The real resource is trustworthy learning signal. Generation creates candidates for that signal; selection, verification and mixture design determine how much of it survives into the trained system.