AI evaluation needs a dataset lifecycle

The dataset is part of the measuring instrument

A score is not a property of a model in isolation. It is the result of applying a task definition, a set of inputs, a reference policy and a scoring procedure to a particular system. Changing any of those components can change the number without improving the deployed product. I would therefore version an evaluation dataset with the care given to a measurement instrument. Its documentation should explain the population of cases it represents, the kinds of failure it can detect and the decisions for which its results provide useful evidence.

Gebru and colleagues propose datasheets that describe a dataset's motivation, composition, collection process and recommended uses, among other characteristics. That proposal addresses a real gap between a downloadable file and an interpretable research artifact. My operational extension is to make those descriptions participate in the release process. A dataset revision should identify which part of the measurement contract changed, and a release report should name that revision. A chart labelled accuracy without a stable dataset and scoring identity can remain visually convincing long after the underlying comparison has stopped being meaningful.

References: [1] Gebru et al. — Datasheets for Datasets

Related rows are not independent cases

Suppose an illustrative collection contains two hundred original tasks, each expanded into five paraphrases. An independent eighty-percent training and twenty-percent test assignment for each row seems ordinary, but it allows members of the same task family into both partitions. Under that Bernoulli assignment, the probability that a five-row family crosses the boundary is one minus 0.8 to the fifth power minus 0.2 to the fifth power, which equals 0.672. The expected number of crossing families is therefore 134.4. This is an expectation across random splits, not a claim that a particular split contains a fractional family.

The leakage is conceptual: the test may ask whether the system recognizes a reformulation of a task it has already encountered. That can be useful if reformulation is the intended target, but it should not be presented as performance on independent tasks. Split at the level of the dependency you want to exclude, such as source document, customer session, repository or original question. A stable group identifier makes that policy enforceable. It also exposes cases with several parents, which may require connected-component grouping rather than arbitrarily choosing one parent and leaving the other dependency across the boundary.

from hashlib import sha256

leak_probability = 1 - 0.8**5 - 0.2**5
assert abs(leak_probability - 0.672) < 1e-12

def partition(group):
    digest = sha256(('split-v1:' + group).encode()).digest()
    return 'test' if int.from_bytes(digest, 'big') % 5 == 0 else 'train'

rows = [(f'task-{group}', variant) for group in range(200) for variant in range(5)]
train_groups = {group for group, _ in rows if partition(group) == 'train'}
test_groups = {group for group, _ in rows if partition(group) == 'test'}
assert train_groups.isdisjoint(test_groups)
assert len(train_groups | test_groups) == 200

A manifest should explain a content hash

A hash is useful for identifying bytes, but it cannot explain why those bytes constitute a good test. I would keep a small manifest containing source revisions, inclusion rules, group identities, label-policy version, split procedure and scorer version. Hash a canonical representation of the actual evaluated records, and retain the manifest beside the result. If the same dataset name later resolves to different bytes, the discrepancy becomes detectable. If the bytes stay identical while the scoring policy changes, the separate scorer identity explains why a new number can still be legitimate.

Dataset cards on the Hugging Face Hub provide a practical place to document content and context alongside structured metadata. A card does not certify the truth of every label, and completing its fields does not replace inspection. Its value is making assumptions visible to a future user who did not attend the collection process. For an internal evaluation, an ordinary versioned text file can serve the same purpose. The important property is that documentation travels with the artifact and describes the version that produced the result, rather than an evolving idealized description of the project.

References: [2] Hugging Face Hub — Dataset Cards

Corrections need a trail rather than an eraser

Evaluation cases will contain mistakes. A reference answer may be stale, a supposedly impossible request may have become supported, or a scorer may reject an equivalent valid response. Refusing to correct those defects preserves an unreliable instrument. Silently correcting them makes old results difficult to interpret. I would treat a suspected issue as a record with evidence, a proposed resolution and a decision. Cases can be quarantined while the ambiguity is investigated, but published comparisons should state whether quarantined cases were excluded and retain the earlier result's original definition.

When a correction is accepted, release a new dataset or scorer revision and rerun the relevant systems on both versions where feasible. A small paired comparison can show whether the apparent improvement comes from the model or the repaired instrument. Preserve difficult cases that are valid, even when they make a release look worse. Otherwise the collection gradually becomes a catalogue of cases the current system happens to solve. For privacy-sensitive source material, the trail can retain a reason and a stable non-content identifier without preserving material that should have been deleted.

A fresh test asks a different question

Recht and colleagues assembled new test sets following the original CIFAR-10 and ImageNet collection processes as closely as their study allowed and observed performance drops. Their analysis attributes the gap to harder images rather than finding evidence that adaptive overfitting explains it. That distinction matters: a result can change because the sampled population changes, without contamination or misconduct. I would use the paper to motivate fresh measurement and careful interpretation of sampling differences, not as a shortcut proof that every repeatedly used benchmark has been memorized or rendered useless.

An application benefits from both a stable regression suite and periodically refreshed evidence. The stable suite detects whether known behaviours changed. New samples examine whether those behaviours still cover the situations users encounter. Report the two separately before aggregating them. If a new product feature introduces a class of requests absent from the old test, label it as expanded coverage rather than a simple continuation of the same time series. A dashboard should make the population boundary legible so readers can tell whether a rising or falling score reflects capability, composition or both.

References: [3] Recht et al. — Do ImageNet Classifiers Generalize to ImageNet?

Protect the boundary during development

A held-out split can become a development set through repeated inspection even if its rows never appear in training. Engineers read failures, alter prompts and select configurations that respond to those particular cases. That process is useful, but the role of the collection has changed. I would distinguish diagnostic cases that can be studied freely from a smaller release holdout governed by an explicit access policy. When a holdout failure becomes an engineering fixture, move its role deliberately and replenish the independent evidence rather than pretending the boundary remains untouched.

Derived data needs lineage for the same reason. A synthetic paraphrase of a held-out question is still connected to that question, even if its wording differs substantially. Exact string deduplication will miss this relationship. Record derivation parents during generation instead of trying to infer every connection afterward. Near-duplicate search can supplement that record, but it cannot prove the absence of conceptual overlap. The standard should be a defensible account of how evaluation independence was protected, with known limitations stated, rather than a confident claim that a single similarity threshold has eliminated every form of leakage.

Keep the process proportional to the decision

The strongest objection is overhead: a small team may not need a dedicated dataset platform to evaluate a modest feature. I agree. A directory of versioned cases, a short manifest, group-aware splitting and a reproducible scoring command can establish most of the useful discipline. Add workflow machinery when coordination problems justify it. The non-negotiable part is conceptual clarity about what a case means and whether a comparison still measures the same thing. A lightweight honest evaluation is more useful than a sophisticated dashboard whose inputs change without a trace.

Before trusting a release result, I would ask whether another engineer can identify the exact cases, understand their origin, reproduce the split and explain the label policy without reconstructing a conversation. Then I would inspect the failures that matter to the product, not only the average. A dataset lifecycle makes these questions answerable over time. It turns evaluation from an occasional ceremony into a maintained source of evidence, while acknowledging that every test has a scope and that useful evidence must evolve when the application and its users change.