Evaluating AI without grading the vibes

Define the decision the evaluation should support

Before choosing a benchmark, I would write the decision it is supposed to inform. Are we replacing a model, changing a prompt, adding retrieval or expanding the set of actions an assistant may perform? Those changes can fail in different ways. A retrieval change might improve factual support while increasing latency. A model change might improve common cases while making rare but serious errors more likely. Without a stated decision, a benchmark score becomes an attractive number that people can interpret however the release discussion requires.

HELM offers a useful research precedent by evaluating language models across multiple scenarios and multiple dimensions rather than reducing capability to one metric. Its particular model comparisons are historical results, not a current ranking for this application. The broader framing matters to me: accuracy, robustness and efficiency can conflict, and gaps in evaluation coverage should remain visible. For a product, I would choose a small set of explicit acceptance conditions tied to its promise, then report the cases those conditions do not cover.

References: [1] Holistic Evaluation of Language Models

Turn a vague task into observable outcomes

Suppose a hypothetical documentation assistant is meant to answer version-specific configuration questions. A useful rubric can ask whether it identified the version, cited material that applies to that version, gave the correct configuration and abstained when the source was insufficient. These are separate outcomes. A fluent answer with the wrong version should fail even if its wording resembles a reference answer. Conversely, an answer that uses different wording can succeed if it communicates the same supported configuration and preserves the relevant conditions.

I would also distinguish task success from presentation quality. Clear writing matters, but it should not compensate for unsupported facts or unauthorized actions. Use deterministic checks where a result is objectively testable, such as parsing an output or executing code in an isolated fixture. Use human or model judgment where interpretation is necessary, with a rubric and examples of borderline cases. The evaluator should know which property it is assessing rather than converting a general impression of quality into a number that appears more precise than the underlying judgment.

Preserve a reproducible experiment

A model name alone does not identify an evaluation. The prompt template, decoding settings, tool definitions, retrieval corpus, document versions and scoring code all influence the outcome. I would record them in a manifest with the test-set version and the exact candidate under comparison. The language-model evaluation harness maintained by EleutherAI is a useful primary reference for treating tasks and evaluation configuration as code. The practical lesson is not that one framework solves evaluation, but that the experimental procedure deserves version control alongside the application.

Repeated runs also matter when outputs are stochastic or external services vary. A result from one sample per question may understate variability, especially for tasks involving long action sequences. I would decide in advance how many repetitions are needed and whether the unit of analysis is a question, a user session or an independent task family. Store enough output and evaluator evidence to investigate disagreement, while respecting the privacy requirements of the underlying data. Reproducibility should not become an excuse to retain sensitive material without a defined need.

References: [2] EleutherAI language-model evaluation harness

Use paired comparisons when cases are shared

Consider an illustrative evaluation of 400 independent questions. Model A succeeds on 320 and model B on 332, an apparent improvement of three percentage points. Suppose A alone succeeds on 20 questions and B alone succeeds on 32. The remaining cases agree. The comparison should use those paired outcomes rather than treating the two success rates as unrelated samples. Assign each question a difference of minus one, zero or one. The mean difference is 0.03, and the disagreement fraction is 52 divided by 400, or 0.13.

A simple large-sample estimate gives variance of the paired difference as approximately 0.13 minus 0.03 squared. Dividing by 400 and taking the square root gives a standard error near 0.018. An approximate 95 percent interval is therefore 0.03 plus or minus 0.035, which includes zero. These invented results do not establish a reliable improvement. The approximation also assumes independent cases; near-duplicate prompts or several questions from one document can violate that assumption. I would use an appropriate paired or clustered analysis for the actual sampling design.

from math import sqrt
n, a_only, b_only = 400, 20, 32
difference = (b_only - a_only) / n
second_moment = (a_only + b_only) / n
standard_error = sqrt((second_moment - difference**2) / n)
print(difference, (difference - 1.96*standard_error, difference + 1.96*standard_error))

A judge is another system to evaluate

A model-based judge can make large evaluations practical, but it introduces another source of error. I would test the judge on deliberately varied answer order, verbosity and style while holding the underlying correctness constant. A rubric that rewards confident explanation can accidentally favor an eloquent wrong answer over a concise correct one. The judge should receive the evidence needed for the criterion it is assessing, and its output should include an interpretable reason or label that can be checked against a human-reviewed sample.

Agreement with human labels is useful but insufficient if both share the same blind spot. Include adversarial pairs where a citation is irrelevant, a code solution passes obvious examples but fails an edge case, or an answer changes a crucial qualifier. Keep the judge version and prompt fixed during a comparison. If the evaluator changes together with the candidate, an apparent improvement can come from moving the measuring instrument. I would treat judge calibration as a separate experiment and report uncertainty or disagreement instead of burying it beneath an aggregate preference rate.

Inspect the errors that averages erase

A single success rate can hide changes in the composition of failure. Suppose a candidate fixes many harmless formatting errors but introduces a few unauthorized writes. Counting each failure equally could make the candidate look better while violating the product’s most important constraint. I would define hard gates for unacceptable behavior and separate severity-weighted outcomes from ordinary quality metrics. The weighting should come from the application’s consequences, not be tuned after seeing which setting makes the preferred model win the comparison.

Slices should be chosen before the release decision where possible: long inputs, missing evidence, ambiguous requests, interrupted tool calls and unsupported versions are plausible examples. Report both sample counts and outcomes for each slice. Small slices can produce unstable percentages, and searching many slices after the fact can generate apparently dramatic differences by chance. Nevertheless, inspecting concrete failures remains essential. Statistical caution should prevent overclaiming, not prevent the team from noticing that every failure in a narrow category comes from the same fixable parsing or state-management defect.

Keep the test from becoming the training objective

A benchmark that is repeatedly consulted during prompt development becomes part of the development process. That is useful, but it weakens its role as an independent estimate of future performance. I would maintain a development set for iteration and a protected evaluation set for decisions, with a clear process for refreshing both. If test cases are copied into prompts or examples, record that change. A system can improve on familiar wording without acquiring the behavior the benchmark was intended to measure.

The strongest counterargument is that small teams cannot afford elaborate evaluation governance. They do not need a research institute’s infrastructure to improve matters. Even a modest set of carefully chosen cases, a saved manifest and a habit of examining paired failures are better than judging a few attractive demonstrations. Start with the failure modes the application can least afford, add representative ordinary cases, and expand as real usage reveals gaps. The goal is a credible decision process, not an impressive number of benchmark rows.

Make release criteria survive contact with reality

Offline evaluation cannot fully capture changing traffic, external tools or user adaptation. I would connect it to a bounded rollout with observable failure indicators and a way to stop or revert the change. The production measurements should use definitions compatible with the offline rubric where possible. Otherwise the team can celebrate an offline improvement while discovering that the live metric measures something entirely different. Latency, cost and escalation rates belong beside task quality because they influence whether users can benefit from the improved behavior.

A useful evaluation report ends with a decision and its limits: the candidate improved these outcomes under these conditions, did not establish an improvement on these others, and has these unresolved failure modes. That statement is more useful than declaring one model generally smarter. I want evaluation to make disagreement concrete. If people disagree about the release, they should be able to point to the cases, costs or assumptions that matter, rather than argue about which set of generated answers felt more capable during a demonstration.