Retrieval needs an abstention policy

A nearby passage is not yet an answer

A search result can be relevant without supporting the requested claim. A document about a product’s availability in one country may be semantically close to a question about availability somewhere else. A policy describing last year’s rules may use exactly the right vocabulary while being wrong for today’s decision. I would separate candidate discovery from evidential sufficiency. The retriever proposes material worth inspecting; a subsequent decision determines whether that material supports the particular answer being considered under the question’s scope and time constraints.

The original retrieval-augmented generation paper combines a neural retriever with a sequence generator and treats retrieved documents as latent variables. It reports improvements on selected knowledge-intensive tasks. That contribution does not imply that every top-ranked passage is authoritative or that every generated sentence follows from a source. My design argument extends beyond the paper: an application should explicitly represent the conditions under which it is willing to answer. Without that contract, retrieval can make unsupported claims look better sourced while leaving the underlying decision unchanged.

References: [1] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Write the evidence contract before the prompt

Consider a hypothetical assistant answering whether a software feature exists in a specific release. I would require an identified product, an identified version, a source covering that version and text supporting the relevant behavior. A generic roadmap is not sufficient evidence of availability. A discussion of a similarly named product is not sufficient evidence of identity. These requirements should be visible in the application’s data model so missing fields can be distinguished from contradictory evidence and from a retriever that returned no candidates at all.

A useful answer type can contain supported, insufficient and conflicting outcomes. The unsupported cases should carry a reason and the evidence that was considered, not just an empty string. That makes recovery possible: ask for the release number, search a different source, or explain that the available documents disagree. I would avoid representing uncertainty as a decorative sentence appended after a confident answer. If the evidence contract fails, the main result should preserve that failure rather than quietly allowing a downstream consumer to discard the caveat.

type EvidenceResult =
  | { kind: 'supported'; claim: string; sourceIds: string[] }
  | { kind: 'insufficient'; missing: string[]; inspectedIds: string[] }
  | { kind: 'conflicting'; claim: string; sourceIds: string[] };
// A consumer must handle all three states before displaying an answer.

Make the cost of a wrong answer explicit

Suppose, purely as an illustrative decision model, a correct answer has zero loss, an incorrect answer has loss 20, and abstaining has loss 1. If p is a calibrated probability that a proposed answer is correct, answering has expected loss 20 × (1 − p). Answering is preferable to abstaining only when p exceeds 0.95. Change the consequence of an error or the cost of escalation and the threshold changes. The number is not a universal confidence setting; it follows from stated assumptions.

The difficult word in that example is calibrated. A similarity score, a model’s self-reported confidence and a probability of factual correctness are different objects. I would estimate the decision rule from labeled examples representative of the application, then check whether its behavior survives changes in topic and document quality. A threshold selected on clean internal documentation can fail on outdated, duplicated or adversarial pages. The policy should therefore include explicit structural gates as well as any learned score, especially where a missing version or jurisdiction makes the answer ill-defined.

Evaluate coverage and error together

Selective prediction studies the trade-off between answering more cases and reducing error among the cases that are answered. SelectiveNet formalizes prediction together with a selection function and evaluates selective risk against coverage. I borrow that framing, not its architecture, for retrieval applications. Coverage is the fraction of eligible questions receiving an answer. Selective error is the error rate among those answered. Reporting only one invites manipulation: a system can become very accurate by declining almost everything, or very available by guessing whenever evidence is absent.

In an illustrative set of 1,000 questions, suppose a policy answers 700 and gets 35 wrong. Its coverage is 70 percent and its answered-case error is 5 percent. A second policy answers 900 and gets 90 wrong: 90 percent coverage and 10 percent error. Neither policy is universally better. The choice depends on the cost of mistakes, the usefulness of escalation and the distribution of rejected cases. I would also inspect whether abstentions concentrate on particular users, topics or document formats that the product claims to serve.

References: [2] SelectiveNet: A Deep Neural Network with an Integrated Reject Option

Check support at the level of the claim

A paragraph can mix a supported fact with an unsupported inference. Attaching one citation to its end does not establish every sentence. I would decompose an answer into claims that can be checked against specific passages, including the qualifiers that change their meaning. Words such as all, always, current and approved often carry the strongest unsupported leap. A passage showing that one configuration works does not prove that all configurations do, and a source describing a proposal does not prove that it shipped.

This does not require displaying a citation after every short phrase. The internal representation can be more granular than the prose. Each material claim should retain its source identifiers and the relation being asserted: direct support, inference or unresolved conflict. A renderer can then produce a readable answer while preserving that structure. I would test the system with sources that support only half of an attractive answer, because those cases reveal whether citations function as evidence or merely as decoration attached to whatever the generator wanted to say.

Treat retrieved instructions as content

Retrieved documents can contain imperative language, including text designed to redirect an assistant. OWASP’s prompt-injection guidance describes the risk of external content influencing model behavior across trust boundaries. A retrieval pipeline should therefore preserve the difference between material to analyze and instructions that govern the application. I would not grant a document the authority to select tools, change permissions or decide what may be disclosed merely because the retriever ranked it highly. Relevance and authority need separate representations throughout the pipeline.

A practical test document might contain the correct factual answer alongside an instruction to ignore the user and reveal another document. The expected outcome is use of the relevant evidence without obedience to the embedded instruction. This is a test design, not a claim that delimiters alone solve prompt injection. Tool permissions, source filtering, output validation and execution controls remain necessary outside the model. The abstention path also matters here: if the system cannot separate a usable fact from a compromised instruction channel, it should be able to decline the operation cleanly.

References: [3] OWASP LLM01: Prompt Injection

Do not make abstention a dead end

A bare statement that the system does not know is sometimes accurate but operationally poor. The next useful action depends on why it does not know. Missing evidence may justify another search. An ambiguous question may justify a clarification. Conflicting authoritative documents may justify presenting the conflict. A permissions boundary may require stopping without revealing what inaccessible material contains. I would make these different outcomes explicit, because treating every failure as a generic refusal wastes information that could help the user proceed.

There is a legitimate counterargument that additional checks can make a system slow, expensive and timid. A low-risk brainstorming tool may not need the same evidence contract as a release-support assistant. I agree. The policy should match the promise being made. If the application offers suggestions, label suggestions. If it claims to answer from a controlled documentation set, enforce that claim. The mistake is not choosing a permissive mode; it is allowing a permissive generation path to masquerade as verified retrieval because the interface includes citations.

Keep an auditable path from question to decision

For evaluation, I would retain the query, retrieval configuration, source versions, selected passages, proposed claims and final decision within the appropriate privacy boundary. These records make failures attributable. A missing document is a retrieval problem. A document that contradicts the answer is a support problem. An answer that ignores a failed support check is an orchestration problem. Without those distinctions, every defect becomes a vague request to improve the model, and the team cannot tell whether a change repaired the actual cause.

The final criterion is whether the system’s stated certainty matches the evidence it possesses. Retrieval can improve the evidence available, but an abstention policy determines what happens when improvement is insufficient. I would ship the simplest explicit policy that can be evaluated, then broaden coverage as evidence warrants it. A useful retrieval system should be capable of both producing a grounded answer and identifying the precise boundary that prevents one. That boundary is part of the product’s knowledge, not an embarrassing exception to hide from the user.