Reranking is a budget allocation problem

The shortlist is an upper bound

A reranker cannot rescue a document that never entered its candidate set. That sounds obvious, yet it changes where I would begin a retrieval performance review. If the required evidence appears in only seventy of one hundred test shortlists, even a perfect ordering stage cannot provide it for the other thirty. Improving the ranking metric on those seventy questions may conceal the more consequential failure. I want candidate recall measured separately from final ordering, with an explicit definition of which documents or spans are sufficient to support each answer rather than a loose label for topical similarity.

Nogueira and Cho’s Passage Re-ranking with BERT applies a joint query-passage model to reorder retrieved candidates. Its architectural distinction is useful: the expensive model can examine the relationship between the query and each passage directly, after a cheaper retrieval stage has narrowed the search. The paper’s reported experiments do not establish a universal candidate count or latency budget for another deployment. My interpretation is that reranking introduces a second resource allocation problem. First-stage breadth buys opportunities to find evidence; second-stage depth buys a more detailed examination of selected opportunities. Those purchases should be evaluated together.

References: [1] Passage Re-ranking with BERT — Nogueira and Cho

Turn the deadline into a candidate budget

Consider an illustrative serial cost model with a one hundred and twenty millisecond reranking allowance. Suppose request preparation costs twenty milliseconds and each candidate adds three milliseconds. At most thirty-three candidates fit: twenty plus ninety-nine is one hundred and nineteen. The thirty-fourth would exceed the allowance. These numbers are invented to make the decision visible, not benchmark measurements. Real accelerators batch candidates, so cost often changes in steps and depends on passage lengths. A useful production model would be a measured cost surface over candidate count, token count, batch shape and the load already occupying the device.

The allowance must also be the time genuinely available to reranking. Subtract retrieval, network transfer and answer generation reserves from the end-to-end deadline, including uncertainty rather than only average durations. If the first stage is unusually slow, the second stage needs a smaller feasible plan. I would express that plan as a bounded candidate and token budget, with a documented reduced-quality outcome when it cannot be met. Silently attempting the normal workload until the deadline kills it produces a different product: some users receive a carefully ranked answer while others receive an unexplained timeout.

budget_ms, fixed_ms, per_candidate_ms = 120, 20, 3
count = max(0, (budget_ms - fixed_ms) // per_candidate_ms)
assert count == 33
assert fixed_ms + count * per_candidate_ms <= budget_ms
assert fixed_ms + (count + 1) * per_candidate_ms > budget_ms

Allocate breadth by the question’s structure

Not every question needs the same shortlist. A precise identifier lookup may have one compelling match, while a comparison can require independent evidence about several alternatives. I would divide the retrieval task into evidence obligations before allocating the budget. For a comparison, reserve opportunities for both sides instead of allowing forty near-duplicate passages about the more popular side to consume the list. This is not a claim that a language model can always infer those obligations reliably. It is a reason to expose known query structure from the application when that structure is already available.

An adaptive policy can use retrieval score gaps, source diversity and query type as signals, but those signals need calibration against actual missing-evidence cases. A large score gap may indicate a clear answer or simply a narrow index that omitted the relevant source. I would cap adaptation and keep a small, fixed fallback policy for unfamiliar inputs. The counterargument is that a constant candidate count is easier to debug and often sufficient. I agree when its cost is affordable and its coverage is stable. Adaptation earns its complexity only when it improves the quality-cost frontier on representative requests.

Compare architectures at the complete pipeline level

Khattab and Zaharia’s ColBERT uses independently encoded representations with a later token-level interaction, offering a different balance from evaluating every query-passage pair through a full joint encoder. That distinction matters because retrieval and reranking are not fixed categories with one cost each. A richer first-stage representation may improve the shortlist enough to reduce later work. A simpler index may be easier to refresh and cheaper to store. I would compare complete pipelines at equivalent evidence coverage, including indexing cost and update latency, rather than declaring an architecture better from one isolated scoring benchmark.

Storage and freshness can reverse an apparently attractive inference tradeoff. Keeping many representations per document may be reasonable for a stable collection and expensive for a rapidly changing one. Joint scoring avoids some stored representation costs but repeats computation at request time. Neither approach removes the need to enforce document permissions before exposing text to the answer model. Filtering unauthorized candidates after a reranker has already sent their contents to an external service is too late for that boundary. The pipeline comparison therefore includes where bytes move and which component is allowed to inspect them.

References: [2] ColBERT — Khattab and Zaharia

Rank evidence diversity, not repeated agreement

High scores on duplicate passages do not constitute independent support. A corpus can contain a source article, its mirrored copy, an excerpt and a summary that all repeat the same original claim. If the top results are treated as four votes, the answer system mistakes publication topology for evidential strength. I would track source lineage where possible and reserve context for genuinely different evidence. Deduplication should not erase meaningful revisions, however. Two almost identical policy documents may differ in the one effective date that determines which rule applies to the question being asked.

This creates a tension between relevance and coverage. The individually highest-scoring passages may not form the best collection for answering a multipart question. A selection layer can optimize a simple coverage objective under a token budget, but its categories must be inspectable. Otherwise an opaque diversity score becomes another untestable preference. My initial implementation would use clear constraints such as one current authoritative source per required subject, followed by additional corroboration if budget remains. That approach can lose an unusually useful second passage from the same source, so the evaluation should include cases that require exactly that exception.

A relevance score is not an answerability score

Rerankers are often trained to distinguish more relevant passages from less relevant ones. A passage can be highly relevant while failing to establish the requested fact. It might describe a proposal rather than an implemented feature, or discuss a rule without its effective date. I would keep answerability and support checks downstream as separate decisions. Normalizing a reranker score into a number between zero and one does not turn it into a probability that the answer is true. The training objective, calibration data and decision target must justify that interpretation before the interface encourages anyone to rely on it.

A useful failure taxonomy distinguishes missing candidates, mistaken ordering, incomplete evidence and incorrect synthesis. Each category suggests a different intervention. Increasing the candidate count addresses the first only if the missing evidence is present in the underlying index. A larger reranker might help the second while making latency worse. Expanding source context addresses the third, and stricter claim checking targets the fourth. Without this separation, teams can spend a substantial compute budget improving an intermediate score while leaving the user-visible error unchanged. I would require each proposed optimization to identify the failure category it is intended to reduce.

Make overload choose an explicit quality level

At high load, reranking needs an admission policy rather than an infinitely growing queue. Possible outcomes include a smaller shortlist, a cheaper scorer, a retrieval-only response or an explicit inability to complete the request. Those are product choices with different guarantees. A retrieval-only response can still be useful when it presents sources without synthesizing an unsupported conclusion. It is less acceptable when the interface quietly presents the same confident answer format. The degraded mode should preserve the evidence contract even if it reduces completeness, and its use should be visible in operational measurements.

I would monitor how often each budget tier is selected, the evidence recall within each tier, and the fraction of requests that finish within their actual deadline. Aggregate ranking quality is not enough if the difficult requests are disproportionately routed to the cheapest path during busy periods. The objective is not to maximize the number of candidates evaluated. It is to spend a limited amount of detailed computation on the candidates most likely to satisfy the question’s evidence requirements, while retaining an honest response when that computation cannot establish enough. That is a resource policy worth making explicit.