Model routing needs a fallback contract

The route is part of the application behavior

A model router decides which implementation will attempt a task. That decision can change latency, available tools, context capacity, output format and the likelihood of different errors. I would treat routing as part of the application’s behavior rather than an invisible billing optimization. The user does not need to understand every model choice, but the service needs a clear contract for what remains consistent across routes. If one path supports a required constraint and another does not, selecting between them is not a transparent substitution. The router must either preserve the constraint or expose a different supported outcome.

Chen, Zaharia and Zou’s FrugalGPT studies cascades for balancing cost and quality, while Ong and colleagues’ RouteLLM studies routing using preference data. Both provide useful evidence that model selection can be optimized under particular objectives and evaluations. Neither establishes that a cheaper and a stronger model are semantically interchangeable on every request. My focus is the surrounding contract: what triggers escalation, what happens when escalation is unavailable and which boundaries must remain unchanged? A routing score becomes useful only when it is connected to those operational decisions rather than treated as a universal estimate of correctness.

References: [1] FrugalGPT — Chen, Zaharia and Zou[2] RouteLLM — Ong and colleagues

Calculate the full cascade cost

Consider an illustrative cascade where the cheap attempt costs one unit, the stronger attempt costs ten units and routing costs one tenth of a unit. If twenty percent of requests escalate after the cheap attempt, expected cost is 0.1 plus 1 plus 0.2 times 10, or 3.1 units. That is lower than a direct ten-unit stronger-model call under these assumptions. It does not include retry loops, verification or any extra tool work, and it assumes the escalation rate is stable. Those omitted costs need to be added before presenting the arithmetic as a deployment forecast.

Latency tells a different story. Suppose routing takes ten milliseconds, the cheap attempt takes one hundred and escalation takes six hundred. In a sequential cascade, non-escalated requests finish in one hundred and ten milliseconds and escalated requests in seven hundred and ten. The expected duration is two hundred and thirty milliseconds at the assumed twenty percent escalation rate. A direct stronger call takes six hundred milliseconds in this toy model, so the cascade improves the mean while making escalated requests slower. The calculation shows why an average improvement cannot stand in for a tail-latency contract.

from math import isclose
escalation = 0.2
expected_cost = 0.1 + 1 + escalation * 10
fast_ms, escalated_ms = 10 + 100, 10 + 100 + 600
expected_ms = (1 - escalation) * fast_ms + escalation * escalated_ms
assert isclose(expected_cost, 3.1)
assert (fast_ms, escalated_ms, expected_ms) == (110, 710, 230.0)

Escalation signals need a decision target

A router can use task type, input length, retrieval quality or a learned score to decide which path to take. Those signals should be evaluated against the actual failure the application wants to avoid. A preference label may capture which answer people like better without proving that either answer satisfies a numerical or authorization constraint. A model’s own confidence can be miscalibrated. I would define escalation conditions around observable requirements where possible: missing evidence, invalid structure, unresolved tool results or task categories outside the cheaper route’s validated scope. Learned signals can complement those checks without replacing their meaning.

There is a cost to conservative escalation. If every uncertain case goes to the stronger model, the savings can disappear, and the stronger model may still fail. That is why the fallback contract needs an abstention or clarification outcome rather than an assumption that more compute always produces an acceptable answer. RouteLLM’s preference-based routing objective is useful within its setting; an application with asymmetric error costs may require a different objective. I would evaluate routing decisions by the loss of the resulting product outcome, including unnecessary escalation and incorrect acceptance, rather than by agreement with a model label alone.

References: [2] RouteLLM — Ong and colleagues

Define which failures are retryable

A transport timeout, a malformed response, an unsupported request and a policy refusal are different events. A fallback policy should not collapse them into try another model until something answers. A timeout may justify a bounded retry if the operation is safe to repeat. Invalid structure may justify regeneration with the same authority limits. A refusal or a scope restriction may be the correct outcome and should remain in force across routes. I would classify the reason for fallback explicitly so that availability recovery cannot quietly become a way to bypass an application rule or change the task’s permitted scope.

Fallback also needs a deadline budget. If the first attempt consumes nearly the entire allowance, starting a slower second attempt can only create a late response and additional cost. The router should know the remaining time and choose an outcome that can still be useful. Sometimes that is a shorter response, a source-only result or an explicit failure. These alternatives must preserve the product’s evidence and authorization requirements. A degraded mode is not a license to answer without support simply because the preferred implementation was unavailable. Its reduced capability should be defined in advance and tested as an ordinary route.

Do not replay committed effects

A routed task may include external actions as well as text generation. If the first model already caused an authorized change, switching models must not repeat that change merely because the response was lost or judged incomplete. The orchestration layer needs an execution ledger and idempotent operation identities where the tool supports them. The fallback model should receive the actual committed state, not a vague transcript of what the first model intended. This is especially important when a tool call times out: uncertainty about the response is not proof that the operation did not happen.

I would keep action commitment outside the replaceable model step. Models can prepare proposals and interpret results, while a trusted workflow manages the state transitions and deduplicates retries. That structure lets a fallback continue from a known boundary instead of restarting the entire conversation as though no work occurred. The counterargument is that some tasks are purely generative and do not need this machinery. That is correct; the contract should scale with consequence. The important rule is not to inherit a text-only retry policy unchanged when the application later adds tools with persistent external effects.

Privacy and capability must survive substitution

A fallback destination may have different data-handling conditions, location constraints or tool integrations. The router should select only among destinations authorized for the current input and task. A service outage does not expand that set automatically. I would attach route eligibility to trusted request metadata and check it before sending any content. The same applies to context length and output guarantees. If a fallback cannot accept the necessary evidence or produce a required format reliably, the application needs a defined alternative rather than silently truncating the input and pretending the original task was completed.

A consistent interface can hide provider-specific details from the user while still preserving them in internal traces. Record the selected model revision, route reason, escalation reason and any reduced-capability mode. Avoid logging sensitive prompt text solely to make routing analysis easier. These records help separate model quality changes from changes in the traffic sent to each model. Without them, a cheaper route can appear to improve because the router stopped sending it difficult requests, or a stronger route can appear to worsen because it receives only the failures of the first stage. Selection changes the meaning of route-level averages.

Evaluate the policy, not only its components

A routing evaluation should replay the complete policy on held-out tasks, including the triggers, verification steps and fallback outcomes. Measuring each model independently is necessary but insufficient because the router determines which errors reach users. Include distribution shifts, unavailable destinations, deadline exhaustion and ambiguous tool results. Compare total cost, completion quality and latency distributions under the same task mix. FrugalGPT’s cascade perspective is useful here: the unit being optimized is the sequence of decisions, not merely the price of one call. The evaluation should reflect the sequence the deployed application actually executes.

My preferred router is explicit about both its savings and its obligations. It can choose an inexpensive route when the task fits, escalate when additional capability is justified and stop when no authorized route can meet the contract. It does not assume that a stronger model makes every failure retryable or that a lower average cost excuses a slower or less reliable tail. Model routing is most useful when the application’s promises remain stable across the choice of implementation. The fallback contract is what makes that stability testable instead of leaving it to the next generated answer.

References: [1] FrugalGPT — Chen, Zaharia and Zou