LLM Model Routing by Task, Cost, and Quality

LLM model routing can cut the cost of a mixed workload. Routine requests go to a smaller model, while harder work goes to a stronger one. The dangerous failure is easy to miss: the bill falls, but answer quality slips, tool support disappears, or the selected model cannot honor the requested schema. An aggregate success rate can hide those mistakes until someone acts on a bad answer.

A prompt classifier alone is not a production router. Add deterministic capability gates, a tested cost and quality threshold, a strong fallback, versioned decisions, and a release process that compares routed output with the result from the unrouted model.

Why model routing fails in production

Routing looks simple when every request is plain text: estimate difficulty, then choose the cheaper or stronger model. Internal workflows carry constraints that a difficulty score cannot capture.

A request may require a specific tool schema, image input, long context window, strict JSON, regional endpoint, or model approved for a particular data class. A cheap model can understand the prompt and still be ineligible. RouteLLM issue 63 records a concrete case: a selected local model rejected a forwarded parameter it did not support. The routing decision succeeded, but the request contract did not survive it.

Quality depends on local traffic too. The RouteLLM documentation recommends calibrating thresholds with data that resembles the requests an application receives. A threshold derived from general chat may not transfer to invoice extraction, support escalation, code repair, or policy interpretation. RouteLLM issue 79 raises a related concern about scores tied to internally fixed model identities rather than the pair deployed by the caller. Treat model names, versions, and routing scores as one versioned unit.

An average quality score can also conceal asymmetric harm. Sending a simple summary to the strong model wastes money. Sending a high-risk approval request to the weak model can produce a bad business action. The policy must consider the consequence of a mistake, not only estimated difficulty.

Define the routing contract before choosing a router

Define each workflow operation before routing it. Do not ask the router to infer every requirement from free text. The application already knows the endpoint, requested output, available tools, caller, and business operation, so pass that context into the policy layer.

For each route, record:

1. The models allowed for its data classification and region.
2. Required capabilities such as tool calling, vision, context size, and structured output.
3. The maximum acceptable latency and cost per request.
4. The consequence of a weak answer.
5. The evaluation set and metric used to judge the route.
6. The fallback model and the conditions that force it.

These fields divide routing into two steps. Hard checks determine eligibility first. The router then chooses among models that can safely execute the request.

Amazon Bedrock makes a similar distinction in its intelligent prompt routing documentation. Configured routers use a fallback model as an anchor and response quality difference as the routing criterion. The documentation also warns that effectiveness depends on training data and may suffer on specialized use cases. Keep application-level capability and risk rules even when a provider supplies the difficulty model.

Use the strong model as the default fallback. Send work to the cheaper model only after every hard constraint passes and the score clears a tested threshold. Unknown routes, missing capability records, unsupported locales, and malformed policies should all select the fallback.

Use deterministic gates before learned scoring

Run cheap, explainable gates first so a learned router cannot choose an impossible or unacceptable target.

A policy function can follow this sequence:

choose_model(request, route_policy, model_catalog):
    candidates = models_allowed_for(request.data_class, request.region)
    candidates = candidates_with_all(route_policy.required_capabilities)

    if route_policy.risk == "high":
        return route_policy.strong_fallback, "high_risk"

    if request.context_tokens > route_policy.weak_context_limit:
        return route_policy.strong_fallback, "context_limit"

    if candidates does not include route_policy.weak_model:
        return route_policy.strong_fallback, "capability_mismatch"

    score = difficulty_router.predict(request.prompt, request.features)

    if score >= route_policy.strong_threshold:
        return route_policy.strong_fallback, "predicted_difficulty"

    return route_policy.weak_model, "eligible_low_difficulty"

Keep the model catalog in versioned configuration instead of scattering assumptions through client code. Store capability flags, context limits, regions, data approvals, provider identifiers, and exact model versions. Any deployment that changes those values should trigger routing regression tests.

Force the strong model when a routing mistake has a large consequence. That includes changing permissions, approving a payment, interpreting a contract clause for action, or generating a command that will run without review. Learned scoring is better suited to bounded, reversible work such as drafting, classification with human review, and summaries that users can inspect.

Calibrate the cost and quality threshold on local tasks

The RouteLLM paper treats routing as a choice between stronger and weaker models under a performance and cost tradeoff. Its implementation uses a threshold to change the share of requests sent to each model. That threshold is specific to the workload.

Build a labeled set from requests shaped like production traffic. Remove sensitive values while preserving task distribution, prompt length, tool use, languages, and failure cases. Run both candidate models on every request. Grade their output with the checks used by the workflow rather than a general preference score.

An extraction route might grade schema validity and field accuracy. A support route might grade category, escalation decision, and policy citations. A tool route should verify tool name, arguments, authorization preconditions, and whether the result requires human approval. Human review is useful for subjective tasks, but deterministic checks should carry the load where a contract exists.

Calculate a separate threshold for task classes that differ materially. With one global threshold, abundant easy traffic can dominate a smaller, difficult class. Report at least these values for each class:

• Weak-model pass rate.
• Strong-model pass rate.
• Router false-weak rate, where the router chose weak but only strong passed.
• Strong-route rate and cost per accepted result.
• Latency at the median and tail.
• Fallback and execution-error rates.

Set the threshold against an explicit false-weak budget. If a workflow can tolerate one weak-model miss per thousand reviewed drafts, test against that target. If the operation cannot tolerate such a miss, remove it from learned routing and force the strong path.

Keep routing separate from execution fallback

A routing decision picks the model that tries first, but that model can still fail during execution. Providers reject parameters, exhaust quota, time out, and return invalid output. Keep an execution fallback after the routing step.

Use fallback selectively. Retry on the strong model when the weak one returns an unsupported-parameter error, invalid schema, tool contract failure, or detectable low-confidence result. Do not automatically replay a side-effecting request after an ambiguous timeout. The first model call may have emitted a tool request that another component started. Such workflows need an idempotency key and a durable record of whether execution began.

Log routing and fallback as separate events. The decision record should contain the route ID, policy version, router version, candidate model versions, selected model, reason code, score band, fallback reason, latency, cost, and any later evaluation outcome. Keep raw prompts out of ordinary telemetry when they contain internal data.

Reason codes help during incidents. When strong-model usage jumps, the team can distinguish harder traffic from a broken capability catalog, a changed threshold, or repeated weak-model execution errors.

Shadow the router before it controls traffic

Let a new router make decisions without enforcing them. Continue serving the current strong model and record what the new policy would have chosen. When cost and data policy permit, run the weak model asynchronously for a sample and grade both outputs.

Shadow mode answers questions that an offline set cannot:

• Does the live task mix match the calibration set?
• Are new prompt shapes routed to the fallback?
• Does one tenant or language receive more weak routes?
• Do tool and schema failures cluster on a candidate model?
• Does the projected saving survive retries and fallbacks?

Enforce routing on a small canary only after the false-weak rate, execution errors, model mix, and cost meet the release criteria. Keep a control group on the strong model. The control separates routing effects from unrelated prompt, retrieval, or tool changes.

Tie rollback to the harm metric as well as spend. A release has failed if it reaches the cost target while doubling schema repair or human escalation.

Detect drift after launch

Traffic, prompt, tool, and model changes can make a routing policy stale. Pin exact model versions where the provider allows it. If a model alias changes behavior, treat it as a new candidate and rerun the paired evaluation.

Monitor routing by task class and policy version. Alert on sudden changes in strong-route share, fallback rate, invalid output, tool errors, reviewer rejection, or task-specific success. Sample accepted weak-model results for delayed review because structural checks will miss some defects.

Recalibrate after meaningful prompt changes, new tools, another language, a model upgrade, or a sustained shift in request distribution. Do not train only on cases the router selected. Once the policy starts avoiding weak-model failures, those failures can disappear from its dataset. Preserve a randomized evaluation sample across both routes.

Amazon notes that its intelligent routing cannot adjust decisions from application-specific performance data. Teams using that service or a self-hosted router still need local outcome measurement and a rollback switch.

Verify the complete routing path

Before rollout, test the router inside the workflow rather than as an isolated classifier.

The fixture set should cover easy and hard requests, every required capability, oversized context, unsupported locales, high-risk operations, malformed policy, unavailable weak models, invalid structured output, provider timeouts, and ambiguous side effects. Assert the selected model, reason code, fallback behavior, final result, and telemetry record for each case.

Then run these release checks:

1. Every route has an owner, policy version, strong fallback, and task-specific evaluation.
2. Every model in the catalog has tested capability metadata and an exact provider identifier.
3. The false-weak rate stays within its class-specific budget.
4. Unsupported capabilities always force an eligible model before learned scoring.
5. Execution fallback cannot duplicate side effects.
6. Shadow and canary results include a strong-model control.
7. One switch can disable learned routing without changing callers.
8. Model, prompt, tool, and router upgrades trigger regression evaluation.

Start with one low-risk, high-volume route. Build fifty to a few hundred representative fixtures and run both models. Set a false-weak budget, then shadow the decision. Add the next route only after the first has a measured quality result, working fallback, and tested rollback.

References

• Amazon Bedrock intelligent prompt routing: provider documentation for routing criteria, fallback models, benefits, and limitations.
• RouteLLM repository: implementation guidance for threshold calibration, serving, and router evaluation.
• RouteLLM paper: primary research on learned routing between stronger and weaker models under a cost and performance objective.
• RouteLLM issue 63: a practitioner report about an unsupported parameter reaching a selected local model.
• RouteLLM issue 79: a practitioner report about routing scores and the deployed model pair.

---