What Is Model Routing?

Model routing is the practice of choosing, for each request, which language model should handle it. Easy requests go to a small, fast, cheap model; hard ones go to a larger and more expensive one; some go to a model that fits a special requirement, such as one hosted in a particular region. The aim is to keep answer quality where it needs to be while paying less and responding faster on average.

How Model Routing Works

There are four common methods, often combined:

  • Rules. Code routes on known attributes: the feature that sent the request, the user’s plan, the input length, whether the data may leave a region. Simple, predictable and easy to test.
  • Classifier. A small model or a trained classifier reads the request and predicts whether a cheaper model will answer it well. RouteLLM, an open research framework, trains such routers on human preference data and reports cost reductions of more than two times in some cases without losing response quality (Ong et al., RouteLLM).
  • Cascade. The request goes to a cheap model first; if a check on its answer fails, it is sent to a stronger one. FrugalGPT studied this approach and reports matching the best individual model of its time with up to 98% lower cost on its benchmarks (Chen et al., FrugalGPT).
  • Fallback. If a provider is down, rate-limited or slow, the request goes to another provider or model. This is routing for availability rather than cost.

Results from papers depend on the benchmarks and the models of the time. A team should measure the savings on its own traffic.

Routing vs the Routing Workflow Pattern

Anthropic’s guide to agent design also uses the word routing for a workflow pattern: a classifier sends each input to a specialised prompt or process, for example refunds to one path and technical questions to another, and it notes the same pattern can send easy questions to smaller models (Building effective agents). Model routing is the special case where the branches differ by model rather than by prompt or tools.

When to Use It and When Not

Model routing pays off when traffic is high and mixed: many short, simple requests alongside fewer hard ones. It adds little when traffic is low, when every request is hard, or when the cost of a wrong answer is far above the cost of the stronger model. In regulated settings, routing can also enforce where data is processed, for instance sending requests with personal data only to a model deployed in the required region. See private LLM and data residency.

Common Failure Modes

  • Router errors hidden in averages. Overall quality looks fine while one category of hard questions is consistently sent to the weak model. Evaluate per category.
  • Prompts tuned for one model. A prompt that works on one model may fail on another. Each route needs its own tested prompt.
  • Inconsistent answers. The same question gets different answers depending on the route. For facts and figures, the answer should come from data and tested code, not from whichever model was chosen.
  • Router cost. A classifier call on every request can eat the savings if the router is itself a large model.

In Practice

Routing starts with plain rules on projects delivered through our LLMOps services: rules are transparent and usually capture most of the savings, and a learned router is added only when the data shows a clear gain. Each route has its own evaluation set from real requests, and the route taken is logged on every trace (see AI observability), so traffic share, cost and quality can be compared route by route. Vendors are treated as interchangeable where possible, with OpenAI, Anthropic, Google, Azure and open-weight models behind one interface, which also gives a fallback path when a provider has an outage.

Back to Glossary

Let's talk about your project

We'll help you harness AI and ML technologies to create innovative software solutions

File requirements: pdf, doc, docx, rtf, ppt, pptx — max 15 MB

No file selected