The Complete Guide to Routing Strategies

Routing starts with a company objective

A routing strategy is the policy that decides which model, provider, region, or service tier handles a request. The best policy is not the one with the most intelligence. It is the one that improves a result the company actually cares about while protecting the constraints it cannot violate.

Those objectives differ. A finance team may want the lowest effective cost above a quality floor. A product team may protect response time at the 95th percentile. A platform team may prioritize uptime and capacity. A security team may require that sensitive prompts stay inside an approved region. The same request can produce a different correct route under each objective.

Start by writing the objective as a constrained decision: minimize cost while quality stays above X, maximize quality while p95 latency stays below Y, or maximize completion rate using only approved deployments. That statement determines which signals the router needs, which strategies fit, and what a successful test looks like.

Separate model selection from request delivery

Routing contains two different decisions. Model-selection strategies choose the capability that should do the work. Delivery strategies choose the provider, deployment, region, or service tier that should serve an already-selected capability. A production policy often uses both, but they solve different problems.

Model selection can change answer quality, style, tool behavior, and context support, so it needs task-level evaluation. Delivery routing tries to preserve the intended capability while improving cost, speed, availability, or policy compliance. Provider endpoints still need equivalence testing because versions, quantization, rate limits, and feature support can differ.

The tables below are maps, not maturity levels. Rules are not automatically inferior to machine learning, and an ensemble is not automatically safer than one well-tested model. Choose the simplest strategy that can observe the signal and enforce the objective you defined.

Model-selection strategies at a glance

Use this table when the open question is which model or set of models should answer. Each strategy links to a detailed section below.

Rules-based and constraint routing

Rules-based routing maps explicit facts to explicit routes. A request can be routed by product feature, customer tier, data sensitivity, required tool, context length, modality, budget, or a caller-supplied task label. The strongest rules act as eligibility filters: remove every route that cannot legally or technically serve the request, then optimize among what remains.

Rules are fast, explainable, easy to test, and often the right first strategy. Their weakness is policy sprawl. Overlapping conditions create surprising precedence, and prompt length or keyword checks are poor substitutes for actual task difficulty. Version the policy, log which rule fired, and maintain a safe default for requests that match nothing.

Known implementations: LiteLLM supports tags, priorities, context-window checks, regional constraints, and custom routing logic. Portkey and Merge Gateway expose policy and conditional routing for centrally managed traffic.

Classifier-based routing

A classifier assigns a request to a discrete label such as extraction, debugging, research, simple, difficult, or high risk. The policy then maps that label to a model or candidate pool. The classifier can be code, a small language model, or a trained model, but it should cost far less and run much faster than the models it selects.

Classifier routing works when the taxonomy is stable and each class has a meaningfully different best route. Measure the confusion matrix, not just overall accuracy. A rare high-risk request misclassified as routine can matter more than many correct low-risk decisions. Add an uncertainty threshold that sends ambiguous requests to a safe default.

Known implementations: OpenRouter Auto uses a lightweight task classifier before ranking eligible models. Morph Model Router classifies difficulty, ambiguity, and domain, then applies caller-defined model and cost policies.

Semantic routing

Semantic routing converts the request into an embedding, a numeric representation of meaning, and compares it with labeled examples or route descriptions. The closest matches vote for a route. This can capture paraphrases that keyword rules miss and can be updated by adding examples instead of retraining a model.

It works best when production requests form clear, stable groups and the example library covers those groups. Similarity is not the same as difficulty, quality, or safety. Keep hard constraints outside the similarity score, inspect nearest-neighbor explanations, and define a fallback when no example is close enough.

Known implementation: RouteLLM includes a similarity-weighted ranking router that weights preference examples by their similarity to the incoming prompt. It is a framework rather than a managed gateway, so teams still own deployment and calibration.

Learned and predictive routing

A learned router predicts how each candidate model will perform on the current request, then selects the route with the highest expected utility under the company's constraints. Targets can include win probability, task score, completion probability, latency, cost, tool success, or a weighted business outcome.

This is valuable when simple labels hide important variation and the company has representative evaluation or production data. It also creates a model-management problem inside the routing layer. Labels can be noisy, new model releases shift the decision boundary, and a router trained on public preferences may not represent a private workload. Track calibration and route regret, replay traffic before updates, and retain an explicit rollback policy.

Known implementations: Not Diamond offers pretrained selection plus custom routers trained on a company's evaluation data. RouteLLM provides matrix-factorization, BERT, and language-model classifiers trained from preference data.

Pareto routing

Pareto routing keeps models that make efficient tradeoffs across two or more objectives. A model is dominated when another candidate is at least as good on every measured axis and better on one. Removing dominated options leaves a frontier of rational choices, such as cheaper models at a given quality level or faster models above a minimum score.

The company still needs to choose where on that frontier to operate. A quality floor, cost ceiling, or latency target turns the frontier into a usable policy. Benchmarks must reflect the real task mix, and the frontier must be refreshed when prices, models, or workloads change.

Known implementations: OpenRouter's Pareto Router selects coding models that meet a requested coding-score tier, then favors the cheapest available candidate or the fastest with its throughput variant. Router.com offers benchmark and cost-aware selection for managed multi-model traffic.

Cascading and escalation

A cascade starts with a cheaper or faster model, evaluates its output, and escalates only when the result fails a confidence or quality check. The validator might check a schema, run tests, verify citations, compare against known facts, or ask a separate judge model. Cascades work best when failure is easy and cheap to detect.

The hard part is knowing when the first answer is good enough. Self-reported confidence is usually weak evidence, and a judge can share the same blind spots as the model it grades. Every escalation adds delay and may pay for both models, so measure completed-task cost rather than the price of the accepted response alone.

Known implementations: General gateways such as LiteLLM and Portkey provide the model fallbacks needed to build a cascade, but quality-based escalation usually requires application-specific validators. Do not confuse this with error fallback, which changes routes after an operational failure rather than a weak answer.

Ensemble and judge routing

An ensemble sends the same task to multiple models, then uses a vote, judge, or synthesis step to choose or combine the results. This is useful when independent approaches reveal uncertainty, cover different specialties, or reduce the chance that one model's blind spot controls a high-value decision.

The strategy is expensive because it deliberately creates more work. Correlated models can agree and still be wrong, the judge can introduce its own bias, and parallel calls increase the operational surface. Reserve ensembles for tasks where the cost of a wrong answer clearly exceeds several extra completions, and evaluate the final synthesized result rather than the panel in isolation.

Known implementation: OpenRouter Fusion runs a panel of models in parallel and gives their responses to an analyst model, which reports agreement, contradictions, coverage gaps, and blind spots for a final model to use.

Delivery and operational strategies at a glance

Use this table after the model or capability has been selected. These strategies decide where, when, and under what operating policy the request should run.

Provider load balancing

Provider load balancing distributes requests across deployments that are expected to provide the same model capability. Common policies include weighted random selection, round robin, priority order, least busy, rate-limit-aware selection, and health-aware selection. The objective is usually capacity, throughput, availability, or commercial diversification rather than answer quality.

Treat provider equivalence as a testable claim. Tool calling, structured output, model versions, quantization, context limits, and safety behavior can differ. Record both the requested model and the actual provider endpoint so quality or billing changes can be traced back to the route.

Known implementations: LiteLLM supports weighted, least-busy, rate-limit-aware, latency-based, and cost-based deployment selection. OpenRouter ranks providers by price, throughput, latency, and policy constraints. Vercel AI Gateway provides managed provider routing and fallback.

Cost, latency, and capacity routing

Metric-based routing scores eligible endpoints using live or recent operating data. Lowest-cost routing favors unit economics. Latency routing favors response time. Least-busy or rate-limit-aware routing protects throughput by steering around saturated deployments. Weighted policies can blend those objectives or reserve capacity for higher-value traffic.

These metrics move quickly and can be misleading. A low token price may lose after retries, long outputs, or cache misses. A latency winner can become overloaded if every router sends it the next request. Use rolling windows, candidate buffers, circuit breakers, and an exploration share so the policy keeps learning about routes it is not currently favoring.

Known implementations: LiteLLM documents cost-based, latency-based, least-busy, rate-limit-aware, and weighted selection. OpenRouter supports provider ordering by price, throughput, and latency after applying endpoint constraints.

Reliability routing with retries and fallbacks

Reliability routing detects a failure, decides whether another attempt is safe, and chooses the next eligible endpoint or model. Retry policies handle transient errors on the same route. Provider fallback preserves the model while changing where it runs. Model fallback changes capability and therefore needs a product decision about acceptable behavior.

Bound the total attempt budget across every layer. Provider SDK retries, gateway retries, and application retries can multiply into a request storm. Non-idempotent tools need a stable operation key so a retry cannot send the same message, charge the same card, or mutate the same record twice. Alert on sustained fallback usage because a high completion rate can hide a primary route that is failing.

Known implementations: LiteLLM, Portkey, OpenRouter, and Vercel AI Gateway all expose retry or fallback controls, with different scopes and defaults.

Geographic, privacy, and compliance routing

Compliance routing filters candidates by region, deployment type, provider policy, zero-data-retention support, contract status, or data classification before any performance optimization occurs. It is the right strategy when a route that violates policy is not an acceptable tradeoff at any price or quality level.

The tradeoff is a smaller candidate pool. Regional capacity can be limited, approved endpoints can cost more, and strict residency requirements reduce failover options. Test the full data path, including router logs, caches, subprocessors, support access, and fallback behavior. A regional model endpoint does not prove that every routing component stays in region.

Known implementations: LiteLLM can filter deployments by region during pre-call checks. Portkey and Requesty expose regional routing and failover. OpenRouter provides data-policy, zero-data-retention, and in-region provider controls.

Session-affinity and cache-aware routing

Session affinity keeps related turns on the same model or provider. That preserves behavioral consistency and can improve prompt-cache hits because repeated context stays on the endpoint that already processed it. Cache-aware policies compare that benefit with the possible quality, price, or latency gain from switching.

Stickiness should be a preference, not a trap. Expire it after inactivity, break it when the route becomes unhealthy or ineligible, and record why a switch occurred. Evaluate whole-session cost and task completion, not isolated prompt prices. A cheap model switch can be a net loss if it discards a large cached prefix or destabilizes a tool-heavy workflow.

Known implementations: OpenRouter Auto and Pareto use best-effort model and provider stickiness for eligible candidates. Factory Router and Devin Adaptive describe cache state or prompt-cache economics as routing signals for coding workflows.

Service-tier routing

Service-tier routing changes the capacity class used by an already-selected model. A router can keep interactive requests on a standard tier, send background work to discounted best-effort capacity, and promote delayed work when its deadline approaches. This can lower cost without introducing the quality uncertainty of switching models.

The policy needs workload metadata that prompt text may not reveal: deadline, urgency, retry budget, idempotency, and whether partial work has side effects. Measure effective cost per completed task, tail completion time, capacity errors, retries, and promotion rate. Vendor terminology is not standardized, so verify whether a tier changes price, priority, throughput, or all three.

Known implementations: OpenRouter can select provider service tiers, and Router.com can include eligible Flex capacity in cost-efficient routing.

How strategies combine in production

Most production routers are layered policies, not one algorithm. The order matters. First apply hard eligibility constraints. Then infer task or difficulty when needed. Score the remaining models against the business objective. Choose a provider using live operating conditions. Apply session affinity if the previous route remains eligible. Finally, execute a bounded retry or fallback plan.

A cost-focused support assistant might filter to approved regions, classify the request, use a learned quality estimate to choose a model above the quality floor, prefer a cached provider, select Flex only for background summaries, and fall back on rate limits. Each layer should emit its inputs, candidate set, decision, and policy version so operators can explain the final route.

Avoid optimizing every layer at once. Add one decision boundary, compare it with a fixed-route baseline, and establish rollback conditions before making the next layer adaptive. That keeps a quality regression attributable instead of turning the route into an opaque chain of guesses.

1. Filter by hard constraints
2. Classify the task or estimate difficulty
3. Score eligible models against the objective
4. Select a provider, region, and service tier
5. Apply session or cache preference
6. Execute bounded retries and fallbacks
7. Log the decision and final outcome

Evaluate and roll out a routing strategy

Build the evaluation around the objective statement from the first section. Use representative prompts, model every attempt in the request chain, and compare against fixed-model, cheapest-model, strongest-model, and random baselines. For learned policies, add an oracle estimate to show how much opportunity remains if the router predicted perfectly.

Track quality by task segment, effective cost per completed task, p50 and p95 latency, completion rate, retry and escalation rates, cache-hit rate, policy violations, and route regret. Route regret is the gap between the chosen route and the best eligible route known after evaluation. Aggregate averages can hide a router that saves money on easy traffic while failing a small but valuable segment.

Replay historical traffic first, then shadow live requests without changing user-visible behavior. Move a small traffic share only after thresholds are set, and keep a one-step rollback to a known fixed route. Routing is an operating process: model releases, prices, capacity, and workload mix keep changing after the first launch.

• Write the primary objective and hard constraints before selecting a strategy
• Log candidates, exclusions, scores, selected route, policy version, attempts, and final outcome
• Evaluate complete tasks and sessions, not only individual model responses
• Segment results by task, customer tier, risk level, and route
• Set expansion and rollback thresholds before looking at production results