Learned routing policy

Learned Routing Policy (LRP) recommends the lowest-cost eligible target predicted to meet an operator-defined quality floor. Different models are good at different jobs and to different degrees. Teams establish the cheapest sufficient model mix using objective outcomes, such as unit tests, extraction accuracy, tool-call correctness, browser tasks or product acceptance tests.

What callers request

Use an allowed deployment-defined model group from Available Models and Access. The same Chat, Responses or Anthropic client request reaches the router. Operators decide which groups use a learned policy and validate each provider, dialect, tool mode and request shape independently. Choosing a group does not bypass access, capability, output-cap, budget or rate-limit checks.

The router filters targets before asking LRP for a recommendation. LRP predicts quality and output-token count for eligible targets and compares their expected costs. If no predicted quality meets the floor, it chooses the highest predicted quality. Unknown pricing cannot make a target appear free. Fallbacks remain within eligible targets. Predictions are estimates; a quality floor is an operator's selection criterion, not a guarantee that every answer passes.

Training and evaluation

The standalone lrp training CLI collects approved datasets, fans requests out to candidate targets, verifies or judges responses, builds shared features, trains per-target LightGBM models, calibrates predictions, and evaluates a held-out split. Session-based splitting prevents a conversation appearing in training and test. Optional near-duplicate embedding deduplication runs before that split so templated prompts do not leak across partitions; operators review removed counts and threshold sensitivity without exporting request content. See the operator evaluation splits note. Small or undertrained models are excluded. Model bundles bind their embedding artifacts and feature definitions for consistent training and serving.

Evaluation compares learned decisions with cheapest, anchor, weighted-random, strength-only and oracle baselines. Operators review quality, stored cost, coverage, floor violations, calibration and performance, including response duration and time to first byte. When aggregator backends expose an actual upstream serving provider, evaluation reports outcome, cost and latency variance by that identity while preserving the exact catalog model id and marking missing provider evidence separately. A cost saving is useful when workload outcomes remain acceptable. Synthetic demonstrations prove wiring and evaluate gates; provider-backed outcomes and actual embedding latency establish promotion evidence. The shipped sample configuration is the source of truth for catalog pricing evidence; live experiments refresh provider metadata before recording costs.

A complete workload-to-routing scenario

LRP trains a routing policy, not the upstream language models. Its Python training tool is packaged as a uv project with locked dependencies. LightGBM learns quality and output-token predictions; isotonic calibration adjusts the quality scores. Training and serving share the same embedding and scalar features. Training is offline: serving loads a versioned bundle and makes predictions without running a training job or asking a third-party judge for each caller request.

Choose workloads before choosing a model mix

Start with the jobs users need completed and how success will be measured. A deployment could distinguish these workloads:

Workload	Representative dataset	Acceptance criterion
Short calculations or classification	Typical and ambiguous requests with exact output constraints	Correct result or approved label
Structured extraction	Documents with missing fields, distractors and realistic lengths	Correct values as well as a valid schema
Code review or repair	Multiple files and representative tools/context	Tests pass or a reviewer records the required outcome

Match the dataset to the intended language, session lengths, system instructions, tool schemas, response formats, output caps and client dialects. Include difficult and failed cases. Hold out whole conversations so the model cannot see another turn from a test conversation during training. More examples of easy requests cannot substitute for evidence about a different workload.

Choose and validate upstream candidates

Choose plausible candidates at different costs and an anchor that defines the comparison bar. Record exact provider/model/API identities, prices, context and output limits, modalities and tool behavior. Validate each exact combination directly and through the router before adding it to a restricted calibration group. A cheap advertised price does not establish sufficient quality.

For router-based collection, use a separate single-target group per candidate and one restricted calibration caller. Each request is tried against each candidate; the outcome record includes the actual target, request ID, usage, stored price/cost, response status, latency and time to first byte. The current v1 fanout client replays OpenAI Chat and obtains exact-model prices from OpenRouter. Other dialects or price sources need independently validated collection integrations. Callers can still use other router API surfaces once operators have separately validated their learned-policy request shapes.

Use objective verifiers when available. Verifiers run in an isolated worker; missing isolation is missing evidence. Third-party judging requires approval for that content transfer, preferably using a judge outside the candidate set. Keep verifier success, relative preference and absolute rubric scores identified because they have different meanings.

Train, calibrate and evaluate

The operational sequence is:

approved workload dataset
  → collect → fanout → verify/judge → featurize → train → evaluate
  → validated bundle → shadow → staging enforcement → workload acceptance

The CLI exposes collect, fanout, judge, featurize, train, eval, validate and serve. Copy-paste command blocks, router YAML, sidecar notes, and the checked-in synthetic evidence tables are in Train And Serve Learned Routing Policy. After installing the service project with uv, each command can be invoked as uv run --project services/learned-routing-policy --locked lrp COMMAND.

Each target needs at least 200 training rows to participate in learned selection. This is a minimum sample count, not a guarantee of workload coverage. Inspect held-out coverage, calibration, quality, spend and performance against all six baselines. Review the quality-floor sweep before choosing a floor. Failed gates remain visible and prevent promotion.

Recorded case study and performance limits

The detailed learned-routing case study follows the complete synthetic workload, training, evaluation and real-process inference scenario. It includes training log excerpts, safe downloadable evidence, actual selected upstreams, per-target quality/cost explanations, the failed synthetic cost gate, and all four real BGE stage-latency measurements.

In that shared-host benchmark, the default one-thread 512-token shape exceeded the default 200 ms service deadline even at median feature latency. Keep long-input enforcement behind workload/hardware validation; short-input results do not establish whole-workload performance. See the case study for measured values, concurrent host-load conditions and implications for BT fallback.

Configurable wait for latency-tolerant workloads and tests

Operators can set top-level deadline_ms in LRP's service config from 1 to 4,500 ms; the default remains 200 ms. For a workload that accepts a longer routing wait, or a controlled inference test, deadline_ms: 500 allows more time for the learned prediction. Set the router group's external_policy.timeout_ms higher, for example 750 ms, to allow transport overhead. The router permits at most 5,000 ms. A router timeout that expires first uses the router's configured policy-error behavior rather than waiting for LRP's fallback.

This increases potential caller latency and worker occupancy. It does not add inference workers, remove admission limits, or make busy-worker fallback impossible. Test learned/fallback label counts and caller latency at the intended request sizes and concurrency. A 500 ms test allowance does not turn a failed 40 ms performance target into a pass. Restore the default deadline and the previous router timeout when reverting a test. The operator runbook shows the configuration and validation steps.

For a temporary run, lrp serve --bundle BUNDLE --config CONFIG --deadline-ms 500 overrides YAML without modifying it. The override accepts the same integer range. Omit it to use YAML, or the 200 ms default when no value is configured. The router timeout still needs to be set separately. This makes a longer inference allowance explicit for testing while preserving normal deployment defaults.

Privacy and operational behavior

LRP is trusted deployment infrastructure. Request-aware routing sends normalized content to the service only when the operator enables include_request. Configured group PII filtering runs first. LRP retains no request content in service logs; training datasets and judgments live in protected operator storage. Third-party judging requires operator approval for the content transfer. Ordinary router request logs contain metadata and cannot reconstruct prompts.

Optional exploration is restricted to explicitly configured calibration projects. Optional session pins are in-memory heuristics based on group, project, environment and first user text. Identical first prompts in a shared project can collide; pins are disabled initially. Image-bearing requests use first eligible order in v1; learned image-quality selection requires separate future validation.

Missing request content uses first eligible order with a degraded diagnostic. Deadline or embedding failure uses strength-based fallback. If no bundle target is known, the service uses first eligible order and records that condition. Policy failures produce 502 routing-policy-error when the operator selected fail-closed behavior. See Error Responses.

Staging and rollback

Native external-policy shadow mode records the learned recommendation while the router serves first eligible configured order. This baseline differs from weighted routing. Operators validate a restricted staging group before enforcement, then require workload, security, client and request-shape acceptance before broad promotion. Baseline mode bypasses LRP and preserves configured eligible order; restoring a prior weighted strategy is a separate configuration rollback.

Operators may optionally require Ed25519-signed model bundles before load or reload. Signing is deployment-owned and transparent to callers; see Operator-signed LRP bundles.

Optional selection constraints (project floors, upstream latency gates, evidence-based cache estimates, and bounded labels) are described for callers in LRP selection constraints.

Deploy the service over loopback in the router's network namespace, such as a sidecar in the same pod. A separate private service on a custom port does not satisfy the router's current egress rules. Step-by-step train, serve, and evidence tables: Train And Serve Learned Routing Policy. Policy requests are authenticated; diagnostics use a separate loopback listener, with authenticated explain/reload disabled initially. Router global metrics continue to require metrics-admin access. See Customer-Controlled Routing for the external-policy contract and contact Metrum for deployment guidance.