Auto Router Feature History

Every release links to its GitHub release and full release notes. Newest first. A feature listed under a version is available from that version onward.

Coming Next

Merged into main after v1.103.0-rc.1 was cut; these changes are not included in that release candidate. Use a build containing the linked PRs until they appear under a tagged release below.

Breaking Changes

Native context compaction is enabled by default. For supported requests with full conversation history, summarize older turns near the selected model's input limit so it can keep serving the request. This works across Chat Completions, Responses, and Messages, requires an eligible native compactor, and adds a separately billed call. Set context_compaction: false to disable it. Stored or opaque native history remains client-managed. #42074

Context-window escalation becomes opt-in. Omitting enable_context_window_escalation no longer moves a long request to a larger tier automatically. Set it to true to preserve automatic escalation. #41872

• Configurable Heuristic v2 threshold. Set heuristic_v2_success_threshold in configuration or the dashboard to control the minimum predicted success rate for tier selection. Clear it to restore the trained artifact's default. #42252
• JEV dashboard setup and connection checks. Configure JEV beside the LLM classifier, preserve its settings through edits, and test the classifier separately from tier models. Paid routing previews enforce virtual-key budgets; encrypted delegated tasks use the configured fallback without a JEV charge. #41886, #41879
• Capability and Fuse forecast details. Request logs show Capability's solve chance and threshold, or both Fuse solvers' chances and the quality gap. Forecasts remain visible when health or modality rules change the final placement. #42057
• More advanced settings in the dashboard. Edit heuristic keyword overrides, housekeeping routing, reminder markers, plan-mode sentinels, output-token caps, and custom-classifier timeouts. Saving unrelated changes preserves these settings. #42293
• Internal-user savings and usage. Admins can inspect an internal user's Savings and Auto-router usage tabs, including traffic attributed through JWTs. Other users can view their own Savings; router usage remains admin-only. #42026
• Prompt-cache affinity across moving breakpoints. With the prompt_caching pre-call check enabled, deployment affinity survives Claude Code moving its cache breakpoint to a newer turn, helping preserve the provider's cached prefix. #42080

v1.103.0-rc.1 (release candidate)

GitHub pre-release, Release notes

Includes the Auto Router improvements from the 1.103.x development builds through this release candidate. Earlier development builds may not include every change below.

• Capability classification. classifier_type: capability forecasts whether the efficient solver can finish the whole task and routes to the capable tier when the estimate falls below the configured threshold. #41270
• Fuse v2 classification (experimental). classifier_type: llm_v2 uses one judge call to forecast both solvers, then selects between them using a quality-gap policy. Invalid forecasts and provider errors fall back to the capable solver. #41272
• Capability and Fuse dashboard setup. Configure efficient, capable, and judge models in dedicated tabs, with forecast policy controls and advanced routing options. Saving either form disables adaptive routing, context-window escalation, and escalation keywords. #41315, #41371
• Fuse model and harness presets. Select maintained solver and harness descriptions in YAML, the API, or the dashboard, preview their text, or replace them with custom descriptions. #41617
• TypeSafe JEV classifier. classifier_type: jev uses JEV to select a complexity tier and records its probabilities, confidence, and classifier cost. Failures use the existing classifier fallback and circuit breaker. #41615
• Per-model Fast mode. Toggle Fast beside reasoning effort for supported tier models, independently for each model. #41282
• Model affinity within each tier. With deployment affinity enabled, a session reuses its chosen eligible model when it returns to a tier. Redis shares those choices across workers while tier reclassification remains active. #41174
• Team-member router management. Admins can enable /auto_router/manage for a team so members can create routers and edit their own configurations, using models they are allowed to access. #41175
• More accurate savings estimates. Baseline costs use durable cache-prefix history and expiry, preserve observed costs before routing diverges, and mark unavailable estimates as unknown. Anthropic Fast mode and geographic pricing modifiers are also included. #41177, #41341
• Heuristic v2 score visibility. Routing details show estimated success rates, the threshold, and the predicted tier even with message logging disabled. The prediction stays separate from the final tier after routing overrides. #42001
• Coding-agent session feedback. LLM API keys can read their own router session stats. Claude Code shows the recorded served model, and both Claude Code and Codex label actual spend with the router's name. #41116, #41186
• CLI command names. Use lite autoroute start and lite autoroute stop; up and down remain deprecated aliases. #41672
• Capability and Fuse license limits. One Capability router and one Fuse v2 router are available without the auto_router entitlement. Additional routers of either type require it. #41326

v1.102.0

GitHub release, Release notes

• Harness-aware classification. Omit Claude Code system text and strip Codex reminder envelopes from classification while preserving delegated tasks and the original routed request. #40655, #40599
• Encrypted delegated tasks. Preserve encrypted task blocks in native OpenAI or Azure OpenAI Responses classifier calls. Unsupported deployments and decryption errors follow classifier_fallback. #40608
• Classifier input logs. Inspect the provider-bound classifier input, masked originating request, and classifier response separately in the Classify row. #40604
• Heuristic v1 tuning. One tuned router stays editable without the auto_router license feature. #39952
• Faster semantic cold start. Build the first route layer once, off the event loop. #39954
• Adaptive router fixes. Read model pricing from model_info and preserve bandit priors across restarts. #39957, #39955
• Cross-provider tool history. /v1/messages can replay tool_use blocks across OpenAI and Anthropic tiers. #39967

Also in this release: declarative custom heuristic dimensions and dashboard weight editing #40156, #40205; an optional NON_REASONING tier #40273; tier-model output limits #40209; healthy-default fallback when a tier cannot serve the request #40757; routed model and session savings in Claude Code and Codex #40330; and tier, cause, score, and reasoning effort in response headers #40792.

Harness-aware routing update

v1.101.0

GitHub release, Release notes

• Heuristic classifiers. heuristic_v2 routes locally; hybrid calls the LLM near a tier boundary. #39276, #39403. Post
• Context and user-turn routing. Fit oversized prompts to a tier and classify only new user turns when configured. #38844, #38861, UI #39042, #39054
• Image routing. Send images to vision-capable tiers and optionally let the classifier read them. #39032, #39454, #39825, UI #39059, #39840
• Stall escalation. Move a request up one tier when an agent repeats tool calls or errors. #39809. Post
• Classifier controls. Set classifier reasoning effort, a total timeout, and a circuit breaker for repeated timeouts. #39372, #39696, #39701
• Tier failover and compression. Use a live peer when a tier is cooled down and choose compression per routing or model hop. #39675, #39823. Post
• Shadow eval targeting. Target teams, users, and model groups, compare up to four routers, and judge tool-call turns. #39015, #39028, #39817, #39818, #39828
• Setup and prompt editing. Configure all tiers from existing models, edit built-in prompts by section, and set session affinity TTL in the UI. #39679, #39688, #39693
• Presets. Add the 1M Context preset, update OpenAI Family, and serve the catalog at runtime. #39412, #39490, #39396, #39797
• Claude Code support. Route subagents through the selected router, expose its mode in /v1/models, and bill routing embeddings to the caller. #39239, #39619, #39532
• Breaking changes. The auto_router license feature meters customization, and shadow eval results rename key fields. #39468, #39674, #39015

Posts: Route on Context Size and Modality, Mid-Task Stall Escalation, Per-Hop Compression.

v1.100.0

GitHub release, Release notes

• Custom tier sets. Define your own tiers for the LLM classifier, preview the exact classifier prompt, keyword rules follow renames. #38602, #38603, #38605.
• Heuristic-first chaining. classifier_type: heuristic_first scores locally and calls the LLM classifier only when needed. #38428.
• Classifier context budget. A character budget across turns replaces the per-turn 200-character clip. #38141, #38145.
• Housekeeping prompts skip the classifier. Client housekeeping messages go to the cheapest tier with no classifier call. #38598.
• Gemini Family preset; per-tier reasoning effort in the Lite and Anthropic presets. #38138, #38482, #38490.
• Dry-run validation before save. The UI validates a config against /auto_router/validate_complexity_router_config; /auto_router/test_routing accepts a real request body. #38595.
• Tier-pinned reasoning effort wins. A tier's reasoning_effort supersedes client carriers; unsupported tier params are dropped instead of failing the tier. #38622, #38698.
• Classifier cost counted. Savings figures, benchmarks, and shadow evals net out the router's own classifier charge. #38835, #38631.
• Router health from its models. A router is flagged when a tier, default, or classifier model cannot serve. #37966, #38174.
• model_group_alias works for auto-routers. #38272, #38382.
• Breaking. Settings placed outside complexity_router_config are rejected (#38570). router_model_name is gone, use return_raw_model_name (#38429). autorouter_savings_baseline_model is deleted; each router derives its baseline from its hardest tier (#38700).

v1.99.0

GitHub release, Release notes

• Operator-defined tier sets for the LLM classifier. #37226.
• Custom classifier plugins. classifier_type: custom with a dotted path to your own classify(). #37249.
• Plan-mode tier floor for coding-agent clients. #37230.
• Per-tier litellm_params and per-model reasoning effort in the tier editor. #37064, #37673.
• Business classification rubric preset. #37534.
• Lite preset (mixed provider) and heuristic scorer settings in the UI. #37068, #37216.
• Shadow evals: several keys per job, budget in dollars. api_key_ids replaces api_key_id, max_budget replaces max_turns (breaking). #37251, #37555.
• Responses API input routed through the auto-router. #37333.
• Savings to callbacks and per key. Per-request savings reach logging callbacks; a Savings tab on the key page. #37894, #37693.

v1.98.0

GitHub release, Release notes

• Shadow evaluations. Sample one key's live traffic, replay it through the router without serving the response, blind LLM judge, reverse mode, /v1/messages and /v1/responses. #36587, #36830, #36865. UI #36588, #36994. Post.
• Calibrated classifier rubric with worked examples, selectable per router; system prompt text no longer scored. #36578, #36721.
• Deployment affinity toggle in the UI; models shown under each tier in the benchmark chart. #36302, #36291.
• Tag routing gates. Required-AND tag prefix, allow_fail_open, untagged requests bypass a tagged pre-routing strategy. #36193, #36627, #36628.

v1.97.0

GitHub release, Release notes

• Deployment affinity on by default (breaking). A session returning to a model group lands on the deployment it used before, so the provider cache stays warm. deployment_affinity: false restores the old behavior. #36146.
• Session affinity off by default, exposed in the UI. #35714.
• Savings and usage tab. Net auto-router savings on Cost Optimization, baseline derived from the hardest tier, per-session rollup, turns per tier. #35522, #35995, #35521, #35907, #35910, #36209. Post.
• Classifier cost per request in routing_decision and the x-litellm-classifier-cost header. #36015.
• 1-click presets and Test Routing. Add Auto Router is name plus template; Test Routing shows the pick before saving; presets match deployments by underlying model ID. #35746, #35859, #35972, #36111. Post.
• Replaceable classifier prompt and tier names. #35855, #35893.

v1.96.0

GitHub release, Release notes

• Classifier context window. The LLM classifier sees prior turns (classifier_context_window_size, default 3). #35185. Post.
• Assistant turns optionally included (classifier_context_include_assistant_turns). #35471.
• Routing decision recorded. Tier, cause, and classifier request body in spend logs and the log drawer; the router's own classifier calls are marked. #35016, #35164, #35300, #35304.
• Auto-routers get their own tab in Models + Endpoints, with the context window fields. #35009, #35315, #35500.

v1.95.0

GitHub release, Release notes

• return_raw_model_name. Put the picked model in the response body model field instead of the alias. #33875.
• Logs show the router. The log drawer and session sidebar mark requests an auto-router served. #34434.

v1.94.0

GitHub release, Release notes

• Auto Router v2. Complexity, semantic, and adaptive routing in one auto_router/complexity_router. Post.
• Router plugins. Router(plugins=[...]), resolvable from proxy config. #32972, #33251, #33644. Post.
• Tier pools. Soft-floor adaptive mode and random-pick multi-model tiers. #32947, #32967.
• Session affinity. Pin a session to its first-turn model. #33126, #33500, #33723.
• Escalation keywords and per-tier semantic keyword prompts. #33656, #33508.
• Cost Optimization page (beta) with an Autorouter tab. #33899.
• Test Connection for the auto router. #32950, #33146.