4 posts tagged with "complexity-router"

Yes, you can use prompt caching with Auto-Routing. The two compound rather than cancel out. We measured it across five datasets, two of which report what the provider's cache actually did.

Auto routing promises a smaller bill without a worse answer. We measured both halves against a baseline that sends every request to claude-opus-5: 8,619 graded prompts and cost simulations over 14,000 real conversations.

Router plugins are now available on LiteLLM. Each plugin receives the routing context, enriches it, and hands it to the next before the router makes the final decision.

The push came from the autorouter discussion (#32168): teams wanted to layer their own signals (language detection, domain classification, tenant policy, budget caps) onto routing without waiting for each one to land in core. This plugin extension lets teams make these changes while keeping LiteLLM's routing core stable.

Auto Router v2 collapses complexity, semantic, and adaptive routing into a single auto_router/complexity_router. One config now covers heuristic scoring, LLM classification, lexical or semantic keyword rules, and Thompson-sampled tier pools.

The push came from the community. On discussion #32168, users pointed out that all three routing strategies should converge into a single Auto Router. One router with configurable signals and weights keeps the API simple while letting the routing engine evolve internally, instead of forcing you to pick a mode up front.

The operational half came from discussion #32172: predictable beats clever for debuggability. A fixed, versioned mapping from capability class to model is what makes "why did this response cost 4x today" answerable after the fact.