Evaluate on Your Traffic

Public benchmarks measure someone else's prompts. Two features measure yours: a shadow evaluation before you switch, and savings accounting after.

Shadow evaluations

• Samples a slice of one key's, team's, or user's live traffic.
• Duplicates each sampled request through the router. The shadow response never reaches the client.
• Judges blind: an LLM compares the router's answer with the one the current model served.
• Runs up to 30 days (default 7) or up to the turn cap (default 200, max 2,000).
• Compares several router configs in one job on the same sampled traffic, so tier map and classifier choices are settled before anything changes.
• On our own traffic: 88.1% matched or beat the current model, 143 judged turns, $1.55 judge spend.

Shadow Evaluations: Test the Auto-Router on Your Own Production Traffic

Setting up a job, defaults for turns and duration, choosing the judge, reading the results card.

Route on Context Size and Modality

Multi-config shadow jobs and the key, team, and user targets.

Savings after you switch

• Per request: what the router picked, why, and what the same request would have cost on the most expensive model in the hardest configured tier. The difference, net of any classifier call, is stamped on the request.
• Rolled up: into the daily spend tables, so it shows per key, team, tag, and organization.
• Usage tab: total estimated savings, sessions and turns, prompt-cache hit rate by turn type. A 30-day window over 400k sessions reads in 38 ms.
• Classifier cost: returned per request in the x-litellm-classifier-cost header when an LLM classifier ran.
• Honest baseline: priced with a warm cache on continuing turns; a fresh cache write after a switch counts against the saving, so a single request can read negative. Formula and every surface: Reported savings.

AutoRouter: Easy Visibility to Your Savings

The usage tab, per-request classifier cost, and preset matching against your own deployments.

Decision log

One greppable line per routing decision: cause, tier, score, signals, routed model.

Reading a single decision

• Each routed request in the logs opens with a routing-decision card: tier, cause (heuristic score, keyword match, LLM classifier, or session pin), and the model that served it.
• Test Routing in the Add Model form shows the same card, so a surprising production decision can be replayed against the form with the same prompt.