Prompt Caching

The objection: a router that switches models mid-conversation must be throwing away the prompt cache.

The measurement: it does not. Router plus caching beat caching alone on one fixed model on every dataset.

Evaluation	Sample	Router + caching vs caching alone
WildChat-1M simulation, general chat	30,769 multi-turn conversations	68.7% cheaper
DevGPT simulation, developer chat	1,011 conversations	46% cheaper
Real agent traces, provider cache accounting	95 sessions, 8,174 API calls	37.4% cheaper
TwinRouterBench static track	81 multi-step instances	44 to 50% cheaper

• A switch is not an eviction. When a session returns to a model it used earlier, the cache is still there.
• 4,684 real switch-backs on live gateway traffic: 97.4% found the cache warm at a 5-minute TTL, 99.3% at a 1-hour TTL.
• The expensive mistake is the opposite one. A router with caching switched off costs about 4x caching one fixed model.
• Savings accounting is honest about it. The all-frontier baseline is priced with a warm cache on continuing turns, and a fresh cache write after a switch counts against the saving. See Reported savings.

Prompt Caching Works with Auto Router

The five datasets, per-dataset cost with and without caching, the switch-back measurement, and the config.

Session affinity and deployment affinity

The two pins that hold a conversation on one model and one deployment when you want them.

When to pin anyway

• session_affinity is off by default. The numbers above show it is not needed for the cache, and pinning forfeits the savings from routing later turns down a tier.
• Turn it on when a tier switch would change behavior the client depends on, such as a long Claude Code session that should stay on one model.
• A tier with several deployments behind it also needs deployment_affinity and the prompt_caching pre-call check in router_settings, so continuing turns return to the deployment holding the cache.
• Both together: Coding agents with load balancing.

Caching across load-balanced deployments

Separate from the router: the prompt_caching pre-call check keeps Anthropic caching working when the same model is load balanced across deployments or AWS accounts.

Claude Code: prompt cache routing

Enable the prompt_caching pre-call check across duplicate deployments and confirm cache reads in the request logs.