1.102.0rc1 - Auto Router Controls, Native OCR & Gateway Reliability
Deploy this version​
- Docker
- Pip
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:1.102.0-rc.1
pip install litellm==1.102.0rc1
The published GitHub tag is v1.102.0-rc.1. These notes compare it with v1.101.0-rc.2. Changes already shipped in RC2 are omitted
Breaking Changes
These callouts cover changes to behavior available in v1.100.1, the latest stable release
Organization endpoints now require an enterprise license. Unlicensed calls to organization APIs return 403 after authentication. Enable the license before using organization APIs. See PR #40613
Responses IDs are authorized regardless of their format. Provider IDs this proxy did not issue are refused by default for retrieval, cancellation, deletion, and chaining. Set allow_unmanaged_response_ids only on deployments that intentionally accept unmanaged IDs. See PR #39548
New model writes no longer accept pricing overrides through model_info. Put custom pricing in litellm_params. Clients that echoed displayed catalog rates into model_info will now keep following the catalog. See PR #36222
Supported OCR uses the native engine by default, and Rust controls are now process-wide. Replace LITELLM_USE_RUST_OCR and per-request or deployment rust settings with LITELLM_RUST or litellm.rust(bool). Use LITELLM_RUST=0 for the Python OCR path. See PR #40734, PR #39928
Auto Router replaces the caller’s output cap with the selected tier model’s limit by default. Per-tier limits still take precedence. Set max_tokens_from_tier_model: false to retain the caller’s cap. See PR #40209
Editing multiple heuristic-v1 routers requires the auto-router license. Existing snapshots and unchanged defaults remain valid; one changed router is available without the license. A second tuned router is refused. See PR #39952, PR #40007
Prompt Security blocks file-modification verdicts by default. Set block_on_file_modify: false if your workflow intentionally accepts rewritten files. See PR #38204
Existing Bedrock and Azure passthrough requests enforce the resolved model’s access rules. Relays must use models granted to the key and team. Update grants for integrations that previously relied on broader passthrough access. See PR #39660, PR #39863
MCP requests enforce server and end-user tool grants more strictly. Initializing with no granted MCP servers returns 403, denied served-MCP tools are no longer silently omitted, and end-user tool permissions constrain listing and calls. Grant the required servers, access groups, and tools. See PR #39234, PR #40616, PR #40865
Policy pipeline ordering is now deterministic. Pipelines run from global to team, key, tag, then model scope, followed by explicitly requested policies. Review workflows whose result depended on the previous unspecified order. See PR #39697
Marengo 2.7 embedding usage reports request and image counts instead of estimated tokens. Update consumers that depended on nonzero token usage. Billing now uses request, image, and duration rates. See PR #40180
MAI image requests validate count and size instead of silently ignoring them. Use n: 1 and supported generation dimensions. Image edits do not support size; enable drop_params to omit unsupported options. See PR #40074
GPT-5.1, GPT-5.2, GPT-5.4, and GPT-5.4 Pro no longer advertise minimal reasoning effort. This includes their dated snapshots. Use a supported effort or drop_params for requests that previously sent minimal. See PR #40581
Headerless Azure Realtime clients use the GA upstream by default. Beta clients must send OpenAI-Beta: realtime=v1 or pin realtime_protocol / LITELLM_AZURE_REALTIME_PROTOCOL. Health checks follow the GA default. See PR #40769
Very large local token counts become sampled estimates. The default exact-count ceiling is 4,000,000 characters per string. Set TOKEN_COUNTER_MAX_EXACT_CHARS to choose a different limit. See PR #40186
Model health checks enforce deployment visibility and limit returned fields. Explicit out-of-scope targets return 403, team-specific deployments follow routing permissions, and results contain an allowlist of display fields. Update monitoring clients that depended on the previous responses. See PR #40765
The existing lite login --config-claude command now writes a static gateway token. Claude Code settings previously used apiKeyHelper to retrieve the login token dynamically. Rerun lite login --config-claude when the stored login key expires. See PR #40330
Key Highlights​
- Auto Router controls and visibility: custom heuristic dimensions, editable scoring weights, an optional NON_REASONING tier, per-tier output limits, healthier fallbacks, and routed-model/session-savings feedback for coding agents
- Native OCR and expanded provider endpoints: native OCR execution across supported providers, Meta Muse Voice realtime transcription, Mistral text-to-speech, Vertex Lyria music, and native Fireworks Responses
- Gateway reliability: optional shared PgBouncer connections and a spend collector, fewer database and Redis calls, stable Redis outage handling, and request/token-based autoscaling controls
- MCP, logging, and guardrails: schema-discovery proxy mode, improved OAuth compatibility and permission enforcement, configurable OTel trace URLs and HTTP/JSON export, PointFive logging, Conduct Guard, and broader post-call pipeline coverage
- 99 new model catalog entries: additions across Bedrock, Azure AI, OpenAI, Fireworks, OpenRouter, Together AI, and other providers, alongside pricing and capability corrections
New Providers and Endpoints​
Expanded provider endpoint support​
| Provider | Endpoint | What you can do |
|---|---|---|
| Meta | /v1/realtime | Stream Muse Voice transcription using OpenAI-compatible realtime events |
| Fireworks AI | /v1/responses | Use Fireworks’ native Responses API, including server-side MCP and chaining |
| Mistral | /v1/audio/speech | Generate speech with Voxtral TTS and supported voice-cloning inputs |
| Vertex AI | /v1/audio/speech | Generate music with Lyria models |
| Hosted vLLM | /v1/images/edits | Use image editing through hosted vLLM deployments |
New Models / Updated Models​
New Model Support (99 new models)​
Counts represent new catalog identifiers, including aliases and regional variants. Prices below are the values bundled in this release, in USD; runtime pricing-map reloads can update them. Input and output columns show base token rates; long-context, cache, image-token, and other specialized rates depend on the model
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features / special pricing |
|---|---|---|---|---|---|
| Azure AI | azure_ai/codex-mini | 200,000 | $1.5 | $6 | Responses; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input; System messages; Parallel tools |
| Azure AI | azure_ai/cohere-command-a | 131,072 | $2.5 | $10 | Chat; Tool calling; Tool choice |
| Azure AI | azure_ai/gpt-6-astra | 922,000 | $10 | $50 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input; Computer use; Web search; System messages; Parallel tools |
| Azure AI | azure_ai/gpt-chat-latest | 272,000 | $5 | $30 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input; Web search; System messages; Parallel tools |
| Azure AI | azure_ai/grok-4-20-non-reasoning | 262,000 | $1.25 | $2.5 | Chat; Vision; Tool calling; Tool choice; Structured output; Web search |
| Azure AI | azure_ai/grok-4-20-reasoning | 262,000 | $1.25 | $2.5 | Chat; Reasoning; Vision; Tool calling; Tool choice; Structured output; Web search |
| Azure AI | azure_ai/model-router | 200,000 | $0.14 | $0 | Chat |
| Azure AI | azure_ai/whisper | - | - | - | Transcription; input_cost_per_second: $0.0001; output_cost_per_second: $0.0001 |
| Amazon Bedrock | bedrock/us-gov-east-1/anthropic.claude-fable-5-1 | 1,000,000 | $12 | $60 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | bedrock/us-gov-east-1/anthropic.claude-opus-5 | 1,000,000 | $6 | $30 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | bedrock/us-gov-east-1/nvidia.nemotron-nano-9b-v2 | 128,000 | $0.072 | $0.276 | Chat; System messages |
| Amazon Bedrock | bedrock/us-gov-west-1/amazon.nova-2-multimodal-embeddings-v1:0 | 8,172 | $0.162 | $0 | Embeddings; Audio input; input_cost_per_audio_per_second: $0.000168; input_cost_per_image: $7.2e-05; input_cost_per_video_per_second: $0.00084 |
| Amazon Bedrock | bedrock/us-gov-west-1/amazon.nova-lite-v1:0 | 300,000 | $0.072 | $0.288 | Chat; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input |
| Amazon Bedrock | bedrock/us-gov-west-1/amazon.nova-micro-v1:0 | 128,000 | $0.042 | $0.168 | Chat; Tool calling; Tool choice; Prompt caching; Structured output |
| Amazon Bedrock | bedrock/us-gov-west-1/anthropic.claude-fable-5-1 | 1,000,000 | $12 | $60 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | bedrock/us-gov-west-1/anthropic.claude-opus-5 | 1,000,000 | $6 | $30 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | bedrock/us-gov-west-1/nvidia.nemotron-nano-9b-v2 | 128,000 | $0.072 | $0.276 | Chat; System messages |
| Amazon Bedrock | eu.twelvelabs.marengo-embed-3-0-v1:0 | 500 | - | $0 | Embeddings; input_cost_per_audio_per_second: $0.00014; input_cost_per_image: $0.0001; input_cost_per_query: $7e-05; input_cost_per_video_per_second: $0.0007 |
| Amazon Bedrock | global.cohere.embed-v4:0 | 128,000 | $0.12 | $0 | Embeddings |
| Amazon Bedrock | global.twelvelabs.pegasus-1-2-v1:0 | - | - | $7.5 | Chat; input_cost_per_video_per_second: $0.00049 |
| Amazon Bedrock | twelvelabs.marengo-embed-3-0-v1:0 | 500 | - | $0 | Embeddings; input_cost_per_audio_per_second: $0.00014; input_cost_per_image: $0.0001; input_cost_per_query: $7e-05; input_cost_per_video_per_second: $0.0007 |
| Amazon Bedrock | us.cohere.embed-v4:0 | 128,000 | $0.12 | $0 | Embeddings |
| Amazon Bedrock | us.twelvelabs.marengo-embed-3-0-v1:0 | 500 | - | $0 | Embeddings; input_cost_per_audio_per_second: $0.00014; input_cost_per_image: $0.0001; input_cost_per_query: $7e-05; input_cost_per_video_per_second: $0.0007 |
| Amazon Bedrock | global.openai.gpt-6-astra | 1,050,000 | $10 | $50 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching |
| Amazon Bedrock | us-gov.anthropic.claude-3-haiku-20240307-v1:0 | 200,000 | $0.3 | $1.5 | Chat; Vision; Tool calling; Tool choice; Structured output; PDF input |
| Amazon Bedrock | us-gov.anthropic.claude-fable-5-1 | 1,000,000 | $12 | $60 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | us-gov.anthropic.claude-opus-5 | 1,000,000 | $6 | $30 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | us-gov.nvidia.nemotron-nano-12b-v2 | 128,000 | $0.24 | $0.72 | Chat; Vision; System messages |
| Amazon Bedrock | us-gov.nvidia.nemotron-nano-3-30b | 262,144 | $0.072 | $0.288 | Chat; Tool calling; Tool choice; System messages |
| Amazon Bedrock | us-gov.nvidia.nemotron-nano-9b-v2 | 128,000 | $0.072 | $0.276 | Chat; System messages |
| Amazon Bedrock | us-gov.nvidia.nemotron-super-3-120b | 256,000 | $0.18 | $0.78 | Chat; Reasoning; Tool calling; Tool choice; System messages |
| Amazon Bedrock | us-gov.openai.gpt-oss-120b-1:0 | 128,000 | $0.18 | $0.72 | Chat; Reasoning; Tool calling; Tool choice; Structured output |
| Amazon Bedrock | us-gov.openai.gpt-oss-20b-1:0 | 128,000 | $0.084 | $0.36 | Chat; Reasoning; Tool calling; Tool choice; Structured output |
| Amazon Bedrock | us-gov.xai.grok-4.6 | 500,000 | $2.64 | $7.92 | Chat; Reasoning; Vision; Tool calling; Tool choice |
| Amazon Bedrock | us.openai.gpt-6-astra | 1,050,000 | $11 | $55 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching |
| Bedrock Mantle | bedrock_mantle/openai.gpt-6-astra | 1,050,000 | $11 | $55 | Responses; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output |
| Bedrock Mantle | bedrock_mantle/openai.gpt-daybreak-blue-5.6-sol | 1,050,000 | $5.5 | $33 | Responses; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output |
| Bedrock Mantle | bedrock_mantle/us-gov-east-1/openai.gpt-oss-120b | 131,072 | $0.18 | $0.72 | Chat; Reasoning; Tool calling; Tool choice; Structured output; Parallel tools |
| Bedrock Mantle | bedrock_mantle/us-gov-east-1/openai.gpt-oss-20b | 131,072 | $0.084 | $0.36 | Chat; Reasoning; Tool calling; Tool choice; Structured output; Parallel tools |
| Bedrock Mantle | bedrock_mantle/us-gov-east-1/xai.grok-4.6 | 500,000 | $2.64 | $7.92 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output |
| Bedrock Mantle | bedrock_mantle/us-gov-west-1/google.gemma-4-26b-a4b | 256,000 | $0.156 | $0.48 | Chat; Reasoning; Vision; Tool calling; Tool choice |
| Bedrock Mantle | bedrock_mantle/us-gov-west-1/google.gemma-4-31b | 256,000 | $0.168 | $0.48 | Chat; Reasoning; Vision; Tool calling; Tool choice |
| Bedrock Mantle | bedrock_mantle/us-gov-west-1/google.gemma-4-e2b | 128,000 | $0.048 | $0.096 | Chat; Reasoning; Vision; Tool calling; Tool choice |
| Bedrock Mantle | bedrock_mantle/us-gov-west-1/openai.gpt-oss-120b | 131,072 | $0.18 | $0.72 | Chat; Reasoning; Tool calling; Tool choice; Structured output; Parallel tools |
| Bedrock Mantle | bedrock_mantle/us-gov-west-1/openai.gpt-oss-20b | 131,072 | $0.084 | $0.36 | Chat; Reasoning; Tool calling; Tool choice; Structured output; Parallel tools |
| Bedrock Mantle | bedrock_mantle/us-gov-west-1/xai.grok-4.6 | 500,000 | $2.64 | $7.92 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output |
| Cerebras | cerebras/qwen-3.8-27b | 65,536 | $0.99 | $1.49 | Chat; Reasoning; Vision; Tool calling; Tool choice; Structured output; Parallel tools |
| ChatGPT subscription | chatgpt/gpt-5.5 | 1,050,000 | - | - | Responses; Vision; Tool calling; Structured output; Parallel tools; Subscription access; no per-token price in catalog |
| ChatGPT subscription | chatgpt/gpt-5.6-luna | 1,050,000 | - | - | Responses; Vision; Tool calling; Structured output; Parallel tools; Subscription access; no per-token price in catalog |
| ChatGPT subscription | chatgpt/gpt-5.6-sol | 1,050,000 | - | - | Responses; Vision; Tool calling; Structured output; Parallel tools; Subscription access; no per-token price in catalog |
| ChatGPT subscription | chatgpt/gpt-5.6-terra | 1,050,000 | - | - | Responses; Vision; Tool calling; Structured output; Parallel tools; Subscription access; no per-token price in catalog |
| Cohere | rerank-v4.0-fast | 32,768 | $0 | $0 | Rerank; input_cost_per_query: $0.002 |
| Cohere | rerank-v4.0-pro | 32,768 | $0 | $0 | Rerank; input_cost_per_query: $0.0025 |
| DeepSeek | deepseek-flash | 1,000,000 | $0.3 | $1.2 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; System messages; Parallel tools; Assistant prefill |
| DeepSeek | deepseek/deepseek-flash | 1,000,000 | $0.3 | $1.2 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; System messages; Parallel tools; Assistant prefill |
| Fireworks AI | fireworks_ai/accounts/fireworks/models/deepseek-v4p1-flash | 1,048,576 | $0.22 | $0.66 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output |
| Fireworks AI | fireworks_ai/deepseek-v4p1-flash | 1,048,576 | $0.22 | $0.66 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output |
| Gemini | gemini/lyria-3.5 | 1,048,576 | $0 | $0 | Chat; Audio output; output_cost_per_image: $0.08 |
| Inception | inception/mercury-2.5 | 260,000 | $0.2 | $0.75 | Chat; Tool calling; Tool choice; Structured output; System messages |
| Meta | meta/muse-voice-transcribe-1.0 | - | - | - | Realtime transcription; Audio input; input_cost_per_second: $5e-05 |
| OpenAI | gpt-image-2.5-flare | - | $5 | - | Image generation; Vision; PDF input; input_cost_per_image_token: $8e-06; output_cost_per_image_token: $3e-05 |
| OpenAI | gpt-image-2.5-flare-2026-09-08 | - | $5 | - | Image generation; Vision; PDF input; input_cost_per_image_token: $8e-06; output_cost_per_image_token: $3e-05 |
| OpenAI | gpt-image-2.5-sunburst | - | $5 | - | Image generation; Vision; PDF input; input_cost_per_image_token: $8e-06; output_cost_per_image_token: $3e-05 |
| OpenAI | gpt-image-2.5-sunburst-2026-09-08 | - | $5 | - | Image generation; Vision; PDF input; input_cost_per_image_token: $8e-06; output_cost_per_image_token: $3e-05 |
| OpenAI | gpt-live-1 | - | - | - | Realtime; Tool calling; Audio input; Audio output; input_cost_per_second: $0.000833333333333 |
| OpenRouter | openrouter/deepseek/deepseek-v4.1-flash | 1,048,576 | $0.15 | $0.6 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output |
| OpenRouter | openrouter/openai/gpt-5.6-luna-pro | 1,050,000 | $0.2 | $1.2 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input |
| OpenRouter | openrouter/openai/gpt-5.6-sol | 1,050,000 | $2 | $10 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input |
| OpenRouter | openrouter/openai/gpt-5.6-sol-pro | 1,050,000 | $2 | $10 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input |
| OpenRouter | openrouter/openai/gpt-5.6-terra-pro | 1,050,000 | $2 | $12 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input |
| OpenRouter | openrouter/openai/gpt-6-astra-pro | 1,050,000 | $10 | $50 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input |
| OpenRouter | openrouter/openai/gpt-chat-latest | 400,000 | $5 | $30 | Chat; Vision; Tool calling; Tool choice; Prompt caching; Structured output; PDF input |
| OpenRouter | openrouter/qwen/qwen3.8-max-0902 | 1,000,000 | $2 | $6 | Chat; Reasoning; Vision; Tool calling; Tool choice; Prompt caching; Structured output |
| Together AI | together_ai/MiniMaxAI/MiniMax-M2.7 | 196,608 | $0.3 | $1.2 | Chat |
| Together AI | together_ai/Qwen/QwQ-32B | 131,072 | $1.2 | $1.2 | Chat |
| Together AI | together_ai/Qwen/Qwen3-Coder-Next-FP8 | 262,144 | $0.5 | $1.2 | Chat |
| Together AI | together_ai/Qwen/Qwen3-VL-32B-Instruct | 262,144 | $0.5 | $1.5 | Chat |
| Together AI | together_ai/Qwen/Qwen3-VL-8B-Instruct | 262,144 | $0.18 | $0.68 | Chat |
| Together AI | together_ai/deepseek-ai/DeepSeek-R1-0528 | 163,840 | $3 | $7 | Chat |
| Together AI | together_ai/mistralai/Ministral-3-14B-Instruct-2512 | 262,144 | $0.2 | $0.2 | Chat |
| Together AI | together_ai/mistralai/Mistral-7B-Instruct-v0.3 | 32,768 | $0.2 | $0.2 | Chat |
| Together AI | together_ai/moonshotai/Kimi-K2.5-fp4 | 262,144 | $0.5 | $2.8 | Chat |
| Together AI | together_ai/moonshotai/Kimi-K2.6 | 262,144 | $1.2 | $4.5 | Chat |
| Together AI | together_ai/nvidia/NVIDIA-Nemotron-Nano-9B-v2 | 131,072 | $0.06 | $0.25 | Chat |
| Together AI | together_ai/zai-org/GLM-5 | 202,752 | $1 | $3.2 | Chat |
| Together AI | together_ai/zai-org/GLM-5.1 | 202,752 | $1.4 | $4.4 | Chat |
| Vertex AI | vertex_ai/gemini-3.5-live-translate-preview | - | $3.5 | $21 | Realtime; Audio input; Audio output; input_cost_per_audio_token: $3.5e-06; output_cost_per_audio_token: $2.1e-05 |
| Vertex AI | vertex_ai/lyria-002 | - | - | - | Audio generation; Audio output; output_cost_per_image: $0.06 |
| Vertex AI | vertex_ai/lyria-3-clip-preview | 131,072 | $0 | $0 | Audio generation; Audio output; output_cost_per_image: $0.04 |
| Vertex AI | vertex_ai/lyria-3-pro-preview | 131,072 | $0 | $0 | Audio generation; Audio output; output_cost_per_image: $0.08 |
| Vertex AI | vertex_ai/xai/grok-4.3 | 200,000 | $1.25 | $2.5 | Chat; Reasoning; Vision; Tool calling; Tool choice; Structured output |
| Vertex AI | vertex_ai/xai/grok-4.6 | 524,288 | $2 | $6 | Chat; Reasoning; Vision; Tool calling; Tool choice; Structured output |
| Voyage | voyage/voyage-multilingual-2 | 32,000 | $0.12 | $0 | Embeddings |
| Weights & Biases | wandb/deepseek-ai/DeepSeek-V4-Pro-0813 | - | $1.31 | $3.96 | Chat; Reasoning; Prompt caching |
| Weights & Biases | wandb/ibm-granite/granite-4.2-8b | - | $0.1 | $0.15 | Chat; Reasoning; Prompt caching |
| xAI | xai/grok-imagine-video | - | - | - | Video generation; input_cost_per_image: $0.002; output_cost_per_second: $0.05; output_cost_per_second_480p: $0.05; output_cost_per_second_720p: $0.07 |
| xAI | xai/grok-imagine-video-1.5 | - | - | - | Video generation; input_cost_per_image: $0.01; output_cost_per_second: $0.08; output_cost_per_second_1080p: $0.25; output_cost_per_second_480p: $0.08; output_cost_per_second_720p: $0.14 |
| xAI | xai/grok-imagine-video-1.5-2026-05-30 | - | - | - | Video generation; input_cost_per_image: $0.01; output_cost_per_second: $0.08; output_cost_per_second_1080p: $0.25; output_cost_per_second_480p: $0.08; output_cost_per_second_720p: $0.14 |
| xAI | xai/grok-imagine-video-1.5-preview | - | - | - | Video generation; input_cost_per_image: $0.01; output_cost_per_second: $0.08; output_cost_per_second_1080p: $0.25; output_cost_per_second_480p: $0.08; output_cost_per_second_720p: $0.14 |
A dash means the catalog does not define that token-based price or context field. Non-token pricing fields above use their named billing unit
Updated pricing​
| Provider / model | Changed token prices (USD per 1M tokens) |
|---|---|
bedrock_mantle/openai.gpt-oss-20b | Input: $0.075 to $0.07 |
bedrock_mantle/openai.gpt-oss-safeguard-20b | Input: $0.075 to $0.07; Output: $0.3 to $0.2 |
deepseek-v4-flash | Input: $0.44 to $0.3; Output: $1.32 to $1.2; Cache read: $0.014 to $0.006 |
deepseek-v4-flash-vision-exp | Input: $0.44 to $0.3; Output: $1.32 to $1.2; Cache read: $0.014 to $0.006 |
deepseek/deepseek-v4-flash | Input: $0.44 to $0.3; Output: $1.32 to $1.2; Cache read: $0.014 to $0.006 |
deepseek/deepseek-v4-flash-vision-exp | Input: $0.44 to $0.3; Output: $1.32 to $1.2; Cache read: $0.014 to $0.006 |
eu.anthropic.claude-3-5-haiku-20241022-v1:0 | Input: $0.25 to $0.8; Output: $1.25 to $4; Cache read: $0.025 to $0.08; Cache write: $0.3125 to $1 |
eu.twelvelabs.marengo-embed-2-7-v1:0 | Input: $70 to not set |
gpt-5.4-pro | Cache read: $3 to not set |
gpt-5.4-pro-2026-03-05 | Cache read: $3 to not set |
gpt-5.5-pro | Cache read: $3 to not set |
gpt-5.5-pro-2026-04-23 | Cache read: $3 to not set |
jina-reranker-v2-base-multilingual | Input: $0.018 to $0.05; Output: $0.018 to $0 |
openrouter/deepseek/deepseek-chat-v3.1 | Input: $0.2 to $0.25; Output: $0.8 to $0.95; Cache read: not set to $0.13 |
openrouter/deepseek/deepseek-v4-flash | Input: $0.08778 to $0.0854; Output: $0.17556 to $0.1708; Cache read: $0.017556 to $0.01708 |
openrouter/deepseek/deepseek-v4-pro | Input: $1.32 to $0.859908; Output: $3.96 to $1.719816; Cache read: not set to $0.071659 |
openrouter/deepseek/deepseek-v4-pro-0813 | Input: $1.32 to $0.57948; Output: $3.96 to $1.73844; Cache read: not set to $0.019316 |
openrouter/google/gemma-4-26b-a4b-it | Input: $0.07 to $0.042; Output: $0.34 to $0.22 |
openrouter/moonshotai/kimi-k2.7-code | Input: $0.66 to $0.71; Output: $3.4 to $3.5; Cache read: $0.18 to $0.15 |
openrouter/moonshotai/kimi-k3 | Input: $3 to $2.1; Output: $15 to $10.53; Cache read: $0.3 to $0.235 |
openrouter/qwen/qwen3-14b | Input: $0.12 to $0.2275; Output: $0.24 to $0.91 |
openrouter/qwen/qwen3-235b-a22b-2507 | Input: $0.0875 to $0.22; Output: $0.35 to $0.88 |
openrouter/qwen/qwen3-30b-a3b-instruct-2507 | Input: $0.04815 to $0.09; Output: $0.19305 to $0.3 |
openrouter/qwen/qwen3-next-80b-a3b-instruct | Input: $0.1 to $0.09 |
openrouter/qwen/qwen3.5-122b-a10b | Input: $0.29 to $0.26; Output: $2.4 to $2.08 |
openrouter/qwen/qwen3.5-35b-a3b | Input: $0.25 to $0.3125; Cache read: not set to $0.15625 |
openrouter/z-ai/glm-4.6 | Input: $0.55 to $0.43; Output: $2.2 to $1.75; Cache read: not set to $0.08 |
openrouter/z-ai/glm-5.2 | Input: $0.966 to $0.6; Output: $3.036 to $2; Cache read: $0.1932 to $0.15 |
openrouter/z-ai/glm-5.3-flash | Input: $0.075 to $0.15; Output: $0.25 to $0.5; Cache read: $0.015 to $0.03 |
together_ai/Qwen/Qwen2.5-72B-Instruct-Turbo | Input: not set to $1.2; Output: not set to $1.2 |
together_ai/Qwen/Qwen2.5-7B-Instruct-Turbo | Input: not set to $0.3; Output: not set to $0.3 |
together_ai/mistralai/Mistral-7B-Instruct-v0.1 | Input: not set to $0.2; Output: not set to $0.2 |
together_ai/mistralai/Mistral-Small-24B-Instruct-2501 | Input: not set to $0.1; Output: not set to $0.3 |
twelvelabs.marengo-embed-2-7-v1:0 | Input: $70 to not set |
us.twelvelabs.marengo-embed-2-7-v1:0 | Input: $70 to not set |
vertex_ai/gemini-3.5-transcribe-preview | Input: $2.5 to $2 |
vertex_ai/mistral-small-2503 | Input: $1 to $0.1; Output: $3 to $0.3 |
vertex_ai/mistral-small-2503@001 | Input: $1 to $0.1; Output: $3 to $0.3 |
vertex_ai/openai/gpt-oss-20b-maas | Input: $0.075 to $0.07; Output: $0.3 to $0.25; Cache read: not set to $0.007 |
vertex_ai/xai/grok-4.20-non-reasoning | Input: $2 to $1.25; Output: $6 to $2.5 |
vertex_ai/xai/grok-4.20-reasoning | Input: $2 to $1.25; Output: $6 to $2.5 |
The registry also updates capability flags, context/output limits, non-token rates, and deprecation dates. No model entries were removed
Amazon Bedrock​
- Stop leaking Anthropic
thinking/reasoning_effortinto DeepSeek Converse requests - PR #33409 - Preserve Bedrock’s
x-amzn-RequestIdon chat errors - PR #40089 - Support TwelveLabs Marengo Embed 3.0 on Bedrock and bill Marengo embeddings by requests, images, and duration - PR #40180
- Pass deployment-configured
aws_session_tagsto STS AssumeRole for Bedrock and SageMaker - PR #40446
Anthropic​
- Skip one-shot Claude Code cache injection - PR #40175
- Key the /v1/messages prompt cache on Claude Code's session_id only - PR #40342
Azure AI​
- Route Azure AI passthrough calls to the configured deployment, enforce model access, and track spend across supported endpoint types - PR #39863
- Validate MAI image count and size options; unsupported values return 400 or are dropped with
drop_params- PR #40074
Azure OpenAI​
- Honor
DEFAULT_MAX_RETRIESwhen creating Azure SDK clients - PR #40464
Bedrock Mantle​
- Validate Bedrock Mantle
reasoning.summary; unsupported values are dropped only whendrop_paramsis enabled - PR #40798
DashScope​
- Remap chat-shaped api_base to the live rerank route - PR #39237
Databricks​
- Translate
reasoning_effortcorrectly for Databricks-hosted Gemini 2.5 models - PR #32786, PR #40909 - Keep top-level reasoning_content from OpenAI-compatible gateway models - PR #40449
- Route Databricks Unity model services through AI Gateway - PR #40492
Fireworks AI​
- Call Fireworks’ native Responses API, including server-side MCP and response chaining - PR #39826
- Fold instructions and developer items into one leading system message on the Responses path - PR #40268
- Keep reasoning_content on replayed assistant messages - PR #40682
General​
- Drop unsupported Responses reasoning parameters when
drop_paramsis enabled, while preserving supported reasoning models - PR #38842
Meta​
- Stream Meta Muse Voice transcription through the Realtime API, with binary PCM input, turn completion, and duration-based billing - PR #39395
Mistral​
- Use Mistral Voxtral text-to-speech through
/v1/audio/speech, with voice mapping, binary audio responses, voice cloning, and per-character billing - PR #38755
Model catalog and pricing​
- Add Bedrock GPT-6 Astra, GPT Image 2.5, Cohere Rerank 4, Vertex Grok and live translation, Lyria, Voyage, ChatGPT, and xAI video catalog entries; correct web-search fees and model capability metadata - PR #31884
- Add missing Bedrock GovCloud pricing, including in-region and cross-region inference model IDs - PR #39764
- Add azure_ai/gpt-6-astra Foundry pricing - PR #39983
- Price Azure AI Foundry catalog aliases and charge the model-router fee once - PR #40189
- Correct Jina reranker pricing and OpenAI reasoning-effort metadata, and add OpenRouter GPT-5.6 Sol - PR #40581
- Registry audit rolling PR: deepseek-flash, gpt-live-1, xAI/Groq deprecation dates, Perplexity Nemotron reasoning - PR #40606
- Add Cerebras, Inception, Together AI, and OpenRouter catalog entries; correct existing prices and Bedrock OpenAI reasoning capabilities - PR #40740
- Add DeepSeek V4.1 Flash on Fireworks - PR #40812
- Update Azure and Together AI deprecation dates and correct Computer Use and OpenRouter capability metadata - PR #40855
- Preserve reasoning support for unmapped OpenAI-family model IDs through capability fallbacks - PR #40902
OpenAI​
- Remove tool-schema regex patterns that OpenAI cannot compile while preserving the rest of the schema - PR #40485
Oracle Cloud Infrastructure​
- Keep stable OCI streaming response IDs and avoid duplicate Cohere tool-call answers - PR #39507, PR #39965
Vertex AI​
- Generate Lyria music through
/v1/audio/speech, with correct audio formats and music pricing - PR #30856 - Support fine-tuned Gemini endpoints in managed batches - PR #39668
- Return 400 with a clear parameter error for unsupported Vertex AI reasoning efforts - PR #40748
Voyage​
- Accept a single string or flat list of documents for Voyage contextual embeddings, while preserving explicit chunking options - PR #35091
Weights & Biases​
- Preserve reasoning_effort in chat completions - PR #39190
- Treat unmapped W&B models as reasoning-capable while allowing exact catalog entries to override the fallback - PR #40625
LLM API Endpoints​
Agent-to-Agent​
- Forward caller identity headers on message/send and message/stream - PR #40305
Chat Completions, Responses, and Messages​
- Handle empty choices and missing roles when building responses, including partial streams used for fallback - PR #37781, PR #40294
- Preserve refusal text and refusal stop reasons when translating Responses or Chat Completions to Anthropic Messages - PR #39723
- Keep provider_specific_fields off the native /v1/messages wire - PR #39967
- Keep provider id and metadata on Responses API bridged chat completions - PR #39981
- Keep background polling alive after the client disconnects - PR #40114
- Keep reasoning_effort for mode: responses bridge deployments - PR #40249
- Keep mid-conversation system messages in input instead of folding them into instructions - PR #40269
- Replay OpenAI encrypted reasoning byte for byte through the /v1/messages bridge - PR #40451
- Echo a named tool_choice in the Responses API shape on the chat-completions bridge - PR #40462
- Preserve hosted web search calls - PR #40828
- Accept supported deferred stream-logging formats on native routes - PR #40869
Files​
- Delete Bedrock files through signed requests to configured buckets and preserve deployment credentials and managed file IDs during deletion - PR #40161
Image Edits​
- Add image edit support - PR #40329
OCR​
- Preserve native OCR provider errors and run post-call logging hooks - PR #40061, PR #40154
- Run supported OCR through the native engine by default for Mistral, Azure Mistral, Azure Document Intelligence, Reducto, Vertex Mistral, and Vertex DeepSeek, with bounded document fetching and provider-specific authentication; use
LITELLM_RUST=0for the Python path - PR #40502, PR #40507, PR #40509, PR #40530, PR #40532, PR #40533, PR #40534, PR #40535, PR #40734
Realtime​
- Dial Azure's GA realtime upstream for GA clients - PR #40769
Search​
- Propagate GET provider HTTP errors - PR #40779
Vector Search and retrieval​
- Expose vector-store retrieval failures in responses; opt into request failure with
vector_store_search_failure_mode: error- PR #39516 - Scope emulated file_search to the request's vector stores - PR #39972
- Inject Headroom retrieval tools only for service-declared cache hashes and preserve assistant content blocks - PR #39974
Management Endpoints / UI​
Admin UI​
- Show batch IDs, partial-success counts, models, reasoning tokens, and cost breakdowns in Request Logs, and attribute batch spend to organizations - PR #39626
- Keep Playground conversations and session state when refreshing Virtual Keys, and hide Create Key from view-only roles - PR #39991
- Add key-scoped auto-router usage tab - PR #39999
- Send empty vector_stores when the last team vector store is removed - PR #40144
- Make automatic auto-router setup discoverable and show what it configured - PR #40146
- List the ChatGPT subscription provider in the Add Model form - PR #40170
- Guard playground cost metric against null and NaN - PR #40257
- Repair pass-through delete confirm dialog and disable delete for config endpoints - PR #40303
- Scope shadow eval models to configured chat groups - PR #40488
- Jump straight to the last Request Logs page instead of advancing one page - PR #40644
- Open related users, teams, organizations, and creators directly from entity tables across Keys, Teams, Deleted Keys, Deleted Teams, Memory, and Prompts - PR #40646, PR #40647, PR #40749, PR #40750, PR #40751, PR #40752, PR #40753
- Search Key Activity by key alias, key hash, user id, or email - PR #40652
- Show loading state instead of stale rows while a table search is pending - PR #40656
- Search, sort and role filter for the team member table - PR #40659
- Clarify blank TPM/RPM hint on budget modals - PR #40697
- Persist cleared selections and default choices across shared selectors and local forms - PR #40795, PR #40826
- Make the env-credential login warning banner dismissible - PR #40831
Authentication and management​
- Cascade-delete JWT key mappings when their virtual key is deleted - PR #33703
- Let internal users read request/response for their own spend logs - PR #35448
- Keep model edits from turning displayed catalog prices into permanent deployment overrides; set custom prices through
litellm_params- PR #36222 - Report model limits from the resolved deployment in
/v1/models, including aliased models - PR #39296 - Check ownership for every addressed Responses API ID; unmanaged provider IDs require
allow_unmanaged_response_ids- PR #39548 - Apply key and team model allowlists to Bedrock passthrough requests - PR #39660
- Apply team model aliases on the JWT auth path - PR #39985
- Disable shared environment-credential login with
general_settings.disable_env_credential_loginand guide admins through account setup - PR #40116 - Show Create Vector Store only to proxy admins - PR #40148
- Let authorized internal users open vector store details - PR #40150
- Keep a body litellm_session_id in SpendLogs under missing_session_id omit - PR #40379
- Ignore team_id="" on /key/update so team-less keys can be updated and imported - PR #40421
- Resolve team-scoped auto-routers by their public name - PR #40432
- Retain metadata when retrieving public team aliases - PR #40554
- Require an enterprise license for organization endpoints - PR #40613
- Support explicit project detachment - PR #40836
- Persist clearing user model budgets - PR #40837
- Allow non-admin users to search key aliases by substring within their authorized scope - PR #40907
Coding agents and skills​
- Run the pi coding agent through the gateway with
lite pi, including model limits and key-based access - PR #36841 - Search accessible hosted skills by semantic similarity through
GET /v1/skills?query=...and theskill_searchMCP tool - PR #39401 - Configure Claude Code, Codex, or both with a gateway key and reversible settings; restore Claude settings with
lite unconfigure claude- PR #40319, PR #40829 - Let the apiKeyHelper supply Claude Code's key under lite claude - PR #40489
- Register HTTPS ZIP archives as Claude Code marketplace plugin sources, with an optional SHA-256 checksum - PR #40496
- Expose gateway models under Claude-compatible IDs for Claude Code model discovery and requests - PR #40515
- Grant key- and team-scoped access to private Claude Code marketplace plugins through Allowed Skills - PR #40518
- Set
litellm_settings.public_skills_index: trueto publish all stored skills through a public Agent Skills index and downloadable archives - PR #40770
Terraform management​
- Read team per-model rpm/tpm limits from metadata and clear them on removal - PR #40439
- Recover Terraform state when a virtual key no longer exists, including cascade deletion - PR #40443, PR #40880
- Encode Terraform per-model key budgets in the structure the proxy accepts - PR #40450
- Allow custom team_id on litellm_team - PR #40459
- Recompute Terraform-managed key expiry when its duration changes - PR #40511
- Preserve prior Terraform key state when an update is rejected - PR #40512, PR #40527
- Read key settings stored in metadata back into Terraform state - PR #40513
- Preserve server-side key metadata that Terraform does not manage - PR #40514
AI Integrations​
Guardrails​
- Block Prompt Security file-modification verdicts by default; set
block_on_file_modify: falseto permit rewriting - PR #38204 - Apply post-call policy pipelines to supported Responses and streaming paths, including background retrieval, text and tool-call rewrites, and legacy post-call hooks - PR #38721, PR #38788, PR #39233, PR #40271, PR #40274, PR #40284
- Resolve generateContent routes and async-first passthrough call types - PR #38869
- Apply key and team guardrails to MCP tool calls - PR #39629
- Run attached policy pipelines in deterministic order: global, team, key, tag, then model - PR #39697
- Stop logging the request payload as guardrail_response on pre_call hooks - PR #39699
- Add inspect_embeddings toggle for AIM and Cato - PR #39918
- Detect short credentials assigned to credential-named fields without changing the configured entropy thresholds - PR #40190
- Log expected skip and deny events below WARNING - PR #40208
- Preserve guardrail results and scan identifiers in telemetry, including streamed responses and request rewrites - PR #40211, PR #40327, PR #40806, PR #40882
- Allow framework-supported logging-only mode - PR #40267
- Pass original request object to post-call guardrail hooks - PR #40414
- Scan and rewrite Responses custom_tool_call output items - PR #40461
- Fail closed with a named error when a Responses input rewrite cannot be applied - PR #40609
- Log only scan time as streaming post_call guardrail duration - PR #40760
- Add Conduct Guard through YAML and the Admin UI, including supported hooks, request parameters, and warning/advisory telemetry; requires
conduct-litellm-guard>=0.2.5- PR #40785
Logging​
- Initialize string success/failure callbacks at startup after config load - PR #38226
- Export batched request logs to PointFive from YAML or the Admin UI, with message redaction on successes and failures - PR #38509
- Surface runtime-registered callbacks in UI Logging page - PR #38974
- Prevent streaming span leaks in MLflow logging and restore compatibility with MLflow 2.x trace completion - PR #39049
- Finish response metadata before synchronous logging callbacks read it - PR #39869
- Split Azure Sentinel log batches below ingestion limits and retry oversized batches - PR #39880
- Include the upstream provider request ID in failure logs - PR #40045
- Bucket latency by input sequence length - PR #40059
- Give each call in a session header its own trace instead of upserting one trace per session - PR #40177
- Freeze refreshable credentials before signing and retry 403 uploads with a fresh signature - PR #40187
- Configure the complete OTel v2 trace export URL and HTTP/JSON protocol through environment settings or the Admin UI - PR #40286, PR #40290
- Accept non-string callback settings in
default_team_settings- PR #40458 - Export team max and remaining budget gauges to the Metric API - PR #40542
- Keep call_type and request start time on failed-request spend logs - PR #40558
- Keep tool call and result structure under redaction and emit tool output tokens - PR #40666
- Name Langfuse traces from the langfuse_trace_name header or metadata.trace_name - PR #40793
Secret Managers​
- Encrypt virtual keys stored in AWS Secrets Manager with a customer-managed KMS key using
kms_key_id- PR #40475
Spend Tracking, Budgets and Rate Limiting​
- Honor deployment-specific custom prices for OCR calls - PR #36609, PR #40767
- Don't reserve budget on token counting routes - PR #36718
- Attach v3 priority rate limit headers on /v1/messages - PR #37228
- Record spend for native Responses API WebSocket sessions - PR #38856
- Allow editing soft budget on existing keys - PR #39002
- Bill realtime reasoning tokens nested in text_tokens once - PR #39850
- Account a batch's cost once, from the first retrieve that sees it final - PR #39980
- Price caching savings on the billed request basis - PR #40160
- Itemize auto-router classification spend - PR #40168
- Estimate cache-read, cache-write, and reasoning-token costs in
/cost/estimate, including custom pricing and consistent rate breakdowns - PR #40174 - Expose the loaded pricing-map revision and ETag through the cost-map APIs and Price Data Reload page - PR #40179
- Show auto-router classification rate - PR #40192
- Compare auto-router targets by deployment identity - PR #40206
- Recover key alias for session tokens from spend logs - PR #40275
- Keep team member budget enforced at the cap and across Redis counter expiry - PR #40304
- Emit internal user budget webhook alerts - PR #40396
- Prevent spend counter double counting - PR #40572
- Skip priority headers that cannot be encoded in HTTP responses instead of failing Anthropic Messages requests - PR #40636
- Reset linked end-user budgets correctly - PR #40639
- Price recovered tokens when a /v1/messages client disconnects mid-stream - PR #40766
- Persist cleared budget caps and reset intervals and show the cleared values when forms reopen - PR #40895
MCP Gateway​
- Validate and bind per-user OAuth credentials to the authenticated caller when identity binding is enabled, including credential refresh - PR #38724
- Forward staged credentials on /mcp-rest/test/connection like /test/tools/list - PR #38806
- Apply agent MCP grants and toolsets, and return a clear 403 when a request names a gateway MCP server the key cannot access - PR #39234
- Start the named server's OAuth directly for a resource-scoped gateway flow - PR #39933
- Add opt-in per-server oauth relay discovery - PR #39936
- Honor explicit
nullvalues when updating MCP toolsets - PR #40022 - Show inherited MCP servers on the internal user editor and flag access groups with no members - PR #40036
- Accept on_violation block and alert for mcp_security - PR #40155
- Encrypt stored static headers and stdio environment - PR #40164
- Discover and call tools on demand through
/mcp/proxyusingsearch_tools,get_tool_schema, andcall_tool, with existing MCP access controls - PR #40298 - Emit completed MCP proxy-call logs with the correct upstream metadata - PR #40337
- Log proxy tool dispatch exceptions - PR #40351
- Surface connection failures across transports - PR #40359
- Log upstream request method, body and response on tool-list and OAuth2 token failures - PR #40440
- Challenge and scope gateway-owned server authentication - PR #40453
- Report resolved upstream authentication in debug headers - PR #40454
- Preserve dotted MCP tool argument names - PR #40494
- Refresh MCP tool previews after connection edits and require explicit credentials when changing the destination origin - PR #40498
- Respect optional discovery capabilities and quiet unsupported methods - PR #40525
- Write failure spend log for guardrail-blocked /mcp-rest/tools/call - PR #40555
- Reject initialize with 403 when the key grants no MCP servers - PR #40616
- Accept VS Code OAuth registration callbacks - PR #40664
- Check OpenAPI specifications without native MCP handshakes - PR #40665
- Return actionable OAuth-registration errors and bound MCP discovery retries - PR #40679
- Restore MCP catalog provider logos - PR #40781
- Cache upstream discovery lists - PR #40790
- Use gateway authentication for root discovery - PR #40791
- Match per-server OAuth metadata issuers - PR #40808
- Enforce end-user MCP tool permissions on both tool listing and execution - PR #40865
Performance / Loadbalancing / Reliability improvements​
Auto Router​
- Preserve existing heuristic-v1 router tuning across upgrades and enforce the free one-router editing quota consistently - PR #39952, PR #40007, PR #40140
- Build semantic routing configuration without blocking the event loop - PR #39954
- Restore adaptive-routing scores from both their initial prior and persisted learning - PR #39955
- Use deployment
model_infopricing when scoring adaptive routes by cost - PR #39957 - Add declarative custom dimensions to the heuristic scorer - PR #40156
- Edit built-in and custom heuristic dimensions in the dashboard, rebalance weights, and opt into match-count scoring - PR #40205
- Set auto-routed output limits from the selected tier model; retain caller limits with
max_tokens_from_tier_model: false- PR #40209 - Add an optional
NON_REASONINGauto-router tier belowSIMPLE, enabled withenable_non_reasoning_tier- PR #40273 - Discard incompatible encrypted reasoning when switching auto-router tiers so the request can continue - PR #40280
- Show the routed model and session savings in Claude Code and Codex, with key-scoped data from
GET /auto_router/session- PR #40330 - Refresh auto-router reasoning presets for supported model families - PR #40341
- Resolve routing candidates consistently, including provider-qualified patterns, team routes, and default deployments - PR #40491
- Classify the actual coding-agent task, including encrypted delegated tasks, without unrelated Codex envelopes or Claude Code system text - PR #40599, PR #40608, PR #40655
- Record the exact classifier input and masked source request for auto-router diagnostics - PR #40604
- Fall back to a healthy default model when the selected auto-router tier cannot serve the request - PR #40757
- Expose auto-router tier, cause, score, and reasoning effort in response headers - PR #40792
- Skip hosted-web-search requests in shadow evaluation to avoid incomplete comparisons - PR #40827
- Restore inherited compression settings when an override is cleared - PR #40839
Deployment, caching, and database reliability​
- Authenticate to Amazon ElastiCache using IAM credentials - PR #38413
- Recover database indexes left invalid by an interrupted migration - PR #39384
- Clean up the full migration process tree after a startup migration timeout - PR #39509
- Share worker database connections through optional PgBouncer, including rotating RDS IAM/Azure Entra tokens and the spend collector - PR #39683, PR #40623, PR #40660
- Serve metrics through a dedicated process or sidecar in Helm, AWS Terraform, and GCP Cloud Run deployments - PR #40163, PR #40614
- Keep cost-based routing data separate from latency samples - PR #40225
- Honor idle-connection defaults, CA bundles, and SSL settings in componentized database deployments - PR #40285, PR #40428, PR #40815
- Reduce repeated database and Redis work in authentication, budget reservation, and post-call spend tracking - PR #40371, PR #40593, PR #40834, PR #40841
- Bound concurrent key and spend database lookups with
PROXY_DB_LOOKUP_MAX_CONCURRENCYto reduce overload during cache misses - PR #40387 - Opt into request-per-second and token-per-second gateway autoscaling in the supported Helm and Terraform deployments - PR #40479
- Offload spend tracking to an optional pod-local collector; workers resume local processing if the collector is unavailable - PR #40545
- Reconnect the Prisma writer when its database session becomes read-only - PR #40610
- Keep Redis circuit breakers stable during outages, count pool-wait timeouts, reduce repeated logs, and use available local routing counters - PR #40620, PR #40624, PR #40764
- Separate virtual-key caching from other management objects and make management-cache capacity configurable - PR #40713, PR #40725
- Run migrations when the Prisma command is available as a Python module but absent from PATH - PR #40768
- Attribute requests rejected before dispatch to the correct endpoint in cache analytics - PR #40824
Load balancing and retries​
- Handle deployments with no recorded latency samples without failing routing - PR #39970
- Share in-flight counts, failure thresholds, and cooldowns across workers so load balancing and deployment recovery stay consistent - PR #40009, PR #40025, PR #40224
- Skip a deployment that refused a retryable request across router entrypoints - PR #40014, PR #40306
- Keep provider response headers on streaming chat completions - PR #40091
- Rank streaming latency routes using raw time to first token - PR #40202
- Use configured weight, RPM, or TPM from any deployment in simple-shuffle routing - PR #40222
- Prevent deployment tags from changing request tag routing during retries and fallbacks - PR #40226
- Keep request-specific routing strategies from accumulating global callbacks - PR #40229
- Route streaming requests by a chosen time-to-first-token percentile using
ttft_percentile- PR #40352
Request handling and runtime​
- Honor string-valued
drop_paramssettings loaded from YAML or database deployments - PR #33738 - Serve one deployment under several ingress prefixes with
SERVER_ROOT_PATHS, including correctly prefixed MCP OAuth discovery - PR #35935 - Register skill injection when the proxy starts, avoiding callback side effects from SDK imports - PR #38914
- Restore the standalone AI Gateway release-image build - PR #39523
- Restore secure upstream WebSocket connections in the Rust gateway - PR #39530
- Return proper nullable error fields on non-LLM routes instead of the literal string
"None"- PR #39536 - Reduce repeated database writes and duplicate rows for model health checks - PR #39539
- Keep spend-log flush notifications local to their worker event loop - PR #39556
- Move remote media and template fetching, plus asynchronous Bedrock request signing, off inference workers’ event loops - PR #39839, PR #40270, PR #40311
- Load database-backed credentials before reconciling models on each worker - PR #39876
- Control optional Rust execution through
litellm.rust(bool)orLITELLM_RUST; request-level controls are removed - PR #39928 - Redact provider credentials from passthrough failure tracebacks - PR #39964
- Allow spend-log partition maintenance to run within its configured statement timeout - PR #40098
- Log the budget-reservation configuration notice once at startup - PR #40167
- Bound concurrent token counting and move large counts off the event loop; strings above
TOKEN_COUNTER_MAX_EXACT_CHARSuse sampled estimates - PR #40186 - Load the first remote pricing map synchronously and run failed-fetch retries in the background - PR #40350
- Batch gateway request-count rollups to reduce database writes - PR #40362
- Start
liteCLI commands without downloading the remote pricing map - PR #40372 - Optionally count supported budget-admission input tokens in Rust, with Python fallback for unsupported requests - PR #40381
- Record unknown-model failures with a fixed model label and sanitized stored error text - PR #40622, PR #40820
- Prioritize liveness and core inference routes in request dispatch - PR #40687
- Load provider passthrough routes on demand while preserving route precedence - PR #40691
- Track Bedrock passthrough streaming usage incrementally, avoiding full-response buffering - PR #40724
- Scope health checks to accessible models and access groups, and return only approved display fields - PR #40765
- Resolve included files in bucket-hosted proxy configurations - PR #40772
- Expose authenticated gateway memory summaries on the data plane, including pod identity and Linux memory readings without psutil - PR #40773
- Extend optional Rust budget-admission counting to supported
cl100k_basetokenizers - PR #40777 - Deduplicate matched policy attachments in one pass - PR #40883
PR roll-up by ownership area​
Customer-visible PRs: 317
- Performance: 92
- Models & Providers: 40
- Auth & Management: 37
- LLM API Endpoints: 31
- MCP: 29
- Guardrails: 24
- UI: 24
- Spend / Budgets / Rate Limits: 22
- Logging: 17
- Secret Managers: 1
New Contributors​
- @Atharva-Kanherkar
- @CaptainAni187
- @ZXT-zjbiliy
- @clement-paradex
- @dclarksymmetry
- @gym-cmd
- @haydster7
- @jon-walton
- @joshua-berri
- @jrlprost
- @kerry-berri
- @krth1k
- @shotsan
- @yinonkahta-p5