LiteLLM is moving to Rust 🦀Read the latest updates.›

1.107.0rc1 - Gemini 4 Argon, Microsoft 365 Copilot, OpenAI Live Sessions & Azure PTU Sharing

Deploy this version

Docker

docker run \
-e LITELLM_MASTER_KEY=sk-<paste-a-long-random-key> \
-e DATABASE_URL=postgresql://<user>:<password>@<host>:5432/<dbname> \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:1.107.0-rc.1

Pip

pip install litellm==1.107.0rc1

This release is published as ghcr.io/berriai/litellm:v1.107.0-rc.1. See the GitHub release and the full releases page

The published GitHub tag is v1.107.0-rc.1. These notes compare it with v1.106.0-rc.1, the previous release candidate cut from main. Changes backported onto rc/1.106.0 and already shipped in v1.106.0 are omitted

Customer-facing changes come first. Test, CI and internal changes are listed at the bottom

Breaking Changes

These callouts cover user-facing behavior that differs from v1.106.0, the latest stable release

Fallbacks to models the caller cannot use are now skipped. enforce_fallback_model_access defaults to true. Set general_settings.enforce_fallback_model_access: false to restore the old behavior. See PR #46127

The proxy no longer starts when a configured logging callback cannot load, for example with an unsupported langfuse SDK. Install a supported version or remove the callback. See PR #46378

OpenRouter models are no longer in the built-in cost map. Spend still comes from OpenRouter's usage.cost, but litellm.get_model_info("openrouter/...") now raises, so set any context window or price you rely on in model_info. See PR #45665, PR #45666, PR #45667, PR #45668, PR #45669

refusal moved from provider_specific_fields to message.refusal on chat completion responses. See PR #44788

Token counting returns an error instead of a local estimate when the provider's counting call fails or the deployment has no provider key, on /v1/messages/count_tokens, /v1/responses/input_tokens and Gemini countTokens. See PR #46121, PR #46815

The Bedrock guardrail returns 400 for attachments it cannot scan, such as PDFs, audio or images sent by URL. Set skip_unscannable_attachments: true to let them through. See PR #43383

Key Highlights

• New models: Gemini 4 Argon on the Gemini API and Vertex AI with automatic continuation of long answers, Meta Muse Spark Open on Together AI, Fireworks AI and Baseten, MiniMax H3 video, xAI Grok Voice 3.0 and Wan 3.0 video on DashScope, among 187 new catalog entries
• New providers: Inference APIs, CoralBricks, ScaleDown and Microsoft 365 Copilot, plus Databricks Agent Platform agents in the agent gateway
• OpenAI Live and SDK 3: OpenAI Live sessions on /v1/live/sessions with the same auth, budgets and spend logging as Realtime, LiteLLM installs next to OpenAI SDK 3, and prompt cache breakpoints work on gpt-5.6
• Shared Azure PTU capacity: one Azure PTU deployment can be split across teams with per-team token ceilings, flat-cost splitting in the daily rollup and PTU-hour reporting
• Admin and observability: an Errors tab on Usage, opt-in telemetry controls under Settings, per-user GitHub OAuth for GitHub Copilot, and disable_requester_ip_address_logging to keep caller IPs out of spend logs
• MCP Gateway: ID-JAG grants on the gateway token endpoint and a shared gateway client for DCR bridge servers

New Providers and Endpoints

New Providers (4 new providers)

Provider	Supported LiteLLM Endpoints	Description
Inference APIs	/v1/chat/completions, /v1/responses, /v1/messages	OpenAI-compatible provider under inferenceapis/, reading INFERENCEAPIS_API_KEY
CoralBricks	/v1/chat/completions, /v1/responses, /v1/messages	OpenAI-compatible provider under coralbricks/, reading CORALBRICKS_API_KEY
ScaleDown	/v1/chat/completions	Native classify, extract, summarize, decisions and compress operations under scaledown/
Microsoft 365 Copilot	/v1/chat/completions	Graph Copilot Chat under microsoft_365_copilot/, exchanging the caller's JWT for a delegated Entra token

New LLM API Endpoints (1 new endpoint)

Endpoint	Method	Description
/v1/live/sessions	POST	OpenAI Live sessions, also on /live/sessions and /openai/v1/live/sessions, billed from usage.seconds

Expanded provider endpoint support

Provider	Endpoint	What you can do
Databricks Agent Platform	/a2a/<agent>, agent gateway	Call Model Serving agent endpoints and Databricks Apps agents with PAT or OAuth M2M auth
DashScope, Qwen Cloud, Qwen AI Platform	/v1/videos	Wan 3.0, Wan 2.7 and HappyHorse video generation
MiniMax	/v1/videos	MiniMax-H3 and H3-Max video generation

New Models / Updated Models

New Model Support (187 new models)

Counts represent new catalog identifiers compared with v1.106.0, including aliases and regional variants. Prices are the values bundled in this release, in USD, and runtime pricing-map reloads can update them. The 496 openrouter rows and 11 retired github_copilot rows were removed, see the Breaking Changes above

All 187 new catalog entries

Provider	Model	Context Window	Input ($/1M tokens)	Output ($/1M tokens)	Features / special pricing
Amazon Bedrock	amazon.nova-premier-v1:0	1,000,000	$2.5	$12.5	Chat; Vision; Tool calling; Prompt caching; Structured output; PDF input
Amazon Bedrock	jp.amazon.nova-2-lite-v1:0	1,000,000	$0.396	$3.311	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Video input
Amazon Bedrock	twelvelabs.pegasus-1-5-v1:0	98,304	-	$7.5	Chat; Video input; input_cost_per_video_per_second: $0.00049
Amazon Bedrock	us.stability.stable-conservative-upscale-v1:0	77	-	-	Image edit; output_cost_per_image: $0.4
Amazon Bedrock	us.stability.stable-creative-upscale-v1:0	77	-	-	Image edit; output_cost_per_image: $0.6
Amazon Bedrock	us.stability.stable-fast-upscale-v1:0	77	-	-	Image edit; output_cost_per_image: $0.03
Amazon Bedrock	us.stability.stable-image-control-sketch-v1:0	77	-	-	Image edit; output_cost_per_image: $0.07
Amazon Bedrock	us.stability.stable-image-control-structure-v1:0	77	-	-	Image edit; output_cost_per_image: $0.07
Amazon Bedrock	us.stability.stable-image-erase-object-v1:0	77	-	-	Image edit; output_cost_per_image: $0.07
Amazon Bedrock	us.stability.stable-image-inpaint-v1:0	77	-	-	Image edit; output_cost_per_image: $0.07
Amazon Bedrock	us.stability.stable-image-remove-background-v1:0	77	-	-	Image edit; output_cost_per_image: $0.07
Amazon Bedrock	us.stability.stable-image-search-recolor-v1:0	77	-	-	Image edit; output_cost_per_image: $0.07
Amazon Bedrock	us.stability.stable-image-search-replace-v1:0	77	-	-	Image edit; output_cost_per_image: $0.07
Amazon Bedrock	us.stability.stable-image-style-guide-v1:0	77	-	-	Image edit; output_cost_per_image: $0.07
Amazon Bedrock	us.stability.stable-outpaint-v1:0	77	-	-	Image edit; output_cost_per_image: $0.06
Amazon Bedrock	us.stability.stable-style-transfer-v1:0	77	-	-	Image edit; output_cost_per_image: $0.08
Azure AI	azure_ai/claude-opus-5-5-fast	1,000,000	$8	$40	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use
Azure AI	azure_ai/Cohere-Embed-V5-Fast	128,000	$0.08	$0	Embedding; input_cost_per_image_token: $0.4/1M
Azure AI	azure_ai/Cohere-Embed-V5-Pro	128,000	$0.12	$0	Embedding; input_cost_per_image_token: $0.4/1M
Baseten	baseten/meta-models/muse-spark-open	1,048,576	$1.25	$4.25	Chat; Reasoning; Vision; Tool calling; Prompt caching; Audio input
Baseten	baseten/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B	1,048,576	$0.2	$0.5	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output
ChatGPT	chatgpt/gpt-rosalind-discovery	-	-	-	Responses
Coral Bricks	coralbricks/deepseek-v4.1-flash	1,048,576	$0.01	$1.2	Chat; Reasoning; Vision; Tool calling; Prompt caching
Coral Bricks	coralbricks/deepseek-v4.1-flash-fast	1,048,576	$0.01	$1.2	Chat; Reasoning; Vision; Tool calling; Prompt caching
Coral Bricks	coralbricks/deepseek-v4.1-flash-fast-fp4	1,048,576	$0.01	$1.2	Chat; Reasoning; Vision; Tool calling; Prompt caching
Coral Bricks	coralbricks/glm-5.3	1,048,576	$1.12	$4.4	Chat; Reasoning; Tool calling; Prompt caching
Coral Bricks	coralbricks/glm-5.3-fast	1,048,576	$1.12	$4.4	Chat; Reasoning; Tool calling; Prompt caching
Coral Bricks	coralbricks/glm-5.3-fp4	1,048,576	$1.12	$4.4	Chat; Reasoning; Tool calling; Prompt caching
DashScope	dashscope/happyhorse-1.0-i2v	-	-	-	Video generation; output_cost_per_second: $0.220034; output_cost_per_second_720p: $0.123769; output_cost_per_second_1080p: $0.220034
DashScope	dashscope/happyhorse-1.0-r2v	-	-	-	Video generation; output_cost_per_second: $0.220034; output_cost_per_second_720p: $0.123769; output_cost_per_second_1080p: $0.220034
DashScope	dashscope/happyhorse-1.0-t2v	-	-	-	Video generation; output_cost_per_second: $0.220034; output_cost_per_second_720p: $0.123769; output_cost_per_second_1080p: $0.220034
DashScope	dashscope/happyhorse-1.1-i2v	-	-	-	Video generation; output_cost_per_second: $0.165026; output_cost_per_second_480p: $0.061885; output_cost_per_second_720p: $0.123769
DashScope	dashscope/happyhorse-1.1-r2v	-	-	-	Video generation; output_cost_per_second: $0.165026; output_cost_per_second_480p: $0.061885; output_cost_per_second_720p: $0.123769
DashScope	dashscope/happyhorse-1.1-t2v	-	-	-	Video generation; output_cost_per_second: $0.165026; output_cost_per_second_480p: $0.061885; output_cost_per_second_720p: $0.123769
DashScope	dashscope/wan2.7-i2v	-	-	-	Video generation; output_cost_per_second: $0.143353; output_cost_per_second_720p: $0.086012; output_cost_per_second_1080p: $0.143353
DashScope	dashscope/wan2.7-r2v	-	-	-	Video generation; output_cost_per_second: $0.143353; output_cost_per_second_720p: $0.086012; output_cost_per_second_1080p: $0.143353
DashScope	dashscope/wan2.7-t2v	-	-	-	Video generation; output_cost_per_second: $0.143353; output_cost_per_second_720p: $0.086012; output_cost_per_second_1080p: $0.143353
DashScope	dashscope/wan3.0-video	-	-	-	Video generation; output_cost_per_second: $0.165025; output_cost_per_second_480p: $0.041256; output_cost_per_second_720p: $0.082513
DashScope	dashscope/wan3.0-video-prime	-	-	-	Video generation; output_cost_per_second: $0.254399; output_cost_per_second_480p: $0.0636; output_cost_per_second_720p: $0.127199
Databricks	databricks/ai_decide	-	$0	$0	Decisions
Databricks	databricks/system.ai.deepseek-v4-flash-0731	-	-	-	Chat
Databricks	databricks/system.ai.kimi-k3	-	-	-	Chat
Fireworks AI	fireworks_ai/accounts/fireworks/models/muse-spark-open	1,048,576	$1.25	$4.25	Chat; Vision; Tool calling
Fireworks AI	fireworks_ai/accounts/fireworks/routers/rune	262,144	$0.042	$0	Decisions
Gemini	gemini/gemini-4-argon	1,048,576	$2	$10	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search; Audio input; Video input; google_maps_grounding_cost_per_query: $0.014; output_cost_per_reasoning_token: $10/1M
GitHub Copilot	github_copilot/claude-fable-5	200,000	-	-	Chat; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/claude-fable-5.1	200,000	-	-	Chat; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/claude-haiku-5.5	128,000	$0.1	$0.5	Chat; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/claude-opus-4.8	200,000	-	-	Chat; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/claude-opus-4.8-fast	200,000	-	-	Chat; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/claude-opus-5	200,000	-	-	Chat; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/claude-opus-5.5	200,000	-	-	Chat; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/claude-sonnet-5	200,000	-	-	Chat; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/claude-sonnet-5.5	200,000	-	-	Chat; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/gemini-3.7-flash	200,000	-	-	Chat; Reasoning; Vision; Tool calling
GitHub Copilot	github_copilot/gemini-3.8-flash	200,000	-	-	Chat; Reasoning; Vision; Tool calling
GitHub Copilot	github_copilot/gemini-4-argon	200,000	$2	$10	Chat; Reasoning; Vision; Tool calling
GitHub Copilot	github_copilot/gpt-5.4	272,000	-	-	Chat; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/gpt-5.4-mini	272,000	-	-	Responses; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/gpt-5.5	272,000	-	-	Responses; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/gpt-5.6-luna	200,000	-	-	Responses; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/gpt-5.6-sol	272,000	-	-	Responses; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/gpt-5.6-terra	272,000	-	-	Responses; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/gpt-6-astra	272,000	-	-	Responses; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/gpt-6-luna	272,000	-	-	Responses; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/gpt-6-sol	272,000	-	-	Responses; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/gpt-6.1-sol	272,000	-	-	Responses; Reasoning; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/kimi-k3	917,504	-	-	Chat; Vision; Tool calling; Structured output
GitHub Copilot	github_copilot/mai-code-1.1-flash	128,000	-	-	Responses; Vision; Tool calling; Structured output
InferenceAPIs	inferenceapis/deepseek-ai/DeepSeek-V3	1,048,576	$0.19	$0.38	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-ai/DeepSeek-V4-Flash	1,048,576	$0.19	$0.38	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-ai/DeepSeek-V4-Flash-0731	1,048,576	$0.19	$0.38	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-ai/DeepSeek-V4-Pro	1,048,576	$1.72	$5.15	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-ai/DeepSeek-V4-Pro-0813	1,048,576	$1.72	$5.15	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-ai/DeepSeek-V4.1-Flash	1,048,576	$0.4	$1.6	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-chat	1,048,576	$0.19	$0.38	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-flash	1,048,576	$0.4	$1.6	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-reasoner	1,048,576	$0.19	$0.38	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-v3	1,048,576	$0.19	$0.38	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-v4-flash	1,048,576	$0.19	$0.38	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-v4-pro	1,048,576	$1.72	$5.15	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/deepseek-v4.1-flash	1,048,576	$0.4	$1.6	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/glm-5.3	1,048,575	$1.82	$5.72	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/glm-5.3-flash	1,048,575	$0.2	$0.66	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/gpt-oss-120b	131,072	$0.2	$0.8	Chat; Reasoning; Tool calling
InferenceAPIs	inferenceapis/gpt-oss-20b	131,072	$0.04	$0.19	Chat; Reasoning; Tool calling
InferenceAPIs	inferenceapis/kimi-k2.7-code	262,144	$0.89	$4.42	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/llama-3.1-8b-instant	131,072	$0.2	$0.5	Chat; Tool calling
InferenceAPIs	inferenceapis/llama-3.3-70b	131,072	$0.2	$0.5	Chat; Tool calling
InferenceAPIs	inferenceapis/llama-3.3-70b-instruct	131,072	$0.2	$0.5	Chat; Tool calling
InferenceAPIs	inferenceapis/llama-3.3-70b-versatile	131,072	$0.2	$0.5	Chat; Tool calling
InferenceAPIs	inferenceapis/meta-llama/Llama-3.3-70B-Instruct	131,072	$0.2	$0.5	Chat; Tool calling
InferenceAPIs	inferenceapis/meta-llama/Llama-3.3-70B-Instruct-Turbo	131,072	$0.2	$0.5	Chat; Tool calling
InferenceAPIs	inferenceapis/minimax-m3	524,288	$0.39	$1.56	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/MiniMaxAI/MiniMax-M3	524,288	$0.39	$1.56	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/moonshotai/Kimi-K2.7-Code	262,144	$0.89	$4.42	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/openai/gpt-oss-120b	131,072	$0.2	$0.8	Chat; Reasoning; Tool calling
InferenceAPIs	inferenceapis/openai/gpt-oss-20b	131,072	$0.04	$0.19	Chat; Reasoning; Tool calling
InferenceAPIs	inferenceapis/Qwen/Qwen3-VL-235B-A22B-Instruct	262,144	$0.26	$1.15	Chat; Vision; Tool calling
InferenceAPIs	inferenceapis/Qwen/Qwen3.8-Flash	1,000,000	$0.15	$0.5	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/qwen3-vl	262,144	$0.26	$1.15	Chat; Vision; Tool calling
InferenceAPIs	inferenceapis/qwen3-vl-235b	262,144	$0.26	$1.15	Chat; Vision; Tool calling
InferenceAPIs	inferenceapis/qwen3.8-flash	1,000,000	$0.15	$0.5	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/zai-org/GLM-5.3	1,048,575	$1.82	$5.72	Chat; Reasoning; Tool calling; Prompt caching
InferenceAPIs	inferenceapis/zai-org/GLM-5.3-Flash	1,048,575	$0.2	$0.66	Chat; Reasoning; Tool calling; Prompt caching
Meta	meta/muse-wonder-1.0	1,048,576	$4	$20	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search
Microsoft 365 Copilot	microsoft_365_copilot/chat	-	$0	$0	Chat
MiniMax	minimax/MiniMax-H3	-	-	-	Video generation; output_cost_per_second: $0.08; output_cost_per_second_768p: $0.08; output_cost_per_second_2k: $0.13
MiniMax	minimax/MiniMax-H3-Max	-	-	-	Video generation; output_cost_per_second: $0.08; output_cost_per_second_480p: $0.05; output_cost_per_second_768p: $0.08
OCI	oci/openai.gpt-oss-120b	128,000	$0.15	$0.6	Chat; Reasoning; Tool calling
OCI	oci/openai.gpt-oss-20b	128,000	$0.07	$0.3	Chat; Reasoning; Tool calling
OCI	oci/xai.grok-4.20-0309-non-reasoning	1,000,000	$1.25	$2.5	Chat; Tool calling
OCI	oci/xai.grok-4.20-0309-reasoning	1,000,000	$1.25	$2.5	Chat; Reasoning; Tool calling
OCI	oci/xai.grok-4.20-non-reasoning	1,000,000	$1.25	$2.5	Chat; Tool calling
OCI	oci/xai.grok-4.20-reasoning	1,000,000	$1.25	$2.5	Chat; Reasoning; Tool calling
OpenAI	computer-use-preview-2025-03-11	8,192	$3	$12	Chat; Reasoning; Vision; Tool calling; PDF input
OpenAI	gpt-4o-mini-search-preview-2025-03-11	128,000	$0.15	$0.6	Chat; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search
OpenAI	gpt-4o-search-preview-2025-03-11	128,000	$2.5	$10	Chat; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search
OpenAI	gpt-5.5-cyber-preview	-	$12.5	$75	Chat; Reasoning
OpenAI	gpt-audio-mini-2025-10-06	128,000	$0.6	$2.4	Chat; Tool calling; Audio input; Audio output; input_cost_per_audio_token: $10/1M; output_cost_per_audio_token: $20/1M
OpenAI	gpt-rosalind-discovery	-	$5	$25	Chat
OpenAI	o3-deep-research-2025-06-26	200,000	$10	$40	Responses; Reasoning; Vision; Prompt caching; PDF input
OpenAI	o4-mini-deep-research-2025-06-26	200,000	$2	$8	Responses; Reasoning; Vision; Prompt caching; PDF input
Perplexity	perplexity/openai/gpt-5-nano	-	$0.05	$0.4	Responses
Perplexity	perplexity/openai/gpt-5.3-codex	-	$3.5	$28	Responses
Perplexity	perplexity/openai/gpt-6-astra	-	$10	$50	Responses
Perplexity	perplexity/perplexity/deepseek-v4-pro-0813	-	$1.32	$3.96	Responses
Perplexity	perplexity/pplx-decider-v1.1-27b	262,144	$0.02	$0	Decisions
Qwen AI Platform	qwen_ai_platform/happyhorse-1.0-i2v	-	-	-	Video generation; output_cost_per_second: $0.220034; output_cost_per_second_720p: $0.123769; output_cost_per_second_1080p: $0.220034
Qwen AI Platform	qwen_ai_platform/happyhorse-1.0-r2v	-	-	-	Video generation; output_cost_per_second: $0.220034; output_cost_per_second_720p: $0.123769; output_cost_per_second_1080p: $0.220034
Qwen AI Platform	qwen_ai_platform/happyhorse-1.0-t2v	-	-	-	Video generation; output_cost_per_second: $0.220034; output_cost_per_second_720p: $0.123769; output_cost_per_second_1080p: $0.220034
Qwen AI Platform	qwen_ai_platform/happyhorse-1.1-i2v	-	-	-	Video generation; output_cost_per_second: $0.165026; output_cost_per_second_480p: $0.061885; output_cost_per_second_720p: $0.123769
Qwen AI Platform	qwen_ai_platform/happyhorse-1.1-r2v	-	-	-	Video generation; output_cost_per_second: $0.165026; output_cost_per_second_480p: $0.061885; output_cost_per_second_720p: $0.123769
Qwen AI Platform	qwen_ai_platform/happyhorse-1.1-t2v	-	-	-	Video generation; output_cost_per_second: $0.165026; output_cost_per_second_480p: $0.061885; output_cost_per_second_720p: $0.123769
Qwen AI Platform	qwen_ai_platform/wan2.7-i2v	-	-	-	Video generation; output_cost_per_second: $0.143353; output_cost_per_second_720p: $0.086012; output_cost_per_second_1080p: $0.143353
Qwen AI Platform	qwen_ai_platform/wan2.7-r2v	-	-	-	Video generation; output_cost_per_second: $0.143353; output_cost_per_second_720p: $0.086012; output_cost_per_second_1080p: $0.143353
Qwen AI Platform	qwen_ai_platform/wan2.7-t2v	-	-	-	Video generation; output_cost_per_second: $0.143353; output_cost_per_second_720p: $0.086012; output_cost_per_second_1080p: $0.143353
Qwen AI Platform	qwen_ai_platform/wan3.0-video	-	-	-	Video generation; output_cost_per_second: $0.165025; output_cost_per_second_480p: $0.041256; output_cost_per_second_720p: $0.082513
Qwen AI Platform	qwen_ai_platform/wan3.0-video-prime	-	-	-	Video generation; output_cost_per_second: $0.254399; output_cost_per_second_480p: $0.0636; output_cost_per_second_720p: $0.127199
Qwen Cloud	qwencloud/happyhorse-1.0-i2v	-	-	-	Video generation; output_cost_per_second: $0.24; output_cost_per_second_720p: $0.14; output_cost_per_second_1080p: $0.24
Qwen Cloud	qwencloud/happyhorse-1.0-r2v	-	-	-	Video generation; output_cost_per_second: $0.24; output_cost_per_second_720p: $0.14; output_cost_per_second_1080p: $0.24
Qwen Cloud	qwencloud/happyhorse-1.0-t2v	-	-	-	Video generation; output_cost_per_second: $0.24; output_cost_per_second_720p: $0.14; output_cost_per_second_1080p: $0.24
Qwen Cloud	qwencloud/happyhorse-1.1-i2v	-	-	-	Video generation; output_cost_per_second: $0.18; output_cost_per_second_480p: $0.07; output_cost_per_second_720p: $0.14
Qwen Cloud	qwencloud/happyhorse-1.1-r2v	-	-	-	Video generation; output_cost_per_second: $0.18; output_cost_per_second_480p: $0.07; output_cost_per_second_720p: $0.14
Qwen Cloud	qwencloud/happyhorse-1.1-t2v	-	-	-	Video generation; output_cost_per_second: $0.18; output_cost_per_second_480p: $0.07; output_cost_per_second_720p: $0.14
Qwen Cloud	qwencloud/wan2.7-i2v	-	-	-	Video generation; output_cost_per_second: $0.15; output_cost_per_second_720p: $0.1; output_cost_per_second_1080p: $0.15
Qwen Cloud	qwencloud/wan2.7-r2v	-	-	-	Video generation; output_cost_per_second: $0.15; output_cost_per_second_720p: $0.1; output_cost_per_second_1080p: $0.15
Qwen Cloud	qwencloud/wan2.7-t2v	-	-	-	Video generation; output_cost_per_second: $0.15; output_cost_per_second_720p: $0.1; output_cost_per_second_1080p: $0.15
Qwen Cloud	qwencloud/wan3.0-video	-	-	-	Video generation; output_cost_per_second: $0.2; output_cost_per_second_480p: $0.05; output_cost_per_second_720p: $0.1
Qwen Cloud	qwencloud/wan3.0-video-prime	-	-	-	Video generation; output_cost_per_second: $0.28; output_cost_per_second_480p: $0.068; output_cost_per_second_720p: $0.14
ScaleDown	scaledown/classify	-	$0.05	$0	Chat
ScaleDown	scaledown/compress	-	$0.05	$0	Chat
ScaleDown	scaledown/decisions	-	$0.05	$0	Chat
ScaleDown	scaledown/extract	-	$0.05	$0	Chat
ScaleDown	scaledown/summarize	-	$0.05	$0	Chat
SCX AI	scx-ai/DeepSeek-V4.1-flash	1,000,000	$0.15	$0.6	Chat; Reasoning; Vision; Tool calling; Prompt caching
SCX AI	scx-ai/GLM-5.3	1,000,000	$0.98	$3.08	Chat; Reasoning; Tool calling; Prompt caching; Structured output
SCX AI	scx-ai/Kimi-K3	1,000,000	$2.83	$14.13	Chat; Reasoning; Tool calling; Prompt caching; Structured output
Tencent	tencent/deepseek-v4-pro-0813	1,000,000	$1.32	$3.96	Chat; Reasoning; Tool calling; Prompt caching; Structured output
Tencent	tencent/deepseek-v4.1-flash	1,000,000	$0.3	$1.2	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output
Tencent	tencent/glm-5.2	1,000,000	$1.4	$4.4	Chat; Reasoning; Tool calling; Prompt caching; Structured output
Tencent	tencent/glm-5.3	1,000,000	$1.4	$4.4	Chat; Reasoning; Tool calling; Prompt caching; Structured output
Tencent	tencent/glm-5.3-flash	1,000,000	$0.15	$0.5	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; Video input
Tencent	tencent/glm-5.3-flashx	1,000,000	$0.37	$1.25	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; Video input
Tencent	tencent/kimi-k2.7-code	256,000	$0.95	$4	Chat; Reasoning; Tool calling; Prompt caching; Structured output
Tencent	tencent/kimi-k2.7-code-highspeed	256,000	$1.9	$8	Chat; Reasoning; Tool calling; Prompt caching; Structured output
Tencent	tencent/kimi-k2.8-preview	1,000,000	$1	$4	Chat; Reasoning; Tool calling; Prompt caching; Structured output
Tencent	tencent/kimi-k3	1,000,000	$3	$15	Chat; Reasoning; Tool calling; Prompt caching; Structured output
Together AI	together_ai/meta-models/Muse-Spark-Open-816B-A42B	1,048,576	$1.25	$4.25	Chat; Tool calling; Prompt caching; Structured output
Together AI	together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL	262,144	$0.15	$0.5	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output
Vertex AI	gemini-robotics-er-2-preview	131,072	$1	$5	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search; Audio input; Video input; input_cost_per_audio_token: $1/1M; output_cost_per_reasoning_token: $5/1M
Vertex AI	vertex_ai/deepseek-ai/deepseek-v4.1-flash-maas	1,048,576	$0.3	$1.2	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output
Vertex AI	vertex_ai/gemini-4-argon	1,048,576	$2	$10	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search; Audio input; Video input; google_maps_grounding_cost_per_query: $0.014; input_cost_per_audio_token: $2/1M; output_cost_per_reasoning_token: $10/1M
Vertex AI	vertex_ai/gemini-4-argon-cyber-permissive	1,048,576	$4	$20	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search; Audio input; Video input; google_maps_grounding_cost_per_query: $0.014; input_cost_per_audio_token: $4/1M; output_cost_per_reasoning_token: $20/1M
Vertex AI	vertex_ai/gemini-robotics-er-2-preview	131,072	$1	$5	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search; Audio input; Video input; input_cost_per_audio_token: $1/1M; output_cost_per_reasoning_token: $5/1M
Vertex AI	vertex_ai/semantic-ranker-default	-	$0	$0	Rerank; input_cost_per_query: $0.001
Vertex AI	vertex_ai/semantic-ranker-default-002	-	$0	$0	Rerank; input_cost_per_query: $0.001
Vertex AI	vertex_ai/semantic-ranker-default-003	-	$0	$0	Rerank; input_cost_per_query: $0.001
Vertex AI	vertex_ai/semantic-ranker-default-004	-	$0	$0	Rerank; input_cost_per_query: $0.001
Vertex AI	vertex_ai/semantic-ranker-default@latest	-	$0	$0	Rerank; input_cost_per_query: $0.001
Vertex AI	vertex_ai/semantic-ranker-fast	-	$0	$0	Rerank; input_cost_per_query: $0.001
Vertex AI	vertex_ai/semantic-ranker-fast-004	-	$0	$0	Rerank; input_cost_per_query: $0.001
Vertex AI	vertex_ai/semantic-ranker-fast@latest	-	$0	$0	Rerank; input_cost_per_query: $0.001
Weights & Biases	wandb/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B	1,049,000	$0.15	$0.75	Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output
xAI	xai/grok-voice-2.0	-	-	-	Realtime; Tool calling; Audio input; Audio output; input_cost_per_second: $0.001333; input_cost_per_text_message: $0.004
xAI	xai/grok-voice-3.0	-	-	-	Realtime; Tool calling; Audio input; Audio output; input_cost_per_second: $0.001333; input_cost_per_text_message: $0.004
xAI	xai/grok-voice-latest	-	-	-	Realtime; Tool calling; Audio input; Audio output; input_cost_per_second: $0.001333; input_cost_per_text_message: $0.004

New providers

• Add Inference APIs as an OpenAI-compatible provider under the inferenceapis/ prefix, reading INFERENCEAPIS_API_KEY, listed in the Add Model form, with 12 priced chat models and native /v1/responses and /v1/messages - PR #41669
• Add ScaleDown as a native text provider under the scaledown/ prefix with classify, extract, summarize, decisions and compress operations, x-api-key authentication, tracked cost, and an Add Model entry in the Admin UI - PR #44168
• Add CoralBricks as an OpenAI-compatible provider under the coralbricks/ prefix, reading CORALBRICKS_API_KEY, listed in the Add Model form, with pricing for glm-5.3-fast and deepseek-v4.1-flash-fast - PR #44700
• Add Microsoft 365 Copilot as a chat provider under the microsoft_365_copilot/ prefix that exchanges the caller's JWT for a delegated Graph token through RFC 8693 or JWT bearer on-behalf-of, configured with an LLM credential in the Add Model form - PR #45158

Decisions providers

• Add Laya and Bespoke Nimble to the Add Model form and serve laya/* and bespoke/* deployments on the Decisions routes through <api_base>/v1/systemone - PR #45260
• Call the ai_decide AI Function route at <workspace>/api/2.0/ai-functions/ai-decide for databricks/ai_decide, so Decisions and the Databricks Auto Router classifier no longer get a 404 - PR #45722
• Bill /v1/decisions and /v1/systemone from <field>_decisions cost map keys, so gpt-6-luna Decisions are priced at 0.10 USD per 1M input tokens with no output or cache charges - PR #45904
• Add Fireworks AI accounts/fireworks/routers/rune on /v1/decisions and /v1/systemone, reading FIREWORKS_AI_API_KEY and billing cached tokens at the cheaper rate - PR #46057
• Answer 400 on a malformed provider_specific_header in litellm.decisions and litellm.systemone instead of silently dropping it - PR #46619
• Keep box answers from OpenAI-format decisions providers instead of parsing them as refusals - PR #46633
• Read a deployment's own supports_box_questions flag before the cost map, so a flagged deployment accepts box questions - PR #46696

Amazon Bedrock

• Map Bedrock Mantle's OpenAI-shaped context_length_exceeded errors to ContextWindowExceededError, so Claude Code compacts and context_window_fallbacks fire - PR #41747
• List the models the account can invoke behind bedrock/* when check_provider_endpoint: true, instead of the static catalog or no Bedrock models at all - PR #43981
• Send max_completion_tokens instead of max_tokens to Bedrock Mantle gpt-5.x and gpt-6 chat completions, which rejected max_tokens with a 400 - PR #43982
• Turn each tool_reference in a tool result into a text block holding the tool name on Bedrock Converse, so ToolSearch results no longer reach Bedrock as empty toolResult.content and fail with a 400 - PR #44590
• Default Bedrock Grok 4.6 and 4.7 to native Chat Completions, remint positional call_<n> tool call ids as unique call_<uuid> and drop stop for Grok, so Claude Code no longer 400s after a ToolSearch turn - PR #45473
• Place images from a tool result beside the tool result on Bedrock OpenAI GPT models, which rejected images nested in toolResult.content with a 400 - PR #45579
• Add the twelvelabs.pegasus-1-5-v1:0 row and set the context window of the Bedrock GPT-5.x rows to 1,050,000 tokens from the model cards - PR #45621
• Drop stop for Bedrock GPT 5.6 and newer instead of sending stopSequences that Bedrock rejects with a 400, which unblocks Claude Code auto mode - PR #45664
• Reprice the regionless bedrock/moonshotai.kimi-k2.5 and bedrock/moonshotai.kimi-k2-thinking rows to the us-east-1 Bedrock prices - PR #45836
• Raise the max output tokens of the Bedrock zai.glm-4.7 and zai.glm-4.7-flash rows from 4,000 to 128,000, from the model cards - PR #45976
• Take context windows, max output tokens and end-of-life dates for 46 Bedrock rows from the Bedrock model cards - PR #45980
• Replay redacted_thinking on Converse and return its data as Claude's own string, so a Converse turn replays on Invoke and anthropic/ instead of getting a 400 - PR #46532
• Use base_model for Claude capability checks on application inference profile ARNs, so adaptive thinking on an Opus 4.8 ARN is no longer rewritten into a fixed budget - PR #46569
• Grant sts:GetWebIdentityToken and sts:TagGetWebIdentityToken in the web identity session policy, so bedrock/claude_platform/* models using OIDC web identity auth no longer return 403 - PR #46670
• Type PerformanceConfigBlock.latency as standard or optimized to match the values Bedrock accepts, instead of throughput - PR #46698
• Add the jp.amazon.nova-2-lite-v1:0 Japan geo inference profile to the model catalog - PR #46721
• Add Priority and Flex tier prices to openai.gpt-oss-120b-1:0 and openai.gpt-oss-20b-1:0 - PR #46728
• Reject malformed Nova Sonic realtime session settings, keep cache read and write tokens on streamed cache hits, keep Prometheus rate limit gauges on /v1/messages, and price exact cost-map names so Bedrock calls stop billing zero - PR #47090

Anthropic

• Add an empty "" placeholder when the last Claude user message has null, [] or missing content, so these requests no longer fail with a prefill 400 - PR #41397
• Keep stop_reason and stop_details from Anthropic refusals on the response and the logged response across /chat/completions, /v1/responses and /v1/messages, so callbacks see more than content_filter - PR #46104
• Stream json_schema answers as delta.content on Bedrock Mantle and Claude Platform, and join answers that Claude splits into one tool block per field into a single object - PR #46331
• Check forced tool support on the real Vertex model, skip a non-string metadata.user_id instead of returning a 500, and drop speed under drop_params for providers that reject it - PR #46337
• Keep signed thinking blocks with empty text on chat completions, return the 400 when a /v1/messages caller sets block_binding.prefix_mismatch_behavior to error, and keep thinking.block_binding on Bedrock Invoke - PR #46342
• Translate OpenAI's tool_search tool on /v1/chat/completions into Anthropic's regex tool search for Claude models instead of answering 500, and answer 400 naming the fix where a route cannot run it - PR #46389
• Keep the tool_search_tool_result block in provider_specific_fields["tool_results"] and replay it after its server_tool_use, so the second call of a Claude tool search loop no longer gets a 400 - PR #46440
• Forward service_tier on chat completions to Anthropic and Azure Foundry Claude, map OpenAI's default to standard_only, and answer 400 with a drop_params hint for tiers Claude does not take - PR #46461
• Count the cache mark on a replayed server tool call the way modify_params sends it, so the cache control hook no longer adds a fifth mark that Anthropic rejects with a 400 - PR #46591
• Drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking, and drop budget_tokens from the rebuilt block, so Claude no longer returns 400 - PR #46741

Azure

• Add azure_ai/Cohere-Embed-V5-Fast and azure_ai/Cohere-Embed-V5-Pro embedding models to the cost map - PR #45764
• Move the deprecation_date of azure/gpt-realtime-translate to 2027-11-09, matching the Azure Models API - PR #45849
• Move the deprecation_date of azure_ai/kimi-k2.7-code to 2027-04-16, matching the Azure Models API - PR #45945
• Sync Azure text-embedding-3-large, text-embedding-3-small, text-embedding-ada-002, tts-1-hd and legacy GPT-4 Turbo and GPT-3.5 Turbo Instruct prices with the Azure OpenAI pricing page - PR #46200
• Add azure_ai/claude-opus-5-5-fast at twice the azure_ai/claude-opus-5-5 prices with a 2027-10-01 deprecation date - PR #46760

Baseten

• Add baseten/meta-models/muse-spark-open and baseten/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B to the model catalog - PR #46794

Crusoe

• Default Crusoe calls to https://api.inference.crusoecloud.com/v1, since the legacy host's firewall rejected some prompts with a 403, and CRUSOE_API_BASE still overrides it - PR #33121

DashScope

• Add Wan 3.0, Wan 2.7 and HappyHorse video generation on /v1/videos for dashscope/, qwencloud and qwen_ai_platform, with per-region pricing - PR #43706

Databricks

• Keep #/$defs refs as the client sent them in Databricks json_schema response formats, so nested types no longer fail with a 400 on non-Claude models - PR #45659
• Send a single plain text block as a string to non-Claude Databricks models, so JSON mode no longer fails with a 400 on model_info.mode: responses deployments - PR #45686
• Accept service_tier on Databricks requests, including /v1/responses calls to chat models, and return the served service_tier on non-streamed responses so pricing can see it - PR #46111

Gemini and Vertex AI

• Add litellm_settings.gemini_include_thoughts_default, and set it to false to stop requesting Gemini thought summaries while thinking and reasoning tokens stay on - PR #36460
• Send a tool result containing a string $ref to Gemini and Vertex AI as raw JSON text in content, so the next turn is no longer rejected with a 400 - PR #38284
• Report an unreadable vertex_credentials file as a file read failure that points to the log, instead of saying the JSON is invalid - PR #40173
• Keep every non-null anyOf branch when stripping null types for Gemini and Vertex AI, so doubled or single null unions no longer leak a null branch or fail with a 500, and ["integer","null"] fields stop coming back as 0 - PR #41683
• Route Vertex AI Model Garden and partner models such as xai/grok-* through the chat completions adapter on :generateContent, :streamGenerateContent and :countTokens, instead of 404 and 500 errors from the native Google path - PR #41836
• Stop adding temperature: 1.0 to Gemini 3 and newer requests that omit it, and keep forwarding an explicit temperature as before - PR #45107
• Map 12 OpenAI voice names such as alloy to Gemini prebuilt TTS voices, so Gemini TTS no longer fails with a 400 and passes the default health check - PR #45303
• Unwrap Vertex AI Claude json_schema answers nested under keys such as parameters or input, and send output_config.format instead of the deprecated output_format, so structured outputs stop failing with a 400 - PR #45871
• Price Vertex AI semantic-ranker rerank at 1.00 USD per 1,000 queries, instead of logging every rerank request at zero spend - PR #46061
• Add vertex_ai/gemini-robotics-er-2-preview to the model catalog - PR #46126
• Add gemini/gemini-4-argon to the model catalog with Gemini API prices - PR #46227
• Add vertex_ai/gemini-4-argon to the model catalog with Vertex AI prices and capability flags - PR #46228
• Raise gemini/gemini-4-argon max_output_tokens from 32768 to 65536 to match the default output limit in the model docs - PR #46236
• Add vertex_ai/gemini-4-argon standard, priority and flex/batch pricing with the non-global uplift, so Vertex AI calls return a response cost - PR #46237
• Continue Gemini 4 long answers automatically with Google's continuationToken until max_tokens is spent, sum usage across the calls, and cancel interactions at {id}/cancel - PR #46240
• Remove the duplicate vertex_ai/gemini-4-argon rows and the bare gemini-4-argon row, so only the vertex_ai/ prefixed name resolves - PR #46246
• Add vertex_ai/gemini-4-argon-cyber-permissive to the model catalog with Vertex AI prices - PR #46249
• Pass the per-request timeout to Vertex AI and Google AI Studio Gemini streaming calls, so a stalled stream raises instead of waiting for the 6000s default - PR #46349
• Add the 2025-11-14 shutdown date to gemini-2.0-flash-exp-image-generation and its gemini/ row - PR #46489
• Add Flex prices for input, cached input and output to gemini-nano-banana-2.1 and vertex_ai/gemini-nano-banana-2.1 - PR #46512
• Add vertex_ai/deepseek-ai/deepseek-v4.1-flash-maas to the model catalog with Vertex AI prices - PR #46729
• Price vertex_ai/deepseek-ai/deepseek-v4.1-flash-maas at zero while Vertex AI offered it at no charge - PR #46753
• Move vertex_ai/deepseek-ai/deepseek-v4.1-flash-maas from $0 to its published paid price now that the free period has ended - PR #46793
• Drop cache_read_input_token_cost_flex from the four Vertex AI Gemini 3 preview rows, since Vertex AI lists cached Flex input as unavailable - PR #46826

GitHub Copilot

• Add a Per-user GitHub OAuth auth type for GitHub Copilot credentials, so each github_copilot/ request uses the calling user's own GitHub login and Internal Users connect their account under LLM Credentials, Your connections - PR #45241
• Sync the GitHub Copilot model catalog with the models Copilot serves, removing 11 retired ids that returned model_not_supported and adding 22 current ones - PR #45386
• Add github_copilot/claude-haiku-5.5 to the cost map - PR #45656
• Add github_copilot/gemini-4-argon to the model catalog at Copilot list pricing - PR #46280
• Add 2026-11-16 deprecation dates to github_copilot/claude-opus-5 and github_copilot/gpt-5.6-terra - PR #46792

Meta

• Add meta/muse-wonder-1.0 with Meta's pricing and support for max and xhigh reasoning effort - PR #46072

MiniMax

• Add MiniMax-H3 and H3-Max video generation on /v1/videos, with per-second prices per resolution and billing for input images past the free allowance - PR #43705

OCI

• Drop reasoning_effort on OCI Grok 4.20 and non-function tools on every OCI model under drop_params, so Codex and Claude Code no longer fail every turn with a 400 - PR #40732

Ollama

• Give Ollama OpenAI-compatible responses a unique chatcmpl- id instead of Ollama's reused one, which silently dropped successful requests from spend logs, and keep the original in provider_specific_fields.original_response_id - PR #39227

OpenAI

• Add gpt-rosalind-discovery to the cost map with the prices on OpenAI's pricing page - PR #45653

SageMaker

• Send model_id on SageMaker embedding deployments as InferenceComponentName, and add a sagemaker/openai/<endpoint> route that sends an OpenAI input body and parses data[].embedding responses - PR #44985
• Stop sending the deployment's AWS credentials in the sagemaker_chat request body, while the request is still signed with them - PR #45724

SCX.ai

• Add scx-ai/DeepSeek-V4.1-flash, scx-ai/GLM-5.3 and scx-ai/Kimi-K3 to the model catalog with SCX.ai's published prices, so their spend is no longer logged at $0 - PR #46496

Soniox

• Decode Soniox context, translation and flag form fields sent through /v1/audio/transcriptions into typed values, and fill language_hints from OpenAI language when none are set - PR #40605

Together AI

• Restore together_ai/moonshotai/Kimi-K3 to Together's standard serverless price now that the launch promotion has ended - PR #46226
• Set together_ai/Qwen/Qwen3.7-Max to its published price and restore the 2026-09-15 removal dates on the affected Together models - PR #46281
• Add together_ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL to the model catalog with Together's published prices - PR #46693

Weights and Biases

• Add wandb/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B to the model catalog - PR #46795

xAI

• Add xai/grok-voice-3.0, xai/grok-voice-2.0 and xai/grok-voice-latest realtime rows priced per second - PR #46602

Model catalog and pricing

• Register text pricing for five ScaleDown models at 0.05 USD per million input tokens with free output, so cost calculation no longer reports them as unmapped - PR #44167
• Add a 2026-10-23 deprecation_date to gemini/deep-research-pro-preview-12-2025 and drop the withdrawn shutdown date from gemini/gemini-3.1-flash-image, matching Google's changelog - PR #45491
• Raise together_ai/Qwen/Qwen3.7-Max to Together's current 2.50 USD input and 7.50 USD output per 1M tokens, which was underbilled by 40%, and correct the azure_ai/kimi-k2.7-code deprecation_date - PR #45595
• Add the Perplexity Agent API models and set claude-sonnet-4-5 to a 200,000 token context, since Anthropic retired its 1M context beta for that model - PR #45647
• Add a 2026-10-22 deprecation_date to together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo and together_ai/deepseek-ai/DeepSeek-V4-Flash-0731 - PR #45657
• Remove the first 105 openrouter rows from the model cost map, part 1 of 5, since OpenRouter bills from its own usage.cost - PR #45665
• Remove 29 more openrouter rows from the model cost map, part 2 of 5 - PR #45666
• Remove 139 more openrouter rows from the model cost map, part 3 of 5, and drop the *_above_32k_tokens, *_above_128k_tokens and *_above_256k_tokens fields only they used - PR #45667
• Remove 202 more openrouter rows from the model cost map, part 4 of 5, including the openrouter/openai/gpt-image-* rows - PR #45668
• Remove the last 21 openrouter rows from the model cost map, part 5 of 5, so no openrouter row remains - PR #45669
• Correct six Bedrock Titan Text and Nova 2 Lite prices to the AWS Price List, add amazon.nova-premier-v1:0 and clear two rescinded Together retirement dates - PR #45793
• Add six OpenAI dated snapshots with a 2026-07-23 deprecation_date, price both Perplexity decider models at 0.02 USD per 1M input tokens and drop a stale cached price from a Together model - PR #45842
• Fix tencent/deepseek-v4-pro TokenHub pricing, add ten TokenHub DeepSeek, GLM and Kimi models, correct Fireworks context windows and Bedrock Claude 3.5 Sonnet limits, and update four Together removal dates - PR #46206
• Add 2027-01-14 shutdown dates to five OpenAI GPT-4o and GPT-4.1 snapshots, move Azure gpt-chat-latest to 2027-05-18, bill the per text input charge on xAI voice models and refresh Sail DeepSeek V4.1 Flash prices - PR #46675
• Add 16 missing priced OpenAI, Together, Fireworks and Bedrock models to the model catalog, so their spend is no longer unmapped - PR #46742

LLM API Endpoints

Responses API

• Keep the provider's request id header on mid-stream errors from /v1/responses and Bedrock event streams, so the failed log row carries error_provider_request_id - PR #45453
• Map structured OpenAI cyber_policy errors, including streamed error events, to ContentPolicyViolationError so configured content-policy fallbacks run - PR #45828
• Translate OpenAI's tool_search tool to Anthropic's regex tool search for Claude models on /v1/responses, and return Claude's searches as tool_search_call items, streaming included - PR #45882
• Keep the provider's code and message from top-level stream error events instead of reporting Response API in-stream error - PR #46438
• Map Anthropic's bm25 and regex tool search tools to OpenAI's server tool_search on OpenAI Responses requests, so a Claude to OpenAI fallback no longer answers 400 - PR #46441
• Open faked Responses stream items empty so text and tool arguments arrive once, which stops streamed /v1/messages tool calls from getting doubled JSON input - PR #46533
• Use a deployment's own provider and api_key for /v1/responses over WebSocket, like HTTP, instead of falling back to the global OpenAI key - PR #46724
• Write one spend log row per response.create turn on a native Responses WebSocket, keyed by the resp_ id, instead of one summed row at socket close - PR #46768

Chat Completions over the Responses API bridge

• Return every message, reasoning item and tool call of a bridged reply on choices[0], so clients that only read the first choice run the tool - PR #46099
• Number bridged chat tool calls 0, 1 and so on instead of 0 for every call, matching native chat - PR #46109
• Keep defer_loading on function tools sent to the Responses API and route requests with a tool_search tool to Responses on OpenAI and Azure, so GPT-5.5 tool search stops answering 400 - PR #46439
• Round-trip tool_search_call and tool_search_output items between bridged chat turns through provider_specific_fields.tool_search_items, so GPT-5.5 keeps each search and its namespace - PR #46444
• Return only the first message item as content for json_schema and json_object requests, so structured output no longer comes back as two JSON documents glued together - PR #46508

Anthropic Messages API

• Stop writing extra keys such as input_examples into the caller's own tool input_schema while translating Anthropic tools, so strict tools no longer get a 400 under a guardrail and a later fallback to Claude keeps every field - PR #34653
• Keep thinking blocks with empty text and a valid signature on /v1/messages requests, so replayed turns on Claude models that omit thinking text no longer lose their reasoning - PR #41362
• Add general_settings.safeguards_classifier_model, which judges each tool call on non-Claude /v1/messages routes and returns Anthropic-shaped safeguard_results, so Claude Code auto mode keeps its server-side check - PR #45600
• Forward service_tier on /v1/messages to Anthropic, Azure AI and Bedrock Mantle, so a standard_only request no longer lands on Priority Tier capacity - PR #46370
• Emit one tool_use block per call when a /v1/messages stream chunk carries several tool calls, so parallel tool calls from Mistral models no longer merge into invalid JSON - PR #46492
• Send detail: auto on input_image parts bridged from /v1/messages to Responses and in the Admin UI playground, so image requests to vLLM-backed openai/ models stop returning 400 - PR #46720
• Honor litellm_settings.request_timeout and REQUEST_TIMEOUT on native Messages calls instead of timing out long turns after 600 seconds - PR #46749

Count tokens

• Return the provider's own status and message when its token counting call fails on /v1/messages/count_tokens, /v1/responses/input_tokens and the Gemini countTokens route, instead of a 200 with a local tiktoken estimate - PR #46121
• Count system prompts and tools on Vertex Gemini and Google AI Studio, honor the deployment's api_base, and fall back to Mantle when a Bedrock Claude count gets a runtime 404 - PR #46430
• Count /v1/messages/count_tokens requests for OpenAI models the way /v1/messages converts them, so PDFs, images, tool blocks and list-form system prompts are included - PR #46759
• Count /v1/messages/count_tokens requests for Gemini models the way /v1/messages converts them, so PDFs, images and tool turns are included and AI Studio Gemini returns a count instead of a 500 - PR #46767
• Return a 400 naming the Anthropic blocks that /v1/messages/count_tokens can't convert for OpenAI and Gemini models, instead of a low count with a 200 - PR #46772
• Return a 401 naming the missing provider key from the count endpoints instead of a local estimate, with errors in Anthropic's envelope on /v1/messages/count_tokens and Google's on :countTokens - PR #46815
• Add token_count_model and token_count_framing settings and report litellm_count_source and litellm_count_model on count responses, so Bedrock Claude Sonnet 5.5 and Kimi K3 counts work, and answer 400 for malformed tools instead of 500 - PR #47209

Prompt caching

• Carry cache_control on /v1/responses message, function_call and function_call_output items onto the bridged chat messages, so Claude on Bedrock, Vertex AI and Azure AI gets cache hits - PR #32511
• Keep a deployment's configured prompt cache breakpoint when chat completions and /v1/messages are bridged to Responses for Bedrock Mantle GPT-5.6, so explicit cache writes happen again - PR #45500
• Rewrite cache_control marks to prompt_cache_breakpoint on the /v1/messages to Responses bridge for OpenAI gpt-5.6 and newer, and keep the system prompt as a developer message so its mark survives - PR #46082
• Keep the prompt caching affinity pin for 3600 seconds when the cacheable prefix has a "ttl": "1h" breakpoint, instead of a fixed 300 seconds, so the prefix stays on the deployment that holds its cache - PR #46084
• Rewrite cache_control marks to prompt_cache_breakpoint on /v1/chat/completions for OpenAI gpt-5.6 and newer, instead of stripping them - PR #46101
• Rewrite cache_control marks to prompt_cache_breakpoint on native /v1/responses for OpenAI gpt-5.6 and newer, instead of stripping them - PR #46105
• Cache screenshots inside tool_result blocks on the /v1/messages to Responses bridge, and honor cache_control on blocks inside a tool result - PR #46324
• Strip prompt_cache_breakpoint from messages, content parts, tool calls and tools under drop_params for OpenAI models that cannot read it - PR #46453
• Keep the 1 hour routing pin on a response cache hit for a Bedrock application inference profile, matching the base model's 1h cache TTL instead of 5 minutes - PR #47174

Realtime

• Proxy OpenAI Live sessions on /v1/live/sessions, /live/sessions and /openai/v1/live/sessions, with the same key auth, model access checks, routing, budgets and spend logging as Realtime, and Live voice billed from usage.seconds - PR #43621
• Send response.create after a transcript only when a transcript guardrail turned off the auto-response, so clients with create_response: false stop getting a response they never asked for - PR #43791
• Forward Nova Sonic tool calls with their arguments over /v1/realtime, accept flat and nested tool shapes, and return an error for a session.update Bedrock cannot apply instead of ending the session - PR #46214

Batches and files

• Return a 400 that names the line number when a Vertex AI batch upload has a malformed JSONL row, instead of a 500 pointing at the wrong line - PR #40097
• Add opt-in batch_line_item_callbacks (s3_v2, gcs_bucket) that writes each batch line's request, response, tokens, cost and creator to the bucket after a managed batch is billed - PR #41691
• Parse Vertex AI batch and file timestamps as UTC, so created_at is no longer shifted by the host timezone and a whole-second createTime no longer fails batch create and retrieve with a 500 - PR #45550
• Read a completed xAI batch id as a file from the batch itself instead of xAI's files API, which always answered 404, so the results file is named at once and its size fills in from the batch cost job - PR #45559
• Check input file ownership when creating batches and fine-tuning jobs from managed files, returning 404 for a missing file and 403 for another owner's file, and keep a regenerated key's own files with it - PR #45855

Vector Stores

• Give each default-constructed vector store registry its own list, so a fresh VectorStoreRegistry() no longer inherits stores registered on an earlier one - PR #38879
• Send the provider exactly the caller's metadata when creating or updating vector stores through the proxy, so stores stop filling OpenAI's 16 metadata slots and later updates no longer fail with 400 - PR #47159

Pass-through endpoints

• Let configured pass-through endpoints take priority over a built-in /<prefix>/{param:path} provider route with the same prefix, and remove an endpoint's route when it is deleted or dropped from config - PR #44133
• Classify streamed /v1/messages pass-through calls as Anthropic on any host, so a custom ANTHROPIC_API_BASE gateway logs real tokens, cost and the provider message id instead of zero spend - PR #45602
• Return AWS-shaped errors on the Bedrock passthrough, so AWS SDKs read the error code and message of a guardrail block or a Bedrock failure instead of an empty ClientError - PR #46522

Agents and Agent-to-Agent

• Add opt-in general_settings.agent_access_default_deny so a key with no agent grants on itself or its team can reach no agents - PR #43979
• Add a per-agent litellm_params.a2a_protocol_version (0.3 or 1.0) that overrides the agent card's declared version, so LangGraph agents on langgraph-api 0.15 stop failing with has no field named "kind" - PR #44463
• Add the databricks_agent provider so Databricks Agent Platform agents (Model Serving endpoints, the unified responses URL and Databricks Apps) work in the agent gateway with PAT or OAuth M2M auth, plus a Databricks Agent card in the Admin UI - PR #45681

SDK

• Keep the provider's response headers on streaming SDK errors, so a streamed Gemini 429 exposes them on litellm_response_headers like a non-streaming call - PR #41528
• Stop prompt-based tool calling from editing the caller's messages, so a list-content system message no longer gains another copy of the tool list every turn - PR #41676
• Resolve litellm.proxy lazily on attribute access, so litellm.proxy._types works without import litellm loading proxy modules - Commit 691ca03
• Add CustomStreamWrapper.close() and release the upstream connection when a sync stream ends or fails, so Anthropic and other sync streams stop pinning pooled connections - PR #46354

General

• Honor an explicit custom_llm_provider and api_base on async image generation, speech, transcription and every realtime route, so bare custom model names no longer fail with LLM Provider NOT provided - PR #36346
• Show the target model's metadata for a listed team or key alias on GET /v1/models, and stop returning 404 for aliases, including router model_group_alias names, on GET /v1/models/{model_id} - PR #41041
• Strip trailing slashes from api_base before adding the route for Perplexity, Jina, Tavily and other providers, so URLs no longer get a // that answers 404 or 403 - PR #42887
• Pass refusal, logprobs and system_fingerprint from the provider response through on /v1/chat/completions, including explicit nulls, instead of moving refusal into provider_specific_fields - PR #44788
• Keep the mapped status and message when a provider error hits the assistants and threads routes, so a provider 404 returns 404 instead of a blank 500, with file paths and secrets scrubbed from the body - PR #44994
• Drop stream_options from the forced non-streaming call in web search, code interpreter and headroom interception and replay it on the final stream, so streamed requests no longer fail with 400 on Azure - PR #45433
• Return provider-matching 400s instead of 500s or a silently changed 200 for OCR requests with no file, reversed daily activity date ranges, max_output_tokens of 0 on /v1/responses, and more than 16 vector store file attributes - PR #45537
• Keep request fields and headers that a pre-call hook deletes off the provider request on native chat, Responses, Messages and OCR routes - PR #45692
• Add an index to streamed thinking_blocks from Anthropic, Bedrock Converse and Databricks, so the OpenAI SDK client.chat.completions.stream() helper no longer crashes on reasoning output - PR #46611
• Intercept OpenAI's bare {"type": "web_search"} tool on chat completions, so Claude models no longer answer 500 and OpenAI models no longer answer 400 - PR #46736

Management Endpoints / UI

Admin UI

Auto-Routers

• Group classifier, tier and advanced settings in auto-router setup, and let a new decision router pick an onboarded decision model instead of entering a provider and raw model name - PR #45913

General

• Add a Settings, Telemetry page with a switch per telemetry group and a destination picker, and record page views and tab clicks only when that group is on - PR #45494
• Restore the background blur behind dashboard modals - PR #45866
• Drop the beta marker from the dark mode toggle - PR #46305
• Remove the deprecation banner from Memory, Prompts, Agent Builder and MCP Network Settings, and make it permanent on Workflows, the old Usage page and the API Reference tab - PR #46314
• Move the new version notice from a banner into the notifications bell, which now lists announcements with a read state per item - PR #46334
• Replace the stacked deployment warning banners with a header health icon, one alert banner and a side panel where proxy admins snooze, ignore or restore each warning - PR #46498
• Add a Cost per Query (USD) field to the search tool add and edit forms, and keep a stored price when a tool is edited - PR #46723
• Read the signed-in user's role from the new GET /management/v1/users/me instead of the session cookie, and sign out when it cannot be loaded - PR #47224

Home and LiteAdmin

• Let LiteAdmin detach a key from its team through a confirmation-gated tool, and preselect an accessible Sonnet 5.5 model in new chats - PR #45625
• Add a LiteAdmin composer to Home and independent LiteAdmin chat tabs, so New chat opens a tab instead of clearing the current conversation - PR #46068
• Simplify LiteAdmin on Home to a standalone heading and one composer with editable suggestions, and place the virtual key action beside the Home title - PR #46301
• Widen the LiteAdmin model selector on Home so longer model names fit - PR #46446
• Place the Home Create New Virtual Key button under the title in the default style, like the Virtual Keys page - PR #46778

Logs

• Skip failure rows for expired Admin UI session keys on dashboard info routes such as /key/list and /team/list, so a stale tab no longer floods Request Logs with 401 rows - PR #45881

Models + Endpoints

• Fix Add Model showing an empty model field for 14 providers, including MiniMax, Cursor and SageMaker, by resolving the provider key to the slug the backend declares - PR #43707
• Add a Status filter with All, Active and Paused options to Models and Endpoints - PR #43710
• Show the public JWKS of a LiteLLM-signed Anthropic credential in its edit modal with a Copy JWKS button - PR #45527
• Show the Microsoft 365 Copilot logo instead of the Azure one for Microsoft 365 Copilot models - PR #45687
• Fix credential edits saving onto a different credential, merge PATCH /credentials onto the writer's row, refuse POST /credentials for a name defined in config.yaml with 409, and show each credential's name and source in the Admin UI - PR #45835
• Disable the add, edit and delete controls for model group aliases that config.yaml owns, with a read-only notice on the alias tab - PR #45857
• Gray out the router settings and fallbacks that config.yaml owns, with a badge and tooltip, and skip them on save - PR #45861
• Run the Add Model validation when Test Connect is clicked so an incomplete form shows inline errors instead of crashing with t[0] is undefined, and clear provider inputs after a successful add - PR #46834

Playground and Decisions

• Show cost-map decision models as decisions in the public Model Hub and list the supported ones in the AI Hub, with docs links and a prefilled Add Model form - PR #45965
• Give the Decisions playground a labeled setup card, one editor card with Form and cURL tabs, and an always visible Response card - PR #45981
• Show the legacy evaluation mode as decisions in the Model Hub mode filter, Mode column and guardrail decision-model picker - PR #46098
• Label Decisions playground issues by question key instead of an index, and fix the alignment of the Response card header - PR #46145

Teams, users and access groups

• Page the team and organization members tables at 25 rows and look up budget rows from a map built once per team, so teams with thousands of members open quickly - PR #26667
• Show $X of $Y (team member) on keys that have no budget of their own in the Virtual Keys table, the team Virtual Keys tab and the key detail page, instead of Unlimited - PR #43927
• Rename the key edit Team ID field to Team, list teams by alias, and add a clear button that detaches a key from its team - PR #45629
• Accept email addresses containing # in the member forms, so budget and rate limit edits work for Entra ID guests - PR #45688

Usage

• Add GET /user/daily/activity/aggregated/users, a Top Users by Spend panel on the Usage Overview and a User Activity tab, so JWT-authenticated users show up in usage - PR #45310
• Add an admin-only Errors tab to Usage with error rate over time, failed requests by status code, and failures by key, team, user and model - PR #45396
• Title the Usage Overview chart Daily usage (or Weekly usage) and label the ranked model list Top models - PR #45604

Keys, teams and authentication

• List team access group models on /v1/model/info, and keep keys with their own model list from seeing team models they cannot call on the listing routes and GET /v1/models/{model_id} - PR #32599
• Return a plain Invalid API key provided. 401 for an unknown key, without the key table name or the hash of the submitted key - PR #39787
• Name the rejected control characters and .. path segments in the 400 returned for an invalid key_alias - PR #41412
• Return HTTP 200 with the blocked end users from POST /customer/block instead of a 500 after the block was already saved - PR #41677
• Add opt-in personal_key_model_access_from_teams so personal keys can call, and list on /v1/models, the union of models from their owner's current teams, checked on every request - PR #43692
• Expand the all-team-models value in project model access checks, so keys in a project set to All Team Models stop getting 403 project_model_access_denied - PR #43734
• Check project ownership when generating, updating, regenerating and bulk updating keys, and refuse moving a project to another team while other teams' keys are attached - PR #44165
• Add a proxy-admin team setting require_trace_id that rejects LLM, MCP and A2A requests from the team with HTTP 400 when they carry no trace ID, with a switch on the team pages - PR #44933
• Add a raise_max_budget team admin editable field, off by default, that lets team admins raise their team's max_budget up to the organization's cap - PR #45404
• Add opt-in create_member_keys under team admin editable fields so team admins can create LLM API keys for internal user and viewer teammates through /key/generate and the Create Key form - PR #45870
• Enforce enforce_fallback_model_access by default, so configured and request-body fallbacks to models outside the caller's allowed models are skipped and the primary model's error is returned - PR #46127
• Add four opt-in Team admin editable fields for the member default budget, budget duration, TPM limit and RPM limit, so team admins can manage them from the team Settings tab - PR #46353
• Add an opt-in soft_budget entry under Team admin editable fields, so team admins can set their team's soft budget from the team Settings tab - PR #46485
• Evict and broadcast the cached user on every /user/update and /user/bulk_update, so a models-only or role-only change takes effect on every proxy right away instead of after the cache entry expires - PR #46500
• Add an opt-in metadata entry under Team admin editable fields so team admins can edit their team's metadata, while proxy-controlled metadata keys stay with proxy admins - PR #46506
• Honor end_user_id_jwt_field when a JWT maps to a virtual key, so spend rows, customer budgets and limits use the verified end-user claim over a body user or the x-litellm-end-user-id header - PR #46637
• Give /team/delete transactions a 30 second timeout, land concurrent first SSO sign-ins on one account instead of a 500, stop an intermittent 500 in the spend metadata merge, and keep key settings drafts when the form refreshes - PR #47043

SSO and SCIM

• Refuse an SSO sign-in that resolves no user id with HTTP 401, and fail a Microsoft sign-in when Graph returns an error instead of a profile, so a blank account session is never created - PR #44098
• Write a SCIM group's whole roster in one locked transaction, so POST, PUT and PATCH on /scim/v2/Groups with hundreds of members no longer time out at the load balancer - PR #45197
• Store the SSO exchange token with the same helper as password login, so the UI session cookie keeps its Secure flag on https deployments - PR #45840
• Fall back to the id_token claims when the generic SSO userinfo response is not JSON, and return a 401 auth_error naming the userinfo endpoint instead of a 500 when no usable id_token exists - PR #46187
• Resolve SCIM group member identities with one chunked query per push instead of one query per member, cutting an 851-member no-change PUT from about 11s on a 20k-user table - PR #46412
• Route SSO token exchange and userinfo calls for the generic, Google and Microsoft providers through LiteLLM's httpx client, so non-PKCE logins honor ssl_verify and SSL_VERIFY and IdP connection failures return a 401 naming the endpoint - PR #46547
• Share SAML sign-in state over Redis whenever Redis is configured, without enable_redis_auth_cache, so a callback that lands on another replica or worker no longer returns 401 - PR #46807

Model management

• Name the replacement for a deprecating deployment with model_info.successor_model, shown in GET /v1/model/deprecations and in the daily Slack deprecation alert - PR #41670
• Drop model capability flags such as supports_max_reasoning_effort from a deployment's litellm_params before calling the provider, and warn at load to move them to model_info - PR #46647

Usage and analytics

• Record failed requests by HTTP status and by key, team, user and model group, and serve them from the admin-only GET /gateway/errors/activity, with by_status_code added to GET /gateway/daily/activity - PR #45244
• Scope /global/activity/cache_hits to the caller's own and permitted-team logs for non-admin callers, including its key alias and model filter options - PR #46210

Proxy configuration

• Refuse POST /fallback and DELETE /fallback/{model} with a 400 when router_settings.fallbacks is set in config.yaml, matching /config/update - PR #45860
• Treat blank numeric environment variables as unset so the proxy no longer crash-loops at boot, and resolve blank guardrail litellm_params env values to the field default instead of silently dropping the guardrail - PR #46341
• Store list-valued audit log changes as JSON text, so changes that carry a list, such as ignoring or snoozing a deployment warning, are written to the audit log instead of failing - PR #46555
• Keep JSON nulls as null in audit log values instead of storing the string None, so a cleared budget reads as an empty value while keys stay masked - PR #46577
• Revert the LITELLM_FIPS_MODE startup gate and its loud password migration failure, because its FIPS probe misreports real enforcing runtimes - PR #46740

CLI and coding agents

• Add an installation_preference field (available, auto_install or required) to Claude Code marketplace plugins and emit it as installationPreference in marketplace.json, so Claude Desktop can install archive plugins automatically - PR #41430
• Grant private Claude Code skills through access groups with access_skill_names, so a group's teams and keys see them on /claude-code/plugins, and show group-granted skills as inherited on the team Skills card - PR #46294
• Run group subcommands such as credentials list in the lite shell under their real name, keep quoted JSON arguments together, and open the bare lite shell on the server from lite login - PR #46511

Deployment

• Add GET /health/deployment_warnings to list active warnings with severity, and PATCH /health/deployment_warnings/{warning_id} to ignore or snooze one for up to 30 days for every admin, with changes written to the audit log - PR #46347
• Chain RDS IAM database auth through the pod's IRSA role when STS refuses the direct web identity exchange for AWS_ROLE_NAME, so a database role that only trusts the pod role connects at startup - PR #46540
• Forward AWS_SESSION_TOKEN to the STS AssumeRole and direct RDS clients for RDS IAM auth, so the proxy starts with temporary AWS credentials instead of failing with InvalidClientTokenId - PR #46559

Terraform

• Stop sending blocked on litellm_user updates in the Terraform provider, so changing a user no longer fails with a 400 from /user/update - PR #45410
• Add an optional team_member_model_max_budget JSON string to litellm_team, read back from /team/info so out-of-band edits show as drift, and clear the caps when the attribute is removed - PR #46274
• Add opt-in write-only *_wo secret attributes with *_wo_version companions, such as credential_values_wo, to keep secrets out of state, and let litellm_mcp_server set client_id, client_secret_wo and auth_value_wo - PR #46538
• Accept any 2xx from the proxy on MCP server, model, key and organization requests, so a 201 or 202 no longer fails terraform apply or terraform destroy or leaves the resource out of state - PR #46550
• Stop the perpetual plan diff on litellm_mcp_server by making alias computed, keeping the configured spec_version when the proxy returns none, and storing an mcp_info with only unmodeled fields as empty - PR #46781
• Send additional_drop_params to the proxy as the deployment's drop list instead of deleting the keys it names, and read it back so drift shows in terraform plan - PR #46782

AI Integrations

Lens and Agent Traces

• Embed the independent Lens UI in the dashboard and delegate authenticated /lens/* requests to the standalone Lens service, so Lens deploys and releases on its own - PR #45529
• Load the Lens package stylesheet in the gateway, so embedded Lens shows its layout styles - PR #45829

Guardrails

• Add a built-in airia guardrail that checks prompts before the call and responses after it, blocking or redacting as Airia decides and failing closed on a malformed Gateway reply - PR #40784
• Scan inline PNG and JPEG images and text documents with the Bedrock guardrail, and block requests carrying attachments it cannot scan with a 400 unless skip_unscannable_attachments: true is set - PR #43383
• Refresh the HiddenLayer guardrail JWT off the event loop, so a 401 no longer blocks unrelated requests - PR #45437
• Stream detect-only post_call policy pipeline steps live when streaming_buffer_until_moderated is false, still scanning the full answer at the end of the stream - PR #45516
• Guardrail each content block of a streamed Bedrock Converse response on its own, so a masked block no longer corrupts the stream or breaks parallel tool call JSON - PR #45822
• Resolve a decision guardrail's model in the caller's team scope, so a guardrail on a team's own model no longer returns 502 and the call is charged to that team - PR #46085
• Send the request's top-level user string to Zscaler AI Guard in the execute-policy body - PR #46094
• Add a wingback guardrail that sends chat requests to Wingback for checks - PR #46136
• Add the Airia guardrail card and logo to the Guardrail Garden and the guardrail provider list - PR #46562
• Read the four streaming_* keys from the Noma v2 guardrail config for both noma_v2 and noma with use_v2: true, so buffering and scan cadence settings take effect - PR #46606
• Record every content filter match in logging_only mode as guardrail_flagged with risk score 10, keep scanning the response, and offer logging_only in the Admin UI mode picker - PR #46673
• Relay Bedrock passthrough converse-stream frames while post_call guardrails scan them, and scan Claude invoke-with-response-stream output too, ending a blocked stream with an AWS exception the SDK reads - PR #47157
• Send Bedrock Converse messages and tools to guardrails in the chat completions shape and write their masks back into the Converse request, returning a 500 when an edit cannot be written back safely - PR #47234

Logging and observability

Azure Storage

• Upload audit log events to Azure Storage as JSON files under audit_logs/ when audit_log_callbacks includes azure_storage - PR #43838

Datadog

• Tag Datadog spans with litellm.user_email for JWT-authenticated requests, and show model_info.description on /model_group/info and in the Model Hub table - PR #42749

FOCUS and Ternary

• Add a ternary callback that exports LiteLLM spend to Ternary as a daily FOCUS CSV, loading polars only on the first export and keeping rows by their own date under any Postgres timezone - PR #46106

General

• Redact the nested request body and the reasoning in provider_specific_fields from callback payloads when message redaction is on - PR #45923
• Accept every known logging callback on /health/services, so the Admin UI Test button works for loggers like zerobus instead of returning 400 Service must be in list, and return 400 for a known logger that is not configured - PR #46297
• Fail proxy startup with an error naming the logger when a configured logging callback's SDK is missing or outside the supported version range, instead of returning a 500 on the first request - PR #46378

Langfuse

• Log token usage for streamed /v1/responses calls, and chat calls bridged to the Responses API, in Langfuse instead of showing 0 input and 0 output tokens - PR #42212

OpenTelemetry

• Return the W3C traceresponse header on sampled responses when LITELLM_OTEL_V2=1, naming the trace and the gateway's server span, which carries the response's litellm.call_id - PR #42079
• Let a team or key set callback_vars.capture_message_content to no_content or span_only for its own OTel v2 destination, overriding the global setting - PR #44264
• Let a team or key set callback_vars.excluded_services to override the global OTel v2 excluded services for its own destination, where none excludes nothing - PR #45880

Prometheus

• Collect litellm_redis_circuit_breaker_state as one series per state across worker processes under PROMETHEUS_MULTIPROC_DIR, so recycled Celery or gunicorn workers stop piling up series - PR #46357
• Report the Redis circuit-breaker metrics through the Prometheus services logger, so a Redis cache no longer registers litellm_redis_circuit_breaker_* collectors on the default registry without being asked - PR #46484

S3

• Bound the S3 object key filename to 255 UTF-8 bytes, keeping the start of the response ID plus a SHA-256, so s3_v2 logs for long /v1/responses IDs upload to filesystem-backed stores and download with aws s3 cp - PR #40825
• Bound each s3_v2 upload attempt with S3_UPLOAD_ATTEMPT_TIMEOUT_SECONDS (default 30), add jitter to the backoff and retry 429s and transport errors within the same flush - PR #46340

Sentry

• Start Sentry at boot once SENTRY_DSN is set and keep request bodies and frame locals out of events unless SENTRY_SEND_DEFAULT_PII=true, return rate_limit_error on cooldown 429s, and export the Lens row counters to Prometheus - PR #46344

Telemetry

• Add opt-in telemetry in litellm.telemetry with a fixed allowlist of record fields and consent groups, where a group only counts when all its parent groups are on - PR #45484
• Aggregate telemetry records into fixed-bucket histogram rows and export each window as one JSON report over HTTP, retrying on 429 and 5xx responses - PR #45487
• Record request latency, status and provider attempts for the enabled telemetry groups, flushed every 5 minutes, with LITELLM_TELEMETRY_DISABLED=true turning everything off - PR #45489
• Store telemetry reports in the LiteLLM_TelemetryReport table when LITELLM_TELEMETRY_DESTINATION=local_table, keep one instance ID across restarts and workers, and add GET /telemetry/reports for admins to export them - PR #45490

Spend Tracking, Budgets and Rate Limiting

Cost tracking

• Bill Responses API image_generation tool calls and web search reported only in tool_usage.web_search.num_requests as tool_usage_cost - PR #41877
• Split one Azure PTU deployment across teams with model_info.ptu_shares, enforce each share as a per-team token ceiling, split the flat cost across teams in the daily rollup, and report PTU-hours in team usage - PR #43043
• Accept <provider>/<model-pattern> keys such as vertex_ai/claude-* in cost_discount_config, with a Model pattern field in the Admin UI Provider Discounts form - PR #43335
• Keep an explicit zero hourly cache write rate in pricing tiers instead of billing those writes at the ordinary write rate - PR #45545
• Bill a background Responses call once on a proxy with a database, to the creator's key, team, user, end user and tags, instead of again on every read and on the background poll - PR #45703
• Bill Gemini tool-use prompt tokens from URL context, code execution and File Search as input, so a response reporting 14,138 prompt tokens no longer records an input cost for only 23 - PR #46021
• Apply the Anthropic fast multiplier to the cache read and cache creation costs in cost_breakdown, so they add up to the billed input_cost - PR #46112
• Bill azure_ai Claude web searches per usage.server_tool_use.web_search_requests instead of once per response - PR #46114
• Bill a deployment with custom input_cost_per_token and output_cost_per_token at its own rates instead of the upstream x-litellm-response-cost header, so chained LiteLLM proxies honor the operator's price - PR #46120
• Bill a Bedrock inference-profile ARN that ends in an eu. or us. prefixed model id at that regional price row instead of the base model row - PR #46128
• Bill azure_ai Claude web searches per request for deployments that carry their own prices too, instead of once per response - PR #46135
• Bill Gemini and Vertex AI explicit context cache creation tokens from the cachedContents response as cache_creation_input_tokens, so cache creation is no longer under-billed - PR #46144
• Keep the Vertex AI cache creation token count out of request kwargs and clamp negative counts to 0, so a request body cannot change the billed tokens - PR #46152
• Scope /spend/tags to the caller's own and permitted-team logs for non-admin callers, and make its start_date and end_date parameters filter the aggregation window - PR #46194
• Price a chatgpt/ deployment through model_info.base_model when its own provider row has no rates, instead of logging $0 spend - PR #46318
• Split xai spend rows across input_cost and output_cost instead of booking the whole total as output - PR #46320
• Report a response_cost of 0 in x-litellm-response-cost on a response cache hit, so the header matches /spend/logs - PR #46471
• Price agentcore/search at 0.007 USD per query, matching AWS's 7 USD per 1,000 queries, instead of 0 - PR #46571
• Bill xAI Grok Voice realtime sessions by audio seconds plus client text messages, using the new optional input_cost_per_text_message pricing field, instead of logging $0 spend - PR #46600
• Price reasoning tokens of a bridged /v1/messages call from the chat completion the bridge translated, so cost_breakdown.reasoning_cost is always present and output_cost agrees with spend - PR #46726
• Bill a search tool that fails over to an unpriced backup at the backup's own price instead of the failed primary's input_cost_per_query - PR #46734
• Bill bridged /v1/messages Responses replies like native /v1/responses for reasoning tokens and the flex service tier - PR #47009

Spend logs and daily rollups

• Honor litellm_settings.disable_end_user_cost_tracking in spend logs, ClickHouse rows and the end-user tables, and stop failed requests from creating customer rows - PR #40945
• Bound spend log cleanup delete batches to the Postgres statement timeout instead of Prisma's 5 second transaction timeout, so slow batches stop failing with a 422 and expired logs get deleted - PR #43279
• Run POST /global/spend/refresh on the writer instead of the read replica, so it no longer fails and the Old Usage page shows the fresh monthly total - PR #45699
• Write spend log rows for /vllm chat and text completion calls to router models, which left key, team and user spend at zero, and for /azure and /azure_ai calls on proxies with a sync success callback - PR #45733
• Add general_settings.disable_requester_ip_address_logging to store a null caller IP in new spend log rows and mask IPs in stored headers and errors, also toggleable in Admin UI General Settings - PR #45804
• Move the call-id index aside with the other six indexes in db_scripts/partition_spend_logs.sql, so the next migration job no longer fails with an index-already-exists error after spend logs are partitioned - PR #46131

Budgets

• Enforce the personal max_budget of a user whose JWT carries the proxy admin role on LLM routes, as is already done for other users - PR #43070
• Add team_member_model_max_budget on /team/new and /team/update and model_max_budget on /team/member_update for per-model team member budgets, with over-budget requests following router_settings.fallbacks and matching editors in the Admin UI - PR #43887
• Report budget_reset_at on each model_max_budget_usage entry, and end monthly budget windows one calendar month after they open, counted in UTC - PR #45797
• Reset end-user spend in committed pages of RESET_BUDGET_JOB_BATCH_SIZE rows, so the budget reset job still commits when LiteLLM_EndUserTable has millions of rows - PR #46502

Rate limiting

• Add general_settings.disable_fallbacks_on_per_model_rate_limits so per-model key, team, organization and project limits (model_rpm_limit, model_tpm_limit) return 429 instead of retrying on a fallback - PR #45459
• Release the legacy rate limiter's slot only from an admitted Responses WebSocket, so rejected realtime attempts no longer lower a key's in-flight count under LEGACY_MULTI_INSTANCE_RATE_LIMITING - PR #46822

MCP Gateway

Auth and OAuth

• Request offline access and consent from Google upstream OAuth providers, so Google MCP connections receive refresh tokens - PR #41197
• Skip budget checks on zero-spend MCP discovery routes and JSON-RPC methods such as /mcp-rest/tools/list, /v1/mcp/tools, initialize and tools/list, so over-budget keys can still start MCP clients while tools/call stays enforced - PR #41276
• Keep the parsed key when the litellm_key_header_name header is absent from the request, so clients using Authorization or x-litellm-api-key no longer get a 401 and MCP toolsets load - PR #44469
• Scope gateway OAuth consent on aggregate /mcp to the servers in x-mcp-servers, and add opt-in general_settings.mcp_require_explicit_server_scope that answers 400 mcp_server_scope_required on a bare /mcp - PR #44957
• Let MCP server create and update bodies on /v1/mcp/server carry the four token_exchange_* fields, so saving an MCP server with OAuth token exchange no longer returns 401 - PR #45723
• Pass only the JSON-RPC method to MCP admission auth, so a tools/call that adds an extra model field is still checked against the key's budget - PR #45844
• Preserve existing scopes when a managed OAuth server asks for more permissions, and relay insufficient-scope denials as a 403 OAuth challenge so MCP clients can request the missing scope - PR #45886
• Add optional id_jag_resource_client_id and id_jag_resource_client_secret so ID-JAG leg 2 can use its own confidential client instead of the IdP client - PR #46211
• Persist the first DCR client for config-declared MCP servers whoever registers it when PROXY_BASE_URL is set, so non-admin users stop being signed out each time the access token expires - PR #46544
• Accept a validated JWT as the OBO subject token after virtual-key mapping, and answer 401 with IdP authentication instructions when it is missing or invalid - PR #46549
• Accept an ID-JAG as an RFC 7523 jwt-bearer grant on the gateway token endpoint, and keep the IdP id_token, SAML assertion or OIDC refresh token from login as the subject for oauth2_id_jag servers - PR #47058
• Cap an ID-JAG access token to the MCP routes when its signed resource is only {base}/mcp, and refuse an unknown resource with 400 invalid_target - PR #47138
• Add Authorization URL, Token URL and Scopes fields to the OAuth Delegate form for MCP servers, and register one shared gateway client on {PROXY_BASE_URL}/callback for DCR bridge servers so upstreams that restrict redirect domains accept sign-in - PR #47153

Rate limits and guardrails

• Pass the caller's bearer privately to MCP pre-call guardrails after JWT-to-key mapping, so the Agent 365 guardrail stops rejecting valid tool calls - PR #45746
• Exclude the gateway's own key from the caller bearer passed to MCP guardrails when it is sent in both x-litellm-api-key and Authorization, while caller JWTs still reach the guardrail - PR #45897
• Stop proxy-started legacy MCP listeners from using up a server's rpm, so a client gets its full configured budget - PR #46982

Protocol and catalog

• Keep one upstream MCP session per gateway mcp-session-id and server across tool calls, so stateful upstream servers remember state between calls - PR #42741
• Leave out tool texts too long to embed instead of failing every MCP tool search with an embedding length error - PR #43554
• Forward MCP progress messages through the gateway with output guardrails applied, and cancel in-flight upstream tool calls promptly instead of waiting for the session lock - PR #46134
• Relay upstream tool, prompt and resource change notifications and resource subscriptions through caller-scoped streams, rechecking access before delivery - PR #46153
• Route a bare tool name on a single-server scope (/{server_name}/mcp or x-mcp-servers) to that server, so tools/call stops answering Tool not found on a replica that has not listed it - PR #46225
• Advertise supported notification capabilities of authorized upstream servers in server/discover, keep legacy initialize flags, and limit optional probes to the remaining listing budget - PR #46264
• Resolve get_tool_schema and call_tool on /mcp/proxy from the server that owns the tool_id, so slow sibling servers no longer delay the call - PR #46545
• Add general_settings.mcp_disabled_capabilities to stop advertising prompts and resources, with matching switches on the Admin UI MCP network settings card - PR #46563
• Read the first Router stream chunk through __aiter__, so streamed /v1/chat/completions with an auto-run MCP tool no longer returns a 500 - PR #46578

Admin MCP and UI

• Add mcp_tool_permissions to access groups, a per-server tool allowlist that keys, teams and agents inherit, where an empty list denies every tool on that server, with create, edit and detail pages in the Admin UI - PR #46300
• Scope the MCP server selector on team, key and user forms to the servers they can reach, and add user_id to GET /v1/mcp/server to list servers for one user - PR #46565

Performance / Loadbalancing / Reliability improvements

Auto Router and model routing

• Keep the region, RPM and param checks running when the router cannot count a request's tokens and skip only the context-window check, so an EU-pinned end user's audio or image prompt no longer reaches a US deployment - PR #44544
• Let an exact same-model base_model widen a non-Azure deployment's context window in the router, so a prompt that fits the window shown by /model/info is no longer rejected - PR #46197
• Report a fallback skipped for access or budget as skipped instead of failed, so the 500 no longer repeats the primary error as if the fallback had been called - PR #46291
• Add fallback_on_unsupported_params to Router and router_settings, so setting it to False raises a param rejection instead of answering from a fallback model - PR #46350
• Read the first stream chunk inside Router chat completions, so a stream that fails before any token spends num_retries and walks the fallbacks like a non-streaming request - PR #46493

Caching, database and runtime

• Track management audit log writes and drain them for up to 5 seconds on shutdown, so a SIGTERM during a deploy no longer drops audit rows for calls that returned 200 - PR #43594
• Sign RDS IAM database tokens for the region in the RDS hostname, with AWS_RDS_REGION and AWS_RDS_READ_REPLICA_REGION overrides, so cross-region writers and read replicas connect - PR #44782
• Build the served text of a finished /v1/messages stream in a worker thread and stringify Redis cache values only when set_verbose is on, so other requests no longer wait on the event loop - PR #45252
• Roll back and retry a boot-time migration that times out waiting for a table lock, so a long LiteLLM_SpendLogs reader no longer blocks every pod's spend log writes during a rollout - PR #45389
• Start at most one cold MCP schema validation worker per CPU the proxy can run on - PR #45791
• Broadcast a cache invalidation to every instance on team writes, so team edits apply on all instances right away instead of after up to 60 seconds - PR #46113
• Print a startup warning when --max_requests_before_restart runs with a single worker, and say in its help text that it is only safe with 2 or more workers - PR #46185
• Bound the SIGTERM wait on uvicorn and gunicorn by GRACEFUL_SHUTDOWN_TIMEOUT, so a request still running after the timeout is cancelled and the proxy's shutdown teardown still runs - PR #46186
• Start new migrations with SET LOCAL lock_timeout = '10s' in their SQL instead of a connection option, so pooled database URLs boot and a migration stuck behind a table lock times out and retries - PR #46329
• Resend a POST or PATCH on a fresh connection only when nothing reached the upstream, so a request the provider already read is no longer run and billed twice after a dropped connection - PR #46336
• Restore the 15 second default for AIOHTTP_KEEPALIVE_TIMEOUT in the Python and Rust HTTP clients, so gateways that close idle connections stop causing 500s on the first request after a quiet spell - PR #46355
• Keep response cache hits free of fields the provider never sent, so a hit no longer adds system_fingerprint: null to the body - PR #46359
• Compress gzip responses over 16 KiB in a worker thread instead of on the event loop, so large embeddings responses no longer stall health checks and other requests - PR #46362
• Run prisma migrate deploy inside the pinned migration coordinator lock and turn off Prisma's session advisory lock, so a restart behind a transaction pooler no longer waits 600 seconds on a stranded lock - PR #46368
• Start the telemetry_reports migration with SET LOCAL lock_timeout, so a boot behind a held LiteLLM_TelemetryReport lock gives up and retries instead of waiting forever - PR #46386
• Default every Redis client to a 5 second socket timeout when none is set and raise the REDIS_SOCKET_TIMEOUT default from 0.1 to 5 seconds, so a hung Redis no longer blocks requests indefinitely - PR #46501
• Finish Prisma engine requests when their caller is cancelled during shutdown, so the final spend flush no longer fails and daily user and team spend for 200 responses is kept - PR #46503
• Accept database passwords with spaces or quotes in the v2 migration lock, redact the password from lock errors and report P1001 when the database is unreachable - PR #46592
• Build the /public/providers/fields and /public/agents/fields responses once per process and serve them from memory instead of re-reading and re-validating the provider file on every request - PR #46617
• Hold only Prisma's own advisory lock during prisma migrate deploy, so a booting pod no longer times out behind a replica building request log indexes - PR #46660
• Limit gzip to 35 dashboard analytics GET routes and compress every eligible response in a worker thread, so inference and other API responses are no longer compressed on the proxy - PR #46712

Docker and Helm

• Support a separate hostname per component in the componentized Helm chart with ingress.componentHosts, so the UI, backend and gateway each get their own DNS name and the UI finds the backend and gateway at runtime - PR #46360
• Read the database and Redis host or full URL from a Kubernetes Secret with the new hostKey and urlKey fields on each passwordSecret block, so host is no longer required in the values file - PR #46410
• Add a litellm-gateway command to quickstart installs with status, up, down, restart, logs, open, update and uninstall, and open a menu when the install command is rerun on an existing install - PR #46486
• Add db.useIAMAuth and db.useAzureEntraAuth to the litellm-helm chart, so RDS IAM and Azure Entra users start the proxy with a minted token and no password Secret - PR #46534
• Expose redis.ssl, redis.sslCertReqs and redis.sslCaCerts in the helm/litellm chart, rendered as REDIS_SSL, REDIS_SSL_CERT_REQS and REDIS_SSL_CA_CERTS for every component - PR #46536

Dependency updates

• Allow openai>=2.20.0,<4.0.0 so LiteLLM installs next to OpenAI SDK 3, while SDK 2 users keep the same minimum - PR #44927
• Bump mlflow to 3.17.0 in the lockfile and drop its osv-scanner suppression - PR #45777

Documentation Updates

• Point the GCP one-click deploy button and tutorial at the terraform-google-litellm mirror repository - Commit 320536d

Tests, CI and Internal Changes

These 218 changes touch tests, CI, contributor tooling, refactors, Rust runtime scaffolding that is not yet wired into a user-facing path, or product changes that were reverted before the cut. They do not change proxy or SDK behavior on their own. Changes listed as commits have no public PR

Integration and e2e tests (66)

• Add an e2e case that checks persisted spend for configured pricing - PR #41566
• Cover the agents_a2a endpoint contracts with integration tests - PR #44650
• Cover the images endpoint contracts with integration tests - PR #44653
• Cover the small_inference endpoint contracts with integration tests - PR #44654
• Cover the scim endpoint contracts with integration tests - PR #44656
• Cover the embeddings endpoint contracts with integration tests - PR #44660
• Cover the mcp_runtime endpoint contracts with integration tests - PR #44671
• Cover the keys_auth endpoint contracts with integration tests - PR #44674
• Cover the google_genai endpoint contracts with integration tests - PR #44680
• Cover the model_management endpoint contracts with integration tests - PR #44684
• Cover the prompts_tags_caching endpoint contracts with integration tests - PR #44686
• Cover the mcp_management endpoint contracts with integration tests - PR #44687
• Cover the chat_completions endpoint contracts with integration tests - PR #44688
• Cover the realtime endpoint contracts with integration tests - PR #44689
• Cover the pass_through endpoint contracts with integration tests - PR #44690
• Cover the videos_containers endpoint contracts with integration tests - PR #44691
• Cover the fine_tuning endpoint contracts with integration tests - PR #44697
• Cover the audio endpoint contracts with integration tests - PR #44699
• Cover the files_batches endpoint contracts with integration tests - PR #44703
• Cover the anthropic_messages endpoint contracts with integration tests - PR #44704
• Cover the teams endpoint contracts with integration tests - PR #44705
• Cover the guardrails_policies endpoint contracts with integration tests - PR #44722
• Cover the orgs_users_customers endpoint contracts with integration tests - PR #44725
• Cover the responses endpoint contracts with integration tests - PR #44733
• Pin the assistants and evals endpoint contracts in integration tests - PR #44734
• Cover the budgets_spend endpoint contracts with integration tests - PR #44736
• Cover the config_utils endpoint contracts with integration tests - PR #44737
• Cover the vector_stores endpoint contracts with integration tests - PR #44739
• Cover the health_routing endpoint contracts with integration tests - PR #44744
• Make MCP scope-discovery server aliases digits-only in the integration test - PR #45675
• Model the team membership lookup in MCP DCR admission fixtures and assert Decisions call types per route - PR #45917
• Pin the What's new catalog to a local copy in the canary egress rig - PR #45925
• Assert lock-timeout retry and exhaustion in migration tests instead of an unbounded wait - PR #45955
• Retry the owned Redis test helper on another port when its port is already served - PR #46019
• Attribute pass-through probe traffic in MCP tests to the caller instead of the whole peer log - PR #46020
• Give the ownership test proxy its own DB pool headroom - PR #46025
• Isolate every multi-worker integration suite by default - PR #46027
• Migrate before the timed owned proxy launch and gate the credential probe on every worker - PR #46030
• Give chaos and timing tests a harness API and gate raw signals - PR #46031
• Cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies - PR #46037
• Rename the e2e load tests to perf and consolidate the opt-in into E2E_PERF_TESTS - PR #46077
• Wait for SIGSTOP to take effect before handing back a suspended proxy process in integration tests - PR #46088
• Isolate shared proxy state behind six order-dependent flaky integration tests - PR #46172
• Remove six test setup causes of CircleCI integration failures - PR #46490
• Update the breakpoint cache mark census expectation after a cache control change - PR #46509
• Fix three more CircleCI integration failures caused by test setup - PR #46510
• Delete the flaky team model TPM concurrency test - PR #46541
• Update stale CircleCI integration expectations - PR #46570
• Cache the two-turn tool_search guardrail replays in integration tests - PR #46614
• Expect the list-position index on streamed thinking blocks in the integration suite - PR #46618
• Extend runtime regression contracts for Messages and the SDK - PR #46641
• Pin the xAI reported-cost split and the coordinator-held Prisma lock wait in CI tests - PR #46645
• Expect signed empty thinking blocks to survive /v1/messages replay in the integration suite - PR #46686
• Add an e2e perf rig with out-of-process mocks, a PerformanceTestCase runner, per-request limits and the small_bodies_5rps case - PR #46713
• Add the mixed_size_requests_2rps perf case with coding-agent bodies, tools, JWT and key callers and two guardrails - PR #46717
• Remove BUG skips for three integration tests that pass on main - PR #46725
• Drop inherited provider API keys from the integration runner env - PR #46733
• Wait on the right signal in five flaky integration contracts - PR #46737
• Hand every harness server a port from one per-worker block - PR #46761
• Attach e2e perf case metrics and limits as JUnit properties - PR #46774
• Start perf users staggered with jitter and give tool-call streams a first token - PR #46776
• Build OpenAI and Anthropic SDK models before integration tests start threads - PR #46797
• Count the Postgres statements and Redis commands a request waits on in e2e perf datastore limits, and report background work on its own - PR #46798
• Send the Connection close header on every integration browser request - PR #46805
• Wait for the gen-AI span before reading the excluded-services operator trace - PR #46846
• Drain SDK logging in the integration harness so lifecycle tests stay stable - PR #47143

Refactors and typing (43)

• Clear 670 basedpyright Any errors by validating provider payloads at the boundary - PR #45388
• Replace Any with proven types in 3 files - PR #45565
• Import LiteLLM_SkillsTable from litellm.models - Commit 6c71452
• Import team and user table models from litellm.models - Commit 3fafd00
• Move LitellmUserRoles and KeyManagementRoutes into litellm.types.proxy.auth.user_roles - Commit a3f05fb
• Move the typesafe pass-through logging handler out of the proxy package - Commit 396e8d1
• Move Anthropic pass-through SSE stream assembly into litellm/llms - Commit 7f5abff
• Move CommonProxyErrors into litellm/types/proxy - Commit 38880fd
• Move is_jwt into litellm.types.proxy.auth.user_api_key_auth - Commit 7831d8a
• Move the prompt cache observation lookup out of the proxy hook - Commit ebf2156
• Move raw SSE stream assembly helpers out of proxy guardrails - Commit 3981228
• Move Slack alerting leaf types out of proxy/_types.py - Commit 9c8083e
• Move the prompt cache pricing helper out of the proxy package - Commit c397b57
• Move PassThroughGuardrailSettings into the pass-through types module - Commit 8f047e0
• Move stream_chunk_builder into streaming_chunk_builder_utils - Commit 9e3cd2f
• Move NewSkillRequest to litellm.models.skills - Commit 7383f59
• Clean up typing and duplicated export job code in per-user Copilot auth, the Ternary FOCUS export and telemetry, with no behavior change - PR #45769
• Move cache token extractors out of proxy savings - Commit 8d510a6
• Move responses_api_bridge_check out of litellm.main - Commit d02dc43
• Import CustomLogger and CustomLLM under TYPE_CHECKING in adapter TypedDicts - Commit 13191e5
• Move path_utils from proxy common_utils to litellm_core_utils - Commit 9d559d8
• Import litellm lazily in Prometheus types helpers - Commit 4606bf5
• Move AUTH_OBJECTS_TARGET into litellm.constants - Commit 12c36b0
• Move complexity router request models next to their base config - Commit 0fc6d4d
• Import proxy helpers inside the repository methods that use them - Commit d1c4d06
• Move ColdStorageHandler out of the proxy into litellm_core_utils - Commit c9e0ec8
• Type UserAPIKeyAuth, its token view and hash_token in place - Commit dec8ec5
• Move UserAPIKeyAuth, its token view and hash_token into litellm.types.proxy.auth.user_api_key_auth - Commit 9e7e5c2
• Annotate GenerateRequestBase containers and check_max_budget - Commit b7d48ec
• Move GenerateRequestBase and LiteLLM_ObjectPermissionBase into litellm.types.proxy.management_endpoints.request_base - Commit 981710f
• Move resource_ownership out of the proxy into litellm_core_utils - Commit c582bc6
• Import UserAPIKeyAuth from litellm.types.proxy.auth in the SDK - Commit a1e5c90
• Annotate user request metadata and check_user_info validators - Commit a859d84
• Annotate key request containers and drop_blank_team_id - Commit 2374b0c
• Move key request models into litellm.types.proxy.management_endpoints.key_requests - Commit c9d706a
• Move user request models into litellm.types.proxy.management_endpoints.user_requests - Commit 513e7a8
• Move MemberDeleteRequest into litellm.types.proxy.management_endpoints.team_endpoints - Commit 2011004
• Replace Any with proven types in 12 files - PR #45943
• Replace Any with proven types in 6 files - PR #46042
• Rename is_claude_4_5_on_bedrock to supports_bedrock_prompt_cache_ttl - PR #46058
• Name the pre-content stream retry helpers for both stream paths and let the retry loop take its attempt function - PR #46474
• Clear 1143 basedpyright Any errors across 273 files - PR #46574
• Replace Any with proven types in 24 files - PR #46663

Rust core (28)

• Bump the Rust workspace's serde_with to 3.21.0 to clear osv-scan - PR #45498
• Type the AWS connection params in auth-aws - PR #45610
• Give Messages configs typed LiteLLM params - PR #45611
• Add native DeepSeek /v1/messages support to the Rust core, opted in with LITELLM_RUST - PR #45614
• Add native Vertex AI Claude /v1/messages support to the Rust core, opted in with LITELLM_RUST - PR #45626
• Type the Vertex AI connection params in auth-gcp - PR #45641
• Use one LitellmParams type for a config and a caller - PR #45642
• Route Python state reads through host modules in the Rust bridge - PR #45689
• Add native EdenAI /v1/messages support to the Rust core and pass the provider's billed cost to spend logging - PR #45698
• Add a live provider harness for the Messages route - PR #45823
• Share provider constants through llms-types and restructure the Rust AGENTS.md - PR #45850
• Add native MiniMax /v1/messages support to the Rust core, including SSE streaming and cache hints - PR #45896
• Add native Tencent /v1/messages support to the Rust core, including SSE streaming and cache hints - PR #45899
• Add native Bedrock Claude Platform and Bedrock Mantle /v1/messages support to the Rust core - PR #45908
• Add native OpenRouter /v1/messages support to the Rust core - PR #45910
• Use one stream codec shape per API format - PR #45996
• Add native OpenRouter /v1/responses support to the Rust core - PR #46000
• Run the Python driver as one async body in host-python - PR #46235
• Use a generator-style execution protocol with stream step names in the Rust bridge - PR #46282
• Return an unstarted Execution from native entrypoints and drive it in Python - PR #46299
• Move host-python driver tests out of the inline module - PR #46325
• Collect host-python failures in a thiserror Error enum - PR #46361
• Pin Messages and Responses callback contracts on the native path - PR #46443
• Assert native serving through the x-litellm-rust response marker - PR #46531
• Select native routes per test without LITELLM_RUST - PR #46620
• Remove the LoggerRule concept from the Rust bridge catalog - PR #46711
• Move the route host modules under host/routes - PR #46786
• Let each route host pick the body params - PR #46787

CI and release tooling (22)

• Shard the integration suites by recorded timings and give their workers private databases and fixtures - PR #44146
• Fix stale and flaky tests across CircleCI, GHA and Buildkite - PR #45530
• Finish the lint, unit and smoke check layout - PR #45706
• Move packaging and install checks off the CircleCI build_and_test job - PR #45757
• Use CircleCI native timing splits and fewer parallel nodes for integration suites - PR #45837
• Gate undocumented POST routes in the proxy OpenAPI document - PR #45843
• Give heavy integration suite workers private proxies, databases, Redis and provider edges - PR #45863
• Run every CircleCI check from the integration workflow - PR #45938
• Trigger the gateway-dev deploy on every push to main - PR #45958
• Remove pass-through and user_api_key_auth coverage scripts that no CI job runs - PR #45969
• Move redis-compat and MCP unit suites from CircleCI to GitHub Actions - PR #46026
• Run integration shards on large machines and size workers per suite - PR #46034
• Bump litellm to 1.107.0, litellm-enterprise to 0.1.77 and litellm-proxy-extras to 0.4.109 - PR #46239
• Run the two slowest unit shards on 4 workers - PR #46743
• Run the mcp-elicitation shard with dist load so the 74 params of test_interactions.py spread over its 4 workers - PR #46770
• Run the MCP interactions integration contract in its own unit shard so the mcp-elicitation shard stops timing out - PR #46773
• Run MCP interactions and Redis cluster IAM integration tests only in CircleCI - PR #46780
• Restore loadscope on the mcp-interactions shard so its MCP subscription cases do not race across workers - PR #46783
• Classify token_count_framing and token_count_fallback_model in the cost map schema generator - PR #46784
• Bump claude-agent-sdk to 0.2.165 in the CI dependency group to clear osv-scan - PR #46976
• Bump litellm-enterprise to 0.1.78 and litellm-proxy-extras to 0.4.110 so the release candidate publishes its new migrations - PR #47230
• Rebuild the Admin UI bundle for rc/1.107.0 - PR #47232

Cost calculation tests (20)

• Check that a streamed request bills the same SpendLogs row as the non-streamed request - PR #45730
• Move the OpenAI cost tracking cases from cost_tracking_cases.json to CostTrackingTestCase literals - PR #45879
• Bill Decisions spend from captured provider responses on /v1/decisions and /v1/systemone in cost tests - PR #45902
• Add tier 1 Claude 5 and gpt-6 native cost cases - PR #46124
• Add BridgeCostParityTestCase with a gpt-6-sol /v1/messages to Responses pair - PR #46130
• Add Azure gpt-6 /v1/messages bridge cost parity pairs - PR #46139
• Add Claude 5 /v1/responses bridge cost parity pairs - PR #46140
• Add Bedrock Converse Claude 5 /v1/responses bridge cost parity pairs - PR #46141
• Add gpt-6 chat to Responses bridge cost parity pairs - PR #46143
• Add long context tier cost cases for gpt-6 (272k) and claude-haiku-5-5 (100k) - PR #46147
• Add xAI grok-4.7, Bedrock runtime chat completions and Bedrock Mantle cost cases - PR #46154
• Add tier 1 gpt cache_read cases and recapture input_text usage shapes - PR #46623
• Let CostMapEntry hold flex, priority and ultrafast cache and tier prices - PR #46629
• Add best-guess tier 1 cases for vertex_ai and azure_ai Claude and gpt-6 cache_write_5m - PR #46687
• Add tier 1 Claude Messages stream twins and recapture drifted usage shapes - PR #46690
• Add azure_ai claude-haiku-5-5 cases and gpt-6 service tier cache prices - PR #46694
• Cover turn-2 usage shapes across bridged previous_response_id, cached streams and pass-through cache writes - PR #46703
• Cover custom deployment aliases, off-peak pricing, bridge usage shapes and duplicate spend rows - PR #46708
• Pin LiteLLM's own prompt count over a tool-call history when the stream carries no usage - PR #46710
• Assert the streamed usage.cost matches the spend row on every cost case - PR #46762

Translation test matrix (11)

• Add claude-haiku-5-5 basic translation cases to the integration suite - PR #45176
• Add StreamingTranslationTestCase with a Claude Sonnet 5.5 chat completions case - PR #46151
• Share rich request and response templates across translation tests - PR #46546
• Recapture Bedrock Invoke claude-sonnet-4-6 bases after a usage shape change - PR #46579
• Add gpt-6-astra translation cases - PR #46580
• Add claude-fable-5-1 translation cases - PR #46581
• Share rich fixtures across Claude, GPT and Gemini providers in translation tests - PR #46585
• Compare coding-agent response contracts across model tiers - PR #46597
• Cover response contracts across translation endpoints - PR #46632
• Add a claude-sonnet-5-5 Anthropic Messages service_tier case - PR #46678
• Recapture Azure gpt-6-astra content filters - PR #46691

Test suite migration (11)

• Move offline router, retry and latency tests from local_testing to tests/unit - PR #45551
• Move offline logging, secret manager and provider tests from legacy directories to tests/unit - PR #45554
• Move offline proxy auth, hook, spend and pass-through tests to tests/unit - PR #45568
• Move offline Anthropic prompt caching tests to tests/unit - PR #45616
• Migrate SDK and proxy-booting legacy CircleCI tests to their tiers - PR #45758
• Migrate tests/llm_translation into the unit, integration and e2e tiers - PR #45759
• Migrate local_testing SDK and provider tests out of build_and_test - PR #45760
• Move routing, caching and callback tests out of local_testing - PR #45761
• Migrate legacy proxy access-check, schema migration and MCP tests to the new tiers - PR #45937
• Migrate tests/proxy_behavior into the integration and unit tiers - PR #45939
• Migrate legacy unit directories and unrun folders into tests/unit, tests/integration and tests/e2e - PR #45940

Unit tests (7)

• Add a unit test that SDK-echoed null tool_calls never reach the Headroom /v1/compress call - PR #39667
• Wait on conditions instead of wall-clock time in guardrail parallelism and scope-option tests - PR #45571
• Let the cold concurrent schema validation test set its own timeout so it stops flaking on a single CPU core - PR #46184
• Wait on the work itself instead of fixed sleeps in unit tests - PR #46261
• Cover the OTel V2 otel callback on /health/services - PR #46394
• Baseline breaker metrics from live state after gc in the Redis cache tests - PR #46652
• Wait for the wedged Redis to be reached instead of a fixed 0.1s sleep in the team roster sync tests - PR #46727

Contributor tooling (7)

• Ignore .mypy_cache in git - Commit 0519882
• Load nested AGENTS.md files as Greptile review context scoped to their directories - PR #45636
• Ban semicolons and colons in human-facing text in the AGENTS.md files - PR #45684
• Remove committed PR screenshots from the repo - PR #45816
• Drop Greptile from the PR template, AGENTS.md and CI - PR #46013
• Steer tests away from implementation details and deliberate reverts in AGENTS.md - PR #46317
• Update the CODEOWNERS entry for a maintainer handle - PR #46452

Superseded within this release (3)

• Serve the bundled whats_new.json on Home by default, replaced before the cut by the blog-sourced What's new cards that already shipped in v1.106.0 - PR #45934
• Add azure_ai/claude-fable-5-5, removed again by a later PR in this release - PR #46201
• Remove azure_ai/claude-fable-5-5 again because Azure does not list the model - PR #46428

PR roll-up by ownership area

Customer-facing changes: 382. Tests, CI and internal changes: 218. Total: 600

• Other (tests, CI, internal): 218
• Models & Providers: 109
• LLM API Endpoints: 65
• Auth & Management: 45
• UI: 35
• Spend / Budgets / Rate Limits: 34
• Performance: 33
• MCP: 27
• Logging & Tracing: 20
• Guardrails: 13
• Docs: 1

New Contributors

• @Acacian made their first contribution in PR #41528
• @adityajoy-1902 made their first contribution in PR #42079
• @airia-open-source made their first contribution in PR #40784
• @akram1089 made their first contribution in PR #43594
• @Apeksha9389 made their first contribution in PR #46471
• @Arjun-Aravind made their first contribution in PR #45437
• @Aryan-Pardeshi made their first contribution in PR #36460
• @debanshd made their first contribution in PR #41683
• @ErenAta16 made their first contribution in PR #34653
• @jsvisa made their first contribution in PR #46021
• @le-shi made their first contribution in PR #32599
• @link89 made their first contribution in PR #40825
• @marcusgreer made their first contribution in PR #45410
• @matthowardcohere made their first contribution in PR #46274
• @mayank-affirm made their first contribution in PR #41747
• @narmstrong-berri made their first contribution in PR #46492
• @rameswaran-mohan made their first contribution in PR #41670
• @Raushankumar0720 made their first contribution in PR #42887
• @RonP3 made their first contribution in PR #43791
• @ryuto-takemoto made their first contribution in PR #44544
• @setarm01 made their first contribution in PR #46197
• @shubhamjain0594 made their first contribution in PR #45579
• @soham96 made their first contribution in PR #44167
• @sukhdeepjohar made their first contribution in PR #45107
• @Sushit-prog made their first contribution in PR #38284
• @TaylorHawkes made their first contribution in PR #41669
• @thementalcoding made their first contribution in PR #43710
• @VimarshDwivedi made their first contribution in PR #46184
• @WongAnneLister made their first contribution in PR #36346
• @yuefdev made their first contribution in PR #41677

Full Changelog

https://github.com/BerriAI/litellm/compare/v1.106.0-rc.1...v1.107.0-rc.1