LiteLLM v1.106.0 adds Claude Haiku 5.5, a Rust Lens service and MCP rate limits
Deploy this version​
- Docker
- Pip
docker run \
-e LITELLM_MASTER_KEY=sk-<paste-a-long-random-key> \
-e DATABASE_URL=postgresql://<user>:<password>@<host>:5432/<dbname> \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:1.106.0
pip install litellm==1.106.0
This release is published as ghcr.io/berriai/litellm:v1.106.0. See the GitHub release and the full releases page
These notes cover everything since v1.105.0. Changes added to the release line after rc.1, through rc.4 and the fixes backported before the stable tag, are under Included after the v1.106.0-rc.1 cut
Customer-facing changes come first. Test, CI and internal changes are listed at the bottom
These callouts cover user-facing behavior that differs from v1.105.0, the previous stable release
The response cache skips requests with more than 4 messages by default. The new cache_params.max_messages defaults to 4, so longer conversations and Responses inputs are neither looked up nor stored, exact or semantic. Raise it, or set it to null to remove the limit. See PR #43878
Responses WebSocket sessions are capped at 60 minutes by default. Sessions on /v1/responses close with code 1000 after general_settings.responses_websocket_session_limit_seconds (default 3600, maximum 7200), idle or not. The old 30 second first-message timeout is gone. See PR #44433
The caller's LiteLLM key is never forwarded to MCP servers, including x-litellm-api-key configured under a server's Extra Headers. Give upstream servers their own credential through a static header or OAuth. See PR #45401
PATCH /credentials/{name} no longer renames a credential. A different credential_name returns 400. Use the new display_name to relabel a credential, or create a new one to change its name. See PR #43148
Decisions requests that send media a model cannot read now return 400. images on /v1/systemone, audio and videos on either route, and TypeSafe assets used to be dropped with a 200. They now return 400 naming the field unless drop_params is set, and OpenAI decision models now receive System One images, which are billed. See PR #46628
/v1/responses/compact now checks previous_response_id the same way /v1/responses does. Another user's or team's id returns 403, and so does an id the proxy never issued unless general_settings.allow_unmanaged_response_ids is true. Ids handed out by the websocket /v1/responses route are refused for every key. See PR #46827
Key Highlights​
- New models on day one: Claude Haiku 5.5 across Anthropic, Bedrock, Bedrock Mantle, Vertex AI, Azure AI and Perplexity, Kimi K3 and GLM 5.3 on Bedrock, Mistral Large 4, and Gemini Nano Banana 2.1, among 59 new catalog entries
- Lens as a standalone Rust service: trace ingestion and investigations move out of the gateway, with always-on investigations, a Findings inbox, signals, end-user feedback and datasets built from real traces
- Admin UI Home page: the dashboard opens on a Home page with a usage summary, recent launches, blog updates and a model leaderboard, alongside a Decisions tab in the Playground, a redesigned Usage page and a Cmd+K command palette
- More Decisions providers and media: Laya, Bespoke Nimble and Databricks serve Decisions and act as Auto Router classifiers, Cloudflare Clef reads images, Clef-omni reads audio and video, and TypeSafe Jev 1.14 takes images. Reka joins as an OpenAI-compatible provider
- MCP Gateway controls: rate limits cover every MCP operation with a new server-level rpm, OAuth adds Client ID Metadata Document support, and the caller's LiteLLM key is never forwarded upstream
- Observability: Arize on OTel v2 works per team, Prometheus can cap series per metric, and
litellm.agent()gains an in-process tool-calling loop
Included after the v1.106.0-rc.1 cut​
- Cloudflare Clef reads images on both Decisions routes, the new
cloudflare/clef-omnireads audio and video, and TypeSafe Jev 1.14 (typesafe/jev-1.14.0) takes images throughassets,boxquestions and the opt-inTypeSafe-Danger-Zone-*headers (rc.4) - PR #46628 - Deployments can set
model_info.mode: decisions, which behaves the same asmode: evaluationfor health checks and the Admin UI (rc.4) - PR #46628 - Home page What's new cards show the newest LiteLLM blog posts tagged
day 0 support, so new launches appear without a proxy upgrade (rc.2) - PR #46309 - The yes / no Decisions answer type is labelled "Yes / No" instead of
noul(rc.2) - PR #45933 - Setting
custom_prometheus_metadata_labels: ["project_alias"]no longer breaks the project per-model rate limit gauges or skips the other metrics for project keys (rc.3) - PR #46491 streaming_buffer_until_moderated: trueon the Decision Model guardrail holds a streamed answer until the decision model passes it (rc.3) - PR #46383/v1/responses/compactworks with the caller's ownprevious_response_id, which used to fail with a 400 from OpenAI - PR #46827- Redis Cluster with
max_connectionsset waits for a free connection under load instead of switching Redis off for 60 seconds - PR #47006
New Providers and Endpoints​
New Providers (3 new providers)​
| Provider | Supported LiteLLM Endpoints | Description |
|---|---|---|
| Reka | /v1/chat/completions, /v1/responses, /v1/messages | Reka as an OpenAI-compatible provider under reka/, with Responses and Messages bridged to its chat completions |
| Laya | /v1/decisions, Auto Router | Laya decision models and an OSS classifier for the Auto Router |
| Bespoke | /v1/systemone, Auto Router | Bespoke Nimble decision models and an OSS classifier for the Auto Router |
Expanded provider endpoint support​
| Provider | Endpoint | What you can do |
|---|---|---|
| Databricks | /v1/decisions, Auto Router | Call Databricks ai_decide as a Decisions provider and Auto Router classifier |
New Models / Updated Models​
New Model Support (59 new models)​
Counts represent new catalog identifiers compared with v1.105.0, including aliases and regional variants. Prices below are the values bundled in this release, in USD; runtime pricing-map reloads can update them. Input and output columns show base token rates; long-context, cache, image-token, and other specialized rates depend on the model
| Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features / special pricing |
|---|---|---|---|---|---|
| Amazon Bedrock | amazon.nova-2-5-sonic | - | $0.33 | $2.75 | Realtime; input_cost_per_audio_token: $3/1M; output_cost_per_audio_token: $12/1M |
| Amazon Bedrock | anthropic.claude-haiku-5-5 | 1,000,000 | $0.1 | $0.5 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | apac.anthropic.claude-haiku-5-5 | 1,000,000 | $0.11 | $0.55 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | au.anthropic.claude-haiku-5-5 | 1,000,000 | $0.11 | $0.55 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | bedrock/us-gov-east-1/anthropic.claude-haiku-5-5 | 1,000,000 | $0.12 | $0.6 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | bedrock/us-gov-west-1/anthropic.claude-haiku-5-5 | 1,000,000 | $0.12 | $0.6 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | eu.anthropic.claude-haiku-5-5 | 1,000,000 | $0.11 | $0.55 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | global.anthropic.claude-haiku-5-5 | 1,000,000 | $0.1 | $0.5 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | global.twelvelabs.pegasus-1-5-v1:0 | 98,304 | - | $7.5 | Chat; input_cost_per_video_per_second: $0.00049 |
| Amazon Bedrock | global.zai.glm-5.3 | 1,000,000 | $1.68 | $5.28 | Chat; Reasoning; Tool calling; Prompt caching; Structured output |
| Amazon Bedrock | in.moonshotai.kimi-k3 | 1,000,000 | $3.3 | $16.5 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output |
| Amazon Bedrock | jp.anthropic.claude-haiku-5-5 | 1,000,000 | $0.11 | $0.55 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | us-gov.anthropic.claude-haiku-5-5 | 1,000,000 | $0.12 | $0.6 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | us.anthropic.claude-haiku-5-5 | 1,000,000 | $0.11 | $0.55 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Amazon Bedrock | us.twelvelabs.pegasus-1-5-v1:0 | 98,304 | - | $7.5 | Chat; input_cost_per_video_per_second: $0.00049 |
| Amazon Bedrock | us.zai.glm-5.3 | 1,000,000 | $1.848 | $5.808 | Chat; Reasoning; Tool calling; Prompt caching; Structured output |
| Anthropic | claude-haiku-5-5 | 1,000,000 | $0.1 | $0.5 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search; Computer use |
| Azure AI | azure_ai/claude-haiku-5-5 | 1,000,000 | $0.1 | $0.5 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Azure AI | azure_ai/grok-4.7 | 500,000 | $2 | $6 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; Web search |
| Azure AI | azure_ai/kimi-k2-thinking | 262,144 | $0.6 | $2.5 | Chat; Reasoning; Tool calling |
| Azure OpenAI | azure/apac/gpt-6.1-sol | 922,000 | $2.4 | $12 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search; Computer use |
| Azure OpenAI | azure/eu/gpt-6.1-sol | 922,000 | $2.4 | $12 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search; Computer use |
| Azure OpenAI | azure/model-router | 200,000 | $0.14 | $0 | Chat; Vision |
| Azure OpenAI | azure/us/gpt-6.1-sol | 922,000 | $2.2 | $11 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search; Computer use |
| Azure OpenAI | azure/whisper | - | - | - | Transcription; input_cost_per_second: $0.0001; output_cost_per_second: $0.0001 |
| Bedrock Mantle | bedrock_mantle/anthropic.claude-haiku-5-5 | 1,000,000 | $0.11 | $0.55 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Bedrock Mantle | bedrock_mantle/us-gov-west-1/anthropic.claude-haiku-5-5 | 1,000,000 | $0.12 | $0.6 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Bespoke | bespoke/bespokelabs/Bespoke-Nimble-9B | 8,192 | $0 | $0 | Decisions |
| Bespoke | bespoke/nimble | 8,192 | $0 | $0 | Decisions |
| Bespoke | bespoke/nimble-latest | 8,192 | $0 | $0 | Decisions |
| ChatGPT | chatgpt/gpt-6-astra | 922,000 | - | - | Responses; Reasoning; Vision; Tool calling; Structured output |
| ChatGPT | chatgpt/gpt-6-luna | 922,000 | - | - | Responses; Reasoning; Vision; Tool calling; Structured output |
| ChatGPT | chatgpt/gpt-6-sol | 922,000 | - | - | Responses; Reasoning; Vision; Tool calling; Structured output |
| ChatGPT | chatgpt/gpt-6.1-sol | 922,000 | - | - | Responses; Reasoning; Vision; Tool calling; Structured output |
| Cohere | cohere/embed-v5.0-fast | 128,000 | $0.08 | $0 | Embedding |
| Cohere | cohere/embed-v5.0-pro | 128,000 | $0.12 | $0 | Embedding |
| Databricks | databricks/databricks-openjev-qwen35-4b | - | $0 | $0 | Decisions |
| Gemini | gemini/gemini-nano-banana-2.1 | 131,072 | $1.5 | $7.5 | Image generation; Reasoning; Vision; PDF input; Web search; output_cost_per_image: $0.0336; output_cost_per_image_token: $30/1M |
| Laya | laya/english | - | $0 | $0 | Decisions |
| Laya | laya/multilingual | - | $0 | $0 | Decisions |
| Laya | laya/typed-decisions | - | $0 | $0 | Decisions |
| Mistral | mistral/mistral-large-4 | 1,048,576 | $0.68 | $2.09 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output |
| Mistral | mistral/mistral-large-4-0 | 1,048,576 | $0.68 | $2.09 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output |
| OpenRouter | openrouter/anthropic/claude-haiku-5.5 | 1,000,000 | $0.1 | $0.5 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search |
| OpenRouter | openrouter/anthropic/claude-haiku-5.5:batch | 1,000,000 | $0.05 | $0.25 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Web search |
| OpenRouter | openrouter/google/gemini-nano-banana-2.1 | 65,536 | $1.5 | $7.5 | Chat; Reasoning; Vision; Tool calling; Structured output; Web search; output_cost_per_image_token: $30/1M |
| OpenRouter | openrouter/inclusionai/ling-3.0-flash-sante | 262,144 | $0.042 | $0.1232 | Chat; Reasoning; Tool calling; Prompt caching |
| OpenRouter | openrouter/inclusionai/ling-3.1-flash | 262,144 | $0 | $0 | Chat; Reasoning; Tool calling |
| OpenRouter | openrouter/mistralai/mistral-large-4-0 | 1,048,576 | $0.68 | $2.09 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output |
| OpenRouter | openrouter/openai/gpt-image-2 | - | $5 | - | Image generation; Vision; input_cost_per_image_token: $8/1M; output_cost_per_image_token: $30/1M |
| OpenRouter | openrouter/openai/gpt-image-2.5-flare | - | $5 | - | Image generation; Vision; input_cost_per_image_token: $8/1M; output_cost_per_image_token: $30/1M |
| OpenRouter | openrouter/openai/gpt-image-2.5-sunburst | - | $5 | - | Image generation; Vision; input_cost_per_image_token: $8/1M; output_cost_per_image_token: $30/1M |
| OpenRouter | openrouter/stepfun/step-5-preview | 1,000,000 | $1 | $2.7 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output |
| Perplexity | perplexity/anthropic/claude-haiku-5-5 | - | $0.1 | $0.5 | Responses; Tool calling; Web search |
| Vertex AI | gemini-nano-banana-2.1 | 131,072 | $1.5 | $7.5 | Image generation; Reasoning; Vision; Prompt caching; PDF input; Web search; input_cost_per_image: $0.00168; output_cost_per_image: $0.0336 |
| Vertex AI | vertex_ai/claude-haiku-5-5 | 1,000,000 | $0.1 | $0.5 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Vertex AI | vertex_ai/claude-haiku-5-5@default | 1,000,000 | $0.1 | $0.5 | Chat; Reasoning; Vision; Tool calling; Prompt caching; Structured output; PDF input; Computer use |
| Vertex AI | vertex_ai/gemini-nano-banana-2.1 | 131,072 | $1.5 | $7.5 | Image generation; Reasoning; Vision; Prompt caching; PDF input; Web search; input_cost_per_image: $0.00168; output_cost_per_image: $0.0336 |
| xAI | xai/grok-imagine-video-1.5-lite | - | - | - | Video generation; input_cost_per_image: $0.01; output_cost_per_second: $0.02 |
New providers​
- Add Reka as an OpenAI-compatible provider under the
reka/prefix, readingREKA_API_KEYandREKA_API_BASE, listed in the Add Model form, with/v1/responsesand/v1/messagesbridged to its chat completions - PR #44278
Decisions providers​
- Add Bespoke Nimble as a Decisions provider behind
BESPOKE_API_BASE, served onPOST /bespoke/v1/systemoneand selectable as the Auto Router OSS Classifier - PR #44246 - Add Databricks
ai_decideas a/v1/decisionsprovider and Auto Router classifier (opensource_classifier_config.provider: databricks), keyed byDATABRICKS_API_KEYorDATABRICKS_TOKEN, with a Databricks option in the Add Auto Router form - PR #45200 - Add Laya as a Decisions provider and Auto Router OSS classifier: routers set
classifier_type: oss_classifierwithopensource_classifier_config.provider: laya(orjevfor Jev-compatible self-hosted classifiers), and existing Jev configurations keep working - PR #43626
Amazon Bedrock​
- Route
bedrock_mantle/anthropic.claude-*chat completions to Mantle's native Messages endpoint, and price Mantle calls from their ownbedrock_mantle/cost-map row when one exists - PR #43646 - Add an
anthropic_messageshealth check mode, used by default for Bedrock Mantle Claude deployments, so they stop showing unhealthy on/healthand Test Connect - PR #44419 - Count tokens for Claude Opus 4.8, 5 and 5.5 on Bedrock Mantle when bedrock-runtime cannot count them, instead of falling back to the local tokenizer - PR #45317
- Send the project id to Bedrock Mantle as the documented
anthropic-workspace-idheader, soaws_bedrock_project_idtakes effect - PR #44173 - Honor the per-request
timeouton Converse and Invoke streaming calls - PR #44134 - Drop lookaround regex patterns from tool schemas for Converse models that reject them (Kimi K3, Grok 4.6 and 4.7), controlled by the new
supports_regex_lookaroundcost-map andmodel_infoflag - PR #44138 - Stop sending Converse
cachePointblocks to Kimi K3, gated by a newsupports_prompt_cache_breakpointflag - PR #44292 - Drop or reject reasoning effort levels a Bedrock GPT model does not support, following
drop_params, instead of forwarding them for a Bedrock 400 - PR #44183 - Split a leading
<reasoning>tag out of the answer only for gpt-oss on native Chat Completions, so GPT 5.6 and Grok answers keep it - PR #44432 - Forward each Nova Sonic assistant sentence once over the realtime API instead of twice - PR #44987
- Report Scheduled Bedrock batch jobs as
in_progresson retrieve instead ofvalidating- PR #45157 - A converse-stream 200 made only of unrecognized event frames now ends as a 502 naming the event types, and frames named after a Bedrock exception raise with that exception's status, instead of returning a blank
end_turnat $0 - PR #44112 - Send OpenAI explicit prompt cache breakpoints for Bedrock Mantle GPT-5.6 and newer, honor
supports_prompt_cache_breakpointset on a deployment, and stop malformedcache_control_injection_pointsfrom failing every request with a 500 - PR #38729
Anthropic​
- Stop repeating streamed thinking text in the signature chunk, so saved and replayed thinking blocks carry the text once - PR #44127
- Forward the
inline-tools-2026-09-15beta header to Anthropic and Vertex AI, so mid-session tool changes no longer fail with a 400 - PR #45297 - Fail a token-file or inline-token federation credential that is missing its rule or organization id with a 401 naming the missing id - PR #45178
- Pass a leading system message through count_tokens'
systemparameter, so/v1/responses/input_tokenswithinstructionsand/v1/messages/count_tokensget Anthropic's count - PR #45463 - Normalize images before provider token counting on Anthropic and Azure AI Anthropic - PR #45185
- Anthropic: workload identity federation with short-lived tokens minted from an OIDC assertion (token file, env token, LiteLLM-signed issuer or Keycloak), set up through config, env vars or a named credential, plus
GET /credentials/{name}/jwks- PR #44448
ChatGPT and GitHub Copilot​
- Refuse to start the ChatGPT or GitHub Copilot device-code login inside a proxy request, answering 400 instead of freezing the worker for up to 15 minutes - PR #39585
- Honor the caller's
streamflag onchatgpt/*models, so non-streaming/v1/responses,/v1/chat/completionsand/v1/messagescalls return JSON instead of raw SSE or a 500 - PR #41235 - Preserve the requested
service_tieron ChatGPT Responses calls, mappingfasttopriority- PR #40108 - Add
chatgpt/catalog rows forgpt-6-sol,gpt-6-luna,gpt-6-astraandgpt-6.1-sol, so chat completions calls to them go through the Responses bridge - PR #44758
Exa AI​
- Fall back to highlights or summary for the search result snippet when Exa returns no
text- PR #42213
Fireworks AI​
- Add the
fireworks_forward_user_iddeployment setting, which sends the key's LiteLLM user id to Fireworks asuseron chat completions, Responses and Messages - PR #45265
Gemini and Vertex AI​
- Stop replaying Claude thinking block signatures to Gemini, so mixed Claude and Gemini model groups no longer fail with
Invalid thought signature- PR #44661 - Forward
systemandtoolsto Vertex partner model count_tokens, and fall back to the local tokenizer on a Vertex count error - PR #43900 - Dial the multi-region Live API host for Vertex realtime sessions with
vertex_location: usoreu, and allow plainws://api_base values - PR #45166 - Accept file content blocks with
video_metadata(fps,start_offset,end_offset) on Gemini multimodal embeddings - PR #45305 - Apply the regional endpoint uplift on the Vertex image generation and edit cost path, and add a 1.1x uplift to
vertex_ai/gemini-3.1-flash-image- PR #44679, PR #44673
Mistral​
- Add Mistral Large 4 (
mistral/mistral-large-4,mistral/mistral-large-4-0) with a 524,288-token context window, pricing, prompt caching and reasoning - PR #44870
OCI​
- Resolve the OCI GenAI endpoint realm from the compartment OCID, so Government realm regions such as
us-luke-1reachoci.oraclegovcloud.com- PR #43180
Ollama​
- Turn streamed and fenced prompt-based JSON tool calls on
ollama/models into real tool calls - PR #45053 - Answer in text after a tool result on
ollama/models instead of repeating the tool call - PR #45079 - Use one id per
ollama/stream, return 400 for non-text tool content, separate consecutive tool results, and drop the empty message item from Responses streams - PR #45177
OpenAI​
- Leave
modeloff the upstream OpenAI URL for realtimeintent=transcriptionsessions, which OpenAI rejected withinvalid_model- PR #43854
OpenAI-compatible providers​
- Send provider attribution headers on the default SDK path, fixing Novita's
X-Novita-Sourceand adding Perplexity'sX-Pplx-Integration- PR #44291
SAP​
- Forward
cache_controlon SAP messages and tools, and stop sendingcache_control: nullthat SAP rejected with a 400 - PR #39122
Scaleway​
- Add Scaleway rerank support on
/v1/rerank- PR #44160
VoyageAI by MongoDB​
- Rename the Voyage AI provider to VoyageAI by MongoDB, and send MongoDB-issued
al-keys toai.mongodb.comfor embeddings and rerank; thevoyageprovider id is unchanged - PR #41812
Model catalog and pricing​
- Add
claude-haiku-5-5pricing, then consolidate its over-100k token rates and capability flags across providers, with corrected forced tool use and thinking flags on Azure AI and Vertex AI - PR #45108, PR #45151, PR #45114, PR #45116 - Lower the Claude Sonnet 5.5 cache read price on Anthropic, Bedrock, Bedrock Mantle and Vertex AI - PR #45113, PR #45216, PR #45223, PR #45217
- Add the US data residency multiplier to anthropic
claude-opus-5-5- PR #45078 - Add OpenAI TTS and GPT-5.x deprecation dates - PR #44175
- Add openai gpt-6.1-sol ultrafast tier prices - PR #45408
- Add xAI grok-voice-transcribe-1.0 deprecation date - PR #44237
- Registry audit 2026-10-03, adding Cohere Embed v5, xAI grok-imagine-video-1.5-lite, OpenRouter GPT Image rows and Gemini 3.5 Transcribe limits - PR #44376
- Set Gemini Deep Research input limits, Vertex Gemini 3.6 and 3.7 Flash retirement dates, and Azure data zone gpt-6.1-sol pricing - PR #44633
- Add Bedrock flex prices, Anthropic web search flags, Gemini shutdown dates, OpenRouter alias fixes and Mistral Large 4 context - PR #44910
- Remove malformed, duplicate and decommissioned PaLM cost-map entries - PR #44880
- Correct
supported_endpointson Gemini and Vertex image rows and drop/v1/batchfromgemini/gemini-nano-banana-2.1- PR #44891, PR #44940 - Add gemini/gemini-nano-banana-2.1 from the Gemini pricing page - PR #44868
- Add Vertex AI gemini-nano-banana-2.1 and Nano Banana 2 priority pricing - PR #44892
- Add the priority tier audio input price to three Gemini rows - PR #44632
- Add a deprecation date to vertex_ai gemini-3.1-flash-lite-image - PR #44639
- Correct
input_cost_per_secondon Vertex gemini-3.5 transcribe-live and live-translate - PR #45002 - Mark Azure
us/andeu/Responses-only models asmode: responses, so their chat completions calls are bridged to Azure Responses - PR #44323 - Add
max_input_tokensto Azure MAI-Image and azure_ai/flux.2-pro rows, and set Azure text-embedding max input to 8192 - PR #44375, PR #44405, PR #44449 - Add azure_ai/kimi-k2-thinking and its retirement date - PR #44382, PR #44455
- Add azure_ai/grok-4.7 and update the azure_ai/grok-4.6 input price - PR #44937
- Add azure gpt-6-sol priority processing prices - PR #44829
- Sync Azure OpenAI audio, realtime and image prices with the Azure pricing page - PR #44992
- Update Azure retirement dates for gpt-realtime-2.1, gpt-4o-realtime-preview-2024-10-01, jamba-instruct, gpt-4o-transcribe, kimi-k2.7-code and sora-2 - PR #44526, PR #44567, PR #44585, PR #44986, PR #45355
- Date azure_ai claude-opus-5-5 and claude-sonnet-5-5 and fill azure/eu/gpt-6-astra limits - PR #45018
- Price Bedrock Amazon Nova 2 Pro preview at the standard tier - PR #44302
- Add Bedrock priority and flex prices for Grok 4.3, 4.6, 4.7 and Kimi K3 - PR #44670
- Add Bedrock GLM 5.3 cross-region rows, Nova 2.5 Sonic and the Kimi K3 India cross-region row - PR #44710, PR #44990
- Add Bedrock TwelveLabs Pegasus 1.5 inference profiles - PR #45477
- Update Bedrock GovCloud OpenAI prices, add the GPT-6 Astra ultrafast tier and the Titan Image v2 EOL date - PR #45077
- Add Bedrock gpt-6.1-sol ultrafast tier prices and its 1M context window - PR #45482, PR #45488
- Take Bedrock and Bedrock Mantle context limits, output limits and EOL dates from the Bedrock model cards - PR #45091, PR #45409, PR #45417
- Add Bedrock deprecation dates for DeepSeek R1, Qwen3 Coder and Claude Sonnet 4.5 - PR #45089, PR #45496
- Remove the bare Bedrock
xai.grok-4.7andopenai.gpt-6.1-solrows that AWS cannot invoke, sous.profile ARNs bill at theus.price - PR #45460, PR #45479 - Sync OpenRouter prices, limits and deprecation dates from the models API - PR #44287, PR #44466, PR #44533, PR #44975, PR #45469
- Update together_ai Kimi-K3 and Qwen3.8-Flash prices to published rates - PR #44864
- Update the baseten DeepSeek-V4.1-Flash-Fast cache read price and max output - PR #45468
litellm.supports_audio_output()reads thesupports_audio_outputcost map key instead ofsupports_audio_input- PR #45294- Azure: add
azure/model-routerpricing (the router's own $0.14 per 1M input token fee, charged on top of the routed model) andazure/whisperaudio transcription pricing - PR #44872
LLM API Endpoints​
Responses API​
- Let Anthropic
/v1/responsesfollow-ups withprevious_response_idsend a tool result plusinstructionswithout a 400; a follow-up'sinstructionsnow replace the earlier turns' instead of stacking behind them - PR #45322 - Drop LiteLLM-minted reasoning items when replaying Responses history to OpenAI and Azure, so a conversation can switch from Claude to GPT - PR #44132
- Idle Responses WebSockets stay open instead of closing after 30s, and every session is capped by the new
general_settings.responses_websocket_session_limit_seconds(default 3600, 60 to 7200) - PR #44433
Chat Completions over the Responses API bridge​
- Return a bridged reply's text and tool calls as one choice with
finish_reason: "tool_calls", so clients reading onlychoices[0](e.g. Google ADK) run the tool; the gpt-5.4/5.5 routing change from the first attempt was reverted - PR #44295, PR #44344, PR #44346 - Keep
prompt_cache_breakpointmarkers on text, image and file blocks so tool-calling chats on gpt-5.6 get prompt-cache reads, and drop them when the model does not support breakpoints - PR #44119 - Keep cache breakpoints on blocks the bridge stringifies, drop malformed markers under
drop_params, and accept thettl: "30m"breakpoint field - PR #40032 - Send
input_audioparts as Responsesinput_audioinstead of billing the base64 as text; withoutdrop_paramsthe provider's own error now comes back - PR #45224 - Honor the request's
cachecontrols (e.g."no-cache": true) on chat completions bridged to the Responses API - PR #44676 - Surface the provider's
RateLimitErrorinstead of a nestedMidStreamFallbackErrorwhen a bridged stream hits a 429 - PR #44989 - Report truncated bridged Responses output as
incomplete, end those streams withresponse.incomplete, and echo requesttemperature,top_pandmax_output_tokens- PR #44460
Prompt caching​
- Default gateway-injected prompt cache breakpoints to implicit lookup on
/v1/chat/completions,/v1/responsesand/v1/messages, and keep breakpoints through the chat to Responses bridge, so growing agent histories reuse earlier cached prefixes - PR #42281, PR #45121 - Count client
cache_controlmarks on assistant tool calls when injecting breakpoints, so Claude requests stay within the 4-breakpoint limit - PR #43556
Batches and files​
- Save file details for every batch output and error file, so they appear in
GET /v1/filesand Bedrock batch output files retrieve without a 500 - PR #41761
Vector Stores​
- Return LiteLLM managed file ids from
GET /v1/vector_stores/{id}/files, so listed ids can be retrieved, deleted and used asafter/beforecursors - PR #43800
Pass-through endpoints​
- Use OpenAI workload identity federation tokens on
/openai_passthroughand/openaiwhen no static OpenAI key is set - PR #44140 - Resolve
oidc/pass-through credentials on every request, so rotating projected Kubernetes service account tokens work for pass-through auth - PR #44577
SDK​
- Add
litellm.run_tool_loop()andlitellm.arun_tool_loop()to run a completion plus tool-execution loop until the model stops calling tools - PR #44381 - Build an independent
litellm-coredistribution from its own packaging manifest viascripts/build_core_distribution.py; legacy packaging is unchanged - PR #44340 - Add
Harness.TOOL_LOOPtolitellm.agent(), an in-process tool-calling loop overlitellm.acompletion()that turns plain Python functions into tools from their type hints and docstrings, returns tool errors to the model, and supportsoutput=,permissions="ask",max_turns, sessions, cost and usage; gateway calls are taggedharness,tool_loopin spend logs - PR #44391 - Add a public
hidden_paramsproperty on SDK response objects alongside_hidden_params, which keeps working; proxy response headers are unchanged - PR #44668
General​
- Local token counting prices a base64 PDF document per page (text plus rendered image) instead of as one image, so
count_tokensfallbacks track the billed input - PR #45301 GET /v1/models?client_version=...returns a Codex-native catalog, withmodel_info.service_tiersexposed per model so Codex can offer tiers like/ultrafast; the OpenAI list shape is unchanged withoutclient_version- PR #44136GET /v1/modelswithcheck_provider_endpoint: truediscovers upstream models forlitellm_proxy/*andhosted_vllmwildcard deployments - PR #38617- Return 400 naming the missing field instead of 500 on LLM routes, 400 for
pageorpage_sizebelow 1 on activity and vector store lists, and the provider's 404 for unknown video, eval, vector store and skill ids - PR #43787
Management Endpoints / UI​
Admin UI​
Home page​
- New Home page at
/ui/homeis the dashboard root, with a 7-day usage strip, What's new launch cards, blog updates and a model leaderboard; logging in or opening/uinow lands on Home, and Virtual Keys keeps its sidebar entry and/ui/api-keys- PR #45858 - Home What's new cards show the three newest LiteLLM blog posts tagged
day 0 support, and/public/litellm_blog_postsreturns each post's tags and up to 50 posts - PR #46309
Playground and Decisions​
- New Decisions tab in the Playground (
?tab=system-one) for trying decision models on/v1/decisionsor/v1/systemonewith the session key or a virtual key, keeping a separate draft per endpoint and showing each choice, yes / no and score answer with its confidence plus usage and latency - PR #44043, PR #44664, PR #45729, PR #45817 - Build decision requests in a Form tab, with an Input box and one card per question, instead of hand-writing the JSON body - PR #45739
- Pick one of four example presets (bug triage, support ticket routing, comment moderation, lead scoring) or Clear the request to start from empty; the picker reads Custom once the request is edited - PR #45848
- Label the yes / no question type "Yes / No" in the request form and the Add Model form, and explain it on hover - PR #45841, PR #45933
- Decisions playground lays out the form first with a large Send button under it, shows results only once a request runs, and replaces the editable JSON tab with a read-only cURL tab showing the exact request - PR #45903
- Models + Endpoints shows a decision models banner in place of the cost optimization feedback banner, provider search finds decision model providers, and the playground model picker lists decision models - PR #45735
Models + Endpoints​
- Add Tencent to the Add Model provider dropdown with its logo, an optional API base and a required API key - PR #40924
- Add Model lists only catalog models instead of also showing deployment aliases registered by custom pricing, using the new
catalog_only=trueoption onGET /public/litellm_model_cost_map- PR #44877 - Show the Add Model model picker once the model catalog finishes loading, even when the provider was picked first - PR #45426
- Add the Evaluation mode (
Evaluation - /v1/decisions) to the Add Model form so decision models can be added and tested from the UI;POST /health/test_connectionnow acceptsmode: "evaluation"- PR #45481 - Decision models banner links to the supported providers docs in place of the Add a decision model button, and the dashboard calls the decisions playground Decisions instead of System One - PR #45732
- Drop the per-provider decision models sublabel from the Add Model provider dropdown - PR #45755
- Configure Anthropic workload identity federation from Add Credential, Edit Credential and Add Model, with federated credentials marked in the credentials table; Edit Credential now saves only the values the admin changed - PR #44889
- Configure OpenAI workload identity federation from LLM Credentials and Add Model, swapping API Key for Identity Provider ID, Service Account ID and Identity Token File - PR #45528
Auto-Routers​
- Opt into cache-aware routing under auto-router Advanced settings, Sessions and efficiency, with optional output-token estimates and a prediction timeout - PR #43396
- Rename the Jev classifier to OSS Classifier and add Laya as a provider choice with checkpoint and timeout settings - PR #43768
- Auto-router setup groups classifier, tier and advanced settings; a Decisions Model classifier picks an onboarded decision model deployment, and tiers start prefilled with editable recommendations - PR #45916
- Add a per-router tokens, cost, cost per million tokens and savings table to Cost Optimization, Auto-Router, Usage, backed by new daily token records - PR #45152
Usage and Model Leaderboard​
- Redesign the Usage page with a stat strip with sparklines, a stacked daily chart by model, a Top agents panel built from User-Agent tags on the admin global view, and a provider share bar, shared across the global, team, organization, tag and customer views - PR #45221, PR #45229
- Move Key Activity search back to the top of the tab and add a model name search to Model Activity - PR #44521
- Model Leaderboard's Top models chart starts at the first day with usage so bars stay wide and readable - PR #44249
- Remove the Top models by task card from Model Leaderboard - PR #44502
Teams, users and access groups​
- Keys declared in
team_metadata_schemashow as fixed labels in team metadata, with only the value editable - PR #41482 - The model picker in access group, team, organization and default user settings lists saved models that are no longer available in an Unavailable group so they can be removed - PR #43944
- Model access group chips on team and key pages link to the access group filter on Models + Endpoints - PR #45402
- Invitation and reset password links show in a read-only field with a Copy button beside them - PR #44454
Logs​
- Logs tabs share page spacing, show refresh status in the toolbar and keep pagination at the bottom of full-height tables - PR #44306
General​
- Add a Cmd+K (Ctrl+K) command palette and top bar search that finds virtual keys by alias or ID and jumps to any sidebar page - PR #45456
- LiteAdmin moves from the floating button into a header pill that opens a docked side panel, toggled with Cmd/Ctrl+J, and shows on the Playground again - PR #44293
- LiteAdmin is now enterprise-only - PR #44399
- Below 768px the sidebar starts hidden and opens over the page, and secondary header tools move into a More menu - PR #44603
- Add Moyai to the product switcher with a launch page and Quick connect, which issues Moyai a scoped virtual key; UI Settings shows the connection with a Disconnect button - PR #45196
- Restore the background blur behind dashboard modals - PR #45877
Keys, teams and authentication​
- Show Projects in the nav to org and team admins, scope
/project/listand/project/infoto the org admin's org, and offer Projects in the page-visibility picker - PR #41325 - Add
auto_register_map_existing_keytolitellm_jwtauthso a JWT user's first call reuses their existing virtual key instead of minting a second one (off by default) - PR #42375 - Limit which models an end user can call with a
modelslist on/customer/new,/customer/updateand/customer/info; calls outside it get 403customer_model_access_denied- PR #43904 - Apply path-less SCIM group PATCH ops (Okta Push Groups renames) instead of storing them under an empty metadata key, and let affected teams save again in the UI - PR #43978
- Add opt-in
general_settings.vector_store_deny_by_defaultso keys, teams and users can only query vector stores listed in theirobject_permission- PR #44244 disable_model_add_for_internal_usersnow blocks team admins and opted-in members from creating models and auto routers through the API as well as the UI - PR #44438- Add
denied_passthrough_routeson keys and teams to carve sub-paths out of anallowed_passthrough_routesgrant - PR #44924 - A team alias listed on an end user wins over a same-named key alias in the customer model check, matching routing - PR #44677
- Add
search_tool_deny_by_defaultundergeneral_settings, so keys need an explicit search tool grant on/v1/search,/search, websearch interception and/search_tools/list- PR #44490
Credentials​
- Add an optional credential
display_nameand makecredential_nameimmutable onPATCH /credentials/{name}; config-defined credentials are read-only and GET returnssource- PR #43148 - Keep credential edits on the credential being edited, reject names that collide with config credentials (409), and add a Source column to the credentials list - PR #45890
- Encrypt search tool
litellm_paramsat rest and cover them in master key rotation - PR #43631
Model management​
- Accept team-scoped models by their public name on
POST /fallback, and read stored fallback rules fresh so concurrent writes stop dropping rules - PR #45455
Usage and analytics​
- Stop caching failed spend-log key alias lookups as confirmed misses, and survive a dropped DB connection during the lookup on daily activity routes - PR #43560
- Count Auto-Router savings by selected UTC request day so the Auto-Router and Overall tabs agree, and remove the "Avg saved per session" tile - PR #44115
- Write per-router Auto-Router daily rollups with
disable_spend_logs: trueand for requests without a session id - PR #44226 - ROI calculator connects GitLab as well as GitHub, attributes cost to branches from paired
repo:andbranch:tags, and splits Overview, People and Branches into tabs - PR #44324 - ROI calculator measures shipping velocity, quality signals and recorded spend from repository activity without an estimator model, and links several provider usernames to one email - PR #44426
- ROI calculator people, merged-change and branch lists default to Matched people only - PR #44465
- Retire the integrated ROI Calculator: its dashboard page, Observability nav entry, API routes, OAuth callbacks and scheduled refresh are removed,
/ui/roi-calculator/returns 404, and stored calculator configuration stays in the database. The standalone litellm-roi-calculator project remains available - PR #44477
CLI and coding agents​
lite pipins compat flags so pi stops sendingstoreand prompt cache fields that Anthropic models reject - PR #40739- Let
lite loginand Claude Code gateway sign-in complete onDISABLE_ADMIN_UI=truenodes - PR #44620 - Claude Code gateway sign-in returns a rotating 14-day refresh token, and
POST /claude_code_gateway/oauth/revokerevokes it - PR #44635 - Statusline cost comparison shows full-session gateway spend, including requests without a baseline estimate - PR #44959
- Fix 401 "user no longer exists" on the first request with a key from a first-time SSO
lite login- PR #45020
Deployment​
- Bundle the optional LiteAdmin Slack worker in the Enterprise gateway image, with Slack accounts connected through the gateway's existing SSO login - PR #44444
- Stop logging license values, decoded payloads and license-bearing URLs during license verification - PR #44956
- Allowlist
/moyai/connect/*for split gateway and backend deployments and tag the Moyai UI settings Redis write asconfig_params- PR #45220
Terraform​
- Keep an omitted
allowed_routesonlitellm_keyknown at plan time and stop updates from re-sending stale routes from state - PR #44487
AI Integrations​
Lens and Agent Traces​
Lens service and deployment​
- Move trace ingestion, ClickHouse lifecycle and trace reads into a standalone Rust Lens service; agents upload directly to Lens with dedicated tracing keys, the old gateway upload URLs return 410, and Python workers are replaced - PR #45148
- Simplify Lens installation: Helm supplies private credentials, persistent ClickHouse and ingress-derived endpoints, Compose generates and keeps credentials, and setup shows install guidance or copyable framework configuration - PR #45230
- Configure trace storage with one
general_settings.tracing.storeblock (type,url,database,retention_days), use one ClickHouse URL for writes and reads, and apply one 14-day default TTL to trace and spend tables - PR #43941 - Select the Lens worker image matching the running LiteLLM release, reject incompatible workers before they claim work, and expose a private worker token for Compose and Helm installs - PR #44428
- Pin the Helm-installed Lens worker to its approved image digest and harden the worker image with pinned base images and hashed Python dependencies - PR #44467
- Align the Lens source setup guide and tracing Compose file with worker images that are actually published - PR #44476
- Fix Lens investigations failing on Python 3.10 with
module 'asyncio' has no attribute 'timeout'- PR #44911 - Track ClickHouse tracing migrations in a checksummed
_sqlx_migrationsledger so restarts apply only pending migrations and reapply the currentretention_daysto every trace table; an edited, already-applied migration now disables tracing at startup while the proxy keeps serving traffic - PR #44580
Trace ingestion​
- Support Claude Agent SDK traces with prompts, tool arguments and outputs, token counts, the agent name and a Claude logo - PR #44248
- Accept native Claude Code response logs on the authenticated
/v1/logsendpoint to rebuild replies, failed-tool output and subagent branches; group a whole session withlens.session.capture=true- PR #44711 - Keep native Claude Code logs that arrive without trace context, and recover headless tool results that precede trailing system messages - PR #44894
- Use configured framework agent names instead of service names for the Agent column and filter, and render GenAI messages that use
partsinstead of crashing the Traces page - PR #44218 - Show the first user message as a run's input instead of the latest follow-up - PR #44958
- Link a trace to the Slack, Teams, Discord, Linear, GitHub or Jira thread that started it via
agent.source.type,agent.source.urlandagent.source.titlespan attributes - PR #45169 - Keep request logs that carry numeric or mixed-type tags, and return meaningful RPC error codes in rejected OTLP batches - PR #45206
Trace cost correlation​
- Keep distinct spend rows, persist gateway call IDs and match gateway and provider IDs with ownership checks so agent traces show the correct cost across 12 agent SDKs - PR #44421
- Stop summing costs from conflicting spend IDs on one span, and stop unrelated tool HTTP calls from hiding a model call's cost - PR #44456
- Price native Claude Code calls in Lens by forwarding and storing the provider request ID, show partial totals with the number of priced calls, and link each step to its spend log - PR #44738
- Accept historical spend rows that have no provider request ID - PR #44930
- Keep refreshing runs every five seconds until gateway costs arrive, instead of showing unknown cost for up to ten minutes - PR #45105
- Redact credential-like custom metadata in ClickHouse spend rows, drop custom metadata when message logging is off, and keep spend null when no cost was calculated - PR #44363
Trace API and SQL queries​
- Scope trace reads to the caller's user, key or permitted teams to match log visibility, and return 400 with a
trace_refhint when a trace ID is shared across runs - PR #44228 POST /v1/traces/querynow returns only{"data": [...]}, dropping the raw ClickHousemeta,rowsandstatisticsfields - PR #44609- Serve the curated SQL examples in
/v1/traces/query/helpfrom shared query files - PR #44388 - Add
GET /v1/traces/agentsand an agent picker in the Lens header that scopes Traces to one agent and remembers the choice - PR #45202 - Load large trace lists and details in bounded pages within ClickHouse limits instead of failing with HTTP 500, with Load more steps for long runs - PR #44384
- Batch run-list reads, cap the pagination cache at 128 MiB, and restart Refresh trace from the first page - PR #44398
- Cache trace reads in process so repeated opens, pages and list polls share one ClickHouse read, and report typed read failures the UI can retry - PR #44422
- Open large traces and list pages faster, and queue concurrent trace reads instead of rejecting the ninth with 503 - PR #45228
- Preserve ASGI lifespan state across all proxy entrypoints, so Agent Traces no longer returns 501 on split gateway and backend deployments or with eager route loading - PR #44214
- Share spend log ownership checks with the Trace API and remove its API key access grants - PR #44239