Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.78.0.rc.2

pip install litellm==1.78.0.rc.2

Key Highlights​

• MCP Gateway - Control Tool Access by Team, Key - Control MCP tool access by team/key.
• Performance Improvements - 70% Lower p99 Latency
• GPT-5 Pro & GPT-Image-1-Mini - Day 0 support for OpenAI's GPT-5 Pro (400K context) and gpt-image-1-mini image generation
• EnkryptAI Guardrails - New guardrail integration for content moderation
• Tag-Based Budgets - Support for setting budgets based on request tags

MCP Gateway - Control Tool Access by Team, Key​

Proxy admins can now control MCP tool access by team or key. This makes it easy to grant different teams selective access to tools from the same MCP server.

For example, you can now give your Engineering team access to list_repositories, create_issue, and search_code tools, while Sales only gets search_code and close_issue tools.

This makes it easier for Proxy Admins to govern MCP Tool Access.

Get Started

Performance - 70% Lower p99 Latency​

This release cuts p99 latency by 70% on LiteLLM AI Gateway, making it even better for low-latency use cases.

These gains come from two key enhancements:

Reliable Sessions

Added support for shared sessions with aiohttp. The shared_session parameter is now consistently used across all calls, enabling connection pooling.

Faster Routing

A new model_name_to_deployment_indices hash map replaces O(n) list scans in _get_all_deployments() with O(1) hash lookups, boosting routing performance and scalability.

As a result, performance improved across all latency percentiles:

• Median latency: 110 ms → 100 ms (−9.1%)
• p95 latency: 440 ms → 150 ms (−65.9%)
• p99 latency: 810 ms → 240 ms (−70.4%)
• Average latency: 310 ms → 111.73 ms (−64.0%)

Test Setup​

Locust

• Concurrent users: 1,000
• Ramp-up: 500

System Specs

• Database was used
• CPU: 4 vCPUs
• Memory: 8 GB RAM
• LiteLLM Workers: 4
• Instances: 4

Configuration (config.yaml)

View the complete configuration: gist.github.com/AlexsanderHamir/config.yaml

Load Script (no_cache_hits.py)

View the complete load testing script: gist.github.com/AlexsanderHamir/no_cache_hits.py

New Models / Updated Models​

New Model Support​

Features​

• OpenAI

  • Add GPT-5 Pro model configuration and documentation - PR #15258
  • Add stop parameter to non-supported params for GPT-5 - PR #15244
  • Day 0 Support, Add gpt-image-1-mini - PR #15259
  • Add gpt-realtime-mini support - PR #15283
  • Add gpt-5-pro-2025-10-06 to model costs - PR #15344
  • Minimal fix: gpt5 models should not go on cooldown when called with temperature!=1 - PR #15330

• Snowflake Cortex

  • Add function calling support for Snowflake Cortex REST API - PR #15221

• Gemini

  • Fix header forwarding for Gemini/Vertex AI providers in proxy mode - PR #15231

• Azure

  • Removed stop param from unsupported azure models - PR #15229
  • Fix(azure/responses): remove invalid status param from azure call - PR #15253
  • Add new Azure AI models with pricing details - PR #15387
  • AzureAD Default credentials - select credential type based on environment - PR #14470

• Bedrock

  • Add Global Cross-Region Inference - PR #15210
  • Add Cohere Embed v4 support for AWS Bedrock - PR #15298
  • Fix(bedrock): include cacheWriteInputTokens in prompt_tokens calculation - PR #15292
  • Add Bedrock AU Cross-Region Inference for Claude Sonnet 4.5 - PR #15402
  • Converse → /v1/messages streaming doesn't handle parallel tool calls with Claude models - PR #15315

• Vertex AI

  • Implement Context Caching for Vertex AI provider - PR #15226
  • Support for Vertex AI Gemma Models on Custom Endpoints - PR #15397
  • VertexAI - gemma model family support (custom endpoints) - PR #15419
  • VertexAI Gemma model family streaming support + Added MedGemma - PR #15427

• OCI

  • Add OCI Cohere support with tool calling and streaming capabilities - PR #15365

• Watson X

  • Add Watson X foundation model definitions to model_prices_and_context_window.json - PR #15219
  • Watsonx - Apply correct prompt templates for openai/gpt-oss model family - PR #15341

• OpenRouter

  • Fix - (openrouter): move cache_control to content blocks for claude/gemini - PR #15345
  • Fix - OpenRouter cache_control to only apply to last content block - PR #15395

• Together AI

  • Add new together models - PR #15383

Bug Fixes​

• General
  • Bug fix: gpt-5-chat-latest has incorrect max_input_tokens value - PR #15116
  • Fix reasoning response ID - PR #15265
  • Fix issue with parsing assistant messages - PR #15320
  • Fix litellm_param based costing - PR #15336
  • Fix lint errors - PR #15406

LLM API Endpoints​

Features​

• Responses API

  • Added streaming support for response api streaming image generation - PR #15269
  • Add native Responses API support for litellm_proxy provider - PR #15347
  • Temporarily relax ResponsesAPIResponse parsing to support custom backends (e.g., vLLM) - PR #15362

• Files API

  • Feat(files): add @client decorator to file operations - PR #15339

• /generateContent

  • Fix gemini cli by actually streaming the response - PR #15264

• Azure Passthrough

  • Azure - passthrough support with router models - PR #15240

Bugs​

• General
  • Fix x-litellm-cache-key header not being returned on cache hit - PR #15348

Management Endpoints / UI​

Features​

• Proxy CLI Auth

  • Proxy CLI - dont store existing key in the URL, store it in the state param - PR #15290

• Models + Endpoints

  • Make PATCH /model/{model_id}/update handle team_id consistently with POST /model/new - PR #15297
  • Feature: adds Infinity as a provider in the UI - PR #15285
  • Fix: model + endpoints page crash when config file contains router_settings.model_group_alias - PR #15308
  • Models & Endpoints Initial Refactor - PR #15435
  • Litellm UI API Reference page updates - PR #15438

• Teams

  • Teams page: new column "Your Role" on the teams table - PR #15384
  • LiteLLM Dashboard Teams UI refactor - PR #15418

• UI Infrastructure

  • Added prettier to autoformat frontend - PR #15215
  • Adds turbopack to the npm run dev command in UI to build faster during development - PR #15250
  • (perf) fix: Replaces bloated key list calls with lean key aliases endpoint - PR #15252
  • Potentially fixes a UI spasm issue with an expired cookie - PR #15309
  • LiteLLM UI Refactor Infrastructure - PR #15236
  • Enforces removal of unused imports from UI - PR #15416
  • Fix: usage page >> Model Activity >> spend per day graph: y-axis clipping on large spend values - PR #15389
  • Updates guardrail provider logos - PR #15421

• Admin Settings

  • Fix: Router settings do not update despite success message - PR #15249
  • Fix: Prevents DB from accidentally overriding config file values if they are empty in DB - PR #15340

• SSO

  • SSO - support EntraID app roles - PR #15351

Logging / Guardrail / Prompt Management Integrations​

Features​

• PostHog
  • Feat: posthog per request api key - PR #15379

Guardrails​

• EnkryptAI
  • Add EnkryptAI Guardrails on LiteLLM - PR #15390

Spend Tracking, Budgets and Rate Limiting​

• Tag Management

  • Tag Management - Add support for setting tag based budgets - PR #15433

• Dynamic Rate Limiter v3

  • QA/Fixes - Dynamic Rate Limiter v3 - final QA - PR #15311
  • Fix dynamic Rate limiter v3 - inserting litellm_model_saturation - PR #15394

• Shared Health Check

  • Implement Shared Health Check State Across Pods - PR #15380

MCP Gateway​

• Tool Control

  • MCP Gateway - UI - Select allowed tools for Key, Teams - PR #15241
  • MCP Gateway - Backend - Allow storing allowed tools by team/key - PR #15243
  • MCP Gateway - Fine-grained Database Object Storage Control - PR #15255
  • MCP Gateway - Litellm mcp fixes team control - PR #15304
  • MCP Gateway - QA/Fixes - Ensure Team/Key level enforcement works for MCPs - PR #15305
  • Feature: Include server_name in /v1/mcp/server/health endpoint response - PR #15431

• OpenAPI Integration

  • MCP - support converting OpenAPI specs to MCP servers - PR #15343
  • MCP - specify allowed params per tool - PR #15346

• Configuration

  • MCP - support setting CA_BUNDLE_PATH - PR #15253
  • Fix: Ensure MCP client stays open during tool call - PR #15391
  • Remove hardcoded "public" schema in migration.sql - PR #15363

Performance / Loadbalancing / Reliability improvements​

• Router Optimizations

  • Fix - Router: add model_name index for O(1) deployment lookups - PR #15113
  • Refactor Utils: extract inner function from client - PR #15234
  • Fix Networking: remove limitations - PR #15302

• Session Management

  • Fix - Sessions not being shared - PR #15388
  • Fix: remove panic from hot path - PR #15396
  • Fix - shared session parsing and usage issue - PR #15440
  • Fix: handle closed aiohttp sessions - PR #15442
  • Fix: prevent session leaks when recreating aiohttp sessions - PR #15443

• SSL/TLS Performance

  • Perf: optimize SSL/TLS handshake performance with prioritized cipher - PR #15398

• Dependencies

  • Upgrades tenacity version to 8.5.0 - PR #15303

• Data Masking

  • Fix - SensitiveDataMasker converts lists to string - PR #15420

General AI Gateway Improvements​

Security​

• General
  • Fix: redact AWS credentials when redact_user_api_key_info enabled - PR #15321

Documentation Updates​

• Provider Documentation

  • Update doc: perf update - PR #15211
  • Add W&B Inference documentation - PR #15278

• Deployment

  • Deletion of docker-compose buggy comment that cause config.yaml based startup fail - PR #15425

New Contributors​

• @Gal-bloch made their first contribution in PR #15219
• @lcfyi made their first contribution in PR #15315
• @ashengstd made their first contribution in PR #15362
• @vkolehmainen made their first contribution in PR #15363
• @jlan-nl made their first contribution in PR #15330
• @BCook98 made their first contribution in PR #15402
• @PabloGmz96 made their first contribution in PR #15425

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.77.7.rc.1

pip install litellm==1.77.7.rc.1

Key Highlights​

• Dynamic Rate Limiter v3 - Automatically maximizes throughput when capacity is available (< 80% saturation) by allowing lower-priority requests to use unused capacity, then switches to fair priority-based allocation under high load (≥ 80%) to prevent blocking
• Major Performance Improvements - 2.9x lower median latency at 1,000 concurrent users.
• Claude Sonnet 4.5 - Support for Anthropic's new Claude Sonnet 4.5 model family with 200K+ context and tiered pricing
• MCP Gateway Enhancements - Fine-grained tool control, server permissions, and forwardable headers
• AMD Lemonade & Nvidia NIM - New provider support for AMD Lemonade and Nvidia NIM Rerank
• GitLab Prompt Management - GitLab-based prompt management integration

Performance - 2.9x Lower Median Latency​

This update removes LiteLLM router inefficiencies, reducing complexity from O(M×N) to O(1). Previously, it built a new array and ran repeated checks like data["model"] in llm_router.get_model_ids(). Now, a direct ID-to-deployment map eliminates redundant allocations and scans.

As a result, performance improved across all latency percentiles:

• Median latency: 320 ms → 110 ms (−65.6%)
• p95 latency: 850 ms → 440 ms (−48.2%)
• p99 latency: 1,400 ms → 810 ms (−42.1%)
• Average latency: 864 ms → 310 ms (−64%)

Test Setup​

Locust

• Concurrent users: 1,000
• Ramp-up: 500

System Specs

• CPU: 4 vCPUs
• Memory: 8 GB RAM
• LiteLLM Workers: 4
• Instances: 4

Configuration (config.yaml)

View the complete configuration: gist.github.com/AlexsanderHamir/config.yaml

Load Script (no_cache_hits.py)

View the complete load testing script: gist.github.com/AlexsanderHamir/no_cache_hits.py

MCP OAuth 2.0 Support​

This release adds support for OAuth 2.0 Client Credentials for MCP servers. This is great for Internal Dev Tools use-cases, as it enables your users to call MCP servers, with their own credentials. E.g. Allowing your developers to call the Github MCP, with their own credentials.

Set it up today on Claude Code

Scheduled Key Rotations​

This release brings support for scheduling virtual key rotations on LiteLLM AI Gateway.

From this release you can enforce Virtual Keys to rotate on a schedule of your choice e.g every 15 days/30 days/60 days etc.

This is great for Proxy Admins who need to enforce security policies for production workloads.

Get Started

New Models / Updated Models​

New Model Support​

Features​

• Anthropic
  • Add new claude-sonnet-4-5 model family with tiered pricing above 200K tokens - PR #15041
  • Add anthropic/claude-sonnet-4-5 to model price json with prompt caching support - PR #15049
  • Add 200K prices for Sonnet 4.5 - PR #15140
  • Add cost tracking for /v1/messages in streaming response - PR #15102
  • Add /v1/messages/count_tokens to Anthropic routes for non-admin user access - PR #15034
• Gemini
  • Ignore type param for gemini tools - PR #15022
• Vertex AI
  • Add LiteLLM Overhead metric for VertexAI - PR #15040
  • Support googlemap grounding in vertex ai - PR #15179
• Azure
  • Add azure_ai grok-4 model family - PR #15137
  • Use the extra_query parameter for GET requests in Azure Batch - PR #14997
  • Use extra_query for download results (Batch API) - PR #15025
  • Add support for Azure AD token-based authorization - PR #14813
• Ollama
  • Add ollama cloud models - PR #15008
• Groq
  • Add groq/moonshotai/kimi-k2-instruct-0905 - PR #15079
• OpenAI
  • Add support for GPT 5 codex models - PR #14841
• DeepInfra
  • Update DeepInfra model data refresh with latest pricing - PR #14939
• Bedrock
  • Add JP Cross-Region Inference - PR #15188
  • Add "eu.anthropic.claude-sonnet-4-5-20250929-v1:0" - PR #15181
  • Add twelvelabs bedrock Async Invoke Support - PR #14871
• Nvidia NIM
  • Add Nvidia NIM Rerank Support - PR #15152

Bug Fixes​

• VLLM
  • Fix response_format bug in hosted vllm audio_transcription - PR #15010
  • Fix passthrough of atranscription into kwargs going to upstream provider - PR #15005
• OCI
  • Fix OCI Generative AI Integration when using Proxy - PR #15072
• General
  • Fix: Authorization header to use correct "Bearer" capitalization - PR #14764
  • Bug fix: gpt-5-chat-latest has incorrect max_input_tokens value - PR #15116
  • Update request handling for original exceptions - PR #15013

New Provider Support​

• AMD Lemonade
  • Add AMD Lemonade provider support - PR #14840

LLM API Endpoints​

Features​

• Responses API

  • Return Cost for Responses API Streaming requests - PR #15053

• /generateContent

  • Add full support for native Gemini API translation - PR #15029

• Passthrough Gemini Routes

  • Add Gemini generateContent passthrough cost tracking - PR #15014
  • Add streamGenerateContent cost tracking in passthrough - PR #15199

• Passthrough Vertex AI Routes

  • Add cost tracking for Vertex AI Passthrough /predict endpoint - PR #15019
  • Add cost tracking for Vertex AI Live API WebSocket Passthrough - PR #14956

• General

  • Preserve Whitespace Characters in Model Response Streams - PR #15160
  • Add provider name to payload specification - PR #15130
  • Ensure query params are forwarded from origin url to downstream request - PR #15087

Management Endpoints / UI​

Features​

• Virtual Keys

  • Ensure LLM_API_KEYs can access pass through routes - PR #15115
  • Support 'guaranteed_throughput' when setting limits on keys belonging to a team - PR #15120

• Models + Endpoints

  • Ensure OCI secret fields not shared on /models and /v1/models endpoints - PR #15085
  • Add snowflake on UI - PR #15083
  • Make UI theme settings publicly accessible for custom branding - PR #15074

• Admin Settings

  • Ensure OTEL settings are saved in DB after set on UI - PR #15118
  • Top api key tags - PR #15151, PR #15156

• MCP

  • show health status of MCP servers - PR #15185
  • allow setting extra headers on the UI - PR #15185
  • allow editing allowed tools on the UI - PR #15185

Bug Fixes​

• Virtual Keys

  • (security) prevent user key from updating other user keys - PR #15201
  • (security) don't return all keys with blank key alias on /v2/key/info - PR #15201
  • Fix Session Token Cookie Infinite Logout Loop - PR #15146

• Models + Endpoints

  • Make UI theme settings publicly accessible for custom branding - PR #15074

• Teams

  • fix failed copy to clipboard for http ui - PR #15195

• Logs

  • fix logs page render logs on filter lookup - PR #15195
  • fix lookup list of end users (migrate to more efficient /customers/list lookup) - PR #15195

• Test key

  • update selected model on key change - PR #15197

• Dashboard

  • Fix LiteLLM model name fallback in dashboard overview - PR #14998

Logging / Guardrail / Prompt Management Integrations​

Features​

• OpenTelemetry
  • Use generation_name for span naming in logging method - PR #14799
• Langfuse
  • Handle non-serializable objects in Langfuse logging - PR #15148
  • Set usage_details.total in langfuse integration - PR #15015
• Prometheus
  • support custom metadata labels on key/team - PR #15094

Guardrails​

• Javelin
  • Add Javelin standalone guardrails integration for LiteLLM Proxy - PR #14983
  • Add logging for important status fields in guardrails - PR #15090
  • Don't run post_call guardrail if no text returned from Bedrock - PR #15106

Prompt Management​

• GitLab
  • GitLab based Prompt manager - PR #14988

Spend Tracking, Budgets and Rate Limiting​

• Cost Tracking
  • Proxy: end user cost tracking in the responses API - PR #15124
• Parallel Request Limiter v3
  • Use well known redis cluster hashing algorithm - PR #15052
  • Fixes to dynamic rate limiter v3 - add saturation detection - PR #15119
  • Dynamic Rate Limiter v3 - fixes for detecting saturation + fixes for post saturation behavior - PR #15192
• Teams
  • Add model specific tpm/rpm limits to teams on LiteLLM - PR #15044

MCP Gateway​

• Server Configuration
  • Specify forwardable headers, specify allowed/disallowed tools for MCP servers - PR #15002
  • Enforce server permissions on call tools - PR #15044
  • MCP Gateway Fine-grained Tools Addition - PR #15153
• Bug Fixes
  • Remove servername prefix mcp tools tests - PR #14986
  • Resolve regression with duplicate Mcp-Protocol-Version header - PR #15050
  • Fix test_mcp_server.py - PR #15183

Performance / Loadbalancing / Reliability improvements​

• Router Optimizations
  • +62.5% P99 Latency Improvement - Remove router inefficiencies (from O(M*N) to O(1)) - PR #15046
  • Remove hasattr checks in Router - PR #15082
  • Remove Double Lookups - PR #15084
  • Optimize _filter_cooldown_deployments from O(n×m + k×n) to O(n) - PR #15091
  • Optimize unhealthy deployment filtering in retry path (O(n*m) → O(n+m)) - PR #15110
• Cache Optimizations
  • Reduce complexity of InMemoryCache.evict_cache from O(n*log(n)) to O(log(n)) - PR #15000
  • Avoiding expensive operations when cache isn't available - PR #15182
• Worker Management
  • Add proxy CLI option to recycle workers after N requests - PR #15007
• Metrics & Monitoring
  • LiteLLM Overhead metric tracking - Add support for tracking litellm overhead on cache hits - PR #15045

Documentation Updates​

• Provider Documentation
  • Update litellm docs from latest release - PR #15004
  • Add missing api_key parameter - PR #15058
• General Documentation
  • Use docker compose instead of docker-compose - PR #15024
  • Add railtracks to projects that are using litellm - PR #15144
  • Perf: Last week improvement - PR #15193
  • Sync models GitHub documentation with Loom video and cross-reference - PR #15191

Security Fixes​

• JWT Token Security - Don't log JWT SSO token on .info() log - PR #15145

New Contributors​

• @herve-ves made their first contribution in PR #14998
• @wenxi-onyx made their first contribution in PR #15008
• @jpetrucciani made their first contribution in PR #15005
• @abhijitjavelin made their first contribution in PR #14983
• @ZeroClover made their first contribution in PR #15039
• @cedarm made their first contribution in PR #15043
• @Isydmr made their first contribution in PR #15025
• @serializer made their first contribution in PR #15013
• @eddierichter-amd made their first contribution in PR #14840
• @malags made their first contribution in PR #15000
• @henryhwang made their first contribution in PR #15029
• @plafleur made their first contribution in PR #15111
• @tyler-liner made their first contribution in PR #14799
• @Amir-R25 made their first contribution in PR #15144
• @georg-wolflein made their first contribution in PR #15124
• @niharm made their first contribution in PR #15140
• @anthony-liner made their first contribution in PR #15015
• @rishiganesh2002 made their first contribution in PR #15153
• @danielaskdd made their first contribution in PR #15160
• @JVenberg made their first contribution in PR #15146
• @speglich made their first contribution in PR #15072
• @daily-kim made their first contribution in PR #14764

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.77.5-stable

pip install litellm==1.77.5

Key Highlights​

• MCP OAuth 2.0 Support - Enhanced authentication for Model Context Protocol integrations
• Scheduled Key Rotations - Automated key rotation capabilities for enhanced security
• New Gemini 2.5 Flash & Flash-lite Models - Latest September 2025 preview models with improved pricing and features
• Performance Improvements - 54% RPS improvement

Performance Improvements - 54% RPS Improvement​

This release brings a 54% RPS improvement (1,040 → 1,602 RPS, aggregated) per instance.

The improvement comes from fixing O(n²) inefficiencies in the LiteLLM Router, primarily caused by repeated use of in statements inside loops over large arrays.

Tests were run with a database-only setup (no cache hits).

Test Setup​

All benchmarks were executed using Locust with 1,000 concurrent users and a ramp-up of 500. The environment was configured to stress the routing layer and eliminate caching as a variable.

System Specs

• CPU: 8 vCPUs
• Memory: 32 GB RAM

Configuration (config.yaml)

View the complete configuration: gist.github.com/AlexsanderHamir/config.yaml

Load Script (no_cache_hits.py)

View the complete load testing script: gist.github.com/AlexsanderHamir/no_cache_hits.py

New Models / Updated Models​

New Model Support​

Features​

• Gemini
  • Added Gemini 2.5 Flash and Flash-lite preview models (September 2025 release) with improved pricing - PR #14948
  • Added new Anthropic web fetch tool support - PR #14951
• XAI
  • Add xai/grok-4-fast models - PR #14833
• Anthropic
  • Updated Claude Sonnet 4 configs to reflect million-token context window pricing - PR #14639
  • Added supported text field to anthropic citation response - PR #14164
• Bedrock
  • Added support for Qwen models family & Deepseek 3.1 to Amazon Bedrock - PR #14845
  • Support requestMetadata in Bedrock Converse API - PR #14570
• Vertex AI
  • Added vertex_ai/qwen models and azure/gpt-5-codex - PR #14844
  • Update vertex ai qwen model pricing - PR #14828
  • Vertex AI Context Caching: use Vertex ai API v1 instead of v1beta1 and accept 'cachedContent' param - PR #14831
• SambaNova
  • Add sambanova deepseek v3.1 and gpt-oss-120b - PR #14866
• OpenAI
  • Fix inconsistent token configs for gpt-5 models - PR #14942
  • GPT-3.5-Turbo price updated - PR #14858
• OpenRouter
  • Add gpt-5 and gpt-5-codex to OpenRouter cost map - PR #14879
• VLLM
  • Fix vllm passthrough - PR #14778
• Flux
  • Support flux image edit - PR #14790

Bug Fixes​

• Anthropic
  • Fix: Support claude code auth via subscription (anthropic) - PR #14821
  • Fix Anthropic streaming IDs - PR #14965
  • Revert incorrect changes to sonnet-4 max output tokens - PR #14933
• OpenAI
  • Fix a bug where openai image edit silently ignores multiple images - PR #14893
• VLLM
  • Fix: vLLM provider's rerank endpoint from /v1/rerank to /rerank - PR #14938

New Provider Support​

• W&B Inference
  • Add W&B Inference to LiteLLM - PR #14416

LLM API Endpoints​

Features​

• General
  • Add SDK support for additional headers - PR #14761
  • Add shared_session parameter for aiohttp ClientSession reuse - PR #14721

Bugs​

• General
  • Fix: Streaming tool call index assignment for multiple tool calls - PR #14587
  • Fix load credentials in token counter proxy - PR #14808

Management Endpoints / UI​

Features​

• Proxy CLI Auth
  • Allow re-using cli auth token - PR #14780
  • Create a python method to login using litellm proxy - PR #14782
  • Fixes for LiteLLM Proxy CLI to Auth to Gateway - PR #14836

Virtual Keys

• Initial support for scheduled key rotations - PR #14877
• Allow scheduling key rotations when creating virtual keys - PR #14960

Models + Endpoints

• Fix: added Oracle to provider's list - PR #14835

Bugs​

• SSO - Fix: SSO "Clear" button writes empty values instead of removing SSO config - PR #14826
• Admin Settings - Remove useful links from admin settings - PR #14918
• Management Routes - Add /user/list to management routes - PR #14868

Logging / Guardrail / Prompt Management Integrations​

Features​

• DataDog
  • Logging - datadog callback Log message content w/o sending to datadog - PR #14909
• Langfuse
  • Adding langfuse usage details for cached tokens - PR #10955
• Opik
  • Improve opik integration code - PR #14888
• SQS
  • Error logging support for SQS Logger - PR #14974

Guardrails​

• LakeraAI v2 Guardrail - Ensure exception is raised correctly - PR #14867
• Presidio Guardrail - Support custom entity types in Presidio guardrail with Union[PiiEntityType, str] - PR #14899
• Noma Guardrail - Add noma guardrail provider to ui - PR #14415

Prompt Management​

• BitBucket Integration - Add BitBucket Integration for Prompt Management - PR #14882

Spend Tracking, Budgets and Rate Limiting​

• Service Tier Pricing - Add service_tier based pricing support for openai (BOTH Service & Priority Support) - PR #14796
• Cost Tracking - Show input, output, tool call cost breakdown in StandardLoggingPayload - PR #14921
• Parallel Request Limiter v3
  • Ensure Lua scripts can execute on redis cluster - PR #14968
  • Fix: get metadata info from both metadata and litellm_metadata fields - PR #14783
• Priority Reservation - Fix: Priority Reservation: keys without priority metadata receive higher priority than keys with explicit priority configurations - PR #14832

MCP Gateway​

• MCP Configuration - Enable custom fields in mcp_info configuration - PR #14794
• MCP Tools - Remove server_name prefix from list_tools - PR #14720
• OAuth Flow - Initial commit for v2 oauth flow - PR #14964

Performance / Loadbalancing / Reliability improvements​

• Memory Leak Fix - Fix InMemoryCache unbounded growth when TTLs are set - PR #14869
• Cache Performance - Fix: cache root cause - PR #14827
• Concurrency Fix - Fix concurrency/scaling when many Python threads do streaming using sync completions - PR #14816
• Performance Optimization - Fix: reduce get_deployment cost to O(1) - PR #14967
• Performance Optimization - Fix: remove slow string operation - PR #14955
• DB Connection Management - Fix: DB connection state retries - PR #14925

Documentation Updates​

• Provider Documentation - Fix docs for provider_specific_params.md - PR #14787
• Model References - Update model references from gemini-pro to gemini-2.5-pro - PR #14775
• Letta Guide - Add Letta Guide documentation - PR #14798
• README - Make the README document clearer - PR #14860
• Session Management - Update docs for session management availability - PR #14914
• Cost Documentation - Add documentation for additional cost-related keys in custom pricing - PR #14949
• Azure Passthrough - Add azure passthrough documentation - PR #14958
• General Documentation - Doc updates sept 2025 - PR #14769
  • Clarified bridging between endpoints and mode in docs.
  • Added Vertex AI Gemini API configuration as an alternative in relevant guides. Linked AWS authentication info in the Bedrock guardrails documentation.
  • Added Cancel Response API usage with code snippets
  • Clarified that SSO (Single Sign-On) is free for up to 5 users:
  • Alphabetized sidebar, leaving quick start / intros at top of categories
  • Documented max_connections under cache_params.
  • Clarified IAM AssumeRole Policy requirements.
  • Added transform utilities example to Getting Started (showing request transformation).
  • Added references to models.litellm.ai as the full models list in various docs.
  • Added a code snippet for async_post_call_success_hook.
  • Removed broken links to callbacks management guide. - Reformatted and linked cookbooks + other relevant docs
• Documentation Corrections - Corrected docs updates sept 2025 - PR #14916

New Contributors​

• @uzaxirr made their first contribution in PR #14761
• @xprilion made their first contribution in PR #14416
• @CH-GAGANRAJ made their first contribution in PR #14779
• @otaviofbrito made their first contribution in PR #14778
• @danielmklein made their first contribution in PR #14639
• @Jetemple made their first contribution in PR #14826
• @akshoop made their first contribution in PR #14818
• @hazyone made their first contribution in PR #14821
• @leventov made their first contribution in PR #14816
• @fabriciojoc made their first contribution in PR #10955
• @onlylonly made their first contribution in PR #14845
• @Copilot made their first contribution in PR #14869
• @arsh72 made their first contribution in PR #14899
• @berri-teddy made their first contribution in PR #14914
• @vpbill made their first contribution in PR #14415
• @kgritesh made their first contribution in PR #14893
• @oytunkutrup1 made their first contribution in PR #14858
• @nherment made their first contribution in PR #14933
• @deepanshululla made their first contribution in PR #14974
• @TeddyAmkie made their first contribution in PR #14758
• @SmartManoj made their first contribution in PR #14775
• @uc4w6c made their first contribution in PR #14720
• @luizrennocosta made their first contribution in PR #14783
• @AlexsanderHamir made their first contribution in PR #14827
• @dharamendrak made their first contribution in PR #14721
• @TomeHirata made their first contribution in PR #14164
• @mrFranklin made their first contribution in PR #14860
• @luisfucros made their first contribution in PR #14866
• @huangyafei made their first contribution in PR #14879
• @thiswillbeyourgithub made their first contribution in PR #14949
• @Maximgitman made their first contribution in PR #14965
• @subnet-dev made their first contribution in PR #14938
• @22mSqRi made their first contribution in PR #14972

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.77.3-stable

pip install litellm==1.77.3

Key Highlights​

• +550 RPS Performance Improvements - Optimizations in request handling and object initialization.
• Priority Quota Reservation - Proxy admins can now reserve TPM/RPM capacity for specific keys.

Priority Quota Reservation​

This release adds support for priority quota reservation. This allows Proxy Admins to reserve specific percentages of model capacity for different use cases.

This is great for use cases where you want to ensure your realtime use cases must always get priority responses and background development jobs can take longer.

This release adds support for priority quota reservation. This allows Proxy Admins to reserve TPM/RPM capacity for keys based on metadata priority levels, ensuring critical production workloads get guaranteed access regardless of development traffic volume.

Get started here

+550 RPS Performance Improvements​

This release delivers significant RPS improvements through targeted optimizations.

We've achieved a +500 RPS boost by fixing cache type inconsistencies that were causing frequent cache misses, plus an additional +50 RPS by removing unnecessary coroutine checks from the hot path.

New Models / Updated Models​

New Model Support​

Features​

• OVHCloud AI Endpoints
  • New provider support with comprehensive model catalog - PR #14494
• CompactifAI
  • New provider integration - PR #14532
• SambaNova
  • Added DeepSeek v3.1 and GPT-OSS-120B models - PR #14500
• Bedrock
  • Cross-region inference profile cost calculation - PR #14566
  • AWS external ID parameter support for authentication - PR #14582
  • CountTokens API implementation - PR #14557
  • Titan V2 encoding_format parameter support - PR #14687
  • Nova Canvas image generation inference profiles - PR #14578
  • Bedrock Batches API - batch processing support with file upload and request transformation - PR #14618
  • Bedrock Twelve Labs embedding provider support - PR #14697
• Vertex AI
  • Gemini labels field provider-aware filtering - PR #14563
  • Gemini Batch API support - PR #14733
• Volcengine
  • Fixed thinking parameters when disabled - PR #14569
• Cohere
  • Handle Generate API deprecation, default to chat endpoints - PR #14676
• TwelveLabs
  • Added Marengo Embed 2.7 embedding support - PR #14674

Bug Fixes​

• Bedrock
  • Empty arguments handling in tool call invocation - PR #14583
• Vertex AI
  • Avoid deepcopy crash with non-pickleables in Gemini/Vertex - PR #14418
• XAI
  • Fix unsupported stop parameter for grok-code models - PR #14565
• Gemini
  • Updated error message for Gemini API - PR #14589
  • Fixed 2.5 Flash Image Preview model routing - PR #14715
  • API key passing for token counting endpoints - PR #14744

New Provider Support​

• OVHCloud AI Endpoints
  • Complete provider integration with model catalog and authentication - PR #14494
• CompactifAI
  • New provider support with documentation - PR #14532

LLM API Endpoints​

Features​

• /responses
  • Added cancel endpoint support for non-admin users - PR #14594
  • Improved response session handling and cold storage configuration with s3 - PR #14534
  • Added OpenAI & Azure /responses/cancel endpoint support - PR #14561
• General
  • Enhanced rate limit error messages with details - PR #14736
  • Middle-truncation for spend log payloads - PR #14637

Bugs​

• /chat/completions
  • Fixed completion chat ID handling - PR #14548
  • Prevent AttributeError for _get_tags_from_request_kwargs - PR #14735
• /responses
  • Fixed cost calculation - PR #14675
• General
  • Rate limiter AttributeError fix - PR #14609

Spend Tracking, Budgets and Rate Limiting​

• Responses API Cost Calculation fix - PR #14675
• Anthropic Cache Token Pricing - Separate 1-hour vs 5-minute cache creation costs - PR #14620, PR #14652
• Indochina Time Timezone support for budget resets - PR #14666
• Soft Budget Alert Cache Issues - Resolved soft budget alert cache issues - PR #14491
• Dynamic Rate Limiter v3 - Priority routing improvements - PR #14734
• Enhanced Rate Limit Errors - More detailed error messages - PR #14736

Management Endpoints / UI​

Features​

• Team Member Service Account Keys - Allow team members to view keys they create - PR #14619
• Default Budget for JWT Teams - Auto-assign budgets to generated teams - PR #14514
• SSO Access Control Groups - Enhanced token info endpoint integration - PR #14738
• Health Test Connect Protection - Restrict access based on model creation permissions - PR #14650
• Amazon Bedrock Guardrail Info View - Enhanced logging visualization - PR #14696

Bug Fixes​

• SCIM v2 - Fix group PUSH and PUT operations for non-existent members - PR #14581
• Guardrail View/Edit/Delete behavior fixes - PR #14622
• In-Memory Guardrail update failures - PR #14653

Logging / Guardrail Integrations​

Features​

• DataDog
  • Enhanced spend tracking metrics - PR #14555
  • Stream support with is_streamed_request parameter - PR #14673
  • Fixed tool calls metadata passing - PR #14531
• Langfuse
  • Added logging support for Responses API - PR #14597
• Langsmith
  • Langsmith Sampling Rate - Key/Team-level tracing configuration - PR #14740
• Prometheus
  • Multi-worker support improvements - PR #14530
  • User email labels in monitoring - PR #14520
• Opik
  • Fixed timezone issue - PR #14708

Bug Fixes​

• S3
  • Fixed 404 error when using s3_endpoint_url - PR #14559

Guardrails​

• Tool Permission Guardrail - Fine-grained tool access control - PR #14519
• Bedrock Guardrails - Selective guarding support with runtime endpoint configuration - PR #14575, PR #14650
• Default Last Message in guardrails - PR #14640
• AWS exceptions handling despite 200 response - PR #14658

New Integration​

• PostHog - Complete observability integration for LiteLLM usage tracking and analytics - PR #14610

MCP Gateway​

• MCP Server Alias Parsing - Multi-part URL path support - PR #14558
• MCP Filter Recomputation - After server deletion - PR #14542
• MCP Gateway Tools List improvements - PR #14695

Performance / Loadbalancing / Reliability improvements​

• +500 RPS Performance Boost when sending the user field - PR #14616
• +50 RPS by removing iscoroutine from hot path - PR #14649
• 7% reduction in init overhead - PR #14689
• Generic Object Pool implementation for better resource management - PR #14702

General Proxy Improvements​

• Middle-Truncation for spend log payloads - PR #14637

Security​

• Security Update - Bump aiohttp==3.12.14, fix CVE-2025-53643 - PR #14638

New Contributors​

• @luisfucros made their first contribution in PR #14500
• @hanakannzashi made their first contribution in PR #14548
• @eliasto made their first contribution in PR #14494
• @Rasmusafj made their first contribution in PR #14491
• @LingXuanYin made their first contribution in PR #14569
• @ronaldpereira made their first contribution in PR #14613
• @hula-la made their first contribution in PR #14534
• @carlos-marchal-ph made their first contribution in PR #14610
• @akraines made their first contribution in PR #14637
• @mrFranklin made their first contribution in PR #14708
• @tcx4c70 made their first contribution in PR #14675
• @michaeltansg made their first contribution in PR #14666
• @tosi29 made their first contribution in PR #14725
• @gmdfalk made their first contribution in PR #14735
• @FelipeRodriguesGare made their first contribution in PR #14733
• @mritunjaysharma394 made their first contribution in PR #14678

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.77.2-stable

pip install litellm==1.77.2.post1

Key Highlights​

• Bedrock Batches API - Support for creating Batch Inference Jobs on Bedrock using LiteLLM's unified batch API (OpenAI compatible)
• Qwen API Tiered Pricing - Cost tracking support for Dashscope (Qwen) models with multiple pricing tiers

New Models / Updated Models​

New Model Support​

Features​

• Bedrock
  • Bedrock Batches API - batch processing support with file upload and request transformation - PR #14518, PR #14522
• VLLM
  • Added transcription endpoint support - PR #14523
• Ollama
  • ollama_chat/ - images, thinking, and content as list handling - PR #14523
• General
  • New debug flag for detailed request/response logging PR #14482

Bug Fixes​

• Azure OpenAI
  • Fixed extra_body injection causing payload rejection in image generation - PR #14475
• LM Studio
  • Resolved illegal Bearer header value issue - PR #14512

LLM API Endpoints​

Bug Fixes​

• /messages
  • Don't send content block after message w/ finish reason + usage block - PR #14477
• /generateContent
  • Gemini CLI Integration - Fixed token count errors - PR #14451, PR #14417

Spend Tracking, Budgets and Rate Limiting​

Features​

• Qwen API Tiered Pricing - Added comprehensive tiered cost tracking for Dashscope/Qwen models - PR #14471, PR #14479

Bug Fixes​

• Provider Budgets - Fixed provider budget calculations - PR #14459

Management Endpoints / UI​

Features​

• User Headers Mapping - New X-LiteLLM Users mapping feature for enhanced user tracking - PR #14485
• Key Unblocking - Support for hashed tokens in /key/unblock endpoint - PR #14477
• Model Group Header Forwarding - Enhanced wildcard model support with documentation - PR #14528

Bug Fixes​

• Log Tab Key Alias - Fixed filtering inaccuracies for failed logs - PR #14469, PR #14529

Logging / Guardrail Integrations​

Features​

• Noma Integration - Added non-blocking monitor mode with anonymize input support - PR #14401

Performance / Loadbalancing / Reliability improvements​

Performance​

• Removed dynamic creation of static values - PR #14538
• Using _PROXY_MaxParallelRequestsHandler_v3 by default for optimal throughput - PR #14450
• Improved execution context propagation into logging tasks - PR #14455

New Contributors​

• @Sameerlite made their first contribution in PR #14460
• @holzman made their first contribution in PR #14459
• @sashank5644 made their first contribution in PR #14469
• @TomAlon made their first contribution in PR #14401
• @AlexsanderHamir made their first contribution in PR #14538

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.76.3

pip install litellm==1.76.3

Key Highlights​

• Major Performance Improvements +400 RPS when using correct amount of workers + CPU cores combination
• Video Generation Support - Added Google AI Studio and Vertex AI Veo Video Generation through LiteLLM Pass through routes
• CloudZero Integration - New cost tracking integration for exporting LiteLLM Usage and Spend data to CloudZero.

Major Changes​

• Performance Optimization: LiteLLM Proxy now achieves +400 RPS when using correct amount of CPU cores - PR #14153, PR #14242

  By default, LiteLLM will now use num_workers = os.cpu_count() to achieve optimal performance.

  Override Options:

  Set environment variable:

  DEFAULT_NUM_WORKERS_LITELLM_PROXY=1
  

  Or start LiteLLM Proxy with:

  litellm --num_workers 1
  

• Security Fix: Fixed memory_usage_in_mem_cache cache endpoint vulnerability - PR #14229

Performance Improvements​

This release includes significant performance optimizations. On our internal benchmarks we saw 1 instance get +400 RPS when using correct amount of workers + CPU cores combination.

• +400 RPS Performance Boost - LiteLLM Proxy now uses correct amount of CPU cores for optimal performance - PR #14153
• Default CPU Workers - Changed DEFAULT_NUM_WORKERS_LITELLM_PROXY default to number of CPUs - PR #14242

New Models / Updated Models​

New Model Support​

Features​

• Google Gemini
  • Added 'thoughtSignature' support via 'thinking_blocks' - PR #14122
  • Added support for reasoning_effort='minimal' for Gemini models - PR #14262
• OpenRouter
  • Added GPT-4.1 model family - PR #14101
• Groq
  • Added support for reasoning_effort parameter - PR #14207
• X.AI
  • Fixed XAI cost calculation - PR #14127
• Vertex AI
  • Added support for GPT-OSS models on Vertex AI - PR #14184
  • Added additionalProperties to Vertex AI Schema definition - PR #14252
• VLLM
  • Handle output parsing responses API output - PR #14121
• Ollama
  • Added unified 'thinking' param support via reasoning_content - PR #14121
• Anthropic
  • Added supported text field to anthropic citation response - PR #14126
• OCI Provider
  • Handle assistant messages with both content and tool_calls - PR #14171
• Bedrock
  • Fixed structure output - PR #14130
  • Added initial support for Bedrock Batches API - PR #14190
• Databricks
  • Added support for anthropic citation API in Databricks - PR #14077

Bug Fixes​

• Google Gemini (Google AI Studio + Vertex AI)
  • Fixed Gemini 2.5 Pro schema validation with OpenAI-style type arrays in tools - PR #14154
  • Fixed Gemini Tool Calling empty enum property - PR #14155

New Provider Support​

• Volcengine
  • Added Volcengine embedding module with handler and transformation logic - PR #14028

LLM API Endpoints​

Features​

• Images API
  • Added pass through image generation and image editing on OpenAI - PR #14292
  • Support extra_body parameter for image generation - PR #14211
• Responses API
  • Fixed response API for reasoning item in input for litellm proxy - PR #14200
  • Added structured output for SDK - PR #14206
• Bedrock Passthrough
  • Support AWS_BEDROCK_RUNTIME_ENDPOINT on bedrock passthrough - PR #14156
• Google AI Studio Passthrough
  • Allow using Veo Video Generation through LiteLLM Pass through routes - PR #14228
• General
  • Added support for safety_identifier parameter in chat.completions.create - PR #14174
  • Fixed misclassified 500 error on invalid image_url in /chat/completions request - PR #14149
  • Fixed token count error for Gemini CLI - PR #14133

Bugs​

• General
  • Remove "/" or ":" from model name when being used as h11 header name - PR #14191
  • Bug fix for openai.gpt-oss when using reasoning_effort parameter - PR #14300

Spend Tracking, Budgets and Rate Limiting​

Features​

• Added header support for spend_logs_metadata - PR #14186
• Litellm passthrough cost tracking for chat completion - PR #14256

Bug Fixes​

• Fixed TPM Rate Limit Bug - PR #14237
• Fixed Key Budget not resets at expectable times - PR #14241

Management Endpoints / UI​

Features​

• UI Improvements
  • Logs page screen size fixed - PR #14135
  • Create Organization Tooltip added on Success - PR #14132
  • Back to Keys should say Back to Logs - PR #14134
  • Add client side pagination on All Models table - PR #14136
  • Model Filters UI improvement - PR #14131
  • Remove table filter on user info page - PR #14169
  • Team name badge added on the User Details - PR #14003
  • Fix: Log page parameter passing error - PR #14193
• Authentication & Authorization
  • Support for ES256/ES384/ES512 and EdDSA JWT verification - PR #14118
  • Ensure team_id is a required field for generating service account keys - PR #14270

Bugs​

• General
  • Validate store model in db setting - PR #14269

Logging / Guardrail Integrations​

Features​

• Datadog
  • Ensure apm_id is set on DD LLM Observability traces - PR #14272
• Braintrust
  • Fix logging when OTEL is enabled - PR #14122
• OTEL
  • Optional Metrics and Logs following semantic conventions - PR #14179
• Slack Alerting
  • Added alert type to alert message to slack for easier handling - PR #14176

Guardrails​

• Added guardrail to the Anthropic API endpoint - PR #14107

New Integration​

• CloudZero
  • LiteLLM x CloudZero Integration for Cost Tracking - PR #14296

Performance / Loadbalancing / Reliability improvements​

Features​

• Performance
  • LiteLLM Proxy: +400 RPS when using correct amount of CPU cores - PR #14153
  • Allow using x-litellm-stream-timeout header for stream timeout in requests - PR #14147
  • Change DEFAULT_NUM_WORKERS_LITELLM_PROXY default to number CPUs - PR #14242
• Monitoring
  • Added Prometheus missing metrics - PR #14139
• Timeout
  • Stream Timeout Control - Allow using x-litellm-stream-timeout header for stream timeout in requests - PR #14147
• Routing
  • Fixed x-litellm-tags not routing with Responses API - PR #14289

Bugs​

• Security
  • Fixed memory_usage_in_mem_cache cache endpoint vulnerability - PR #14229

General Proxy Improvements​

Features​

• SCIM Support
  • Added better SCIM debugging - PR #14221
  • Bug fixes for handling SCIM Group Memberships - PR #14226
• Kubernetes
  • Added optional PodDisruptionBudget for litellm proxy - PR #14093
• Error Handling
  • Add model to azure error message - PR #14294

New Contributors​

• @iabhi4 made their first contribution in PR #14093
• @zainhas made their first contribution in PR #14087
• @LifeDJIK made their first contribution in PR #14146
• @retanoj made their first contribution in PR #14133
• @zhxlp made their first contribution in PR #14193
• @kayoch1n made their first contribution in PR #14191
• @kutsushitaneko made their first contribution in PR #14171
• @mjmendo made their first contribution in PR #14176
• @HarshavardhanK made their first contribution in PR #14213
• @eycjur made their first contribution in PR #14207
• @22mSqRi made their first contribution in PR #14241
• @onlylhf made their first contribution in PR #14028
• @btpemercier made their first contribution in PR #11319
• @tremlin made their first contribution in PR #14287
• @TobiMayr made their first contribution in PR #14262
• @Eitan1112 made their first contribution in PR #14252

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.76.1

pip install litellm==1.76.1

Key Highlights​

• Major Performance Improvements - 6.5x faster LiteLLM Python SDK completion with fastuuid integration.
• New Model Support - Gemini 2.5 Flash Image Preview, Grok Code Fast, and GPT Realtime models
• Enhanced Provider Support - DeepSeek-v3.1 pricing on Fireworks AI, Vercel AI Gateway, and improved Anthropic/GitHub Copilot integration
• MCP Improvements - Better connection testing and SSE MCP tools bug fixes

Major Changes​

• Added support for using Gemini 2.5 Flash Image Preview with /chat/completions. 🚨 Warning If you were using gemini-2.0-flash-exp-image-generation please follow this migration guide. Gemini Image Generation Migration Guide

Performance Improvements​

This release includes significant performance optimizations:

• 6.5x faster LiteLLM Python SDK Completion - Major performance boost for completion operations - PR #13990
• fastuuid Integration - 2.1x faster UUID generation with +80 RPS improvement for /chat/completions and other LLM endpoints - PR #13992, PR #14016
• Optimized Request Logging - Don't print request params by default for +50 RPS improvement - PR #14015
• Cache Performance - 21% speedup in InMemoryCache.evict_cache and 45% speedup in _is_debugging_on function - PR #14012, PR #13988

New Models / Updated Models​

New Model Support​

Additional Models Added: Various other Vercel AI Gateway models were added too. See models.litellm.ai for the full list.

Features​

• Google Gemini
  • Added support for gemini-2.5-flash-image-preview with image return capability - PR #13979, PR #13983
  • Support for requests with only system prompt - PR #14010
  • Fixed invalid model name error for Gemini Imagen models - PR #13991
• X.AI
  • Added xai/grok-code-fast model family support - PR #14054
  • Fixed frequency_penalty parameter for grok-4 models - PR #14078
• OpenAI
  • Added support for gpt-realtime models - PR #14082
  • Support for reasoning and reasoning_effort parameters by default - PR #12865
• Fireworks AI
  • Added DeepSeek-v3.1 pricing - PR #13958
• DeepInfra
  • Fixed reasoning_effort setting for DeepSeek-V3.1 - PR #14053
• GitHub Copilot
  • Added support for thinking and reasoning_effort parameters - PR #13691
  • Added image headers support - PR #13955
• Anthropic
  • Support for custom Anthropic-compatible API endpoints - PR #13945
  • Fixed /messages fallback from Anthropic API to Bedrock API - PR #13946
• Nebius
  • Expanded provider models and normalized model IDs - PR #13965
• Vertex AI
  • Fixed Vertex Mistral streaming issues - PR #13952
  • Fixed anyOf corner cases for Gemini tool calls - PR #12797
• Bedrock
  • Fixed structure output issues - PR #14005
• OpenRouter
  • Added GPT-5 family models pricing - PR #13536

New Provider Support​

• Vercel AI Gateway
  • New provider support added - PR #13144
• DataRobot
  • Added provider documentation - PR #14038, PR #14074

LLM API Endpoints​

Features​

• Images API
  • Support for multiple images in OpenAI images/edits endpoint - PR #13916
  • Allow using dynamic api_key for image generation requests - PR #14007
• Responses API
  • Fixed /responses endpoint ignoring extra_headers in GitHub Copilot - PR #13775
  • Added support for new web_search tool - PR #14083
• Azure Passthrough
  • Fixed Azure Passthrough request with streaming - PR #13831

Bugs​

• General
  • Fixed handling of None metadata in batch requests - PR #13996
  • Fixed token_counter with special token input - PR #13374
  • Removed incorrect web search support for azure/gpt-4.1 family - PR #13566

MCP Gateway​

Features​

• SSE MCP Tools
  • Bug fix for adding SSE MCP tools - improved connection testing when adding MCPs - PR #14048

Read More

Management Endpoints / UI​

Features​

• Team Management
  • Allow setting Team Member RPM/TPM limits when creating a team - PR #13943
• UI Improvements
  • Fixed Next.js Security Vulnerabilities in UI Dashboard - PR #14084
  • Fixed collapsible navbar design - PR #14075

Bugs​

• Authentication
  • Fixed Virtual keys with llm_api type causing Internal Server Error for /anthropic/* and other LLM passthrough routes - PR #14046

Logging / Guardrail Integrations​

Features​

• Langfuse OTEL
  • Allow using LANGFUSE_OTEL_HOST for configuring host - PR #14013
• Braintrust
  • Added span name metadata feature - PR #13573
  • Fixed tests to reference moved attributes in braintrust_logging module - PR #13978
• OpenMeter
  • Set user from token user_id for OpenMeter integration - PR #13152

New Guardrail Support​

• Noma Security
  • Added Noma Security guardrail support - PR #13572
• Pangea
  • Updated Pangea Guardrail to support new AIDR endpoint - PR #13160

Performance / Loadbalancing / Reliability improvements​

Features​

• Caching
  • Verify if cache entry has expired prior to serving it to client - PR #13933
  • Fixed error saving latency as timedelta on Redis - PR #14040
• Router
  • Refactored router to choose weights by 'weight', 'rpm', 'tpm' in one loop for simple_shuffle - PR #13562
• Logging
  • Fixed LoggingWorker graceful shutdown to prevent CancelledError warnings - PR #14050
  • Enhanced logging for containers to log on files both with usual format and json format - PR #13394

Bugs​

• Dependencies
  • Bumped orjson version to "3.11.2" - PR #13969

General Proxy Improvements​

Features​

• AWS
  • Add support for AWS assume_role with a session token - PR #13919
• OCI Provider
  • Added oci_key_file as an optional_parameter - PR #14036
• Configuration
  • Allow configuration to set threshold before request entry in spend log gets truncated - PR #14042
  • Enhanced proxy_config configuration: add support for existing configmap in Helm charts - PR #14041
• Docker
  • Added back supervisor to non-root image - PR #13922

New Contributors​

• @ArthurRenault made their first contribution in PR #13922
• @stevenmanton made their first contribution in PR #13919
• @uc4w6c made their first contribution in PR #13914
• @nielsbosma made their first contribution in PR #13573
• @Yuki-Imajuku made their first contribution in PR #13567
• @codeflash-ai[bot] made their first contribution in PR #13988
• @ColeFrench made their first contribution in PR #13978
• @dttran-glo made their first contribution in PR #13969
• @manascb1344 made their first contribution in PR #13965
• @DorZion made their first contribution in PR #13572
• @edwardsamuel made their first contribution in PR #13536
• @blahgeek made their first contribution in PR #13374
• @Deviad made their first contribution in PR #13394
• @XSAM made their first contribution in PR #13775
• @KRRT7 made their first contribution in PR #14012
• @ikaadil made their first contribution in PR #13991
• @timelfrink made their first contribution in PR #13691
• @qidu made their first contribution in PR #13562
• @nagyv made their first contribution in PR #13243
• @xywei made their first contribution in PR #12885
• @ericgtkb made their first contribution in PR #12797
• @NoWall57 made their first contribution in PR #13945
• @lmwang9527 made their first contribution in PR #14050
• @WilsonSunBritten made their first contribution in PR #14042
• @Const-antine made their first contribution in PR #14041
• @dmvieira made their first contribution in PR #14040
• @gotsysdba made their first contribution in PR #14036
• @moshemorad made their first contribution in PR #14005
• @joshualipman123 made their first contribution in PR #13144

Full Changelog​

Deploy this version​

New Models / Updated Models​

Bugs​

• OpenAI
  • Gpt-5 chat: clarify does not support function calling PR #13612, s/o  @superpoussin22
• VertexAI
  • fix vertexai batch file format by @thiagosalvatore in PR #13576
• LiteLLM Proxy
  • Add support for calling image_edits + image_generations via SDK to Proxy - PR #13735
• OpenRouter
  • Fix max_output_tokens value for anthropic Claude 4 - PR #13526
• Gemini
  • Fix prompt caching cost calculation - PR #13742
• Azure
  • Support ../openai/v1/respones api base - PR #13526
  • Fix azure/gpt-5-chat max_input_tokens - PR #13660
• Groq
  • streaming ASCII encoding issue - PR #13675
• Baseten
  • Refactored integration to use new openai-compatible endpoints - PR #13783
• Bedrock
  • fix application inference profile for pass-through endpoints for bedrock - PR #13881
• DataRobot
  • Updated URL handling for DataRobot provider URL - PR #13880

Features​

• Together AI
  • Added Qwen3, Deepseek R1 0528 Throughput, GLM 4.5 and GPT-OSS models cost tracking - PR #13637, s/o  @Tasmay-Tibrewal
• Fireworks AI
  • add fireworks_ai/accounts/fireworks/models/deepseek-v3-0324 - PR #13821
• VertexAI
  • Add VertexAI qwen API Service - PR #13828
  • Add new VertexAI image models vertex_ai/imagen-4.0-generate-001, vertex_ai/imagen-4.0-ultra-generate-001, vertex_ai/imagen-4.0-fast-generate-001  - PR #13874
• Anthropic
  • Add long context support w/ cost tracking - PR #13759
• DeepInfra
  • Add rerank endpoint support for deepinfra - PR #13820
  • Add new models for cost tracking - PR #13883, s/o  @Toy-97
• Bedrock
  • Add tool prompt caching on async calls - PR #13803, s/o  @UlookEE
  • role chaining and session name with webauthentication for aws bedrock - PR #13753, s/o @RichardoC
• Ollama
  • Handle Ollama null response when using tool calling with non-tool trained models - PR #13902
• OpenRouter
  • Add deepseek/deepseek-chat-v3.1 support - PR #13897
• Mistral
  • Add support for calling mistral files via chat completions - PR #13866, s/o  @jinskjoy
  • Handle empty assistant content - PR #13671
  • Support new ‘thinking’ response block - PR #13671
• Databricks
  • remove deprecated dbrx models (dbrx-instruct, llama 3.1) - PR #13843
• AI/ML API
  • Image gen api support - PR #13893

LLM API Endpoints​

Bugs​

• Responses API
  • add default api version for openai responses api calls - PR #13526
  • support allowed_openai_params - PR #13671

MCP Gateway​

Bugs​

• fix StreamableHTTPSessionManager .run() error - PR #13666

Vector Stores​

Bugs​

• Bedrock
  • Using LiteLLM Managed Credentials for Query - PR #13787

Management Endpoints / UI​

Bugs​

• Passthrough
  • Fix query passthrough deletion - PR #13622

Features​

• Models
  • Add Search Functionality for Public Model Names in Model Dashboard - PR #13687
  • Auto-Add azure/ to deployment Name in UI - PR #13685
  • Models page row UI restructure - PR #13771
• Notifications
  • Add new notifications toast UI everywhere - PR #13813
• Keys
  • Fix key edit settings after regenerating a key - PR #13815
  • Require team_id when creating service account keys - PR #13873
  • Filter - show all options on filter option click - PR #13858
• Usage
  • Fix ‘Cannot read properties of undefined’ exception on user agent activity tab - PR #13892
• SSO
  • Free SSO usage for up to 5 users - PR #13843

Logging / Guardrail Integrations​

Bugs​

• Bedrock Guardrails
  • Add bedrock api key support - PR #13835

Features​

• Datadog LLM Observability
  • Add support for Failure Logging PR #13726
  • Add time to first token, litellm overhead, guardrail overhead latency metrics - PR #13734
  • Add support for tracing guardrail input/output - PR #13767
• Langfuse OTEL
  • Allow using Key/Team Based Logging - PR #13791
• AIM
  • Migrate to new firewall API - PR #13748
• OTEL
  • Add OTEL tracing for actual LLM API call - PR #13836
• MLFlow
  • Include predicted output in MLflow tracing - PR #13795, s/o @TomeHirata 

Performance / Loadbalancing / Reliability improvements​

Bugs​

• Cooldowns
  • don't return raw Azure Exceptions to client (can contain prompt leakage) - PR #13529
• Auto-router
  • Ensures the relevant dependencies for auto router existing on LiteLLM Docker - PR #13788
• Model Alias
  • Fix calling key with access to model alias - PR #13830

Features​

• S3 Caching
  • Use namespace as prefix for s3 cache - PR #13704
  • Async S3 Caching support (4x RPS improvement) - PR #13852, s/o @michal-otmianowski
• Model Group header forwarding
  • reuse same logic as global header forwarding - PR #13741
  • add support for hosted_vllm on UI - PR #13885
• Performance
  • Improve LiteLLM Python SDK RPS by +200 RPS (braintrust import + aiohttp transport fixes) - PR #13839
  • Use O(1) Set lookups for model routing - PR #13879
  • Reduce Significant CPU overhead from litellm_logging.py - PR #13895
  • Improvements for Async Success Handler (Logging Callbacks) - Approx +130 RPS - PR #13905

General Proxy Improvements​

Bugs​

• SDK
  • Fix litellm compatibility with newest release of openAI (>v1.100.0) - PR #13728
• Helm
  • Add possibility to configure resources for migrations-job - PR #13617
  • Ensure Helm chart auto generated master keys follow sk-xxxx format - PR #13871
  • Enhance database configuration: add support for optional endpointKey - PR #13763
• Rate Limits
  • fixing descriptor/response size mismatch on parallel_request_limiter_v3 - PR #13863, s/o  @luizrennocosta
• Non-root
  • fix permission access on prisma migrate in non-root image - PR #13848, s/o @Ithanil

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.75.8-stable

pip install litellm==1.75.8

Key Highlights​

• Team Member Rate Limits - Individual rate limiting for team members with JWT authentication support.
• Performance Improvements - New experimental HTTP handler flag for 100+ RPS improvement on OpenAI calls.
• GPT-5 Model Family Support - Full support for OpenAI's GPT-5 models with reasoning_effort parameter and Azure OpenAI integration.
• Azure AI Flux Image Generation - Support for Azure AI's Flux image generation models.

Team Member Rate Limits​

LiteLLM MCP Architecture: Use MCP tools with all LiteLLM supported models

This release adds support for setting rate limits on individual members (including machine users) within a team. Teams can now give each agent its own rate limits—so that heavy-traffic agents don’t impact other agents or human users.

Agents can authenticate with LiteLLM using JWT and the same team role as human users, while still enforcing per-agent rate limits.

New Models / Updated Models​

New Model Support​

Features​

• OpenAI
  • Added reasoning_effort parameter support for GPT-5 model family - PR #13475, Get Started
  • Support for reasoning parameter in Responses API - PR #13475, Get Started
• Azure OpenAI
  • GPT-5 support with max_tokens and reasoning parameter - PR #13510, Get Started
• AWS Bedrock
  • Streaming support for bedrock gpt-oss model family - PR #13346, Get Started
  • /messages endpoint compatibility with bedrock/converse/<model> - PR #13627
  • Cache point support for assistant and tool messages - PR #13640
• Azure AI
  • New Azure AI Flux Image Generation provider - PR #13592, Get Started
  • Fixed Content-Type header for image generation - PR #13584
• CometAPI
  • New provider support with chat completions and streaming - PR #13458
• SambaNova
  • Added embedding model support - PR #13308, Get Started
• Vertex AI
  • Added /countTokens endpoint support for Gemini CLI integration - PR #13545
  • Token counter support for VertexAI models - PR #13558
• hosted_vllm
  • Added reasoning_effort parameter support - PR #13620, Get Started

Bugs​

• OCI
  • Fixed streaming issues - PR #13437
• Ollama
  • Fixed GPT-OSS streaming with 'thinking' field - PR #13375
• VolcEngine
  • Fixed thinking disabled parameter handling - PR #13598
• Streaming
  • Consistent 'finish_reason' chunk indexing - PR #13560

LLM API Endpoints​

Features​

• /messages
  • Tool use arguments properly returned for non-anthropic models - PR #13638

Bugs​

• Real-time API
  • Fixed endpoint for no intent scenarios - PR #13476
• Responses API
  • Fixed stream=True + background=True with Responses API - PR #13654

MCP Gateway​

Features​

• Access Control & Configuration
  • Enhanced MCPServerManager with access groups and description support - PR #13549

Bugs​

• Authentication
  • Fixed MCP gateway key authentication - PR #13630

Read More

Management Endpoints / UI​

Features​

• Team Management
  • Team Member Rate Limits implementation - PR #13601
  • JWT authentication support for team member rate limits - PR #13601
  • Show team member TPM/RPM limits in UI - PR #13662
  • Allow editing team member RPM/TPM limits - PR #13669
  • Allow unsetting TPM and RPM in Teams Settings - PR #13430
  • Team Member Permissions Page access column changes - PR #13145
• Key Management
  • Display errors from backend on the UI Keys page - PR #13435
  • Added confirmation modal before deleting keys - PR #13655
  • Support for user parameter in LiteLLM SDK to Proxy communication - PR #13555
• UI Improvements
  • Fixed internal users table overflow - PR #12736
  • Enhanced chart readability with short-form notation for large numbers - PR #12370
  • Fixed image overflow in LiteLLM model display - PR #13639
  • Removed ambiguous network response errors - PR #13582
• Credentials
  • Added CredentialDeleteModal component and integration with CredentialsPanel - PR #13550
• Admin & Permissions
  • Allow routes for admin viewer - PR #13588

Bugs​

• SCIM Integration
  • Fixed SCIM Team Memberships metadata handling - PR #13553
• Authentication
  • Fixed incorrect key info endpoint - PR #13633

Logging / Guardrail Integrations​

Features​

• Langfuse OTEL
  • Added key/team logging for Langfuse OTEL Logger - PR #13512
  • Fixed LangfuseOtelSpanAttributes constants to match expected values - PR #13659
• MLflow
  • Updated MLflow logger usage span attributes - PR #13561

Bugs​

• Security
  • Hide sensitive data in /model/info - azure entra client_secret - PR #13577
  • Fixed trivy/secrets false positives - PR #13631

Performance / Loadbalancing / Reliability improvements​

Features​

• HTTP Performance
  • New 'EXPERIMENTAL_OPENAI_BASE_LLM_HTTP_HANDLER' flag for +100 RPS improvement on OpenAI calls - PR #13625
• Database Monitoring
  • Added DB metrics to Prometheus - PR #13626
• Error Handling
  • Added safe divide by 0 protection to prevent crashes - PR #13624

Bugs​

• Dependencies
  • Updated boto3 to 1.36.0 and aioboto3 to 13.4.0 - PR #13665

General Proxy Improvements​

Features​

• Database
  • Removed redundant use_prisma_migrate flag - now default - PR #13555
• LLM Translation
  • Added model ID check - PR #13507
  • Refactored Anthropic configurations and added support for anthropic_beta headers - PR #13590

New Contributors​

• @TensorNull made their first contribution in PR #13458
• @MajorD00m made their first contribution in PR #13577
• @VerunicaM made their first contribution in PR #13584
• @huangyafei made their first contribution in PR #13607
• @TomeHirata made their first contribution in PR #13561
• @willfinnigan made their first contribution in PR #13659
• @dcbark01 made their first contribution in PR #13633
• @javacruft made their first contribution in PR #13631

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.75.5-stable

pip install litellm==1.75.5.post2

Key Highlights​

• Redis - Latency Improvements - Reduces P99 latency by 50% with Redis enabled.
• Responses API Session Management - Support for managing responses API sessions with images.
• Oracle Cloud Infrastructure - New LLM provider for calling models on Oracle Cloud Infrastructure.
• Digital Ocean's Gradient AI - New LLM provider for calling models on Digital Ocean's Gradient AI platform.

Risk of Upgrade​

If you build the proxy from the pip package, you should hold off on upgrading. This version makes prisma migrate deploy our default for managing the DB. This is safer, as it doesn't reset the DB, but it requires a manual prisma generate step.

Users of our Docker image, are not affected by this change.

Redis Latency Improvements​

This release adds in-memory caching for Redis requests, enabling faster response times in high-traffic. Now, LiteLLM instances will check their in-memory cache for a cache hit, before checking Redis. This reduces caching-related latency from 100ms for LLM API calls to sub-1ms, on cache hits.

Responses API Session Management w/ Images​

LiteLLM now supports session management for Responses API requests with images. This is great for use-cases like chatbots, that are using the Responses API to track the state of a conversation. LiteLLM session management works across ALL LLM API's (including Anthropic, Bedrock, OpenAI, etc). LiteLLM session management works by storing the request and response content in an s3 bucket, you can specify.

New Models / Updated Models​

New Model Support​

Features​

• OCI
  • New LLM provider - PR #13206
• JinaAI
  • support multimodal embedding models - PR #13181
• GPT-5 (OpenAI/Azure)
  • Support drop_params for temperature - PR #13390
  • Map max_tokens to max_completion_tokens - PR #13390
• Anthropic
  • Add claude-opus-4-1 on model cost map - PR #13384
• OpenRouter
  • Add gpt-oss to model cost map - PR #13442
• Cerebras
  • Add gpt-oss to model cost map - PR #13442
• Azure
  • Support drop params for ‘temperature’ on o-series models - PR #13353
• GradientAI
  • New LLM Provider - PR #12169

Bugs​

• OpenAI
  • Add ‘service_tier’ and ‘safety_identifier’ as supported responses api params - PR #13258
  • Correct pricing for web search on 4o-mini - PR #13269
• Mistral
  • Handle $id and $schema fields when calling mistral - PR #13389

LLM API Endpoints​

Features​

• /responses
  • Responses API Session Handling w/ support for images - PR #13347
  • failed if input containing ResponseReasoningItem - PR #13465
  • Support custom tools - PR #13418

Bugs​

• /chat/completions
  • Fix completion_token_details usage object missing ‘text’ tokens - PR #13234
  • (SDK) handle tool being a pydantic object - PR #13274
  • include cost in streaming usage object - PR #13418
  • Exclude none fields on /chat/completion - allows usage with n8n - PR #13320
• /responses
  • Transform function call in response for non-openai models (gemini/anthropic) - PR #13260
  • Fix unsupported operand error with model groups - PR #13293
  • Responses api session management for streaming responses - PR #13396
• /v1/messages
  • Added litellm claude code count tokens - PR #13261
• /vector_stores
  • Fix create/search vector store errors - PR #13285

MCP Gateway​

Features​

• Add route check for internal users - PR #13350
• MCP Guardrails - docs - PR #13392

Bugs​

• Fix auth on UI for bearer token servers - PR #13312
• allow access group on mcp tool retrieval - PR #13425

Management Endpoints / UI​

Features​

• Teams
  • Add team deletion check for teams with keys - PR #12953
• Models
  • Add ability to set model alias per key/team - PR #13276
  • New button to reload model pricing from model cost map - PR #13464, PR #13470
• Keys
  • Make ‘team’ field required when creating service account keys - PR #13302
  • Gray out key-based logging settings for non-enterprise users - prevents confusion on if ‘logging’ all up is supported - PR #13431
• Navbar
  • Add logo customization for LiteLLM admin UI - PR #12958
• Logs
  • Add token breakdowns on logs + session page - PR #13357
• Usage
  • Ensure Usage Page loads after the DB has large entries - PR #13400
• Test Key Page
  • allow uploading images for /chat/completions and /responses - PR #13445
• MCP
  • Add auth tokens to local storage auth - PR #13473

Bugs​

• Custom Root Path
  • Fix login route when SSO is enabled - PR #13267
• Customers/End-users
  • Allow calling /v1/models when end user over budget - allows model listing to work on OpenWebUI when customer over budget - PR #13320
• Teams
  • Remove user - team membership, when user removed from team - PR #13433
• Errors
  • Bubble up network errors to user for Logging and Alerts page - PR #13427
• Model Hub
  • Show pricing for azure models, when base model is set - PR #13418

Logging / Guardrail Integrations​

Features​

• Bedrock Guardrails
  • Redacted sensitive information in bedrock guardrails error message - PR #13356
• Standard Logging Payload
  • Fix ‘can’t register atextexit’ bug - PR #13436

Bugs​

• Braintrust
  • Allow setting of braintrust callback base url - PR #13368
• OTEL
  • Track pre_call hook latency - PR #13362

Performance / Loadbalancing / Reliability improvements​

Features​

• Team-BYOK models
  • Add wildcard model support - PR #13278
• Caching
  • GCP IAM auth support for caching - PR #13275
• Latency
  • reduce p99 latency w/ redis enabled by 50% - only updates model usage if tpm/rpm limits set - PR #13362

General Proxy Improvements​

Features​

• Models
  • Support /v1/models/{model_id} retrieval - PR #13268
• Multi-instance
  • Ensure disable_llm_api_endpoints works - PR #13278
• Logs
  • Add apscheduler log suppress - PR #13299
• Helm
  • Add labels to migrations job template - PR #13343 s/o @unique-jakub

Bugs​

• Non-root image
  • Fix non-root image for migration - PR #13379
• Get Routes
  • Load get routes when using fastapi-offline - PR #13466
• Health checks
  • Generate unique trace IDs for Langfuse health checks - PR #13468
• Swagger
  • Allow using Swagger for /chat/completions - PR #13469
• Auth
  • Fix JWTs access not working with model access groups - PR #13474

New Contributors​

• @bbartels made their first contribution in https://github.com/BerriAI/litellm/pull/13244
• @breno-aumo made their first contribution in https://github.com/BerriAI/litellm/pull/13206
• @pascalwhoop made their first contribution in https://github.com/BerriAI/litellm/pull/13122
• @ZPerling made their first contribution in https://github.com/BerriAI/litellm/pull/13045
• @zjx20 made their first contribution in https://github.com/BerriAI/litellm/pull/13181
• @edwarddamato made their first contribution in https://github.com/BerriAI/litellm/pull/13368
• @msannan2 made their first contribution in https://github.com/BerriAI/litellm/pull/12169

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.74.15-stable

pip install litellm==1.74.15.post2

Key Highlights​

• User Agent Activity Tracking - Track how much usage each coding tool gets.
• Prompt Management - Use Git-Ops style prompt management with prompt templates.
• MCP Gateway: Guardrails - Support for using Guardrails with MCP servers.
• Google AI Studio Imagen4 - Support for using Imagen4 models on Google AI Studio.

User Agent Activity Tracking​

This release brings support for tracking usage and costs for AI-powered coding tools like Claude Code, Roo Code, Gemini CLI through LiteLLM. You can now track LLM cost, total tokens used, and DAU/WAU/MAU for each coding tool.

This is great to central AI Platform teams looking to track how they are helping developer productivity.

Read More

Prompt Management​

Read More

New Models / Updated Models​

New Model Support​

Features​

• Google AI Studio
  • Added Google AI Studio Imagen4 model family support - PR #13065, Get Started
• Azure OpenAI
  • Azure api_version="preview" support - PR #13072, Get Started
  • Password protected certificate files support - PR #12995, Get Started
• AWS Bedrock
  • Cost tracking via Anthropic /v1/messages - PR #13072
  • Computer use support - PR #13150
• OpenRouter
  • Added Grok4 model support - PR #13018
• Anthropic
  • Auto Cache Control Injection - Improved cache_control_injection_points with negative index support - PR #13187, Get Started
  • Working mid-stream fallbacks with token usage tracking - PR #13149, PR #13170
• Perplexity
  • Citation annotations support - PR #13225

Bugs​

• Gemini
  • Fix merge_reasoning_content_in_choices parameter issue - PR #13066, Get Started
  • Added support for using GOOGLE_API_KEY environment variable for Google AI Studio - PR #12507
• vLLM/OpenAI-like
  • Fix missing extra_headers support for embeddings - PR #13198

LLM API Endpoints​

Bugs​

• /generateContent
  • Support for query_params in generateContent routes for API Key setting - PR #13100
  • Ensure "x-goog-api-key" is used for auth to google ai studio when using /generateContent on LiteLLM - PR #13098
  • Ensure tool calling works as expected on generateContent - PR #13189
• /vertex_ai (Passthrough)
  • Ensure multimodal embedding responses are logged properly - PR #13050

MCP Gateway​

Features​

• Health Check Improvements
  • Add health check endpoints for MCP servers - PR #13106
• Guardrails Integration
  • Add pre and during call hooks initialization - PR #13067
  • Move pre and during hooks to ProxyLogging - PR #13109
  • MCP pre and during guardrails implementation - PR #13188
• Protocol & Header Support
  • Add protocol headers support - PR #13062
• URL & Namespacing
  • Improve MCP server URL validation for internal/Kubernetes URLs - PR #13099

Bugs​

• UI
  • Fix scrolling issue with MCP tools - PR #13015
  • Fix MCP client list failure - PR #13114

Read More

Management Endpoints / UI​

Features​

• Usage Analytics

  • New tab for user agent activity tracking - PR #13146
  • Daily usage per user analytics - PR #13147
  • Default usage chart date range set to last 7 days - PR #12917
  • New advanced date range picker component - PR #13141, PR #13221
  • Show loader on usage cost charts after date selection - PR #13113

• Models

  • Added Voyage, Jinai, Deepinfra and VolcEngine providers on UI - PR #13131
  • Added Sagemaker on UI - PR #13117
  • Preserve model order in /v1/models and /model_group/info endpoints - PR #13178

• Key Management

  • Properly parse JSON options for key generation in UI - PR #12989

• Authentication

  • JWT Fields
    • Add dot notation support for all JWT fields - PR #13013

Bugs​

• Permissions
  • Fix object permission for organizations - PR #13142
  • Fix list team v2 security check - PR #13094
• Models
  • Fix model reload on model update - PR #13216
• Router Settings
  • Fix displaying models for fallbacks in UI - PR #13191
  • Fix wildcard model name handling with custom values - PR #13116
  • Fix fallback delete functionality - PR #12606

Logging / Guardrail Integrations​

Features​

• MLFlow
  • Allow adding tags for MLFlow logging requests - PR #13108
• Langfuse OTEL
  • Add comprehensive metadata support to Langfuse OpenTelemetry integration - PR #12956
• Datadog LLM Observability
  • Allow redacting message/response content for specific logging integrations - PR #13158

Bugs​

• API Key Logging
  • Fix API Key being logged inappropriately - PR #12978
• MCP Spend Tracking
  • Set default value for MCP namespace tool name in spend table - PR #12894

Performance / Loadbalancing / Reliability improvements​

Features​

• Background Health Checks
  • Allow disabling background health checks for specific deployments - PR #13186
• Database Connection Management
  • Ensure stale Prisma clients disconnect DB connections properly - PR #13140
• Jitter Improvements
  • Fix jitter calculation (should be added not multiplied) - PR #12901

Bugs​

• Anthropic Streaming
  • Always use choice index=0 for Anthropic streaming responses - PR #12666
• Custom Auth
  • Bubble up custom exceptions properly - PR #13093
• OTEL with Managed Files
  • Fix using managed files with OTEL integration - PR #13171

General Proxy Improvements​

Features​

• Database Migration
  • Move to use_prisma_migrate by default - PR #13117
  • Resolve team-only models on auth checks - PR #13117
• Infrastructure
  • Loosened MCP Python version restrictions - PR #13102
  • Migrate build_and_test to CI/CD Postgres DB - PR #13166
• Helm Charts
  • Allow Helm hooks for migration jobs - PR #13174
  • Fix Helm migration job schema updates - PR #12809

Bugs​

• Docker
  • Remove obsolete version attribute in docker-compose - PR #13172
  • Add openssl in runtime stage for non-root Dockerfile - PR #13168
• Database Configuration
  • Fix DB config through environment variables - PR #13111
• Logging
  • Suppress httpx logging - PR #13217
• Token Counting
  • Ignore unsupported keys like prefix in token counter - PR #11954

New Contributors​

• @5731la made their first contribution in https://github.com/BerriAI/litellm/pull/12989
• @restato made their first contribution in https://github.com/BerriAI/litellm/pull/12980
• @strickvl made their first contribution in https://github.com/BerriAI/litellm/pull/12956
• @Ne0-1 made their first contribution in https://github.com/BerriAI/litellm/pull/12995
• @maxrabin made their first contribution in https://github.com/BerriAI/litellm/pull/13079
• @lvuna made their first contribution in https://github.com/BerriAI/litellm/pull/12894
• @Maximgitman made their first contribution in https://github.com/BerriAI/litellm/pull/12666
• @pathikrit made their first contribution in https://github.com/BerriAI/litellm/pull/12901
• @huetterma made their first contribution in https://github.com/BerriAI/litellm/pull/12809
• @betterthanbreakfast made their first contribution in https://github.com/BerriAI/litellm/pull/13029
• @phosae made their first contribution in https://github.com/BerriAI/litellm/pull/12606
• @sahusiddharth made their first contribution in https://github.com/BerriAI/litellm/pull/12507
• @Amit-kr26 made their first contribution in https://github.com/BerriAI/litellm/pull/11954
• @kowyo made their first contribution in https://github.com/BerriAI/litellm/pull/13172
• @AnandKhinvasara made their first contribution in https://github.com/BerriAI/litellm/pull/13187
• @unique-jakub made their first contribution in https://github.com/BerriAI/litellm/pull/13174
• @tyumentsev4 made their first contribution in https://github.com/BerriAI/litellm/pull/13134
• @aayush-malviya-acquia made their first contribution in https://github.com/BerriAI/litellm/pull/12978
• @kankute-sameer made their first contribution in https://github.com/BerriAI/litellm/pull/13225
• @AlexanderYastrebov made their first contribution in https://github.com/BerriAI/litellm/pull/13178

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.74.9-stable.patch.1

pip install litellm==1.74.9.post2

Key Highlights​

• Auto-Router - Automatically route requests to specific models based on request content.
• Model-level Guardrails - Only run guardrails when specific models are used.
• MCP Header Propagation - Propagate headers from client to backend MCP.
• New LLM Providers - Added Bedrock inpainting support and Recraft API image generation / image edits support.

Auto-Router​

This release introduces auto-routing to models based on request content. This means Proxy Admins can define a set of keywords that always routes to specific models when users opt in to using the auto-router.

This is great for internal use cases where you don't want users to think about which model to use - for example, use Claude models for coding vs GPT models for generating ad copy.

Read More

Model-level Guardrails​

This release brings model-level guardrails support to your config.yaml + UI. This is great for cases when you have an on-prem and hosted model, and just want to run prevent sending PII to the hosted model.

model_list:
  - model_name: claude-sonnet-4
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY
      api_base: https://api.anthropic.com/v1
      guardrails: ["azure-text-moderation"] # 👈 KEY CHANGE

guardrails:
  - guardrail_name: azure-text-moderation
    litellm_params:
      guardrail: azure/text_moderations
      mode: "post_call" 
      api_key: os.environ/AZURE_GUARDRAIL_API_KEY
      api_base: os.environ/AZURE_GUARDRAIL_API_BASE 

Read More

MCP Header Propagation​

v1.74.9-stable allows you to propagate MCP server specific authentication headers via LiteLLM

• Allowing users to specify which header_name is to be propagated to which mcp_server via headers
• Allows adding of different deployments of same MCP server type to use different authentication headers

Read More

New Models / Updated Models​

Pricing / Context Window Updates​

Features​

• Lambda AI
  • New LLM API provider - PR #12817
• Github Copilot
  • Dynamic endpoint support - PR #12827
• Morph
  • New LLM API provider - PR #12821
• Groq
  • Remove deprecated groq/qwen-qwq-32b - PR #12832
• Recraft
  • New image generation API - PR #12832
  • New image edits api - PR #12874
• Azure OpenAI
  • Support DefaultAzureCredential without hard-coded environment variables - PR #12841
• Hyperbolic
  • New LLM API provider - PR #12826
• OpenAI
  • /realtime API - pass through intent query param - PR #12838
• Bedrock
  • Add inpainting support for Amazon Nova Canvas - PR #12949 s/o @SantoshDhaladhuli

Bugs​

• Gemini (Google AI Studio + VertexAI)
  • Fix leaking file descriptor error on sync calls - PR #12824
• IBM Watsonx
  • use correct parameter name for tool choice - PR #9980
• Anthropic
  • Only show ‘reasoning_effort’ for supported models - PR #12847
  • Handle $id and $schema in tool call requests (Anthropic API stopped accepting them) - PR #12959
• Openrouter
  • filter out cache_control flag for non-anthropic models (allows usage with claude code) https://github.com/BerriAI/litellm/pull/12850
• Gemini
  • Shorten Gemini tool_call_id for Open AI compatibility - PR #12941 s/o @tonga54

LLM API Endpoints​

Features​

• Passthrough endpoints
  • Make key/user/team cost tracking OSS - PR #12847
• /v1/models
  • Return fallback models as part of api response - PR #12811 s/o @murad-khafizov
• /vector_stores
  • Make permission management OSS - PR #12990

Bugs​

1. /batches
   1. Skip invalid batch during cost tracking check (prev. Would stop all checks) - PR #12782
2. /chat/completions
   1. Fix async retryer on .acompletion() - PR #12886

MCP Gateway​

Features​

• Permission Management
  • Make permission management by key/team OSS - PR #12988
• MCP Alias
  • Support mcp server aliases (useful for calling long mcp server names on Cursor) - PR #12994
• Header Propagation
  • Support propagating headers from client to backend MCP (useful for sending personal access tokens to backend MCP) - PR #13003

Management Endpoints / UI​

Features​

• Usage
  • Support viewing usage by model group - PR #12890
• Virtual Keys
  • New key_type field on /key/generate - allows specifying if key can call LLM API vs. Management routes - PR #12909
• Models
  • Add ‘auto router’ on UI - PR #12960
  • Show global retry policy on UI - PR #12969
  • Add model-level guardrails on create + update - PR #13006

Bugs​

• SSO
  • Fix logout when SSO is enabled - PR #12703
  • Fix reset SSO when ui_access_mode is updated - PR #13011
• Guardrails
  • Show correct guardrails when editing a team - PR #12823
• Virtual Keys
  • Get updated token on regenerate key - PR #12788
  • Fix CVE with key injection - PR #12840

Logging / Guardrail Integrations​

Features​

• Google Cloud Model Armor
  • Document new guardrail - PR #12492
• Pillar Security
  • New LLM Guardrail - PR #12791
• CloudZero
  • Allow exporting spend to cloudzero - PR #12908
• Model-level Guardrails
  • Support model-level guardrails - PR #12968

Bugs​

• Prometheus
  • Fix [tag]=false when tag is set for tag-based metrics - PR #12916
• Guardrails AI
  • Use ‘validatedOutput’ to allow usage of “fix” guards - PR #12891 s/o @DmitriyAlergant

Performance / Loadbalancing / Reliability improvements​

Features​

• Auto-Router
  • New auto-router powered by semantic-router - PR #12955

Bugs​

• forward_clientside_headers
  • Filter out content-length from headers (caused backend requests to hang) - PR #12886
• Message Redaction
  • Fix cannot pickle coroutine object error - PR #13005

General Proxy Improvements​

Features​

• Benchmarks
  • Updated litellm proxy benchmarks (p50, p90, p99 overhead) - PR #12842
• Request Headers
  • Added new x-litellm-num-retries request header
• Swagger
  • Support local swagger on custom root paths - PR #12911
• Health
  • Track cost + add tags for health checks done by LiteLLM Proxy - PR #12880

Bugs​

• Proxy Startup
  • Fixes issue on startup where team member budget is None would block startup - PR #12843
• Docker
  • Move non-root docker to chain guard image (fewer vulnerabilities) - PR #12707
  • add azure-keyvault==4.2.0 to Docker img - PR #12873
• Separate Health App
  • Pass through cmd args via supervisord (enables user config to still work via docker) - PR #12871
• Swagger
  • Bump DOMPurify version (fixes vulnerability) - PR #12911
  • Add back local swagger bundle (enables swagger to work in air gapped env.) - PR #12911
• Request Headers
  • Make ‘user_header_name’ field check case insensitive (fixes customer budget enforcement for OpenWebUi) - PR #12950
• SpendLogs
  • Fix issues writing to DB when custom_llm_provider is None - PR #13001

New Contributors​

• @magicalne made their first contribution in https://github.com/BerriAI/litellm/pull/12804
• @pavangudiwada made their first contribution in https://github.com/BerriAI/litellm/pull/12798
• @mdiloreto made their first contribution in https://github.com/BerriAI/litellm/pull/12707
• @murad-khafizov made their first contribution in https://github.com/BerriAI/litellm/pull/12811
• @eagle-p made their first contribution in https://github.com/BerriAI/litellm/pull/12791
• @apoorv-sharma made their first contribution in https://github.com/BerriAI/litellm/pull/12920
• @SantoshDhaladhuli made their first contribution in https://github.com/BerriAI/litellm/pull/12949
• @tonga54 made their first contribution in https://github.com/BerriAI/litellm/pull/12941
• @sings-to-bees-on-wednesdays made their first contribution in https://github.com/BerriAI/litellm/pull/12950

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.74.7-stable.patch.1

pip install litellm==1.74.7.post2

Key Highlights​

• Vector Stores - Support for Vertex RAG Engine, PG Vector, OpenAI & Azure OpenAI Vector Stores.
• Bulk Editing Users - Bulk editing users on the UI.
• Health Check Improvements - Prevent unnecessary pod restarts during high traffic.
• New LLM Providers - Added Moonshot AI and Vercel v0 provider support.

Vector Stores API​

This release introduces support for using VertexAI RAG Engine, PG Vector, Bedrock Knowledge Bases, and OpenAI Vector Stores with LiteLLM.

This is ideal for use cases requiring external knowledge sources with LLMs.

This brings the following benefits for LiteLLM users:

Proxy Admin Benefits:

• Fine-grained access control: determine which Keys and Teams can access specific Vector Stores
• Complete usage tracking and monitoring across all vector store operations

Developer Benefits:

• Simple, unified interface for querying vector stores and using them with LLM API requests
• Consistent API experience across all supported vector store providers

Get started

Bulk Editing Users​

v1.74.7-stable introduces Bulk Editing Users on the UI. This is useful for:

• granting all existing users to a default team (useful for controlling access / tracking spend by team)
• controlling personal model access for existing users

Read more

Health Check Server​

This release brings reliability improvements that prevent unnecessary pod restarts during high traffic. Previously, when the main LiteLLM app was busy serving traffic, health endpoints would timeout even when pods were healthy.

Starting with this release, you can run health endpoints on an isolated process with a dedicated port. This ensures liveness and readiness probes remain responsive even when the main LiteLLM app is under heavy load.

Read More

New Models / Updated Models​

Pricing / Context Window Updates​

Features​

• 🆕 Moonshot API (Kimi)
  • New LLM API integration for accessing Kimi models - PR #12592, Get Started
• 🆕 v0 Provider
  • New provider integration for v0.dev - PR #12751, Get Started
• OpenAI
  • Use OpenAI DeepResearch models with litellm.completion (/chat/completions) - PR #12627 DOC NEEDED
• Azure OpenAI
  • Use Azure OpenAI DeepResearch models with litellm.completion (/chat/completions) - PR #12627 DOC NEEDED
  • Added response_format support for openai gpt-4.1 models - PR #12745
• Anthropic
  • Tool cache control support - PR #12668
• Bedrock
  • Claude 4 /invoke route support - PR #12599, Get Started
  • Application inference profile tool choice support - PR #12599
• Gemini
  • Custom TTL support for context caching - PR #12541
  • Fix implicit caching cost calculation for Gemini 2.x models - PR #12585
• VertexAI
  • Added Vertex AI RAG Engine support (use with OpenAI compatible /vector_stores API) - PR #12752, Get Started
• vLLM
  • Added support for using Rerank endpoints with vLLM - PR #12738, Get Started
• AI21
  • Added ai21/jamba-1.7 model family pricing - PR #12593, Get Started
• Together.ai
  • [New Model] add together_ai/moonshotai/Kimi-K2-Instruct - PR #12645, Get Started
• Groq
  • Add groq/moonshotai-kimi-k2-instruct model configuration - PR #12648, Get Started
• Github Copilot
  • Change System prompts to assistant prompts for GH Copilot - PR #12742, Get Started

Bugs​

• Anthropic
  • Fix streaming + response_format + tools bug - PR #12463
• XAI
  • grok-4 does not support the stop param - PR #12646
• AWS
  • Role chaining with web authentication for AWS Bedrock - PR #12607
• VertexAI
  • Add project_id to cached credentials - PR #12661
• Bedrock
  • Fix bedrock nova micro and nova lite context window info in PR #12619

LLM API Endpoints​

Features​

• /chat/completions
  • Include tool calls in output of trim_messages - PR #11517
• /v1/vector_stores
  • New OpenAI-compatible vector store endpoints - PR #12699, Get Started
  • Vector store search endpoint - PR #12749, Get Started
  • Support for using PG Vector as a vector store - PR #12667, Get Started
• /streamGenerateContent
  • Non-gemini model support - PR #12647

Bugs​

• /vector_stores
  • Knowledge Base Call returning error when passing as tools - PR #12628

MCP Gateway​

Features​

• Access Groups
  • Allow MCP access groups to be added via litellm proxy config.yaml - PR #12654
  • List tools from access list for keys - PR #12657
• Namespacing
  • URL-based namespacing for better segregation - PR #12658
  • Make MCP_TOOL_PREFIX_SEPARATOR configurable from env - PR #12603
• Gateway Features
  • Allow using MCPs with all LLM APIs (VertexAI, Gemini, Groq, etc.) when using /responses - PR #12546

Bugs​

• Fix to update object permission on update/delete key/team - PR #12701
• Include /mcp in list of available routes on proxy - PR #12612

Management Endpoints / UI​

Features​

• Keys
  • Regenerate Key State Management improvements - PR #12729
• Models
  • Wildcard model filter support - PR #12597
  • Fixes for handling team only models on UI - PR #12632
• Usage Page
  • Fix Y-axis labels overlap on Spend per Tag chart - PR #12754
• Teams
  • Allow setting custom key duration + show key creation stats - PR #12722
  • Enable team admins to update member roles - PR #12629
• Users
  • New /user/bulk_update endpoint - PR #12720
• Logs Page
  • Add end_user filter on UI Logs Page - PR #12663
• MCP Servers
  • Copy MCP Server name functionality - PR #12760
• Vector Stores
  • UI support for clicking into Vector Stores - PR #12741
  • Allow adding Vertex RAG Engine, OpenAI, Azure through UI - PR #12752
• General
  • Add Copy-on-Click for all IDs (Key, Team, Organization, MCP Server) - PR #12615
• SCIM
  • Add GET /ServiceProviderConfig endpoint - PR #12664

Bugs​

• Teams
  • Ensure user id correctly added when creating new teams - PR #12719
  • Fixes for handling team-only models on UI - PR #12632

Logging / Guardrail Integrations​

Features​

• Google Cloud Model Armor
  • New guardrails integration - PR #12492
• Bedrock Guardrails
  • Allow disabling exception on 'BLOCKED' action - PR #12693
• Guardrails AI
  • Support llmOutput based guardrails as pre-call hooks - PR #12674
• DataDog LLM Observability
  • Add support for tracking the correct span type based on LLM Endpoint used - PR #12652
• Custom Logging
  • Allow reading custom logger python scripts from S3 or GCS Bucket - PR #12623

Bugs​

• General Logging
  • StandardLoggingPayload on cache_hits should track custom llm provider - PR #12652
• S3 Buckets
  • S3 v2 log uploader crashes when using with guardrails - PR #12733

Performance / Loadbalancing / Reliability improvements​

Features​

• Health Checks
  • Separate health app for liveness probes - PR #12669
  • Health check app on separate port - PR #12718
• Caching
  • Add Azure Blob cache support - PR #12587
• Router
  • Handle ZeroDivisionError with zero completion tokens in lowest_latency strategy - PR #12734

Bugs​

• Database
  • Use upsert for managed object table to avoid UniqueViolationError - PR #11795
  • Refactor to support use_prisma_migrate for helm hook - PR #12600
• Cache
  • Fix: redis caching for embedding response models - PR #12750

Helm Chart​

• DB Migration Hook: refactor to support use_prisma_migrate - for helm hook PR
• Add envVars and extraEnvVars support to Helm migrations job - PR #12591

General Proxy Improvements​

Features​

• Control Plane + Data Plane Architecture
  • Control Plane + Data Plane support - PR #12601
• Proxy CLI
  • Add "keys import" command to CLI - PR #12620
• Swagger Documentation
  • Add swagger docs for LiteLLM /chat/completions, /embeddings, /responses - PR #12618
• Dependencies
  • Loosen rich version from ==13.7.1 to >=13.7.1 - PR #12704

Bugs​

• Verbose log is enabled by default fix - PR #12596

• Add support for disabling callbacks in request body - PR #12762

• Handle circular references in spend tracking metadata JSON serialization - PR #12643

New Contributors​

• @AntonioKL made their first contribution in https://github.com/BerriAI/litellm/pull/12591
• @marcelodiaz558 made their first contribution in https://github.com/BerriAI/litellm/pull/12541
• @dmcaulay made their first contribution in https://github.com/BerriAI/litellm/pull/12463
• @demoray made their first contribution in https://github.com/BerriAI/litellm/pull/12587
• @staeiou made their first contribution in https://github.com/BerriAI/litellm/pull/12631
• @stefanc-ai2 made their first contribution in https://github.com/BerriAI/litellm/pull/12622
• @RichardoC made their first contribution in https://github.com/BerriAI/litellm/pull/12607
• @yeahyung made their first contribution in https://github.com/BerriAI/litellm/pull/11795
• @mnguyen96 made their first contribution in https://github.com/BerriAI/litellm/pull/12619
• @rgambee made their first contribution in https://github.com/BerriAI/litellm/pull/11517
• @jvanmelckebeke made their first contribution in https://github.com/BerriAI/litellm/pull/12725
• @jlaurendi made their first contribution in https://github.com/BerriAI/litellm/pull/12704
• @doublerr made their first contribution in https://github.com/BerriAI/litellm/pull/12661

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.74.3-stable

pip install litellm==1.74.3.post1

Key Highlights​

• MCP: Model Access Groups - Add mcp servers to access groups, for easily managing access to users and teams.
• MCP: Tool Cost Tracking - Set prices for each MCP tool.
• Model Hub v2 - New OSS Model Hub for telling developers what models are available on the proxy.
• Bytez - New LLM API Provider.
• Dashscope API - Call Alibaba's qwen models via new Dashscope API Provider.

MCP Gateway: Model Access Groups​

v1.74.3-stable adds support for adding MCP servers to access groups, this makes it easier for Proxy Admins to manage access to MCP servers across users and teams.

For developers, this means you can now connect to multiple MCP servers by passing the access group name in the x-mcp-servers header.

Read more here

MCP Gateway: Tool Cost Tracking​

This release adds cost tracking for MCP tool calls. This is great for Proxy Admins giving MCP access to developers as you can now attribute MCP tool call costs to specific LiteLLM keys and teams.

You can set:

• Uniform server cost: Set a uniform cost for all tools from a server
• Individual tool cost: Define individual costs for specific tools (e.g., search_tool costs $10, get_weather costs $5).
• Dynamic costs: For use cases where you want to set costs based on the MCP's response, you can write a custom post mcp call hook to parse responses and set costs dynamically.

Get started

Model Hub v2​

v1.74.3-stable introduces a new OSS Model Hub for telling developers what models are available on the proxy.

This is great for Proxy Admins as you can now tell developers what models are available on the proxy.

This improves on the previous model hub by enabling:

• The ability to show Developers models, even if they don't have a LiteLLM key.
• The ability for Proxy Admins to select specific models to be public on the model hub.
• Improved search and filtering capabilities:
  • search for models by partial name (e.g. xai grok-4)
  • filter by provider and feature (e.g. 'vision' models)
  • sort by cost (e.g. cheapest vision model from OpenAI)

Get started

New Models / Updated Models​

Pricing / Context Window Updates​

Features​

• Xinference
  • Image generation API support - PR
• Bedrock
  • API Key Auth support for AWS Bedrock API - PR
• 🆕 Dashscope
  • New integration from Alibaba (enables qwen usage) - PR
• 🆕 Bytez
  • New /chat/completion integration - PR

Bugs​

• Github Copilot
  • Fix API base url for Github Copilot - PR
• Bedrock
  • Ensure supported bedrock/converse/ params = bedrock/ params - PR
  • Fix cache token cost calculation - PR
• XAI
  • ensure finish_reason includes tool calls when xai responses with tool calls - PR

LLM API Endpoints​

Features​

• /completions
  • Return ‘reasoning_content’ on streaming - PR
• /chat/completions
  • Add 'thinking blocks' to stream chunk builder - PR
• /v1/messages
  • Fallbacks support - PR
  • tool call handling for non-anthropic models (/v1/messages to /chat/completion bridge) - PR

MCP Gateway​

Features​

• Cost Tracking
  • Add Cost Tracking - PR
  • Add usage tracking - PR
  • Add custom cost configuration for each MCP tool - PR
  • Add support for editing MCP cost per tool - PR
  • Allow using custom post call MCP hook for cost tracking - PR
• Auth
  • Allow customizing what client side auth header to use - PR
  • Raises error when MCP server header is malformed in the request - PR
• MCP Server
  • Allow using stdio MCPs with LiteLLM (enables using Circle CI MCP w/ LiteLLM) - PR, Get Started

Bugs​

• General
  • Fix task group is not initialized error - PR s/o @juancarlosm
• MCP Server
  • Fix mcp tool separator to work with Claude code - PR, Get Started
  • Add validation to mcp server name to not allow "-" (enables namespaces to work) - PR

Management Endpoints / UI​

Features​

• Model Hub
  • new model hub table view - PR
  • new /public/model_hub endpoint - PR
  • Make Model Hub OSS - PR
  • New ‘make public’ modal flow for showing proxy models on public model hub - PR
• MCP
  • support for internal users to use and manage MCP servers - PR
  • Adds UI support to add MCP access groups (similar to namespaces) - PR
  • MCP Tool Testing Playground - PR
  • Show cost config on root of MCP settings - PR
• Test Key
  • Stick sessions - PR
  • MCP Access Groups - allow mcp access groups - PR
• Usage
  • Truncate long labels and improve tooltip in Top API Keys chart - PR
  • Improve Chart Readability for Tag Usage - PR
• Teams
  • Prevent navigation reset after team member operations - PR
  • Team Members - reset budget, if duration set - PR
  • Use central team member budget when max_budget_in_team set on UI - PR
• SSO
  • Allow users to run a custom sso login handler - PR
• Navbar
  • improve user dropdown UI with premium badge and cleaner layout - PR
• General
  • Consistent layout for Create and Back buttons on all the pages - PR
  • Align Show Password with Checkbox - PR
  • Prevent writing default user setting updates to yaml (causes error in non-root env) - PR