Skip to main content

v1.72.0-stable

Krrish Dholakia
Ishaan Jaffer
info

The release candidate is live now.

The production release will be live on Wednesday.

Deploy this version​

docker run litellm
docker run
-e STORE_MODEL_IN_DB=True
-p 4000:4000
ghcr.io/berriai/litellm:main-v1.72.0.rc

Key Highlights​

LiteLLM v1.72.0-stable.rc is live now. Here are the key highlights of this release:

  • Vector Store Permissions: Control Vector Store access at the Key, Team, and Organization level.
  • Rate Limiting Sliding Window support: Improved accuracy for Key/Team/User rate limits with request tracking across minutes.
  • Aiohttp Transport used by default: Aiohttp transport is now the default transport for LiteLLM networking requests. This gives users 2x higher RPS per instance with a 40ms median latency overhead.
  • Bedrock Agents: Call Bedrock Agents with /chat/completions, /response endpoints.
  • Anthropic File API: Upload and analyze CSV files with Claude-4 on Anthropic via LiteLLM.
  • Prometheus: End users (end_user) will no longer be tracked by default on Prometheus. Tracking end_users on prometheus is now opt-in. This is done to prevent the response from /metrics from becoming too large. Read More

Vector Store Permissions​

This release brings support for managing permissions for vector stores by Keys, Teams, Organizations (entities) on LiteLLM. When a request attempts to query a vector store, LiteLLM will block it if the requesting entity lacks the proper permissions.

This is great for use cases that require access to restricted data that you don't want everyone to use.

Over the next week we plan on adding permission management for MCP Servers.

Aiohttp Transport used by default​

Aiohttp transport is now the default transport for LiteLLM networking requests. This gives users 2x higher RPS per instance with a 40ms median latency overhead. This has been live on LiteLLM Cloud for a week + gone through alpha users testing for a week.

If you encounter any issues, you can disable using the aiohttp transport in the following ways:

On LiteLLM Proxy

Set the DISABLE_AIOHTTP_TRANSPORT=True in the environment variables.

Environment Variable
export DISABLE_AIOHTTP_TRANSPORT="True"

On LiteLLM Python SDK

Set the disable_aiohttp_transport=True to disable aiohttp transport.

Python SDK
import litellm

litellm.disable_aiohttp_transport = True # default is False, enable this to disable aiohttp transport
result = litellm.completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello, world!"}],
)
print(result)

New Models / Updated Models​

  • Bedrock
    • Video support for Bedrock Converse - PR
    • InvokeAgents support as /chat/completions route - PR, Get Started
    • AI21 Jamba models compatibility fixes - PR
    • Fixed duplicate maxTokens parameter for Claude with thinking - PR
  • Gemini (Google AI Studio + Vertex AI)
    • Parallel tool calling support with parallel_tool_calls parameter - PR
    • All Gemini models now support parallel function calling - PR
  • VertexAI
    • codeExecution tool support and anyOf handling - PR
    • Vertex AI Anthropic support on /v1/messages - PR
    • Thinking, global regions, and parallel tool calling improvements - PR
    • Web Search Support PR
  • Anthropic
    • Thinking blocks on streaming support - PR
    • Files API with form-data support on passthrough - PR
    • File ID support on /chat/completion - PR
  • xAI
    • Web Search Support PR
  • Google AI Studio
    • Web Search Support PR
  • Mistral
    • Updated mistral-medium prices and context sizes - PR
  • Ollama
    • Tool calls parsing on streaming - PR
  • Cohere
    • Swapped Cohere and Cohere Chat provider positioning - PR
  • Nebius AI Studio
    • New provider integration - PR

LLM API Endpoints​

Management Endpoints / UI​

  • Vector Stores
    • Permission management for LiteLLM Keys, Teams, and Organizations - PR
    • UI display of vector store permissions - PR
    • Vector store access controls enforcement - PR
    • Object permissions fixes and QA improvements - PR
  • Teams
    • "All proxy models" display when no models selected - PR
    • Removed redundant teamInfo call, using existing teamsList - PR
    • Improved model tags display on Keys, Teams and Org pages - PR
  • SSO/SCIM
    • Bug fixes for showing SCIM token on UI - PR
  • General UI
    • Fix "UI Session Expired. Logging out" - PR
    • Support for forwarding /sso/key/generate to server root path URL - PR

Logging / Guardrails Integrations​

Logging​

  • Prometheus
    • End users will no longer be tracked by default on Prometheus. Tracking end_users on prometheus is now opt-in. PR
  • Langfuse
    • Performance improvements: Fixed "Max langfuse clients reached" issue - PR
  • Helicone
    • Base URL support - PR
  • Sentry
    • Added sentry sample rate configuration - PR

Guardrails​

Performance / Reliability Improvements​

  • aiohttp Transport
    • Handling for aiohttp.ClientPayloadError - PR
    • SSL verification settings support - PR
    • Rollback to httpx==0.27.0 for stability - PR
  • Request Limiting
    • Sliding window logic for parallel request limiter v2 - PR

Bug Fixes​

  • LLM API Fixes
    • Added missing request_kwargs to get_available_deployment call - PR
    • Fixed calling Azure O-series models - PR
    • Support for dropping non-OpenAI params via additional_drop_params - PR
    • Fixed frequency_penalty to repeat_penalty parameter mapping - PR
    • Fix for embedding cache hits on string input - PR
  • General
    • OIDC provider improvements and audience bug fix - PR
    • Removed AzureCredentialType restriction on AZURE_CREDENTIAL - PR
    • Prevention of sensitive key leakage to Langfuse - PR
    • Fixed healthcheck test using curl when curl not in image - PR

New Contributors​

Demo Instance​

Here's a Demo Instance to test changes:

Git Diff​