One post tagged with "thinking content"

These are the changes since v1.63.2-stable.

This release is primarily focused on:

• [Beta] Responses API Support
• Snowflake Cortex Support, Amazon Nova Image Generation
• UI - Credential Management, re-use credentials when adding new models
• UI - Test Connection to LLM Provider before adding a model

Docker Run LiteLLM Proxy​

docker run
-e STORE_MODEL_IN_DB=True
-p 4000:4000
ghcr.io/berriai/litellm:main-v1.63.11-stable

Demo Instance​

Here's a Demo Instance to test changes:

• Instance: https://demo.litellm.ai/
• Login Credentials:
  • Username: admin
  • Password: sk-1234

New Models / Updated Models​

• Image Generation support for Amazon Nova Canvas Getting Started
• Add pricing for Jamba new models PR
• Add pricing for Amazon EU models PR
• Add Bedrock Deepseek R1 model pricing PR
• Update Gemini pricing: Gemma 3, Flash 2 thinking update, LearnLM PR
• Mark Cohere Embedding 3 models as Multimodal PR
• Add Azure Data Zone pricing PR
  • LiteLLM Tracks cost for azure/eu and azure/us models

LLM Translation​

1. New Endpoints

• [Beta] POST /responses API. Getting Started

2. New LLM Providers

• Snowflake Cortex Getting Started

3. New LLM Features

• Support OpenRouter reasoning_content on streaming Getting Started

4. Bug Fixes

• OpenAI: Return code, param and type on bad request error More information on litellm exceptions
• Bedrock: Fix converse chunk parsing to only return empty dict on tool use PR
• Bedrock: Support extra_headers PR
• Azure: Fix Function Calling Bug & Update Default API Version to 2025-02-01-preview PR
• Azure: Fix AI services URL PR
• Vertex AI: Handle HTTP 201 status code in response PR
• Perplexity: Fix incorrect streaming response PR
• Triton: Fix streaming completions bug PR
• Deepgram: Support bytes.IO when handling audio files for transcription PR
• Ollama: Fix "system" role has become unacceptable PR
• All Providers (Streaming): Fix String data: stripped from entire content in streamed responses PR

Spend Tracking Improvements​

1. Support Bedrock converse cache token tracking Getting Started
2. Cost Tracking for Responses API Getting Started
3. Fix Azure Whisper cost tracking Getting Started

UI​

Re-Use Credentials on UI​

You can now onboard LLM provider credentials on LiteLLM UI. Once these credentials are added you can re-use them when adding new models Getting Started

Test Connections before adding models​

Before adding a model you can test the connection to the LLM provider to verify you have setup your API Base + API Key correctly

General UI Improvements​

1. Add Models Page
   • Allow adding Cerebras, Sambanova, Perplexity, Fireworks, Openrouter, TogetherAI Models, Text-Completion OpenAI on Admin UI
   • Allow adding EU OpenAI models
   • Fix: Instantly show edit + deletes to models
2. Keys Page
   • Fix: Instantly show newly created keys on Admin UI (don't require refresh)
   • Fix: Allow clicking into Top Keys when showing users Top API Key
   • Fix: Allow Filter Keys by Team Alias, Key Alias and Org
   • UI Improvements: Show 100 Keys Per Page, Use full height, increase width of key alias
3. Users Page
   • Fix: Show correct count of internal user keys on Users Page
   • Fix: Metadata not updating in Team UI
4. Logs Page
   • UI Improvements: Keep expanded log in focus on LiteLLM UI
   • UI Improvements: Minor improvements to logs page
   • Fix: Allow internal user to query their own logs
   • Allow switching off storing Error Logs in DB Getting Started
5. Sign In/Sign Out
   • Fix: Correctly use PROXY_LOGOUT_URL when set Getting Started

Security​

1. Support for Rotating Master Keys Getting Started
2. Fix: Internal User Viewer Permissions, don't allow internal_user_viewer role to see Test Key Page or Create Key Button More information on role based access controls
3. Emit audit logs on All user + model Create/Update/Delete endpoints Getting Started
4. JWT
   • Support multiple JWT OIDC providers Getting Started
   • Fix JWT access with Groups not working when team is assigned All Proxy Models access
5. Using K/V pairs in 1 AWS Secret Getting Started

Logging Integrations​

1. Prometheus: Track Azure LLM API latency metric Getting Started
2. Athina: Added tags, user_feedback and model_options to additional_keys which can be sent to Athina Getting Started

Performance / Reliability improvements​

1. Redis + litellm router - Fix Redis cluster mode for litellm router PR

General Improvements​

1. OpenWebUI Integration - display thinking tokens

• Guide on getting started with LiteLLM x OpenWebUI. Getting Started
• Display thinking tokens on OpenWebUI (Bedrock, Anthropic, Deepseek) Getting Started

Complete Git Diff​

Here's the complete git diff