Skip to main content

v1.99.0rc1 - Dark Mode, CLI OAuth Login & Batch Billing

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:1.99.0-rc.1
Breaking Changes

Audit logging now defaults to on for proxies running an Enterprise license. store_audit_logs becomes tri-state: unset plus a premium license resolves to enabled, replacing roughly fifteen scattered gate checks with one resolver, and LITELLM_STORE_AUDIT_LOGS is now honoured by the key hooks that previously ignored it. An enterprise proxy that never set the flag starts writing audit rows on upgrade, which will grow the audit table. Set store_audit_logs: false explicitly to keep the prior behavior. See PR #37518.

Shadow eval jobs are budgeted in dollars instead of turns. POST /auto_router/shadow_eval/start now takes max_budget in USD per key; a request that still sends max_turns gets a 422 naming the replacement. The sampler, sweep, stop guard, and derived status all gate on the billed response_cost of the shadow arm and the judge, and admission re-checks the proxy's cross-pod spend counter. Jobs created before the migration keep their configured turn budget, since max_budget stays NULL on those rows. See PR #37555.

A shadow eval job now covers a list of keys rather than one. api_key_ids replaces api_key_id on the start payload, results come back pooled and per key, and sibling rows are tied together by a new group_id column. A caller still sending the singular api_key_id must move to the list form. See PR #37251.

Key Highlights​

  • The Admin UI's migration off antd and Tremor is complete - @tremor/react, antd, and @ant-design/icons are all removed from the dashboard's dependencies, with the last components, the shared primitives, and every form ported onto shadcn (base-vega) and react-hook-form. 115 UI pull requests land in this window, and the dashboard is now on React 19
  • Dark mode ships - a light/dark/system toggle in the top bar, semantic status tokens for success, warning and info, a dark variant of the LiteLLM logo, and an admin-supplied dark variant of a custom logo. Hardcoded Tailwind palette classes across the dashboard are mapped onto theme tokens so surfaces, form controls, inline styles and code blocks all follow the theme
  • lite login is a real OAuth flow - the CLI now authenticates with an authorization code plus PKCE against the proxy, stores the credential and the refresh token in the OS keychain rather than a token.json on disk, and lite login --config-claude wires Claude Code up at login
  • Batch spend is accounted for end to end - enqueued-token rate limiting admits batches against a token budget and refunds on completion or cancellation, cost rows are claimed atomically so multi-pod polling cannot double-bill, cancelled and failed batches that still produced output are billed, a single undecodable output line no longer zeroes a batch's spend, and Bedrock batches can be cancelled through POST /v1/batches/{id}/cancel
  • Provisioned throughput can be declared in config.yaml - a PTU reservation no longer has to be created through the API; the rollup accrues flat cost for config-declared deployments, refuses an incomplete reservation the way the endpoints do, requires an operator-declared id, and warns when a config declares PTU while attribution is switched off
  • The complexity router is operator-configurable - operator-defined tier sets for the LLM classifier, custom classifier plugins via classifier_type: custom, a plan-mode tier floor for coding-agent clients, a business classification rubric preset, and per-model reasoning effort in the tier editor

New Providers and Endpoints​

New Providers (5 new providers)​

ProviderSupported LiteLLM EndpointsDescription
SCX.ai/chat/completionsJSON-configured OpenAI-compatible provider, registered in the provider enum, config map, pricing map and dashboard, shipping GLM-5.2 and Qwen3.8-Max
Cognition/chat/completionsCognition gets its own provider identity rather than riding an OpenAI-compatible alias, with swe-1.6, swe-1.7 and swe-1.7-lightning priced
AWS Bedrock AgentCore/searchBedrock AgentCore registered as a search provider
Amazon Comprehend Medical/* passthroughPassthrough provider for Comprehend Medical, routed and credentialed through the proxy
ValkeyVector storesValkey as a managed vector store provider alongside the existing backends

New LLM API Endpoints (4 new endpoints)​

EndpointMethodDescriptionDocumentation
/model/deprecationsGETLists deployments whose models carry a provider-announced deprecation_date, backing proactive deprecation alertsProxy Config
/team/daily/activity/aggregatedGETPre-aggregated team activity for the Usage tab, replacing the per-row scan it used to do client-sideCost Tracking
/team/{team_id}/callback/{callback_name}DELETERemoves a single team-scoped logging callback without rewriting the whole callback setTeam Logging
/auto_router/validate_complexity_router_configPOSTDry-runs a complexity-router config against the write gate so an invalid tier set is rejected before it is savedAuto Router

New Models / Updated Models​

New Model Support (136 new models)​

ProviderModelContext WindowInput ($/1M tokens)Output ($/1M tokens)Features
OpenAIgpt-5.6-cyber400K$12.50$75.00Reasoning, vision, PDF input, function calling, parallel function calling, tool choice, prompt caching, response schema, web search, computer use, native streaming
OpenAIdaybreak-red-latest400K$12.50$75.00Reasoning, vision, PDF input, function calling, parallel function calling, tool choice, prompt caching, response schema, web search, computer use, native streaming
OpenAIdaybreak-blue-latest1M$4.00$20.00Reasoning, vision, PDF input, function calling, parallel function calling, tool choice, prompt caching, response schema, web search, computer use, native streaming
OpenAIchat-latest400K$5.00$30.00Vision, PDF input, function calling, parallel function calling, tool choice, prompt caching, response schema, web search, native streaming
Azureazure/gpt-audio-mini128K$0.60$2.40Function calling, parallel function calling, tool choice, native streaming
Azureazure/gpt-realtime-mini32K$0.60$2.40Realtime, audio input and output, function calling, parallel function calling, tool choice, $0.0000008 per image
Amazon Bedrockus.openai.gpt-5.6-sol1M$5.50$33.00Vision, function calling, tool choice
Amazon Bedrockglobal.openai.gpt-5.6-sol1M$5.00$30.00Vision, function calling, tool choice
Amazon Bedrockus.openai.gpt-5.6-terra1M$2.20$13.20Vision, function calling, tool choice
Amazon Bedrockglobal.openai.gpt-5.6-terra1M$2.00$12.00Vision, function calling, tool choice
Amazon Bedrockus.openai.gpt-5.6-luna1M$0.22$1.32Vision, function calling, tool choice
Amazon Bedrockglobal.openai.gpt-5.6-luna1M$0.20$1.20Vision, function calling, tool choice
Amazon Bedrockus.xai.grok-4.6500K$2.20$6.60Reasoning, vision, function calling, tool choice, prompt caching
Amazon Bedrockglobal.xai.grok-4.6500K$2.00$6.00Reasoning, vision, function calling, tool choice, prompt caching
Amazon Bedrockbedrock_mantle/xai.grok-4.6500K$2.20$6.60Reasoning, vision, function calling, tool choice, prompt caching, response schema
Amazon Bedrockbedrock/guardrails---Guardrail usage and cost accounting
AWS Bedrock AgentCoreagentcore/search---Search
Databricksdatabricks/databricks-claude-opus-4-61M$5.00$25.00Reasoning, function calling, tool choice
Databricksdatabricks/databricks-claude-sonnet-4-61M$3.00$15.00Reasoning, function calling, tool choice
Databricksdatabricks/databricks-gemini-3-pro1M$2.50$15.00Function calling, tool choice
Databricksdatabricks/databricks-gemini-3-1-pro1M$2.50$15.00Function calling, tool choice
Databricksdatabricks/databricks-gemini-3-flash1M$0.625$3.75Function calling, tool choice
Databricksdatabricks/databricks-gemini-3-1-flash-lite1M$0.3125$1.88Function calling, tool choice
Databricksdatabricks/databricks-gpt-5-4272K$2.50$15.00Chat
Databricksdatabricks/databricks-gpt-5-4-mini272K$0.75$4.50Chat
Databricksdatabricks/databricks-gpt-5-4-nano272K$0.20$1.25Chat
Databricksdatabricks/databricks-gpt-5-2272K$1.75$14.00Chat
Databricksdatabricks/databricks-gpt-5-2-codex272K$1.75$14.00Chat
Databricksdatabricks/databricks-gpt-5-3-codex272K$1.75$14.00Chat
Databricksdatabricks/databricks-gpt-5-1-codex-max272K$1.25$10.00Chat
Databricksdatabricks/databricks-gpt-5-1-codex-mini272K$0.25$2.00Chat
Google Geminigemini/gemini-3.1-flash-lite-image65K$0.25$1.50Image generation, vision, function calling, $0.00028 per image
Google Geminigemini/gemini-3.5-live-translate-preview-$3.50$21.00Audio input and output
Google Vertex AIvertex_ai/gemini-3.1-flash-lite-image65K$0.25$1.50Image generation, vision, PDF input, video input, prompt caching, $0.00028 per image
Google Vertex AIgemini-3.1-flash-lite-image65K$0.25$1.50Image generation, vision, PDF input, video input, prompt caching, $0.00028 per image
Fireworks AIfireworks_ai/kimi-k31M$3.00$15.00Reasoning, vision, function calling, tool choice, response schema
Fireworks AIfireworks_ai/kimi-k3-fast1M$4.50$22.50Reasoning, vision, function calling, tool choice, response schema
Fireworks AIfireworks_ai/kimi-k3-us1M$3.30$16.50Reasoning, vision, function calling, tool choice, response schema
Fireworks AIfireworks_ai/glm-5p2-fast1M$2.10$6.60Reasoning, function calling, tool choice, response schema
Fireworks AIfireworks_ai/glm-5p2-fast-us1M$2.10$6.60Reasoning, function calling, tool choice, response schema
Fireworks AIfireworks_ai/deepseek-v4-flash-07311M$0.14$0.28Reasoning, function calling, tool choice, response schema
Fireworks AIfireworks_ai/qwen3p8-max262K$2.00$6.00Reasoning, vision, function calling, tool choice, response schema
Fireworks AIfireworks_ai/muse-glimmer-30b131K$0.35$1.50Reasoning, vision, function calling, tool choice, response schema
Fireworks AIfireworks_ai/nemotron-3-ultra-nvfp4262K$0.60$2.40Reasoning, function calling, tool choice, response schema
Fireworks AIfireworks_ai/nemotron-lightning-3p5-30b-a3b262K$0.05$0.20Reasoning, function calling, tool choice, response schema
Mistralmistral/glm-5-21M$1.40$4.40Reasoning, function calling, tool choice, prompt caching, response schema
Mistralmistral/zai-glm-5-21M$1.40$4.40Reasoning, function calling, tool choice, prompt caching, response schema
Mistralmistral/mistral-ocr-4-1---OCR
Moonshotmoonshot/kimi-k31M$3.00$15.00Reasoning, vision, video input, function calling, tool choice, response schema
OpenRouteropenrouter/anthropic/claude-opus-51M$5.00$25.00Reasoning, vision, PDF input, function calling, tool choice, prompt caching, response schema, computer use
OpenRouteropenrouter/deepseek/deepseek-v4-pro1M$1.32$3.96Reasoning, function calling, tool choice, prompt caching, response schema
OpenRouteropenrouter/deepseek/deepseek-v4-pro-08131M$1.32$3.96Reasoning, function calling, tool choice, prompt caching, response schema
Perplexityperplexity/perplexity/kimi-k3-$3.00$15.00Responses, reasoning, function calling, web search
Perplexityperplexity/perplexity/glm-5.2-$1.40$4.40Responses, reasoning, function calling, web search
Perplexityperplexity/perplexity/kimi-k2.7-code-$0.95$4.00Responses, function calling, web search
Perplexityperplexity/perplexity/deepseek-v4-flash-0731-$0.13$0.26Responses, reasoning, function calling, web search
Perplexityperplexity/pplx-embed-context-v1-4b32K$0.05-Embedding
Perplexityperplexity/pplx-embed-context-v1-0.6b32K$0.008-Embedding
Voyagevoyage/voyage-4-large32K$0.12-Embedding
Voyagevoyage/voyage-432K$0.06-Embedding
Voyagevoyage/voyage-4-lite32K$0.02-Embedding
Voyagevoyage/voyage-code-432K$0.12-Embedding
Voyagevoyage/voyage-context-4120K$0.12-Embedding
Voyagevoyage/voyage-multimodal-3.532K$0.12-Embedding, image embedding input
SCX.aiscx-ai/GLM-5.21M$0.61$1.98Reasoning, function calling, tool choice, prompt caching, response schema
SCX.aiscx-ai/Qwen3.8-Max1M$1.65$4.99Reasoning, vision, function calling, tool choice, prompt caching, response schema
Cognitioncognition/swe-1.6-$0.50$2.50Function calling, prompt caching
Cognitioncognition/swe-1.7-$0.50$2.50Function calling, prompt caching
Cognitioncognition/swe-1.7-lightning-$2.50$12.50Function calling, prompt caching
fal.aifal_ai/gpt-image-2, fal_ai/openai/gpt-image-2---Image generation and edit, priced per size and quality from request params

The fal.ai entry expands to 57 registry keys: the two base slugs plus every low/medium/high quality crossed with six output resolutions, for both generation and /edit. The Fireworks entries each carry an accounts/fireworks/models/ alias, and the two glm-5p2-fast and two kimi-k3 fast/US variants also carry an accounts/fireworks/routers/ alias, so the slug you already use keeps resolving.

Beyond the new entries, this release is another large cost-map maintenance pass over 288 existing entries. 219 entries gain or correct a provider-announced deprecation_date, 25 gain Batch API input and output rates alongside a regional_endpoint_uplift_multiplier, and 15 gain a prompt_cache_min_tokens floor, including 4096 on Gemini 3.5, 3.6 and 3.7 Flash and 3.1 Pro Preview. Pricing moves in both directions: Gemini 3.6 Flash halves to $0.75 / $3.75 per 1M on both Gemini and Vertex, gpt-5.6 and gpt-5.6-sol drop to $4.00 / $20.00 on a promotional cut, mistral/codestral-latest falls to $0.30 / $0.90 and Vertex DeepSeek V3.1 MaaS to $0.60 / $1.70, while gemini-3.1-flash-image doubles to $0.50 / $3.00 and mistral/mistral-small-latest rises to $0.15 / $0.60. The GPT-5.6 family's max input tokens is corrected from 1.05M to 922K across the Azure, US and EU entries. Flex and priority service-tier rates land on 8 and 5 entries respectively, thinking_always_on on 11, and supports_prompt_cache_breakpoint on 4. No pricing entries were removed.

Features​

  • Amazon Bedrock
    • Serve GPT-5.6 cross-region inference profiles on Bedrock runtime - PR #37821
    • Forward LiteLLM identity and metadata into Bedrock requestMetadata - PR #36861
    • Add a config toggle to disable agent-runtime pass-through - PR #37386
    • Day-0 pricing for Grok 4.6 on Bedrock - PR #37517
  • Cognition
    • Give Cognition its own provider identity, then price swe-1.7 at the standard tier and add swe-1.7-lightning - PR #37743, PR #37763
  • SCX.ai
    • Add SCX.ai as a JSON-configured OpenAI-compatible provider - PR #34752
  • Mistral
    • Add zai-glm-5-2 and glm-5-2 pricing - PR #37110
  • Perplexity
    • Add the Agent API third-party models - PR #37112
  • Databricks
    • Add cost map entries for 14 newer Databricks models - PR #28501
  • Moonshot
  • Anthropic
    • Map cache_control_injection_points to the OpenAI prompt_cache_breakpoint on GPT-5.6+ targets - PR #37628
    • Map metadata.user_id to prompt_cache_key on the /v1/messages bridge - PR #37623
  • General
    • Route /chat/completions through the Rust core for Anthropic and Bedrock - PR #37241
    • Surface TinyFish response headers and top-level response extras - PR #32448
    • Authenticate to Azure Postgres with Microsoft Entra ID tokens - PR #37663

Bug Fixes​

  • Amazon Bedrock
    • Preserve cache token usage when invocationMetrics replaces the usage block, and report provider thinking tokens instead of classifying them as text - PR #36878, PR #35998
    • Forward provider response headers on chat completions, degrade gracefully on malformed tool-call arguments, and validate file-content retrieval against the configured output bucket - PR #37003, PR #33842, PR #31435
    • Read batch usage by payload shape rather than provider name, and report uploaded size in the FileObject returned by managed batch uploads - PR #37078, PR #36392
  • Anthropic
    • Emit tool_use content_block_start without awaiting the next chunk, and resolve the provider exactly once on /v1/messages - PR #37310, PR #37757
    • Fold guardrail-modified leading system rows into the top-level system param - PR #37231
    • Stop emitting empty thinking blocks on the Responses adapter, and preserve optional Responses tool properties - PR #36033, PR #36979
    • Log partial stream spend when a /v1/messages client disconnects mid-stream - PR #37558
    • Gate sampling params on /v1/messages the way /chat/completions does, omit thinking.type=disabled for always-on thinking models, and accept a bool thinking param instead of raising AttributeError - PR #37868, PR #37510, PR #37423
  • Google Vertex AI
    • Apply the regional endpoint uplift to cost tracking, resolving the served location from the passthrough URL, optional_params, and the deployment for native /v1/messages calls - PR #37543
    • Only fall back to a placeholder thought signature on the first parallel function call - PR #37541
    • Convert messages to contents in Gemini count_tokens - PR #36981
  • Google Gemini
    • Price Gemini 3.6 Flash at Google's introductory rates on every service tier, and correct gemini-3.1-flash-lite-image capabilities while deduping its entries - PR #37197, PR #36849
  • Azure
    • Rename max_tokens to max_completion_tokens for gpt-5-chat deployments - PR #36857
    • Strip non-OpenAI-spec message fields before the request on Azure AI - PR #34445
  • Fireworks AI
    • Skip the accounts/ rewrite for FW-* Foundry deployment ids - PR #37242
  • SageMaker
    • Send the inference component header and honor hf_model_name - PR #37766
  • General
    • Stop forwarding a client Anthropic OAuth token to Bedrock and Vertex - PR #37905
    • Stop the Rust flag and the client_side_timeout marker from leaking into upstream provider request bodies - PR #37218, PR #37346
    • Resolve the advisor sub-call through the proxy router - PR #36246
    • Surface provider errors from the container file content endpoint - PR #37737

LLM API Endpoints​

Features​

  • Batches
    • Enqueued-token rate limiting for batches, with a refund on completion and on cancellation - PR #37539
    • Support AWS Bedrock batch cancellation via StopModelInvocationJob - PR #34087
    • Redact or drop individual batch records instead of rejecting the whole file - PR #37561
  • OCR
    • Return Azure Document Intelligence's native payload from /v1/ocr via req_format=native - PR #37194
  • Vector Stores
    • Add Valkey as a managed vector store provider - PR #37002
  • Image Generation
    • Add fal.ai gpt-image-2 image generation and edit support - PR #37729
  • Pass-through Endpoints
    • Add the Amazon Comprehend Medical passthrough provider - PR #37229

Bugs​

  • Responses API
    • Preserve reasoning input items and signed thinking blocks on the bridge - PR #36355
    • Mint Responses API item IDs in the completion bridge - PR #37946
    • Map incomplete responses to finish_reason: length instead of a 500, and map Bedrock Mantle context overflow to ContextWindowExceededError - PR #37710, PR #37862
    • Unwrap object-form tool_choice before calling the Responses API, strip the responses/ routing prefix on the path, and preserve Bedrock Mantle validation errors - PR #36032, PR #37345, PR #36580
  • Batches
    • Stop one bad output line from zeroing an entire batch's spend, and decode a model-encoded output file id so completed batches book spend - PR #37457, PR #37573
    • Bill cancelled and failed batches that still produced an output file - PR #37205
    • Price a retrieved batch from its deployment's model and rates - PR #37219
    • Don't crash logging when a completed batch has no output file, and don't retire a completed batch from cost recovery while output_file_id is lagging - PR #34067, PR #37715
    • Return OpenAI-parity errors: 400 for an out-of-range limit on GET /v1/batches, 400 naming the missing required param on POST /v1/batches, and 404 rather than 500 for unresolvable batch and file ids - PR #37198, PR #37199, PR #37201
    • Read batch records the same way the upload validation does - PR #37776
  • Files
    • List and page unscoped managed files locally - PR #37855
  • Pass-through Endpoints
    • Register WebSocket passthrough for OpenAI prefixes, and forward the Bedrock event-stream content-type on unbuffered passthrough - PR #36151, PR #33767
    • Resolve Vertex live credentials from DB model deployments, and bound Vertex credential resolution so realtime failures are loud - PR #37602, PR #37604
  • General
    • End the turn when the agentic web-search loop hits its ceiling - PR #37911
    • Return SSE (text/event-stream) for A2A message/stream instead of NDJSON, and accept the whole JSON-RPC id union the spec defines - PR #35037, PR #37704
    • Stop leaking stored credentials in direct vector-store search debug logs - PR #37373
    • Price fal.ai gpt-image-2 per size and quality from the request params - PR #37751
    • Return 400 for non-object metadata and litellm_metadata instead of a silent drop or a 500 - PR #37203

Management Endpoints / UI​

Features​

  • Design system
    • Complete the move off Ant Design and Tremor: the last components, shared primitives and common components migrate onto shadcn, @tremor/react and antd are dropped from the dependency tree, and @ant-design/icons is swapped for lucide-react - PR #37569, PR #37574, PR #37394, PR #37521, PR #37553
    • Move every form onto react-hook-form and shadcn, covering virtual keys, add model, teams, users, policies, margins, agents, tags, memory, credentials, auto routers, vector stores, pass-through, projects, access groups, MCP, SSO, SCIM, alerting, fallbacks and the login and onboarding flows - PR #37442, PR #37446, PR #37417, PR #37305, PR #37357, PR #37266, PR #37304, PR #37353, PR #37354, PR #37334, PR #37315
    • Upgrade the dashboard to React 19 - PR #37411
    • Move dashboard toasts from antd message/notification onto sonner, and codemod every call site onto lib/toast - PR #37207, PR #37253
  • Dark mode
    • Add a light/dark/system theme toggle to the top bar, marked beta in the theme menu - PR #37669, PR #37680
    • Add success, warning and info status tokens, and map hardcoded Tailwind palette classes onto semantic tokens - PR #37393, PR #37576
    • Serve a dark-mode variant of the LiteLLM logo, and let admins supply a dark-mode variant of their custom logo - PR #37656, PR #37662
  • Auto Router
    • Multi-key shadow eval picker with a per-key breakdown, and the shadowed key named in job responses and the UI headline - PR #37389, PR #37221
    • Configure the heuristic scorer from the Admin UI, add a Lite mixed-provider preset, and set a plan-mode override tier in the create and edit forms - PR #37216, PR #37068, PR #37319
    • Per-model reasoning effort in the complexity tier editor - PR #37673
  • Virtual Keys
    • Link the key info header to its user, creator, team and organization - PR #37187
    • Add a per-key Savings tab to the key detail page - PR #37693
  • Usage and Logs
    • Add a user ID request-log filter, and a searchable per-user usage filter on the Usage page - PR #36781, PR #36790, PR #37206
    • Add /team/daily/activity/aggregated and switch the Usage team tab to it - PR #36562
  • Teams and Projects
    • Edit project input and output TPM limits from the Projects modal - PR #37676
    • Standardize the Teams page header, and decouple bulk invite from the invite user button - PR #36897, PR #37061
  • Auth and Management
    • DELETE /team/{team_id}/callback/{callback_name} for removing a single team callback - PR #37331
    • Source generic OIDC user claims from the ID or access token when UserInfo is incomplete - PR #37696
    • Proactive model deprecation alerts and a /model/deprecations endpoint - PR #26900
    • Bound the health-check table with maximum_health_check_retention_period - PR #37681
  • CLI
    • Native CLI login with OAuth authorization code plus PKCE - PR #37626
    • Store the lite login credential and the --pkce refresh token in the OS keychain rather than token.json - PR #37566, PR #37665
    • lite login --config-claude wires Claude Code at login - PR #37507

Bugs​

  • Dark mode and theming
    • Make dark-mode form controls visible, give status colours a readable foreground, make hardcoded palette surfaces theme-aware, make inline styles and code blocks follow the theme, and move the policy flow builder onto theme tokens - PR #37648, PR #37649, PR #37650, PR #37651, PR #37654
    • Restore hover feedback and dark-mode variants lost in the token migration, and keep semantic button colours on hover - PR #37579, PR #37580
  • Forms
    • Rebuild nested and list paths in the mounted-field projection, add mounted-field projections for the MCP server form graph, and render optional array and object MCP tool parameters as JSON inputs - PR #37450, PR #37440, PR #37548
    • Show select labels on the trigger instead of raw values, restore the cache control Role and Index field hints, and gate the pass-through guardrail field inputs when the section is disabled - PR #37372, PR #37437, PR #37435
    • Clear pass-through header rows when the create modal is reopened, and stop the Add Model mapping table from looping the page - PR #37549, PR #37741
    • Highlight the first member search match so Enter picks it - PR #37429
    • Restore tab strip styling and panel persistence lost in the shadcn migration - PR #37403
    • Keep completion-mode models in the playground chat dropdown - PR #37954
  • Cost Optimization
    • Draw one Per Day savings bar per date, surface the paginated fallback, and drive auto-router usage from the shared time picker - PR #37643, PR #37659, PR #37871
    • Keep keyword tier rules that target operator-defined tiers when hydrating the edit modal, and pin a default model on the complexity router - PR #37413, PR #36615
    • Stop pairing key spend with the team budget when a key has no budget - PR #37196
  • Auth and Management
    • Make /team/member_delete's four cleanups atomic, and populate team member emails missing from the roster snapshot - PR #37959, PR #37759
    • Stop the team fallback from widening model access, and resolve team object_permission independently in the unresolvable-team fallback - PR #37962, PR #37960
    • Resolve bare model names against wildcard deployments in model access groups, and accept inherited model sentinels in project key limits - PR #37492, PR #37515
    • SCIM: match group members by SSO identity or email before creating a placeholder, fail group sync when a member add or user creation fails, propagate team roster write failures, and keep the matched user_id on a POST /Users email match - PR #37686, PR #37688, PR #37700, PR #37701
    • Retry JWKS fetches, serve stale keys, and return 503 when the IdP is unreachable - PR #37690
    • Cache the team member default budget as a typed model, and capture the requester IP in 401 and auth-time 429 failure logs - PR #37695, PR #37707
    • Let org admins view their organization's usage, and return no rows when the aggregated activity entity filter is empty - PR #37235, PR #37414
    • Split agent inference and management routes so admin nodes can create agents - PR #37730
    • Initialize the secret manager before resolving os.environ config references - PR #37544

AI Integrations​

Logging​

  • OpenTelemetry
    • Emit LLM Call spans for speech, image, moderation, OCR and transcription - PR #37752
    • Bound and shut down credential-scoped tracer providers, and build the credential-scoped Resource once per logger - PR #36591, PR #37542
    • Attribute Prisma database spans to PostgreSQL instead of localhost - PR #36595
    • Route Phoenix traces to per-key and per-team projects under OTel v2 - PR #36706
  • Prometheus
    • Render /metrics off the event loop and coalesce concurrent scrapes - PR #37702
    • Fold auth and pre-call time into litellm_request_total_latency_metric - PR #37958
  • DataDog
    • Normalize alias-derived tag values so metrics match what was sent - PR #37682
  • General
    • Close three secret-leak paths in verbose logging, strip callback credentials from the auth object stamped into request metadata, and preserve uvicorn color_message args during secret redaction - PR #37391, PR #37233, PR #37122
    • Bound oversized error payloads written to stdout, and bound the shared logging executor backlog - PR #37684, PR #37694
    • Stop deepcopying results redaction cannot redact - PR #36638
    • Surface per-request auto-router savings to logging callbacks, and group Codex turns under one session id - PR #37894, PR #37895
    • Copy messages before the router call and raise the judge output cap in shadow eval, then drop the unused judge reasoning field and salvage truncated verdicts - PR #37232, PR #37239

Guardrails​

  • Bedrock Guardrails
    • Track Bedrock guardrail usage units per invocation, and count Bedrock guardrail cost against spend and budgets - PR #37225, PR #37362
  • Azure Content Safety
    • Scan text on /guardrails/apply_guardrail - PR #36894
  • Noma
    • Stop sending the conversation twice in the v2 payload - PR #36764
  • General
    • Close PII and PCI masking gaps in SpendLogs, debug logs and the logging_only response - PR #37965
    • Run pre-call guardrails on batch input file uploads, scan batch records with the content hooks that are not guardrails, and omit litellm_batch_guardrail when no guardrail acted - PR #37519, PR #37786, PR #37964
    • Run policy pipelines when the caller sends its own metadata, covering /v1/messages and Claude Code - PR #36889
    • Record MCP tool guardrail evaluations and blocks - PR #36978
    • Cap the date window accepted by /guardrails/usage, retry usage upserts only on connection errors, and requeue rollup rows dropped after retry exhaustion - PR #37380, PR #37247, PR #37387
    • Read through to the DB on registry misses so just-created models, guardrails and agents resolve on sibling replicas - PR #36263

Prompt Management​

  • General
    • Don't route requests without a prompt_id to prompt managers that cannot run them - PR #37575

Secret Managers​

  • General
    • Store the lite login credential and its refresh token in the OS keychain instead of a file on disk - PR #37566, PR #37665
    • Initialize the secret manager before resolving os.environ references in the config - PR #37544

Spend Tracking, Budgets and Rate Limiting​

  • Provisioned throughput
    • Accrue flat cost for PTU deployments declared in config.yaml, on a source-agnostic deployment record - PR #37556, PR #37501
    • Refuse an incomplete config.yaml reservation the way the endpoints do, require an operator-declared id, never retract a flat charge for a deployment the run cannot see, and warn when a config declares PTU while attribution is off - PR #37703, PR #37794, PR #37793, PR #37898
    • Hand the prune a plain delete filter the query builder can serialise - PR #37571
  • Rate limiting and budgets
    • Add project-level ITPM and OTPM quotas - PR #35110
    • Make per-model budgets track spend, enforce, and report the same counter - PR #37736
    • Tokenize each request once, off the event loop, for large prompts in budget reservation - PR #37683
    • Elect one sweeper per tick and bound the window scan in the reset budget job, and reconnect and retry on transient DB transport errors - PR #36497, PR #37705
    • Add an admin toggle to block requests for models without pricing - PR #35181
  • Spend logs
    • Hash raw API keys before persisting them to spend logs, so a non-sk- or Bearer-prefixed key cannot be stored or shown in plaintext in the Key Hash column - PR #30736
    • Add lifecycle timestamps to spend logs; the endTime backfill migration was reverted before the cut - PR #37361, PR #37554, PR #37875
    • Bound each spend-log write statement by row count as well as bytes - PR #37758
    • Record estimated input tokens and populate deployment attribution on failed-request spend logs - PR #37365, PR #37520
    • Log spend for OpenAI passthrough embeddings with unmapped models - PR #37425
  • Cost calculation
    • Claim batch cost rows atomically so multi-pod polling cannot double-bill - PR #37685
    • Price partial-stream spend rows at the real model and keep the prompt and cache fields - PR #37734
    • Recognize the ultrafast service tier in cost calculation, match streamed Messages usage cost to the recorded spend, and track provider-reported cost when the caller omits include_usage - PR #37355, PR #35114, PR #35013, PR #36593
  • Cost map maintenance
    • Consolidate eleven open registry audits into one changeset, and add provider-announced deprecation_date to 205 registry entries - PR #37658, PR #37283
    • Apply the GPT-5.6 Sol promotional pricing cut, and correct GPT-5.6 max input tokens to 922K - PR #37880, PR #37722
    • Correct Gemini 3.1 flash image and DeepSeek V4 pricing, add OpenAI deprecation dates, and set prompt_cache_min_tokens=4096 on Gemini 3.5/3.6/3.7 Flash and 3.1 Pro Preview - PR #37473, PR #37516
    • Add undated Azure aliases for gpt-audio-mini and gpt-realtime-mini, and supports_mid_conversation_system to bare first-party Claude keys - PR #37867, PR #36969
    • Require pricing-owner approval for the model prices JSON files - PR #37551

MCP Gateway​

  • OAuth
    • Serve token-forwarding servers when OAuth discovery fails, and stop OAuth discovery from causing outages - PR #37399, PR #36599
    • Let a salt-key-orphaned OAuth credential be replaced by re-authorization - PR #37672
    • Deny the interactive dcr_bridge authorize for a user without server access, and resolve admin OAuth sessions to the same server set the connect page shows - PR #37865, PR #37900
    • Scope the authorization server issuer for named MCP servers; note that an earlier attempt at the same fix was reverted in this window - PR #37204, PR #36482, PR #37220
  • Tools and routing
    • Bind the tool existence check to the selected server, and strip root_path before matching the per-server MCP route spelling - PR #37388, PR #35576
    • Forward the per-server auth header on OpenAPI tool calls, and stop reporting failed OpenAPI tool calls as successes - PR #37410, PR #37496
    • Normalize auth schemes so MCP egress emits exactly one prefix - PR #37668
    • Attach the per-user BYOK credential when listing tools for non-OAuth2 auth types - PR #34787

Performance / Loadbalancing / Reliability improvements​

  • Routing
    • Give auto prompt caching deployment affinity, and isolate deployment model info from cached backend metadata - PR #37689, PR #37687
    • Route Responses API input through the auto-router, and let a routed deployment's own litellm_params beat forwarded auto-router marker params - PR #37333, PR #37615
    • Honor key-level tag filtering in pre-routing, keep acreate_file fallbacks inside the requested model group, and forward target_model_names on file uploads to litellm_proxy deployments - PR #37366, PR #37424, PR #36240
    • Add router_model_name to auto-routed response bodies, and stop logging "Could not identify azure model" when the deployment name resolves from the cost map - PR #37725, PR #37869
  • Complexity router
    • Operator-defined tier sets for the LLM classifier, custom classifier plugins via classifier_type: custom, a plan-mode tier floor for coding-agent clients, per-tier litellm_params, and a business classification rubric preset - PR #37226, PR #37249, PR #37230, PR #37064, PR #37534
    • Gate the reasoning override on a non-SIMPLE score - PR #37500
  • Caching
    • Bound the semantic cache embedding lookup so a dead embedding endpoint cannot block requests, and truncate the semantic cache embedding input - PR #37742, PR #37367
    • Preserve the prompt cache for mid-conversation system messages on unflagged Claude models - PR #36968
    • Apply Azure AD and GCP IAM auth to every async Redis client path, and reset only the failed node on a cluster client timeout instead of the whole client - PR #37740, PR #37863
  • Proxy runtime
    • Stop large token counts from blocking the proxy event loop - PR #37697
    • Send SSE keepalives while a slow upstream is still silent, including on assistants runs and A2A streams - PR #37322, PR #37368
    • Stop per-request tag and end-user Postgres reads in auth with registry caches - PR #36801
    • Forward store and prompt_cache_key on chat completions, and map nested prompt_tokens_details.cache_creation_input_tokens to cache_write_tokens - PR #33195, PR #37377
  • Database and deployment
    • Apply the configured connection params to the read replica URL, and compose DATABASE_URL_READ_REPLICA from a reader host secret key in Helm - PR #37691, PR #37109
    • Fail the standalone Prisma migration entrypoint on migration errors, but keep a failed prisma generate from failing it - PR #37692, PR #37947
    • Bound the Helm migrations Job so a blocked migration cannot stall the release, and let USE_V2_MIGRATION_RESOLVER select the v2 migration resolver - PR #36975, PR #36258
    • Bump the wolfi-base digest for busybox 1.38.0-r1 and openssl 3.6.3-r5, and bump sqlparse to 0.6.0 to clear osv-scan findings - PR #37950, PR #37200
  • Typing
    • Drop roughly 2.6k basedpyright errors across 72 Any hotspot files - PR #37073, PR #37439

Documentation Updates​

  • Note that the Terraform provider now ships at the LiteLLM version - PR #37912

Documentation now lives in BerriAI/litellm-docs, so doc changes in this window are counted there rather than in this repository's PR set.

PR roll-up by ownership area​

PRs by ownership area (total: 433)

  • UI: 115
  • Other (CI / chore / tests / build / version bumps): 65
  • Spend / Budgets / Rate Limits: 48
  • Performance: 44
  • Models & Providers: 42
  • LLM API Endpoints: 41
  • Logging: 23
  • Auth & Management: 22
  • MCP: 16
  • Guardrails: 15
  • Prompt Management: 1
  • Docs: 1

End-to-End Testing​

We are investing heavily in end-to-end testing to cut regressions and make LiteLLM more stable release over release. Every version is exercised by a live suite that runs against a real deployed proxy and hits real provider endpoints, not mocks, so the behavior we validate is the behavior you get in production.

This window added 73 test-only pull requests, 14 of them touching the live e2e suite. The headline change is a record-and-replay transport seam: a fixture bundle format, canonical content-based match keys, query params and multipart form fields pinned as part of replay identity, and the seam moved down to the provider edge, so the non-streaming provider flows can be replayed deterministically while the live suite still runs against real providers. New permanent regressions pin prompt-cache, service-tier and cost-header billing, OpenAI passthrough routing and file-list isolation, the OpenAI WebSocket passthrough prefixes, and twelve previously closed issues.

The suite itself was hardened against the failures that were masking real signal: response-cache cross-talk, slow providers and single upstream blips are now tolerated deliberately rather than by luck, blind sleeps are replaced with deadline waits, and live suites were repointed off retired Gemini, Groq, Together AI and Vertex image models. On the ratchet side, six ruff rules now reject tests that cannot fail, pytest.raises(Exception) is banned outright, PT011/PT012/PT014/PT017/RUF043 are enforced so a broad raises block cannot pass on the wrong error, and F811 and F821 are enforced so a duplicate or undefined name cannot silently replace the first. Thirty test files stranded in a second mirror now actually run, tests/old_proxy_tests was retired, and the conftest save/restore inventory is frozen so it can only shrink; the global-state cleanup unwound 182 leaking writes out of the cost-calc suites alone.

New Contributors​

  • @ChenluJi made their first contribution in PR #32448
  • @Sujithr07 made their first contribution in PR #33195
  • @MUSE-CODE-SPACE made their first contribution in PR #34067
  • @ayaangazali made their first contribution in PR #34445
  • @bhuvan2134686 made their first contribution in PR #34752
  • @shivijain2323 made their first contribution in PR #35110
  • @Scott-Wilson-ZocDoc made their first contribution in PR #36032
  • @LHMQ878 made their first contribution in PR #36151
  • @harryzhou2000 made their first contribution in PR #36355
  • @irosh-colombage-ZocDoc2 made their first contribution in PR #36482
  • @itaimodi made their first contribution in PR #36764
  • @brian5021 made their first contribution in PR #36878
  • @oneKn8 made their first contribution in PR #36968
  • @sailikhithk made their first contribution in PR #36981
  • @bruno-olivia made their first contribution in PR #37242
  • @longwind48 made their first contribution in PR #37821

Three fixes in this release reached the repository as maintainer-pushed copies so the full CI pipeline could run against them, with commit authorship preserved: PR #37867, PR #37868 and PR #37869 are @mihidumh's work, and PR #37219 is @marty-sullivan's.

Full Changelog​

https://github.com/BerriAI/litellm/compare/v1.98.0-rc.1...v1.99.0-rc.1

🚅
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails — built for production.
Learn more →