Skip to main content

v1.98.0rc1 - Provisioned Throughput Billing, Shadow Evals & Routing Groups

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:1.98.0-rc.1
Breaking Changes

The Langfuse metadata blob is now sourced from the StandardLoggingPayload allowlist instead of raw request metadata. Roughly 20 fields no longer appear on the generation, measured live at 52 keys down to 38, so any saved Langfuse filter, dashboard, or alert built on model_group, model_info, deployment, deployment_model_name, model_group_alias, model_group_size, litellm_api_version, litellm_received_at, litellm_parent_otel_span, queue_time_seconds, attempted_retries, max_retries, agent_id, caller_tags, inherited_tags, global_max_parallel_requests, user_api_key, or the remaining user_api_key_* budget and permission-id fields will stop matching. model_group and the deployment id are still recoverable from hidden_params. Direct-SDK callers passing flat custom metadata must nest it, metadata={"metadata": {"my_key": "v"}}, and it arrives under requester_metadata; proxy callers are unaffected. debug_langfuse now emits caller scalars rather than the raw metadata dump. See PR #36744.

The Global Control Plane worker registry now requires an Enterprise license. A proxy configured with worker_registry and no valid LITELLM_LICENSE refuses to boot instead of silently running unlicensed. Set a valid license to keep the control plane, or remove the worker_registry key to start without it. See PR #36996.

A litellm_settings.callbacks entry whose dotted path names a class now fails config load. Those entries used to be accepted and silently inert: the proxy booted, served traffic, and never ran the hook. Point the entry at an instance or a function, for example custom_callbacks.proxy_handler_instance, to start again. See PR #36858.

Key Highlights​

  • Reserved capacity is billed as reserved capacity - a deployment can now carry ptu_count and cost_per_ptu_per_hour with an effective window, the daily rollup writes per-model flat cost by active hour, and per-token billing is switched off entirely on that deployment so a team paying for provisioned throughput is no longer charged twice for the same traffic. Attribution is opt-in behind an env var, and a price sent alongside PTU config is rejected with a 400
  • You can measure the auto-router before you adopt it - a shadow eval job samples a slice of one key's successful traffic, replays it through the auto-router in a detached task that never serves a response or adds latency, and has an LLM judge compare both answers blind with randomized A/B labels. Counts, status, judge spend, and win rates by tier and incumbent model are all derived at read time from one append-only row per sample, and the job also runs in reverse
  • Routing groups are callable models - model=<group_name> now routes across the union of member deployments using the group's own strategy, group names appear in /v1/models so Claude Code and Codex discovery surface them, and they are grantable on keys and teams. The Create Group modal has promised this since day one
  • Every response can state its own cost breakdown - six x-litellm-response-cost-* headers ship next to the total, where input, cache read, cache creation, output, and tool usage sum exactly to the total and reasoning is a subset of output, so a platform team attributes spend per component with no local pricing table
  • TPM reservations follow declared output size - expected output tokens are now declarable per key, per team, and per model instead of one static floor for every tenant, so concurrent requests stop overrunning a team's TPM limit and teams whose models emit far less stop being throttled. No config means byte-identical behavior and no migration
  • The Admin UI finished its move off antd and Tremor - 74 UI pull requests in this window carry the dashboard onto shadcn (base-vega) primitives, covering the navbar, playground, guardrails, usage, cost tracking, models and endpoints, team and user surfaces, the log details drawer, the AI Hub, and the shared component library

New Providers and Endpoints​

New Providers (1 new provider)​

ProviderSupported LiteLLM EndpointsDescription
Nimble/searchNimble's Search API as the 18th native search provider, registered in the provider enum, config map, pricing map, and dashboard, priced at $0.005 per query

New LLM API Endpoints (2 new endpoints)​

EndpointMethodDescriptionDocumentation
/v1/indexesGETAdmin-only listing of every registered vector store index, newest first, so indexes created through POST /v1/indexes can be auditedVector Stores
/auto_router/shadow_eval/{start,stop,{job_id}}POST, GETStart, stop, and read a pre-adoption shadow eval job for an auto-router, returning derived counts, judge spend, latest error, and win ratesAuto Router

New Models / Updated Models​

New Model Support (52 new models)​

ProviderModelContext WindowInput ($/1M tokens)Output ($/1M tokens)Features
Anthropicclaude-mythos-51M$10.00$50.00Reasoning, adaptive thinking, xhigh and max reasoning effort, vision, PDF input, computer use, function calling, tool choice, prompt caching, response schema, output config
Anthropicclaude-mythos-preview1M$10.00$50.00Reasoning, adaptive thinking, xhigh and max reasoning effort, vision, PDF input, computer use, function calling, tool choice, prompt caching, response schema, output config
OpenAIgpt-transcribe---Audio transcription, $0.000075 per second
OpenAIgpt-live-transcribe---Audio transcription, $0.00028333 per second
OpenAIgpt-realtime-translate16K--Realtime, audio input and output, $0.00056667 per second
Google Geminigemini/gemini-3.7-flash1M$0.75$3.75Reasoning, vision, audio input, video input, PDF input, function calling, parallel function calling, tool choice, prompt caching, response schema, web search, URL context, native streaming
Google Geminigemini/gemini-3.1-flash-tts-preview8K$1.00$20.00Audio speech
Google Geminigemini/gemini-robotics-er-2-streaming-preview-$2.00$10.00Vision, audio input, video input, function calling, web search
Google Vertex AIvertex_ai/gemini-3.7-flash1M$0.75$3.75Reasoning, vision, audio input, video input, PDF input, function calling, parallel function calling, tool choice, prompt caching, response schema, web search, URL context, native streaming
Google Vertex AIgemini-3.7-flash1M$0.75$3.75Reasoning, vision, audio input, video input, PDF input, function calling, parallel function calling, tool choice, prompt caching, response schema, web search, URL context, native streaming
xAIxai/grok-4.6500K$2.00$6.00Reasoning, vision, function calling, tool choice, prompt caching, response schema, web search, above-200K tier pricing
xAIxai/grok-build-0.1256K$1.00$2.00Reasoning, vision, function calling, tool choice, prompt caching, response schema, above-200K tier pricing
xAIxai/grok-4.20-0309-non-reasoning1M$1.25$2.50Vision, function calling, tool choice, prompt caching, response schema, web search
xAIxai/grok-4.20-multi-agent-03091M$1.25$2.50Reasoning, vision, function calling, tool choice, prompt caching, response schema, web search
Mistralmistral/mistral-small-2603262K$0.15$0.60Reasoning, function calling, tool choice, response schema, assistant prefill
Mistralmistral/labs-leanstral-1-5262K$0.00$0.00Function calling, tool choice, response schema
Mistralmistral/mistral-moderation-2603131K$0.00$0.00Moderation
Mistralmistral/voxtral-mini-2602---Audio transcription, $0.00005 per second
Mistralmistral/voxtral-mini-transcribe-realtime-2602---Audio transcription, $0.0001 per second
Mistralmistral/voxtral-mini-tts-2603---Audio speech, $0.000016 per output character
Groqgroq/qwen/qwen3.6-27b131K$0.60$3.00Reasoning, vision, function calling, tool choice
Groqgroq/meta-llama/llama-prompt-guard-2-22m512$0.03$0.03Chat
Groqgroq/meta-llama/llama-prompt-guard-2-86m512$0.04$0.04Chat
Groqgroq/canopylabs/orpheus-v1-english4K--Audio speech, $0.000022 per character
Groqgroq/canopylabs/orpheus-arabic-saudi4K--Audio speech, $0.00004 per character
Metameta/muse-spark-1.21M$1.25$4.25Reasoning, minimal and xhigh reasoning effort, vision, PDF input, function calling, parallel function calling, tool choice, prompt caching, response schema, web search
Metameta/muse-spark-1.2-contributor1M$0.10$0.20Reasoning, minimal and xhigh reasoning effort, vision, PDF input, function calling, parallel function calling, tool choice, prompt caching, response schema, web search
Azure AIazure_ai/grok-4.3200K$1.25$2.50Reasoning, vision, function calling, tool choice, prompt caching, response schema, web search
Azure AIazure_ai/FW-DeepSeek-V3.2164K$0.62$1.85Reasoning, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-DeepSeek-V4-Pro1M$1.925$3.828Reasoning, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-GLM-5200K$1.10$3.52Reasoning, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-GLM-5.1203K$1.54$4.84Reasoning, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-GLM-5.21M$1.54$4.84Reasoning, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-GLM-5.2-Fast1M$2.10$6.60Reasoning, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-Inkling1M$1.00$4.05Reasoning, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-Kimi-K2.5262K$0.66$3.30Reasoning, vision, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-Kimi-K2.6262K$1.045$4.40Reasoning, vision, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-Kimi-K2.7-Code262K$1.05$4.40Reasoning, vision, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-Kimi-K31M$3.30$16.50Reasoning, vision, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-MiniMax-M2.51M$0.33$1.32Reasoning, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-MiniMax-M3512K$0.33$1.32Reasoning, vision, function calling, tool choice, prompt caching
Azure AIazure_ai/FW-Nemotron-3-Ultra-NVFP4262K$0.60$2.40Reasoning, function calling, tool choice, prompt caching
DashScopedashscope/deepseek-v4-flash1M$0.20$0.40Reasoning, function calling, tool choice, prompt caching, response schema
DashScopedashscope/deepseek-v4-flash-07311M$0.20$0.40Reasoning, function calling, tool choice, prompt caching, response schema
DashScopedashscope/deepseek-v4-pro1M$2.40$4.80Reasoning, function calling, tool choice, prompt caching, response schema
DashScopedashscope/glm-5.1203K$1.40$4.40Reasoning, function calling, tool choice, prompt caching, response schema
DashScopedashscope/glm-5.21M$1.40$4.40Reasoning, function calling, tool choice, prompt caching, response schema
DashScopedashscope/kimi-k2.7-code229K$0.95$4.00Reasoning, vision, function calling, tool choice, prompt caching, response schema
DashScopedashscope/qwen3.8-max992K$2.00$6.00Reasoning, vision, function calling, tool choice, prompt caching, response schema
DeepInfradeepinfra/nvidia/NVIDIA-Nemotron-3.5-Lightning262K$0.05$0.20Reasoning, function calling, tool choice
OpenRouteropenrouter/nvidia/nemotron-3.5-lightning262K$0.05$0.20Reasoning, function calling, tool choice
Nimblenimble/search---Search, $0.005 per query

Beyond the new entries, this release is a large cost-map maintenance pass: 270 existing entries gain or correct a provider-announced deprecation_date across Bedrock, Mistral, Cohere, Gemini, OpenAI, and xAI, and 85 gain a search_context_cost_per_query block. xAI pricing is corrected in both directions: the grok-4.20 reasoning and non-reasoning variants drop from $2.00 / $6.00 to $1.25 / $2.50 per 1M and their context window is corrected from 2M to 1M, while grok-code-fast rises from $0.20 / $1.50 to $1.00 / $2.00. The Bedrock Mantle openai.gpt-5.6-sol, -terra, and -luna entries move from a 272K to a 1M context window with matching above-272K tiers, gpt-5-pro max output rises from 128K to 272K, the DeepSeek V4 entries go from 8K to 393K max output, and the Groq llama-3.1-8b-instant, llama-3.3-70b-versatile, and gpt-oss entries are resynced with Groq's own docs. Native structured output is flagged on 17 entries, supports_tool_search on 43, and reasoning-effort flags on a further 29. No pricing entries were removed.

Features​

Bug Fixes​

  • Amazon Bedrock
    • Use the deployment's credentials for AWS requests, and resolve aliases in batch file records - PR #36160, PR #36159
    • Resolve the managed-batch output bucket on every path that reads it, and stop managed-batch litellm_params leaking to the provider - PR #37047, PR #37048
    • Drop toolSpec.strict for Claude Sonnet 5 on Converse - PR #33196
    • Add a text block to Converse user messages carrying documents - PR #36499
    • Send the tool-search beta header for Haiku 4.5 on Invoke /v1/messages - PR #36502
    • Preserve adaptive thinking effort through the /v1/messages bridge - PR #36507
    • Hoist custom.defer_loading before dropping custom on invoke tools - PR #36855
    • Reject the Anthropic server-side web_search tool with an actionable error instead of a provider failure - PR #36473
    • Enable native structured output for GLM 5 and DeepSeek V3.2, and advertise it on every Bedrock id - PR #35669, PR #36597
    • Give the Bedrock Mantle GPT-5.6 Sol, Terra, and Luna entries a 1M context window with long-context pricing - PR #36698
  • Anthropic
    • Preserve midturn system corrections - PR #34290
    • Preserve speed=fast in usage for /v1/messages and pass-through - PR #36447
    • Flag native structured outputs on Anthropic-direct claude-sonnet-5 and claude-haiku-4-5 - PR #35930
  • Google Vertex AI
    • Translate /v1/embeddings batch rows to the Gemini embedding shape - PR #35092
  • Azure AI
    • Recognize real Search doc endpoints so teams can read and write through passthrough - PR #36798
  • Databricks
    • Surface provider usage, including prompt-cache counts, in streaming chunks - PR #36943
  • NVIDIA NIM
    • Preserve image passages and stop sending top_k to /v1/ranking - PR #34177
  • Groq
    • Sync the Groq registry with Groq's docs - PR #36664
  • OpenAI
    • Return a length-truncated 200 when the output budget fits no token - PR #36859
  • General
    • Ship boto3 with the base SDK so Bedrock works out of the box on pip install litellm - PR #36568
    • Let an explicit provider outrank a known OpenAI model name - PR #36800
    • Refresh deprecation dates, correct xAI pricing, add missing provider models, and correct DeepSeek V4 max output - PR #36403, PR #36538, PR #36788, PR #36925

LLM API Endpoints​

Features​

Bugs​

  • Responses API
    • Preserve Codex namespace tool calls - PR #32536
    • Initialize completed_response on the bridge streaming iterator - PR #35413
    • Map interaction step and turn history onto Responses API roles and content types - PR #36733
  • Batches
    • Stop forwarding custom_llm_provider twice in list and cancel - PR #32813
  • Managed files
    • Stop /{provider}/v1/files capturing /openai_passthrough - PR #36092
    • Scope file list pagination cursors to the caller, and report has_more false on caller-scoped pages - PR #36093, PR #36326
  • Passthrough
    • Stop forwarding the client's Accept-Encoding upstream, which garbled Claude Code's /v1/models and count_tokens bodies on stock Docker images once Anthropic began brotli-compressing JSON - PR #37058
  • Anthropic /v1/models
    • Always emit the token limits, null when unknown - PR #36961
  • Web search
    • Stop leaking interception control fields to providers - PR #36480
  • General
    • Make tool_result images visible to OpenAI-compatible providers - PR #34462
    • Stop a bare 429 in an error body outranking the status code in exception mapping - PR #36705

Management Endpoints / UI​

Features​

  • Virtual Keys
    • Per-key prompt caching toggle via enable_prompt_caching, with an Admin UI switch on key create and edit - PR #36466
    • Add a config_updated_at audit timestamp - PR #36488
  • Proxy CLI
    • Make the hidden lite command list configurable through ~/.litellm/config.json - PR #36816
  • Auto-router screens
    • Shadow evals tab beside auto-router usage, with a direction picker and reverse-mode display - PR #36588, PR #36994
    • Deployment affinity toggle, and models shown under each tier in the routing benchmark chart - PR #36302, PR #36291
    • Highlight Auto Router in the navbar announcement - PR #36315
  • Dashboard
    • Render request metrics on the /ui/chat surface, including provider prompt cache tokens in the chat response metrics - PR #36845, PR #36827
    • Show vector store indexes on the Vector Stores page - PR #36306
    • Surface PTU inputs on the model form and flat cost on the Usage page - PR #35393
    • Migrate the playground chat controls to shadcn - PR #36129
    • Warn in the Admin UI when no Redis is configured - PR #36495
    • Link user detail team names to their team pages - PR #37022

Bugs​

  • Auth & roles
    • Carry team grants in lite login session tokens so team-bound CLI users stop seeing the whole proxy on /v1/models - PR #36826
    • Expand config-defined model access groups when resolving team models for /v2/model/info - PR #34211
    • Treat SAML as configured in UI SSO detection - PR #36196
    • Restore management_v1 query-param validation under fastapi>=0.140.7 - PR #35773
  • Teams & access groups
    • Sync assigned_team_ids and assigned_key_ids from the team and key write paths - PR #36825, PR #36843
    • Sweep dangling team references and cache on team delete - PR #36819
    • Resolve member_delete cleanup by user id rather than the addressed email - PR #36839
    • Stop the duplicate legacy invitation email and fix its onboarding link - PR #36455
  • Proxy CLI
    • Launch agents as a child process on Windows - PR #36822
  • Dashboard
    • Hide admin-only Logs tabs from roles that cannot call their endpoints, and restore the Deleted Teams tab for organization admins - PR #36333, PR #36478
    • Gate organization and agent usage views, policy and prompt lookups, the Old Usage page, and four sidebar pages behind the roles their endpoints allow - PR #36334, PR #36335, PR #36469, PR #36475
    • Scope the Virtual Keys and Logs team lists to the caller - PR #36472
    • Show and edit key-level router settings on a virtual key - PR #36674
    • Stop a deselected MCP server keeping its grant on a virtual key, and match the MCP servers count badge to its sibling permission badges - PR #36840, PR #36984
    • Restore playground model filtering by endpoint, and distinguish hosted from local vLLM in the provider dropdown - PR #36130, PR #36974
    • Add NVIDIA Riva to the model provider list - PR #36769
    • Align the spend and budget columns, and rename the models table Status column to Source - PR #35176, PR #37021
    • Show zeroed auto-router usage stats when a window has no sessions, and open the classifier prompt editor above the edit auto-router form - PR #36868, PR #36438
    • Keep the cost tracking removal confirmation open until it settles, de-duplicate the reset budget option, stop the models tab strip scrolling vertically, and anchor chips-combobox popups to the field - PR #36960, PR #37010, PR #36993, PR #36995
  • Dashboard internals: the shadcn migration

AI Integrations​

Logging​

  • Langfuse
    • Source the emitted metadata blob from the StandardLoggingPayload, so a team's own Langfuse credentials stop arriving inside that team's own traces - PR #36744
    • Restrict trace steering keys to real Langfuse trace fields, and coerce header-sourced mask and trace-update steering values - PR #36862, PR #36740
    • Emit the OTel trace version and release on the keys Langfuse v4 reads - PR #36702
  • Arize
    • Trace MCP tool calls instead of crashing on CallToolResult - PR #36453
  • Slack alerting
    • Dedupe scheduled Slack spend reports across pods - PR #36489
  • General
    • Opt-in session_id and trace_id correlation on JSON log records via contextvars - PR #34418
    • Fail config load when a callbacks entry is not dispatchable, instead of booting with a silently inert hook - PR #36858
    • Log requests rejected for an unparsable body in spend logs - PR #36673

Guardrails​

  • Palo Alto Networks Prisma AIRS
    • Return the full scan response on blocked requests, surface scan_id on allowed requests, and scan tool call arguments as plain text rather than a tool event - PR #37036, PR #37037, PR #37038
  • Bedrock Guardrails
    • Skip ApplyGuardrail when there is no content to scan, so a tool-only turn stops failing the whole request - PR #36441
    • Scan and re-emit raw Anthropic SSE streams in the post-call hook - PR #36598
  • General
    • Isolate guardrail load failures per row so one bad entry stops taking the rest with it - PR #36432
    • Report real token usage on guardrail-blocked /v1/responses replies - PR #36907

Spend Tracking, Budgets and Rate Limiting​

  • Provisioned throughput (PTU)
    • Configure ptu_count, cost_per_ptu_per_hour, and an effective window on a model deployment - PR #35341
    • Write per-model PTU flat cost by active hour in the daily rollup, and surface it on the daily activity read path - PR #35343, PR #35391
    • Gate PTU flat-cost attribution behind an opt-in env var - PR #36138
    • Stop per-token billing, and stop billing for grounded search, on a PTU-configured deployment - PR #36829, PR #37043
  • Rate limiting
    • Declare expected output tokens per key, team, and model so TPM reservation stops using one static floor - PR #36143
    • Reserve the larger declared output budget for TPM limits - PR #37001
  • Cost tracking
    • Emit six per-component x-litellm-response-cost-* headers next to the total - PR #36965
    • Track OpenAI and Azure web search tool cost per call, and bill xAI web search from server_side_tool_usage_details - PR #35286, PR #30817
    • Support cache creation cost in tiered pricing, all-or-nothing - PR #36720
    • Net prompt-caching savings against the cache-write premium - PR #36452
    • Apply the Anthropic regional geo uplift to cached tokens, and aggregate the 5m and 1h cache-write split across the iterations path - PR #34850, PR #34860
    • Price dict-shaped image input token details at the image rate - PR #33490
    • Stop a zero output rate from zeroing transcription cost - PR #36914
    • Forward resolved provider and deployment pricing in /cost/estimate - PR #35880
    • Never price a strategy-router alias, and stop get_router_model_info wiping cached pricing - PR #36691, PR #36985
  • Passthrough and batches
    • Track spend for OpenAI passthrough /v1/embeddings, inject streaming usage cost on OpenAI passthrough streams, and track streamed passthrough Responses cost - PR #36660, PR #36503, PR #36529
    • Carry the budget reservation into passthrough request metadata, so successful requests stop leaking reservations into Redis and tripping false BudgetExceededError - PR #36592
    • Attribute Vertex and Anthropic passthrough batch cost to the creating key, team, and tags - PR #34456, PR #36468
    • Account a managed batch's cost exactly once, give a batch's cost row a primary key of its own, mark a terminal batch with no output file as processed, and stop uncostable batches starving the cost poll page - PR #37050, PR #36876, PR #35360, PR #36714
    • Strip NUL bytes from passthrough batch tags before the managed object write - PR #36688
  • Spend logs and budgets
    • Requeue Redis spend buffer transactions when the DB commit fails, requeue spend logs on a transport error, and stop losing rows when a flush is cancelled - PR #33881, PR #36716, PR #34826
    • Atomic budget cascade with chunked reset scans in the reset budget job - PR #36287
    • Honor an explicit null budget_duration on team and key create, with clearable UI dropdowns - PR #36699
    • Tolerate a concurrent creator when creating spend views - PR #36824

MCP Gateway​

  • Scope gateway session bearers to the RFC 8707 resource - PR #35045
  • Serve the aggregate MCP endpoint on bare /mcp instead of a 307 redirect - PR #34845
  • Resolve admin OAuth sessions from any worker via DB-backed drafts, and keep admin-entered OAuth endpoints in management reads - PR #36844, PR #36888
  • Bound MCP client requests with a session read timeout - PR #36675
  • Expose client HTTP headers to logging callbacks and hooks - PR #36724
  • Drop the caller host and configured upstream headers from logged metadata - PR #36901

Performance / Loadbalancing / Reliability improvements​

  • Router & auto-router
    • Make routing groups callable as virtual models and list them in /v1/models - PR #36519
    • Pre-adoption shadow eval for the auto-router with a blind pairwise judge, extended to sample /v1/messages and /v1/responses traffic and to run reverse-direction jobs - PR #36587, PR #36830, PR #36865
    • Calibrate the complexity classifier rubric with worked examples, selectable per router, and stop scoring system prompt text for code and technical complexity - PR #36578, PR #36721
    • Add a required-AND (&) tag prefix and an allow_fail_open flag, let untagged requests bypass a tagged pre-routing strategy on shared model names, and stop re-applying router-selecting request tags to the routed tier's deployments - PR #36193, PR #36627, PR #36628
    • Forward auto-router alias params from the marker entry rather than the first same-name deployment - PR #36626
    • Per-deployment allowed_fails_policy and cooldown_time overrides, cooldown for failed fallback deployments, and a corrected cooldown TTL after Redis backfill - PR #34416, PR #35104
    • Keep batch fallbacks inside the model group that owns the file - PR #36181
    • Warn when a deployment's credentials contradict its provider - PR #36486
  • Streaming & connections
    • Global litellm_settings.sse_keepalive_ping_interval_seconds, on top of the per-deployment keepalive_seconds SSE heartbeat, so silent streams stop being killed by ingress idle timeouts - PR #36154, PR #34423
    • Refactor HTTP handler initialization with client support - PR #30952
    • Cache Anthropic /v1/messages responses, including streaming - PR #34581
    • Unwrap decorated __init__s when deriving the Redis from_url kwargs allowlist - PR #36654
  • Database & background jobs
    • Write each daily spend batch in one upsert statement - PR #36448
    • Bound spend-logs retention cleanup so one run cannot saturate the database - PR #36594
    • Stagger scheduled background jobs across jobs and pods - PR #36589
    • Force a Prisma recreate on a Postgres cached-plan error - PR #36428
    • Serialize model reconciles so concurrent model writes stop evicting each other - PR #36687
    • Skip prisma-dependent hooks when no database is attached - PR #36273
  • Deployment
    • Make VPC, Aurora, and Redis optional in the AWS Terraform module, so a locked-down account can reuse its own networking, Postgres, and Redis - PR #36676
    • Add startupProbe and hpa.behavior knobs to the componentized Helm chart - PR #36382

Documentation Updates​

  • Describe the Terraform provider release as automatic - PR #36467
  • Require a user flow and a live-proxy proof in bug reports, and a user flow plus a stuck-at proof in feature requests - PR #36498, PR #36500
  • Show only the latest run as Before/After in the proof-of-fix section, with nested cases - PR #37063
  • Replace the Changes PR template section with Caveats - PR #36423
  • Rewrite the CLAUDE.md comment rule with explicit any-of exceptions, require ReadOnly on every TypedDict field, tell agents to let heavy gates queue for machine-wide slots, and drop the @ prefix from the PR template path - PR #36301, PR #36421, PR #37005, PR #37057, PR #36726

PR roll-up by ownership area​

PRs by ownership area (total: 276)

  • UI: 74
  • Models & Providers: 39
  • Spend / Budgets / Rate Limits: 37
  • Other (CI / chore / tests / build / version bumps): 36
  • Performance: 27
  • Auth & Management: 16
  • LLM API Endpoints: 14
  • Docs: 10
  • Logging: 9
  • Guardrails: 7
  • MCP: 7

End-to-End Testing​

We are investing heavily in end-to-end testing to cut regressions and make LiteLLM more stable release over release. Every version is exercised by a live suite that runs against a real deployed proxy and hits real provider endpoints, not mocks, so the behavior we validate is the behavior you get in production.

This window added 18 test-only pull requests, 10 of them touching the live e2e suite. Vendor API coverage was hardened so that provider denials, disconnects, server errors, and missing credentials hard-fail instead of passing, and new cells cover Google-native generateContent framing, Prometheus queue time, the Anthropic web_search server tool on Bedrock, and the model allow-list's positive case rather than only its denials. Admin UI e2e tests no longer stop at a success toast: every mutating flow now asserts the outgoing request body and reads the resource back from the API, with new coverage for Logs, Playground, Usage, and MCP edit and delete. Three reproducers for open gateway gaps in passthrough headers and per-model budgets are checked in skipped, with the product gap named in each skip reason, so removing the skip becomes the regression test once a fix lands. On the maintenance side, OTel assertions now target the attempt that served the stream rather than the span count, live Bedrock tests were repointed off the retired Claude 3 Sonnet, and the stale antd selectors that were failing the UI suite on every run were repointed at what the migrated dashboard actually renders.

New Contributors​

  • @Praveen11558 made their first contribution in PR #30952
  • @geraint0923 made their first contribution in PR #30817
  • @dcadenas made their first contribution in PR #32536
  • @anxkhn made their first contribution in PR #32813
  • @kr0k made their first contribution in PR #33196
  • @vairodp made their first contribution in PR #33490
  • @atomic made their first contribution in PR #34177
  • @heathriel made their first contribution in PR #34257
  • @eugene-yao-zocdoc made their first contribution in PR #34290
  • @alexshtf made their first contribution in PR #35669
  • @HuanQian571 made their first contribution in PR #35773
  • @milesadkins made their first contribution in PR #35969
  • @daleselaji-dev made their first contribution in PR #36160
  • @fancybear-dev made their first contribution in PR #36196
  • @ilchemla made their first contribution in PR #36347
  • @Louis-Vauterin made their first contribution in PR #36382
  • @william-xue made their first contribution in PR #36529
  • @lostmartian made their first contribution in PR #36660
  • @FahimaGold made their first contribution in PR #36705
  • @guptaishaan made their first contribution in PR #36907
  • @pokepoke81 made their first contribution in PR #36943
  • @erensh27 made their first contribution in PR #36965

Full Changelog​

https://github.com/BerriAI/litellm/compare/v1.97.0-rc.1...v1.98.0-rc.1

🚅
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails — built for production.
Learn more →