Skip to main content

4 posts tagged with "observability"

View All Tags

Model Deprecation Management

Ishaan Jaffer
CTO, LiteLLM

MODEL LIFECYCLE MANAGEMENT / NEW IN THE LITELLM UI

ModelDeprecationManagement

The LiteLLM UI now has alerts
for model deprecations.

See how it works
Model deprecation management in LiteLLMProvider deprecation notice, an alert in the LiteLLM UI, migration, and resolution. Model providers: OpenAI, Anthropic, Google, Azure, and AWS.PROVIDER NOTICEA model is retiringUI ALERTFind affected keysMIGRATEMove use casesRESOLVEDConfirm zero trafficOpenAIAnthropicGoogleAzureAWSMODEL LIFECYCLEDeprecationalertReview affected keys

This blog post is about how you can do model lifecycle management with LiteLLM AI Gateway. We are giving you the following primitives to better manage model deprecations.

We heard from a lot of our production gateway users that model deprecation was a problem: for production use cases, it's hard to figure out how to prevent outages.

This helps prevent outages with:

  1. Alerting around model deprecations.
  2. Emails around model deprecations.
  3. Usage tracking for different models that are getting deprecated, so you can validate that traffic is going down once you migrate, and a recommended model to use.

Alerts around model deprecations​

Check the notification bell for model deprecations that need attention. Open Alerts under Observability, then select Model deprecations. You can see the provider, the model, its deprecation date, requests in the last 30 days, and when LiteLLM last saw traffic to it.

Open the alert to investigate the migration across teams. You can keep the alert open while you coordinate with the owners, then return to the same page to check progress.

LiteLLM Model deprecations inbox with provider logo, deadline, recent requests and recommended replacement

The Model deprecations inbox in the LiteLLM UI. Sample usage; the provider deadline and recommendation come from model metadata.

Emails around model deprecations​

From the alert, click Set up email reminders, then Open Email Settings. This opens Logging & Alerts → Email Alerts.

If your gateway already sends email, keep its delivery settings. Otherwise, enter the SMTP connection details from your email provider in Email Server Settings. Set Gateway URL to the public address owners use to reach your gateway and save. Enter TEST_EMAIL_ADDRESS, select Test Email Alerts, and confirm that the test email arrives.

In Email Notifications, turn on Model Deprecations and select Save Changes. This event is off by default.

LiteLLM schedules two reminders: 30 days before and 1 day before the provider's deprecation date, checked in UTC. It selects affected key owners and current team admins using requests from the last 30 days. Each recipient sees usage for their keys and teams they administer.

The email names the provider deprecating the model, shows the current and recommended models with provider logos, and gives the owner the migration steps. It includes the recommendation source, the provider notice, and a link to review usage in the gateway.

Model deprecation email naming Google as the provider, comparing the old and recommended models, and listing migration steps

The 30-day email template with sample usage. Owners can see what changes, when they need to act, and where to review the affected traffic.

Keep the gateway running on the reminder dates. The scheduler does not send a missed reminder later if the gateway is offline for the entire UTC date.

Track usage as you migrate​

Click Review impact to open the model. The Daily requests chart shows requests to that model over time, with a separate stack for each key. Compare the total bar height to check whether usage is falling, then inspect the colored stacks to find the keys that still send traffic.

Below the chart, inspect the affected keys, teams, owners, and tags used with that model. Filter by team when you need to coordinate one part of the migration. Use the key and owner list to follow up with the people responsible for each production use case.

Model deprecation detail with daily requests stacked by key falling to zero, and five use-case keys across two teams

Actual LiteLLM UI with a seeded migration example: five use cases across two teams. Daily requests decline as the example keys move off the model. The final complete days show zero requests; the current day is partial.

You do not need to wait for the 30-day total to reach zero. That total includes historical requests. Compare the recent daily bars instead, and check a period that covers your use case. A weekly batch job needs a longer verification window than a service that sends requests every minute.

In this example, customer support, research, document search, invoice extraction, and weekly reports each have a key. As an owner migrates a use case, its color drops out of the daily bars. The key stays in the 30-day history so you can verify what changed.

The alert shows the model to replace and the Recommended new model. LiteLLM reads the recommendation from its model cost map, or from the deployment's configured replacement. The UI identifies which source supplied it and links to the upstream provider's deprecation notice when one is recorded.

Use the provider notice to check the upstream deadline and migration guidance. The cost map records the recommendation; it does not establish that the new model meets your application's quality, latency, or cost requirements. Test the replacement with the workload behind each affected key before switching production traffic.

You can use Model routing to open the relevant model configuration. If your application chooses the model, update that application. Keep the alert open while owners make these changes.

Mark the alert resolved​

Return to Daily requests after the owners switch their applications or routing. Check that traffic to the old model reaches zero across the affected keys. Confirm the migration with owners of infrequent jobs before closing it.

Click Mark resolved when that work is complete. LiteLLM moves the alert to Resolved, so your open inbox shows the migrations that still need attention. New requests to the model reopen the alert.

LiteLLM model migration marked resolved after the daily requests chart reaches zero

The same example migration after marking it resolved. Closing the alert records the team's decision; it does not change model routing.

For help operating LiteLLM across production teams and use cases, talk to us. Read more about gateway alerting, email notifications, and model routing.

Use Hex to analyze LiteLLM costs, then set budgets and routing

Ishaan Jaffer
CTO, LiteLLM
← BlogCOST REPORTING / HEX TEMPLATE
LiteLLMHex

AI spend.Cost reporting, now in Hex.

Connect Hex to your LiteLLM Postgres data.
Set budgets and routing in LiteLLM.

View live Hex demo
TEAMS / MODELS / BUDGETSOPEN TEMPLATE · V2.0.0
THE HEX BOARDSYNTHETIC DEMO DATA

Explore the live Hex dashboard.

Open the working demo ↗ · Synthetic data · No sign-in

Hex board showing project spend, request volume, unit costs and daily trends with synthetic data

See a model change raise unit cost.

Zoomed Hex demo alternating between project model mix and the cost-change table, where document-review shows an 80 percent unit-cost increase
Model mix and seven-day cost changes in Hex. Synthetic demo data.

The public demo includes synthetic timing metrics. The daily team template reports recorded costs, requests and usage; daily rows do not include latency.

LiteLLM records AI costs in Postgres and maintains daily spend totals. Hex connects to that database and reads the totals. Use the dashboard to find a cost increase, then open LiteLLM to change routing or set a budget.

1. Connect Hex to your LiteLLM Postgres database​

In Hex, open Settings → Data sources → + Connection → Postgres. Add a read-only connection to the database your LiteLLM gateway uses.

READ-ONLY DATA FLOWEXISTING DAILY AGGREGATES
01 / YOUR POSTGRESDaily spendLiteLLM_DailyTeamSpend
date + team_id
Day and team
model + spend
Recorded costs
api_requests
Daily requests
prompt_tokens + completion_tokens
Token usage
02 / HEX SOURCE QUERYTeam namesLiteLLM_TeamTable

Join daily totals to team display names.
Keep ownership by stable team ID.

SQL included in the templateNo database objects created
03 / YOUR HEX APPCost reporting

Team costs. Model mix.
Daily trends. Cost changes.

Refresh with a scheduled run
LiteLLM writes the daily totals in Postgres. Hex reads reporting columns from these two existing tables. You change budgets and routing in LiteLLM; Hex does not write to the gateway.

The template reads two existing tables: LiteLLM_DailyTeamSpend for costs and usage, and LiteLLM_TeamTable for team display names. It groups the results by day, team and model. Give the reader access to those reporting fields. It does not need your key table, prompts or responses.

Connection details and SQL permissions.

2. Add the template and inspect your costs​

Open the live Hex demo first. You can filter it without signing in. Its records are synthetic.

To use your own data, download the Postgres template below. In Hex Home, click + → Upload project, then click Upload in the dialog and choose the .hex.yaml file. Select your Postgres connection on the Source SQL cell and Run all.

Hex dialog for uploading a YAML project

Filter by team. Compare recorded spend, request volume and cost per thousand requests, then open Models and cost changes. In the demo, the Legal team's document-review workload switches models and unit cost rises 80%.

3. Take an action in LiteLLM​

Use Hex to investigate the increase. Apply the change through your LiteLLM gateway.

Your decisionIn the LiteLLM UI
Try a cheaper modelOpen Teams → your team → Settings → Edit Settings → Model Aliases. Change the model behind an alias your application already calls, then save. Evaluate the candidate on your workload first.
Route by request complexityOpen Models + Endpoints → Auto Router → Add Auto Router. Choose the models for each tier, test routing and connections, then save. Configure your application or team alias to call that router.
Set a spending limitOpen Teams → your team → Settings → Edit Settings. Set Max Budget (USD) and Reset Budget, then save.

For example, select Legal in LiteLLM and set Max Budget (USD) to 1000 and Reset Budget to monthly (30 days). This limits new spend; it does not change recorded costs.

Set Legal’s budget in the LiteLLM team settings

Current LiteLLM UI with synthetic local gateway data.

After saving, check the team's budget status. Then refresh Hex after LiteLLM updates its daily totals to compare the next complete period. Set a Hex scheduled run if you want the report to refresh on a schedule.

Legal’s budget status after saving

Auto Router setup · Team budgets

Auto Router v1.97: usage benchmarks and better quality for lower cost

Tin Lo
AI Product Engineer, LiteLLM

LiteLLM Autorouter V2: routing accuracy on complex scenarios, 5.6x more accurate by reading the last N turns of the conversation before picking a model



🚀 Help shape the Auto-Router

Get early access, work directly with the LiteLLM team, and influence the roadmap with your production traffic.

Apply to Become a Design Partner

Already testing it? Share your results in discussion #32168.

v1.97 makes three changes to the auto router.

  • The LLM classifier now receives a window of prior conversation turns, defaulting to three. This improves accuracy of follow-up classifications from 14% to 78%, costs at most $0.61 per 1,000 requests, and no additional latency.
  • A new Benchmarks view prices routed traffic against an all-frontier baseline and reports the difference, and those savings now also appear in the Cost Optimization totals.
  • Session affinity is now off by default, following our previous post showing this was leading to worse quality without cost improvements.
Two defaults changed

classifier_context_window_size now defaults to 3 (LLM classifier only), and session_affinity now defaults to false (all routers). Config files are not modified, but the new defaults apply to any key left unset, so a config that never mentioned session_affinity will reclassify every turn after upgrading. Configs that set either key explicitly are unaffected.

Improve release stability with 24 hour load tests

Alexsander Hamir
Performance Engineer, LiteLLM
Krrish Dholakia
CEO, LiteLLM
Ishaan Jaffer
CTO, LiteLLM

LiteLLM Observatory

As LiteLLM adoption has grown, so have expectations around reliability, performance, and operational safety. Meeting those expectations requires more than correctness-focused tests, it requires validating how the system behaves over time, under real-world conditions.

This post introduces LiteLLM Observatory, a long-running release-validation system we built to catch regressions before they reach users.