Skip to main content

Provider Budget Routing

Use this to set budgets for LLM Providers - example $100/day for OpenAI, $100/day for Azure.

Quick Start​

Set provider budgets in your proxy_config.yaml file

Proxy Config setup​

model_list:
    - model_name: gpt-3.5-turbo
      litellm_params:
        model: openai/gpt-3.5-turbo
        api_key: os.environ/OPENAI_API_KEY

router_settings:
  redis_host: <your-redis-host>
  redis_password: <your-redis-password>
  redis_port: <your-redis-port>
  provider_budget_config: 
    openai: 
        budget_limit: 0.000000000001 # float of $ value budget for time period
        time_period: 1d # can be 1d, 2d, 30d 
    azure:
        budget_limit: 100
        time_period: 1d
    anthropic:
        budget_limit: 100
        time_period: 10d
    vertexai:
        budget_limit: 100
        time_period: 12d
    gemini:
        budget_limit: 100
        time_period: 12d

general_settings:
  master_key: sk-1234

Make a test request​

We expect the first request to succeed, and the second request to fail since we cross the budget for openai

Langchain, OpenAI SDK Usage Examples

curl -i http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-1234" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {"role": "user", "content": "hi my name is test request"}
    ]
  }'

How provider budget routing works​

  1. Budget Tracking:

    • Uses Redis to track spend for each provider
    • Tracks spend over specified time periods (e.g., "1d", "30d")
    • Automatically resets spend after time period expires

  2. Routing Logic:

    • Routes requests to providers under their budget limits
    • Skips providers that have exceeded their budget
    • If all providers exceed budget, raises an error

  3. Supported Time Periods:

    • Format: "Xd" where X is number of days
    • Examples: "1d" (1 day), "30d" (30 days)

  4. Requirements:

    • Redis required for tracking spend across instances
    • Provider names must be litellm provider names. See Supported Providers

Monitoring Provider Remaining Budget​

LiteLLM will emit the following metric on Prometheus to track the remaining budget for each provider

This metric indicates the remaining budget for a provider in dollars (USD)

litellm_provider_remaining_budget_metric{api_provider="openai"} 10

Spec for provider_budget_config​

The provider_budget_config is a dictionary where:

  • Key: Provider name (string) - Must be a valid LiteLLM provider name
  • Value: Budget configuration object with the following parameters:
    • budget_limit: Float value representing the budget in USD
    • time_period: String in the format "Xd" where X is the number of days (e.g., "1d", "30d")

Example structure:

provider_budget_config:
  openai:
    budget_limit: 100.0    # $100 USD
    time_period: "1d"      # 1 day period
  azure:
    budget_limit: 500.0    # $500 USD
    time_period: "30d"     # 30 day period