📄️ Router - Load Balancing
LiteLLM manages:
📄️ [BETA] Request Prioritization
Beta feature. Use for testing only.
📄️ Auto Routing
LiteLLM can auto select the best model for a request based on rules you define.
📄️ Proxy - Load Balancing
Load balance multiple instances of the same model
📄️ UI - Router Settings for Keys and Teams
Configure router settings at the key and team level to achieve granular control over routing behavior, fallbacks, retries, and other router configurations. This enables you to customize routing behavior for specific keys or teams without affecting global settings.
📄️ Budget Routing
LiteLLM Supports setting the following budgets:
📄️ Fallbacks
If a call fails after num_retries, fallback to another model group.
📄️ [New] Fallback Management Endpoints
Dedicated endpoints for managing model fallbacks separately from the general configuration.
📄️ Tag Based Routing
Route requests based on tags.
📄️ Timeouts
The timeout set in router is for the entire length of the call, and is passed down to the completion() call level as well.
📄️ Provider specific Wildcard routing
Proxy all models from a provider
📄️ Health Check Driven Routing
Route traffic away from unhealthy deployments before users hit errors. Background health checks run on a configurable interval, and any deployment that fails gets removed from the routing pool proactively, not after a user request already failed.