Reference Architectures

Reference designs for cloud infrastructure, client access, and availability.

Choose a topic

Cloud

Choose your infrastructure

Cloud Deployments: Azure, AWS, and GCP diagrams with network boundaries, ingress, data stores, and provider egress.

Identity-Aware Proxy

Check identity before traffic reaches LiteLLM

Identity-Aware Proxy: authenticate callers with Cloudflare Access, AWS ALB, Microsoft Entra Application Proxy, or Google Cloud IAP. Forward verified identity to LiteLLM for user and team permissions.

mTLS

Require a client certificate

Client Authentication with mTLS: validate client certificates at ingress and forward the workload identity to LiteLLM. Use virtual keys for model access and budgets. Includes load balancer and API gateway configurations.

High Availability

Keep serving through failures and releases

Plan replicas, availability zones, and data-store availability in High Availability. Use Blue/Green Deployments for a controlled cutover or Canary Deployments for a gradual release.

For more than one region, see Multi-Region Deployment, Routing, and Failover.

Control Plane

Manage independent deployments

The Global Control Plane manages independent deployments from one UI. Each deployment has its own database, Redis, keys, and budgets. This Enterprise feature does not synchronize state or route inference traffic between deployments.