Skip to main content

Production Deployment

Production deployment guide for AWS, Google Cloud, Azure, or any Kubernetes cluster. For a first deployment on a single machine, start with the Quickstart; this page picks up where it ends.

There are two supported paths. If you run Kubernetes, deploy with Helm on EKS, GKE, or AKS; the install is the same on every cloud, only the data stores and ingress differ. If you do not run Kubernetes, AWS and GCP have official Terraform modules that stand up the entire stack; Azure has no Terraform module, so AKS with Helm is the supported path there.

Architecture​

Clients (OpenAI SDK, LangChain, curl)
Application Load BalancerHTTPS, path-based routing
ECS Fargategateway :4000, backend :4001, ui :3000
gatewaybackendui
Aurora PostgreSQLwriter + reader, IAM auth
ElastiCache Redismulti-AZ, TLS
Secrets Managermaster key, provider keys

LiteLLM provides two deployment modes:

  • Monolithic: one litellm image serves LLM traffic, management APIs, and the UI. This is what the litellm-helm chart runs, and the simplest to operate.
  • Microservices: a gateway (LLM traffic, port 4000), backend (management APIs and UI backend, port 4001), and ui (port 3000), each deployed and scaled independently. This is what the componentized litellm chart and both Terraform modules run; see the chart values for the full reference.

The supporting infrastructure is identical in either mode:

ComponentPurposeNotes
LiteLLM servicesOne proxy deployment (monolithic) or gateway + backend + ui (microservices)Stateless; run 2+ replicas behind a load balancer
PostgreSQLKeys, teams, users, spend logs, configRequired for the proxy's auth and tracking features
RedisRate limiting, router state, caching across instancesRequired once you run more than one instance
Migrations jobApplies schema migrations against PostgresRuns once per upgrade; proxy instances set DISABLE_SCHEMA_UPDATE=true

Core configuration​

DATABASE_URL="postgresql://user:password@host:5432/litellm"
LITELLM_MASTER_KEY="sk-..."   # admin key for the proxy
LITELLM_SALT_KEY="sk-..."     # encrypts provider credentials stored in the DB. Set once, never change it
DISABLE_SCHEMA_UPDATE="true"  # proxy instances never run migrations; the migrations job does
STORE_MODEL_IN_DB="True"      # manage models from the Admin UI instead of config files

LITELLM_SALT_KEY cannot be rotated after you add models: it encrypts the provider credentials stored in your database, and changing it makes them unreadable. Generate a strong random value and store both keys in your cloud's secret manager.

Official images are published to ghcr.io/berriai and mirrored at docker.litellm.ai/berriai. Use ghcr.io/berriai/litellm for monolithic deployments, including those with Postgres, since it bundles the Prisma toolchain, and pin a version tag rather than latest or a moving tag, so rollbacks are deterministic. All images are signed; see the Docker Image Security Guide for verification and the non-root variant.

Provision the data stores​

The Helm path needs a PostgreSQL database and a Redis reachable from your cluster. Use the managed services:

Provision RDS PostgreSQL and ElastiCache Redis in the same VPC as your EKS cluster, with security groups permitting the cluster's nodes on ports 5432 and 6379.

Deploy with Helm​

First create the secrets both charts consume:

kubectl create secret generic litellm-masterkey \
  --from-literal=masterkey="sk-$(openssl rand -hex 24)"

kubectl create secret generic litellm-db \
  --from-literal=username=litellm \
  --from-literal=password="<database-password>"

kubectl create secret generic litellm-env \
  --from-literal=LITELLM_SALT_KEY="sk-$(openssl rand -hex 24)" \
  --from-literal=REDIS_PASSWORD="<redis-password>" \
  --from-literal=OPENAI_API_KEY="<provider-key>"

Then pick a deployment mode:

values.yaml
replicaCount: 3

image:
  repository: ghcr.io/berriai/litellm
  tag: "v1.90.2"          # pin your version

masterkeySecretName: litellm-masterkey
masterkeySecretKey: masterkey

db:
  useExisting: true
  deployStandalone: false
  endpoint: "<postgres-endpoint>"
  database: litellm
  secret:
    name: litellm-db
    usernameKey: username
    passwordKey: password

environmentSecrets:
  - litellm-env

proxy_config:
  model_list:
    - model_name: gpt-5.6-terra
      litellm_params:
        model: openai/gpt-5.6-terra
        api_key: os.environ/OPENAI_API_KEY
  router_settings:
    redis_host: "<redis-endpoint>"
    redis_port: 6379
    redis_password: os.environ/REDIS_PASSWORD
helm install litellm oci://ghcr.io/berriai/litellm-helm -f values.yaml

The chart lives at helm/litellm-helm; the published chart versions carry LiteLLM release numbers (for example 1.90.2), and helm show values oci://ghcr.io/berriai/litellm-helm lists every knob. Beyond the values above it supports autoscaling (autoscaling.* or keda.*), PodDisruptionBudgets (pdb.*), a Prometheus ServiceMonitor (serviceMonitor.*), read replica routing (db.readReplicaUrl, see Database Read Replica), graceful drain on shutdown (lifecycle), ArgoCD or Helm hooks for the migrations job (migrationJob.hooks.*, see Helm PreSync hooks), and an optional dedicated Prometheus metrics listener configured through metricsServer.*.

Both charts run the migrations job automatically and keep DISABLE_SCHEMA_UPDATE=true on the proxy pods. Expose the service through your cloud's ingress: the AWS Load Balancer Controller on EKS, GKE Ingress on GKE, or Application Gateway Ingress (AGIC) on AKS, with health checks on /health/readiness, then point your DNS record at the resulting load balancer. For secrets, prefer your cloud's secret manager over plain Kubernetes secrets (Key Vault CSI driver on AKS, for example); the charts consume whatever secret you mount.

Autoscaling​

Both charts can scale themselves, and both ship autoscaling off or conservative by default. For the thresholds to aim at, and why memory is not one of them, see autoscaling in the production checklist.

litellm-helm offers two mutually exclusive mechanisms. autoscaling.* renders a standard HorizontalPodAutoscaler, and keda.* renders a KEDA ScaledObject for scaling on queue depth, Prometheus queries, or anything else KEDA can read. Enabling both renders only the HPA, so pick one.

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage and behavior are also accepted

keda:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  pollingInterval: 30   # seconds between trigger evaluations
  cooldownPeriod: 300   # seconds of quiet before scaling back to minReplicas
  triggers: []          # required; a ScaledObject with no triggers will not scale

keda.triggers is empty by default and has no useful default, so supply the trigger yourself; the chart's values.yaml carries a commented Prometheus example. keda.fallback, keda.behavior, and keda.restoreToOriginalReplicaCount are passed through for controlling what happens when the metric source is unavailable and how replicas settle after a scale event.

The componentized chart scales each component on its own, under gateway.hpa, backend.hpa, and ui.hpa. The gateway and backend autoscale out of the box and the UI does not, with maximums sized to the shape of each component's traffic: the gateway defaults to maxReplicas: 10 at 70 percent CPU and 80 percent memory, the backend to maxReplicas: 4 at 70 percent CPU, and the UI to maxReplicas: 3 at 80 percent CPU with enabled: false. Raising the gateway's ceiling is usually all you need, since it is the only component that sees LLM traffic.

Whichever mechanism you use, set the maximum against what your database can serve. The connection pool is per worker, so the ceiling on replicas is also a ceiling on Postgres connections; litellm-helm defaults maxReplicas to 100, which at the default pool limit of 10 asks for roughly 1000 connections at full scale-out. See bounding database connections.

Scale on requests and tokens per pod​

CPU lags LLM traffic: a pod streaming forty responses is mostly waiting on providers, so its CPU stays low while its capacity is gone. Both charts can scale on the two counters the proxy already exports, litellm_proxy_total_requests_metric_total and litellm_total_tokens_metric_total, expressed as requests per second (RPS) and tokens per second (TPS) per pod, the way load is usually quoted (1k rps, 75M tok/s). The targets are opt-in and empty by default, so nothing changes until you set one, and they sit next to the CPU and memory targets: an HPA follows whichever metric asks for the most replicas. averageValue is a Kubernetes quantity, so "6M" and "6000000" are the same tokens per second.

# litellm-helm
autoscaling:
  enabled: true
  targetRequestsPerSecond: "90"
  targetTokensPerSecond: "6M"
metricsServer:
  enabled: true
serviceMonitor:
  enabled: true

# componentized chart
gateway:
  metricsServer:
    enabled: true
  serviceMonitor:
    enabled: true
  hpa:
    targetRequestsPerSecond: "90"
    targetTokensPerSecond: "6M"

Each target renders an autoscaling/v2 Pods metric, litellm_requests_per_second or litellm_tokens_per_second, with an AverageValue target. Kubernetes cannot read Prometheus by itself, so two things have to be in place. The chart's ServiceMonitor (Prometheus Operator) scrapes every pod on its own so each sample carries the pod label. Turn on the dedicated metrics listener with it (gateway.metricsServer.enabled on the componentized chart, metricsServer.enabled on litellm-helm): the main port serves /metrics/ behind virtual-key auth and answers an unauthenticated scrape with 401, so the componentized chart refuses to render a ServiceMonitor without it. Then a Prometheus Adapter has to serve those two names on custom.metrics.k8s.io, grouped by pod. rate() already returns a per-second value, so there is no * 60:

rules:
  - seriesQuery: 'litellm_proxy_total_requests_metric_total{namespace!="",pod!=""}'
    resources: { overrides: { namespace: { resource: namespace }, pod: { resource: pod } } }
    name: { as: litellm_requests_per_second }
    metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
  - seriesQuery: 'litellm_total_tokens_metric_total{namespace!="",pod!=""}'
    resources: { overrides: { namespace: { resource: namespace }, pod: { resource: pod } } }
    name: { as: litellm_tokens_per_second }
    metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)

The unit is a constant factor and does not make the HPA react any faster. What sets the lag is the rate() window in the adapter rule, the scrape interval, and the HPA sync period (15s by default). Keep the window at [1m] and the ServiceMonitor interval at the chart default of 15s or faster so the window always holds at least four samples: after a traffic step the signal moves on the next scrape and reaches its full value 60s later, where a [2m] window is still at half.

Both counters are split by model, key, and team labels, so the sum by (pod) folds a pod's series into one number. kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/<ns>/pods/*/litellm_tokens_per_second shows what the HPA sees. Worked example: 1,000 rps across 10 gateway pods is 100 rps per pod against a target of 90, so the HPA asks for ceil(10 * 100 / 90) = 12 replicas. The token metric does the same arithmetic: ten pods serving 70,000,000 tokens per second between them average 7,000,000 TPS against a target of 6,000,000, so ceil(10 * 7000000 / 6000000) = 12.

Tokens are counted when a response finishes, so a long stream shows up in TPS only once it completes. RPS reacts first and TPS catches up, which is fine for scale-out but means a burst of long streams is under-counted for as long as they run. Do not set a TPS target alone if your traffic is dominated by multi-minute streams.

On litellm-helm the same signals are available through KEDA without an adapter. keda.prometheus.requestsPerSecond and keda.prometheus.tokensPerSecond are the load one replica should carry, and each adds a Prometheus trigger on sum(rate(<counter>{namespace="<release namespace>",job="<release>-metrics"}[1m])), selected by release namespace and the job label the chart's ServiceMonitor produces. KEDA divides the release-wide rate by the per-replica threshold to pick the replica count, so the result matches the HPA path for the same traffic. Keep keda.pollingInterval at 15s or lower for the same reason as the scrape interval above.

keda:
  enabled: true
  prometheus:
    serverAddress: http://prometheus-operated.monitoring.svc:9090
    requestsPerSecond: "90"
    tokensPerSecond: "6000000"
metricsServer:
  enabled: true
serviceMonitor:
  enabled: true

On AWS ECS the Terraform module takes the same per-second inputs, gateway_target_requests_per_second on ALBRequestCountPerTarget and gateway_target_tokens_per_second once you publish the token counter to CloudWatch, and converts them itself because the ALB publishes a per-minute count. CloudWatch target tracking aggregates every metric over 60-second periods with no period setting, so ECS reacts on a roughly one-minute cadence whatever the unit; the module README covers both policies. Cloud Run scales on request concurrency and has no custom-metric input, so there is no TPS path on GCP outside GKE.

Kubernetes without Helm​

If you manage raw manifests, the equivalent deployment is a ConfigMap for config.yaml, a Secret for keys, a Deployment with health probes, and a Service.

Full manifest (ConfigMap, Secret, Deployment, Service)
apiVersion: v1
kind: ConfigMap
metadata:
  name: litellm-config-file
data:
  config.yaml: |
      model_list:
        - model_name: gpt-5.6-terra
          litellm_params:
            model: openai/gpt-5.6-terra
            api_key: os.environ/OPENAI_API_KEY
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: litellm-secrets
data:
  OPENAI_API_KEY: bWVvd19pbV9hX2NhdA== # your api key in base64
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-deployment
  labels:
    app: litellm
spec:
  replicas: 2
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      containers:
      - name: litellm
        image: docker.litellm.ai/berriai/litellm:v1.90.2 # pin a version, do not use :latest
        args:
          - "--config"
          - "/app/proxy_server_config.yaml"
        ports:
        - containerPort: 4000
        volumeMounts:
        - name: config-volume
          mountPath: /app/proxy_server_config.yaml
          subPath: config.yaml
        envFrom:
        - secretRef:
            name: litellm-secrets
        livenessProbe:
          httpGet:
            path: /health/liveliness
            port: 4000
          initialDelaySeconds: 120
          periodSeconds: 15
        readinessProbe:
          httpGet:
            path: /health/readiness
            port: 4000
          initialDelaySeconds: 120
          periodSeconds: 15
      volumes:
        - name: config-volume
          configMap:
            name: litellm-config-file
---
apiVersion: v1
kind: Service
metadata:
  name: litellm-service
spec:
  selector:
    app: litellm
  ports:
    - protocol: TCP
      port: 4000
      targetPort: 4000
  type: NodePort

To connect the database, add DATABASE_URL and LITELLM_MASTER_KEY to the Secret; nothing else in the manifest changes, because the image already carries the Prisma toolchain.

Deploy with Terraform (AWS and GCP)​

The official modules deploy the full microservices stack (network, database, Redis, object storage, secrets, compute, load balancer, and a migrations job that runs before the services start) and are published to the Terraform Registry:

By default, provisions a VPC with public and private subnets, an Aurora PostgreSQL cluster (writer plus reader, IAM database auth), ElastiCache Redis (multi-AZ, encrypted), an S3 bucket, Secrets Manager entries, an Application Load Balancer, and ECS Fargate services. The networking and both data stores are optional, so you can reuse what your account already runs; see below.

main.tf
module "litellm" {
  source  = "BerriAI/litellm/aws"
  version = "~> 1.90"

  region = "us-east-1"
  azs    = ["us-east-1a", "us-east-1b"]
  tenant = "acme"
  env    = "prod"

  ui_password         = var.ui_password
  litellm_license     = var.litellm_license      # optional, omit for open source
  acm_certificate_arn = var.acm_certificate_arn  # TLS is required by default

  proxy_config = {
    model_list = [{
      model_name = "gpt-5.6-terra"
      litellm_params = {
        model   = "openai/gpt-5.6-terra"
        api_key = "os.environ/OPENAI_API_KEY"
      }
    }]
  }
  gateway_extra_secrets = {
    OPENAI_API_KEY = var.openai_key_secret_arn
  }
}

Before you apply: provision the TLS certificate in AWS Certificate Manager (the module refuses a plaintext ALB unless you explicitly set allow_plaintext_alb = true), and create any provider-key secrets in Secrets Manager first, since gateway_extra_secrets takes their ARNs. After apply, point your DNS record at the ALB hostname.

The module auto-generates the master key into Secrets Manager if you do not supply one. The application connects to Aurora with short-lived IAM tokens, so its DATABASE_URL carries no password (the database master password itself is generated into Secrets Manager and never touches the application). Every resource is named <tenant>-litellm-<env>, and the module declares no provider, so you can for_each it to run one stack per tenant.

Bringing your own VPC, database, or Redis. The networking and both data stores are each optional, so you can deploy into infrastructure your account already has. This is the path to take when your guardrails only allow workloads inside a pre-approved VPC, or when a separate team owns the Postgres and Redis you are expected to use. Set only the pieces you want to reuse; anything you leave at its default is still created for you.

main.tf
module "litellm" {
  source  = "BerriAI/litellm/aws"
  version = "~> 1.90"

  region = "us-east-1"
  tenant = "acme"
  env    = "prod"

  # Existing networking. Drop `azs` when you set these: no VPC, subnets,
  # route tables, internet gateway, or NAT gateway are created.
  vpc_id             = "vpc-0123456789abcdef0"
  public_subnet_ids  = ["subnet-aaa", "subnet-bbb"]  # ALB, 2+ AZs
  private_subnet_ids = ["subnet-ccc", "subnet-ddd"]  # tasks and data stores, 2+ AZs

  # Existing data stores. Each URL is stored in Secrets Manager and reaches
  # the containers as DATABASE_URL / REDIS_URL.
  create_database = false
  database_url    = var.database_url
  create_redis    = false
  redis_url       = var.redis_url

  # Attach a group your database already allows, alongside the module's own.
  additional_task_security_group_ids = ["sg-0123456789abcdef0"]
}

Your existing stores have to accept traffic from the tasks. The module always creates its own tasks security group and reports it as the task_security_group_id output, so either allow that group inbound on the database and Redis, or pass a group they already allow through additional_task_security_group_ids. Private subnets you supply also need their own egress, through a NAT gateway or VPC endpoints, since the module creates no routing of its own in this mode. It reaches Secrets Manager, pulls container images, and calls LLM providers from those subnets.

Supply private subnets in at least two availability zones whenever the module still creates Aurora or ElastiCache, since both of their subnet groups require it. One private subnet is accepted only when you have turned both stores off.

Leaving a create_* at false with an empty URL runs the stack without that component entirely. With no database there is no key management, spend tracking, or UI persistence, so authentication falls back to the master key alone. Without Redis, rate limits, budgets, and router cooldowns are counted per gateway process rather than across the cluster, and the module runs two gateway tasks by default and autoscales to ten, so a caller spread across them receives each process's full allowance. The plan warns when you configure that combination. Hold the gateway to a single process with gateway_autoscaling_enabled = false, gateway_desired_count = 1, and gateway_num_workers = 1 if you need per-key limits to mean anything without Redis.

Missing or inconsistent inputs fail during terraform plan rather than halfway through an apply, so setting vpc_id without the subnet ids, or dropping azs without setting vpc_id, tells you so before anything is created.

The AWS module can run Prometheus collection in a dedicated ECS sidecar using gateway_metrics_port and restrict access with gateway_metrics_scrape_cidrs; see Isolate Prometheus scraping from inference traffic.

To manage LiteLLM resources (keys, teams, models) as code once the stack is up, use terraform-provider-litellm.

Other platforms​

Deploy on Render:

Verify the deployment​

Confirm the proxy is up and can reach its database:

curl -s https://llm.example.com/health/readiness

Then open the Admin UI at https://llm.example.com/ui, log in with your master key, add a model, create a virtual key, and send a Playground message; a response proves the full path through the load balancer, proxy, database, and provider credentials. The Quickstart walks through each of those clicks with screenshots; the flow is identical on a production deployment.

Next steps​

Use the production checklist to configure workers, resources, Redis, graceful degradation, and server tuning. Verify images with the Docker Image Security Guide. Add regions with Multi-Region Deployment. For workloads above 1,000 RPS, enable the Redis transaction buffer and evaluate the high-throughput deployment profile.

🚅
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails — built for production.
Learn more →