--- title: "Server Tuning" url: "/docs/proxy/server_tuning" canonical_url: "https://docs.litellm.ai/docs/proxy/server_tuning" type: "docs" last_updated: "2026-10-06" summary: "Optional deep tuning for the LiteLLM Proxy container; alternative ASGI servers, worker recycling, hitless restarts, TLS, keepalive, and loading config from object storage." related: - "/docs/proxy/rust_gateway" - "/docs/proxy/high_throughput" --- # Server Tuning > Index of all LiteLLM docs: https://docs.litellm.ai/llms.txt Reach for this page only when the defaults are not enough. Most deployments should run one Uvicorn worker per pod and scale horizontally, as described in the [production checklist](./prod.md#sizing-and-workers). The options below matter when you pack multiple workers into one container, terminate TLS at the proxy, serve HTTP/2, or cannot mount a config file on your host. See the [CLI reference](./cli.md) for every flag. ## Uvicorn vs. Gunicorn LiteLLM Proxy runs on [Uvicorn](https://uvicorn.dev/) by default. Passing `--run_gunicorn` instead starts [Gunicorn](https://gunicorn.org/) as a process manager that supervises [Uvicorn worker processes](https://uvicorn.dev/deployment/#gunicorn) (`uvicorn.workers.UvicornWorker`). In both cases your application code still runs on Uvicorn; the difference is which process manages and recycles the workers. | | Uvicorn (default) | Gunicorn (`--run_gunicorn`) | |---|---|---| | **When to use** | Recommended for almost all deployments, especially Kubernetes with one worker per pod. | Choose when you run **multiple workers in a single container** and want a mature process manager to supervise and recycle them. | | **Worker recycling** | Uvicorn's [`limit_max_requests`](https://uvicorn.dev/settings/#resource-limits). | Gunicorn's [`max_requests`](https://gunicorn.org/reference/settings/#max_requests), the battle-tested mechanism Gunicorn has shipped for years. | | **Process supervision** | Uvicorn's built-in multiprocess manager. | Gunicorn's [arbiter](https://gunicorn.org/design/#arbiter), which restarts workers one at a time as they exit. | :::tip[Recommendation] On Kubernetes, run **one Uvicorn worker per pod** and scale **horizontally** (more pods) rather than vertically (more workers per pod). One process per pod keeps latency predictable under load, lets the Horizontal Pod Autoscaler use the [thresholds in the production checklist](./prod.md#autoscaling) accurately, and makes rolling restarts hitless because Kubernetes drains one pod at a time. Reach for Gunicorn only when you must pack multiple workers into one container. ::: ### Recycle workers If you observe gradual memory growth under sustained load, recycle each worker after a fixed number of requests to bound memory usage. `--max_requests_before_restart` maps to Uvicorn's [`limit_max_requests`](https://uvicorn.dev/settings/#resource-limits) (default server) and to Gunicorn's [`max_requests`](https://gunicorn.org/reference/settings/#max_requests) under `--run_gunicorn`. Configure it via CLI flag or environment variable: ```shell # CLI CMD ["--port", "4000", "--config", "./proxy_server_config.yaml", "--num_workers", "1", "--max_requests_before_restart", "10000"] # or ENV (for deployment manifests / containers) export MAX_REQUESTS_BEFORE_RESTART=10000 ``` :::tip When you run **multiple workers in one container** and rely on `--max_requests_before_restart`, prefer `--run_gunicorn`. Gunicorn's [`max_requests`](https://gunicorn.org/reference/settings/#max_requests) recycling is more mature than Uvicorn's, and its [arbiter](https://gunicorn.org/design/#arbiter) restarts workers one at a time so the pod keeps serving traffic while a worker is replaced. ::: ```shell # Multiple workers in one container, with Gunicorn-managed recycling CMD ["--port", "4000", "--config", "./proxy_server_config.yaml", "--num_workers", "4", "--run_gunicorn", "--max_requests_before_restart", "10000"] ``` When several workers boot together and serve a similar amount of traffic, they reach the request threshold at almost the same time and recycle in lockstep, dropping a chunk of capacity at once. Add `--max_requests_before_restart_jitter` to offset each worker's threshold by a random amount in `[0, jitter]` so restarts stagger instead of synchronizing. It maps to Uvicorn's [`limit_max_requests_jitter`](https://uvicorn.dev/settings/#resource-limits) (requires `uvicorn>=0.41.0`) and Gunicorn's [`max_requests_jitter`](https://gunicorn.org/reference/settings/#max_requests_jitter), and has no effect without `--max_requests_before_restart`. ```shell # Stagger recycling so workers don't all restart at once CMD ["--port", "4000", "--config", "./proxy_server_config.yaml", "--num_workers", "4", "--run_gunicorn", "--max_requests_before_restart", "10000", "--max_requests_before_restart_jitter", "1000"] ``` ### Keep restarts hitless A restart is "hitless" when in-flight requests finish before the process exits, so no client sees a dropped connection. Two cases matter in production: **Worker recycling (from `--max_requests_before_restart`).** Both servers stop accepting new connections on the recycled worker and let outstanding requests drain before it exits, then a replacement worker starts. Gunicorn additionally guarantees in-flight requests up to its [`graceful_timeout`](https://gunicorn.org/reference/settings/#graceful_timeout) (30s by default) on [`SIGTERM`](https://gunicorn.org/signals/). With one worker per pod, recycling briefly reduces that pod's capacity, which is why we recommend scaling horizontally so the load balancer can route around it. **Rolling deploys and pod restarts (Kubernetes).** Make restarts hitless at the orchestration layer rather than relying on the server alone: - Use a [`RollingUpdate`](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment) strategy (the Deployment default) so new pods become Ready before old pods are terminated. - Keep a [readiness probe](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/) on `/health/readiness` so Kubernetes only sends traffic to pods that can serve it, and stops routing to a pod as soon as termination begins. - Set [`terminationGracePeriodSeconds`](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination) to comfortably exceed your longest expected request (LiteLLM's request timeout defaults to 600s; see the [recommended config](./prod.md#set-a-request-timeout)). On termination Kubernetes sends `SIGTERM`, and both Uvicorn and Gunicorn shut down [gracefully](https://uvicorn.dev/deployment/) by draining in-flight requests before exiting. - Optionally add a small [`preStop` hook](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) (for example `sleep 5`) to give the load balancer time to deregister the pod before the server begins shutting down, eliminating the brief window where traffic can still arrive at a terminating pod. ```yaml title="Kubernetes Deployment snippet for hitless rolling restarts" spec: strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 0 # never drop below desired replica count maxSurge: 1 # add one new pod at a time template: spec: terminationGracePeriodSeconds: 620 # > your longest request (request_timeout: 600) containers: - name: litellm readinessProbe: httpGet: path: /health/readiness port: 4000 lifecycle: preStop: exec: command: ["sh", "-c", "sleep 5"] ``` ## TLS at the proxy For TLS terminated by the proxy itself (rather than your load balancer), pass the key and cert paths: ```shell docker run docker.litellm.ai/berriai/litellm:latest \ --ssl_keyfile_path ssl_test/keyfile.key \ --ssl_certfile_path ssl_test/certfile.crt ``` ## HTTP/2 with Hypercorn To serve HTTP/2, build an image with hypercorn installed and pass `--run_hypercorn`: ```shell FROM docker.litellm.ai/berriai/litellm:latest WORKDIR /app COPY config.yaml . RUN chmod +x ./docker/entrypoint.sh EXPOSE 4000/tcp RUN uv add hypercorn CMD ["--port", "4000", "--config", "config.yaml"] ``` ```shell docker run \ -v $(pwd)/proxy_config.yaml:/app/config.yaml \ -p 4000:4000 \ -e DATABASE_URL=postgresql://:@:/ \ -e LITELLM_MASTER_KEY="sk-" \ your_custom_docker_image \ --config /app/config.yaml \ --run_hypercorn ``` ## Outbound HTTP/2 to providers Available from v1.103.0. The server flags above only affect the hop from your clients to LiteLLM. Calls from LiteLLM to LLM providers use HTTP/1.1 by default because the default aiohttp transport has no HTTP/2 client. Set `http2: true` under `litellm_settings` (or the `LITELLM_HTTP2` environment variable) to have LiteLLM negotiate HTTP/2 with providers over TLS; upstreams that do not offer `h2` via ALPN fall back to HTTP/1.1 automatically, and plain `http://` upstreams stay on HTTP/1.1. ```yaml litellm_settings: http2: true ``` Enabling this routes provider traffic through httpx instead of aiohttp, which was chosen as the default for its higher HTTP/1.1 throughput. Load test with the flag on before enabling it fleet-wide. Clients you pass in yourself through `litellm.client_session` or `litellm.aclient_session` are used as is and are not switched to HTTP/2. Deployments on the `aiohttp_openai/` provider always use aiohttp and stay on HTTP/1.1; LiteLLM logs a warning if the flag is on for such a request. ## Granian ASGI server [Beta] :::info[Beta feature] `--run_granian` is in **beta**. Uvicorn is still the default server. Try Granian when you need more gateway throughput or see instability under load with uvicorn; report issues on [GitHub](https://github.com/BerriAI/litellm/issues). ::: [Granian](https://github.com/emmett-framework/granian) is a Rust-backed ASGI server. In LiteLLM benchmarks it showed a 10 to 20 RPS improvement over uvicorn with the same worker count, steadier latency under sustained load, and lower error rates (see [PR #26027](https://github.com/BerriAI/litellm/pull/26027)). Scale throughput with `--num_workers`. ```shell docker run docker.litellm.ai/berriai/litellm:latest \ --config /app/config.yaml \ --port 4000 \ --run_granian \ --num_workers 4 ``` Both `--ssl_certfile_path` and `--ssl_keyfile_path` are required when enabling TLS with Granian. Not supported with Granian: `--max_requests_before_restart` (use Gunicorn for per-request worker recycling) and `--ciphers` (Hypercorn only). See [CLI server backend options](/docs/proxy/cli#server-backend-options). ## Per-worker admission control A worker whose event loop is saturated keeps accepting connections, so during a load spike callers wait for seconds with no overload signal, and the liveness probe (which runs on the same loop) slows down enough that Kubernetes restarts the pod and pushes the load onto the remaining replicas. Admission control puts a hard cap on how much work each worker process takes on and turns the excess into an explicit, fast `503` that clients can retry against. ```yaml general_settings: max_in_flight_requests_per_worker: 64 # requests being processed at once, per worker process max_queued_requests_per_worker: 64 # requests waiting for a slot; defaults to the in-flight cap admission_queue_timeout_seconds: 1.0 # a queued request is rejected after waiting this long ``` The feature is off until `max_in_flight_requests_per_worker` is set. When a request arrives and the worker has a free slot it runs immediately. Otherwise it waits in the queue until a slot frees or the timeout elapses. If the queue is already full, or the wait times out, the client gets: ``` HTTP/1.1 503 Service Unavailable retry-after: 1 {"error":{"message":"Worker at capacity: 64 in-flight, 64 queued requests. Retry later.","type":"overloaded_error","code":"503"}} ``` A slot is held for the whole response, so a streaming completion counts as one in-flight request until its last chunk is sent, and a client disconnect (queued or in flight) releases the slot immediately. The probe and metrics paths (`/health/liveliness`, `/health/liveness`, `/health/readiness`, `/health/readiness/details`, `/health/backlog`, `/health/drain`, `/metrics`) bypass the gate, so an overloaded worker still answers its liveness probe quickly while a wedged process does not. Rejections happen before authentication, so they are not attributed to a key in spend logs. The limits are read on the first request and need a restart to change. The cap is per worker process and works the same on uvicorn and Granian. A pod started with `--num_workers 4` and `max_in_flight_requests_per_worker: 64` admits up to 256 concurrent requests, and a deployment of N replicas admits N times that, so size it from the per-worker throughput you measured rather than the deployment total. It is a good fit with an HPA: replicas that are already saturated shed load with `503`s instead of accumulating latency while new replicas come up. It complements `global_max_parallel_requests`, which is a deployment-wide limit coordinated through Redis: use the global limit to bound total load on your providers and the per-worker limit to keep any single event loop from drowning without depending on Redis. Monitor it with `/health/backlog` (fields `in_flight_requests`, `admitted_requests`, `queued_requests`, `rejected_requests`) or the Prometheus metrics `litellm_admission_admitted_requests`, `litellm_admission_queued_requests`, and `litellm_admission_rejected_requests_total{reason="queue_full"|"queue_timeout"}`; see [Pod health metrics](/docs/proxy/prometheus#pod-health-metrics). A steady stream of `queue_timeout` rejections means the worker is at capacity and needs more replicas; `queue_full` rejections mean spikes are arriving faster than the queue can absorb, so raise the queue size or add capacity. ## Keepalive timeout Defaults to 5 seconds; between requests, connections must receive new data within this period or be disconnected. ```shell docker run docker.litellm.ai/berriai/litellm:latest \ --keepalive_timeout 75 ``` Or set `KEEPALIVE_TIMEOUT=75` as an env var. ## Load config.yaml from S3 or GCS Use this if you cannot mount a config file on your deployment service (AWS Fargate, Railway, etc.). LiteLLM reads `config.yaml` from the bucket at startup. **GCS Bucket** ```shell docker run --name litellm-proxy \ -e DATABASE_URL= \ -e LITELLM_CONFIG_BUCKET_TYPE="gcs" \ -e LITELLM_CONFIG_BUCKET_NAME="litellm-proxy" \ -e LITELLM_CONFIG_BUCKET_OBJECT_KEY="proxy_config.yaml" \ -p 4000:4000 \ docker.litellm.ai/berriai/litellm:latest ``` **s3** ```shell docker run --name litellm-proxy \ -e DATABASE_URL= \ -e LITELLM_CONFIG_BUCKET_NAME="litellm-proxy" \ -e LITELLM_CONFIG_BUCKET_OBJECT_KEY="litellm_proxy_config.yaml" \ -p 4000:4000 \ docker.litellm.ai/berriai/litellm:latest ``` ## Disable pulling live model prices Set `LITELLM_LOCAL_MODEL_COST_MAP="True"` to use the bundled [model prices file](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) instead of fetching it at startup, if you see long cold starts or have network egress restrictions. ## Related pages - [[Beta] Rust AI Gateway](https://docs.litellm.ai/docs/proxy/rust_gateway.md) - [Scale for high-throughput workloads](https://docs.litellm.ai/docs/proxy/high_throughput.md)