Skip to main content

Benchmarks

Performance test results for LiteLLM Gateway.

High-throughput performance test results​

13.5 billion tokens per minute (TPM).

MetricResult
Requests per second3,000
Client HTTP 200 rate100.00%
Gateway request latency, p5030.581 ms
Gateway request latency, p9554.029 ms
Gateway request latency, p9991.645 ms
Client time to first token, p5031.667 ms

Methodology​

SettingConfiguration
Deployment33 gateway pods; 4 workers, 4 vCPUs, and 16 GiB memory requested per pod
TrafficDistributed Locust with 30 load-generator workers; 3,000 users at one request per second each
Requests/v1/chat/completions; equal shares of 50K, 75K, and 100K-token prompts; 50% streaming; max_tokens: 16
Model and networkIn-process mock model; public AWS Application Load Balancer; 60-second client timeout
ConfigurationHigh-throughput Helm values; PostgreSQL and Redis; virtual-key budgets, token counting, spend tracking, and metrics; response caching disabled
Duration24 minutes 22 seconds
MeasurementPrometheus: throughput and gateway latency. Locust: client success and time to first token, including upload and load-balancer time

Realtime API performance test results​

1,207 requests per second.

MetricResult
End-to-end latency, p5059 ms
End-to-end latency, p9567 ms
End-to-end latency, p9999 ms
Average end-to-end latency63 ms

Methodology​

SettingConfiguration
Deployment4 gateway instances; 4 workers, 4 vCPUs, and 8 GB RAM per instance
TrafficLocust; 1,000 users with a 0.5 to 1 second pause between requests
Requests/v1/realtime over persistent WebSockets; text requests for a two-sentence story with a random ID
Model and networkMock realtime endpoint
ConfigurationPostgreSQL; Redis disabled
DurationNot recorded
MeasurementLocust script: requests per second and latency through response.done, including connection setup when needed

Short-prompt chat performance test results​

1,170 requests per second with 8 ms p95 gateway overhead across four instances.

Metric2 instances4 instances
Requests per second1,035.71,170
Request latency, p50200 ms100 ms
Request latency, p95630 ms150 ms
Request latency, p991,200 ms240 ms
Average request latency262.46 ms111.73 ms
Gateway overhead, p5012 ms2 ms
Gateway overhead, p9529 ms8 ms
Gateway overhead, p9943 ms13 ms
Average gateway overhead14.74 ms3.32 ms

Methodology​

SettingConfiguration
Deployment2 or 4 instances; 4 vCPUs and 8 GB RAM per instance. The 4-instance test uses 4 workers per instance; the 2-instance worker count is not recorded
TrafficLocust; 1,000 users with a 0.5 to 1 second pause between requests
Requests/chat/completions; the 4-instance script sends text with a random ID repeated 150 times. The 2-instance payload is not recorded
Model and networkMock OpenAI endpoint
ConfigurationPostgreSQL; Redis disabled
DurationNot recorded
MeasurementLocust: request throughput and full request latency. Gateway overhead: x-litellm-overhead-duration-ms response header

Logging callback performance test results​

ConfigurationRequests per secondMedian request latency
Base proxy1,133.2140 ms
GCS bucket logging1,137.3138 ms
LangSmith logging1,135132 ms

Methodology​

SettingConfiguration
DeploymentNot recorded
TrafficNot recorded
RequestsNot recorded
Model and networkNot recorded
ConfigurationBase proxy compared with GCS bucket logging and LangSmith logging
DurationNot recorded
MeasurementRequests per second and median request latency

Per-pod benchmarks​

Plan for approximately 100 combined RPS per four-worker pod on AWS c7a.2xlarge for latency headroom with this short-response workload. Three runs on the same pod and node completed 134,992 measured requests with zero failures, with 80% regular and 20% streaming traffic.

Combined RPS per podRegular overhead p99Streaming overhead p99
10047 ms45 ms
15091 ms93 ms
200651 ms645 ms

P99 overhead was near the 100-ms target at 150 RPS and exceeded it at 200 RPS. At 100 RPS, p99 time to first token was 264 ms, including the mock provider's 200-ms delay.

Methodology​

SettingConfiguration
Version and machineLiteLLM v1.110.0; November 20, 2026 (UTC); dedicated c7a.2xlarge node, 8 vCPUs and 16 GiB RAM, AWS EKS us-west-1a
PodSame pod and node across all three runs; four workers; gateway requests 4 vCPUs and 8 GiB RAM, with limits of 8 vCPUs and 8 GiB; collector and metrics sidecars
AuthenticationSame stored LiteLLM virtual key for all load requests; master key used only to create it before testing
Workload80% regular / 20% streaming; small unique prompts, max_tokens: 64; a separate network mock waits 200 ms and returns 37 content chunks, 15 ms apart for streams
Load and durationIn-cluster Locust FastHttpUser; 100, 150 and 200 RPS; 60 seconds of warmup and five minutes of measurement per run. Startup offsets remain in effect after pacing starts; offered arrivals checked in 100-ms bins; complete streams verified
MeasurementGateway overhead uses the x-litellm-overhead-duration-ms response header and excludes the full streaming duration
Services and scopePostgreSQL, Redis and mock provider on a separate node in the same Availability Zone; PgBouncer, collector, prompt storage, spend tracking and Prometheus enabled. Direct Kubernetes Service traffic