Sizing & Scaling
Size the gateway, PostgreSQL, and Redis for the same peak load.
Start with your expected request rate, token volume, and number of concurrent streams. Test with the models and response lengths your applications use. A long stream keeps a connection open even when gateway CPU use is low.
On Kubernetes, add requests per second (RPS) and tokens per second (TPS) to your HPA's CPU and memory targets. See HPA metrics.
Choose a guide
| What you need to decide | Guide |
|---|---|
| Gateway CPU, memory, and worker count | Sizing and workers |
| Database compute, storage, and connection limits | Database Sizing |
| Which features need shared state | What Needs Redis |
| Redis memory and connection capacity | Redis Sizing |
| Scale gateway replicas with RPS and TPS | Autoscaling |
| How to reduce database pressure at high traffic | High-Throughput Deployment |
| How to send database reads to a replica | Database Read Replica |
| How to tune workers, shutdown, and the server | Server Tuning |
Bound the database connections
Each worker has its own connection pool. Plan for the maximum number of replicas your autoscaler can start:
pool limit × workers per replica × maximum replicas
Leave room for management services, migrations, and other database clients. Use database sizing to set pool and replica limits before you enable autoscaling.
