Skip to main content

Database Sizing

See Production Best Practices for configuration and What is stored in the DB for database contents. Size Redis with Redis Sizing. Gateway benchmarks cover proxy throughput and latency.

Workload​

Postgres serves authentication reads and accounting writes. Authentication uses indexed reads from LiteLLM_VerificationToken, LiteLLM_UserTable, and LiteLLM_TeamTable. Caching reduces repeat reads.

Accounting updates key, user, team, and organization spend. Per-request logs go into LiteLLM_SpendLogs. These writes drive database capacity, so enable batched spend writes and use the Redis transaction buffer above roughly 1K RPS.

Sizing by request rate​

These instance sizes are planning estimates. The 10K–20K tier extrapolates the 5K baseline with unchanged logging, batching, and query mix. Validate it with a load test. These instance families jump from 16 to 32 vCPU, so 10K–20K uses one tier.

Size storage from retention and connections from the worker pool budget below. A read replica can serve supported read traffic, but it does not remove write contention on the primary.

AWS RDS for PostgreSQL​

Sustained RPSInstance typevCPURAM
Up to 1Kdb.m7g.xlarge416 GiB
1K to 5Kdb.m7g.2xlarge832 GiB
5K to 10Kdb.m7g.4xlarge1664 GiB
10K to 20Kdb.m7g.8xlarge32128 GiB

See RDS instance specifications. Use gp3 storage, Multi-AZ, and Database Insights.

Google Cloud SQL for PostgreSQL​

Sustained RPSInstance type (Enterprise Plus)vCPURAM
Up to 1Kdb-perf-optimized-N-4432 GB
1K to 5Kdb-perf-optimized-N-8864 GB
5K to 10Kdb-perf-optimized-N-1616128 GB
10K to 20Kdb-perf-optimized-N-3232256 GB

See Cloud SQL machine specifications. Enable regional high availability and automatic storage increase. SSD throughput depends on both disk size and vCPU limits. Check the storage limits.

Azure Database for PostgreSQL​

Sustained RPSInstance type (General Purpose)vCoreRAM
Up to 1KD4ds_v5416 GiB
1K to 5KD8ds_v5832 GiB
5K to 10KD16ds_v51664 GiB
10K to 20KD32ds_v532128 GiB

See Azure compute options. Use Premium SSD v2 and zone-redundant high availability. The built-in PgBouncer provides connection pooling on port 6432.

Capacity planning​

Connection budgets​

A full connection pool can cause FATAL: sorry, too many clients already before CPU reaches its limit. For workers that connect directly to Postgres, include both the autoscaler's maximum and extra instances alive during a rollout:

direct connections = database_connection_pool_limit × workers_per_instance × (maximum_instances + rollout_overlap)

At a pool limit of 10, one worker per instance, 100 maximum instances, and 25 extra rollout instances, the gateway can open 1,250 connections. Reserve capacity for backend services, migrations, and operators. Set the pool with the connection limit configuration.

When you enable database.connectionPool in Helm, gateway workers share one PgBouncer pool per pod. The gateway's Postgres connection budget becomes maxDbConnections × (maximum_pods + rollout_overlap). Count backend and migration connections separately because they connect directly.

CloudConnection limitReference
AWS RDS and AuroraThe default formula is LEAST({DBInstanceClassMemory/9531392}, 5000). Check the effective parameter for your instance.RDS parameter defaults
GCP Cloud SQLThe default derives from instance memory. It adjusts after a resize unless you set the flag yourself.Cloud SQL flags
Azure flexible serverD4ds_v5 allows 1,718 total connections and 1,703 application connections. D8ds_v5 allows 3,437 total and 3,422 application connections.Azure limits

If application demand exceeds the database connection budget, use RDS Proxy, Azure's built-in PgBouncer, or another pooler. Set database_disable_prepared_statements: true for pooled sessions.

Connections also stay busy while transactions wait for shared spend rows. LiteLLM applies SPEND_ROLLUP_LOCK_TIMEOUT_MS, which defaults to 5,000 ms, to spend rollup transactions. Postgres cancels a statement that exceeds this wait. LiteLLM requeues its rows for the next flush and releases the connection. To inspect lock waits, run:

SELECT query, count(*)
FROM pg_stat_activity
WHERE wait_event_type = 'Lock'
GROUP BY query

Storage and write throughput​

Storage follows request log volume and retention. Start with an estimate of 1–2 KB per logged request without prompts, then measure actual row and index sizes. store_prompts_in_spend_logs can increase this by roughly 10 times.

log data bytes = logged requests per second × 86400 × retention days × bytes per request
Sustained logged RPSLog data per day at 1–2 KB per request
10K0.864–1.728 TB
15K1.296–2.592 TB
20K1.728–3.456 TB

These decimal TB estimates exclude indexes, WAL, and free space for maintenance. Multiply by retention days and add those costs.

Use these starting points for write tests, then increase storage to cover retention. Measure write latency and IOPS at 10K, 15K, and 20K because batching changes the write rate.

Sustained RPSSSD starting capacityIOPS starting point
Up to 1K200 GB3,000
1K to 5K500 GB5,000
5K to 20K1 TB10,000

Keep the database small​

Batch spend writes with proxy_batch_write_at: 60. Keep error logs out of the database and set a spend log retention period. If you do not need per-request rows in the UI, set disable_spend_logs: True and keep cost data in your logging integrations. Enable store_prompts_in_spend_logs only when you need prompt history because it also increases gateway memory use.

What to monitor​

Watch connection count against max_connections, write latency, lock waits, and storage growth. Check litellm_in_memory_spend_update_queue_size and litellm_redis_spend_update_queue_size for queued spend updates. Use litellm_pod_lock_manager_size to check the transaction buffer flush lock.