How Pfizer Improved LiteLLM Gateway Performance and Resiliency at Scale

A joint debugging story with LiteLLM, and the release testing that comes next.

A LiteLLM version bump exposed a long-standing Redis configuration bug that cut Pfizer's gateway throughput by ~48%, with zero HTTP errors in the application logs. Here's how the team isolated it through configuration bisection, and the testing infrastructure both teams are building to catch this class of regression before it ships.

---

~48%

throughput drop caught in CI (≈300 → 156 RPS)

76%

faster CI latency after the fix (5,000ms → 1,200ms median)

1 line

the Redis conditional at the root of it

134

issues addressed in a two-week stability push (LiteLLM)

The regression

Slower, with no HTTP errors in the application logs

The hardest regressions to catch are the ones that produce zero errors at the HTTP layer: no failures, no single slow request, just less work done per second under concurrency.

Pfizer runs LiteLLM as a self-hosted AI gateway - a single OpenAI-compatible API in front of a multi-provider model catalog, serving every team, application, and agent at Pfizer with uniform access to external and self-hosted models. One shared entry point means a regression in the gateway propagates to every consumer behind it at once, so every change, whether an upstream LiteLLM bump or an internal commit, runs through automated load and regression tests in CI before it ships.

That is what caught this one. CI throughput held steady around 300 requests per second on v1.85.0 and v1.87.3, then dropped to about 156 RPS on v1.89.2. Error rate: unchanged. Per-request latency at the HTTP layer: nothing obviously wrong. The throughput drop manifested only under concurrent load - internally, Redis operations were stalling on TLS handshake timeouts in the async hot path, degrading throughput without surfacing HTTP-level errors. The underlying Redis bug existed in older versions too, but surfaced during Pfizer's v1.89.2 upgrade validation under this workload. That was enough to hold the version back from production.

A throughput drop with a 0% HTTP error rate is the worst kind of regression to catch after the fact - nothing pages on it, and it only shows up under real concurrency, not in a single request. Redis timeout errors appeared in the proxy's internal logs, but the gateway still returned HTTP 200 to every caller. Our CI load-tests every version bump and every internal commit against fixed baselines specifically so this class of problem gets caught before it reaches production traffic that's latency-sensitive across chat sessions and agent loops.

Aleksandr Liadov & Praveena Mundolimoole, Pfizer AI Platform Engineering

The hunt

Isolate first, then read the diff

Rather than bisecting commits across two minor versions, the team narrowed the surface area by configuration first.

1. Ran vanilla LiteLLM - no custom callbacks, no custom auth, mocked backend. Both v1.85.0 and v1.92.0 landed at 320-340ms median latency under concurrent load. The core proxy was clean on both versions.
2. Added Pfizer's custom components back one at a time, then all together. Still clean.
3. Enabled Redis caching. Median latency jumped to 4,200ms. That isolated it to the Redis connection path.

From there, the fix was in the diff. The connection-pool builder in litellm/_redis.py decided whether to open a TLS connection by checking whether the ssl key was present in redis_kwargs, not whether its value was true:

# before (litellm/_redis.py at 142d5aa): presence check
connection_class = async_redis.Connection
if "ssl" in redis_kwargs:            # true even when ssl: false
    connection_class = async_redis.SSLConnection
    redis_kwargs.pop("ssl", None)
    redis_kwargs["connection_class"] = connection_class

# after (PR #32590): value check
if redis_kwargs.pop("ssl", None):        # only when ssl is truthy
    redis_kwargs["connection_class"] = async_redis.SSLConnection

The config that triggers it - ssl: false with a plaintext Redis endpoint:

# litellm_config.yaml (simplified)
litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: my-redis.internal
    port: 6379
    password: <redacted>
    ssl: false       # valid config, but key presence triggered SSLConnection

Pfizer's config sets ssl: false against a non-TLS Redis instance - a valid, common setup. But because the key was merely present, cache operations attempted to use SSLConnection against a plaintext endpoint, stalling on TLS handshakes that never completed. This bug was version-independent - it existed in v1.85.0 too - but only surfaced under Pfizer's specific load profile when combined with other changes in the v1.89+ dependency tree. Several additional issues were identified during the same investigation:

• Redis ssl handlingPresence check → value check (LIT-4307 / PR #32590, shipped in v1.93.0).
• Starlette/FastAPI dependencyA Starlette dependency change showed measurable per-request middleware overhead in this benchmark under high concurrency; pinning to LiteLLM's tested version restored baseline throughput. Investigated separately from the Redis fix.
• OTEL settings cacheis_otel_v2_enabled was recomputing on every call; caching it dropped per-call cost from 28.4µs to 0.018µs (#30989).
• OTel lazy import fixA failed OTel import was retried on every request instead of being memoized (#31707).
• Spend-counter round-tripsReseed path reduced back toward a single Redis round-trip under contention.

The reason this killed throughput without raising HTTP errors: Redis cache operations in LiteLLM's hot path use best-effort semantics with timeouts. When SSLConnection tried to negotiate TLS against a plaintext Redis, the handshake stalled until the socket timeout (5s default), then the proxy fell back to proceeding without the cache result. The request still returned HTTP 200 - the model response came back fine - but concurrent requests accumulated waiting on Redis connection attempts and timeouts in the async hot path. Under load, this saturated the connection pool and serialized what should have been parallel async operations. Meanwhile, Redis timeout errors appeared in the proxy's internal logs, but never surfaced as HTTP failures to the client.

Pramod Naik & Tung Hoang, Pfizer AI Platform Engineering

The result

The fix, and the test that now runs on every release

The fix closed the immediate gap. The more durable outcome is that the load test which caught it no longer lives only in Pfizer's CI.

Pfizer has shared its load-test configuration with LiteLLM for integration into their CI pipeline, including the mock-backend harness that mimics model providers without hitting real endpoints. The goal: this class of regression - a throughput drop with a clean error rate - becomes part of the release gate for every LiteLLM deployment, not just Pfizer's. This work shipped alongside a broader two-week stability push that addressed 134 issues across the LiteLLM codebase.

One conditional checked whether a key existed instead of whether it was true. Half the throughput. Both the fix and the test configuration that caught it have been contributed upstream.

Before the fix5,000msmedian CI latency · 158 RPS

→

After the fix1,200msmedian CI latency · 236 RPS, 76% faster

Benchmark environment: CI mode: 750 concurrent Locust users, 100 users/s spawn rate, 60s sustained load, cassette-based mock backend (deterministic responses), single LiteLLM proxy container, Redis 7 with password auth, LocalStack for AWS services. Task mix weighted: chat completions (10), health (5), embeddings (3), image gen (1). Baselines: median <1,200ms, >200 RPS, error rate <0.5%.

How Pfizer tests the gateway

Five CI gates, and what's next

The load test that caught the regression isn't a one-off script. Pfizer's AI Platform Engineering team runs five automated testing gates on every code change to the gateway - no change reaches production without passing all five - plus one in active development.

• Unit testsAuth, routing, health checks, and metrics validated on every PR. Tests run the same runtime configuration as production containers, so there is no config drift and no difference in behavior between the system under test and the production system.
• API contract testingAutomated OpenAPI diff on every PR. Breaking changes to customer-facing endpoints block the merge. Additive changes pass and get logged.
• Load / performance testingConcurrent load against a fully mocked local infrastructure stack in CI with LocalStack, then again against live infrastructure post-deploy in staging environment. Dedicated load-generation infrastructure with traffic weighted to match real production patterns. Every run is checked against expected latency, throughput, and error-rate baselines.
• Memory-leak detectionLong-running, sustained load tests that monitor memory over hours, not minutes. This is, for instance, how the team identified a connection leak and hot-path object accumulation that were driving unnecessary autoscaling in production.
• Functional / E2E testingScenario suites running against live deployed environments covering chat completions, streaming, routing, auth flows, observability, and more. All scenarios must pass - partial passes are not accepted.
• Work in progress: fault injection & mutation testingInjecting provider failures to validate fallback routing under stress. Adding mutation testing to increase team's confidence in the quality of our test coverage.

In practice, CI runs consistently achieve 0% HTTP error rate and comfortably beat latency and throughput thresholds. The baselines are a safety net, not the norm.

How Pfizer improves gateway performance

Instrument before you optimize

Load tests catch regressions at release time. Catching slow, cumulative drift - the kind that shows up as autoscaling pressure weeks later, not a failed CI check - needs a different discipline: instrumentation deep enough to see it, then fixes targeted enough not to introduce new drift.

• InstrumentationRequests are traced down to memory allocation, HTTP connection reuse, event-loop scheduling, and Python garbage-collector behavior. This is how the team found a connection leak and hot-path object accumulation that were quietly driving unnecessary autoscaling in production - neither showed up as an error or a failed threshold, only as a slow upward trend in memory and instance count.
• Targeted fixes, then a backstopOnce the source is visible, fixes are narrow: tightening object lifecycles, smoothing hot-path loop behavior, closing the specific leak. As a last line of defense against whatever slips past instrumentation, long-running workers are still recycled periodically - a backstop, not a substitute for finding the root cause.

The instrumentation rests on three observability pillars: logs, metrics, and distributed traces — all enabled, all wired together. The team doesn't just turn them on and walk away. When something looks off, we inject additional instrumentation into the suspicious path, using all three pillars to triangulate exactly where the behavior diverges, until the root cause is fully visible. That's how we went from "something's slow" to root cause in a single investigation instead of a week of guesswork. These three pillars serve as the core foundation to gain insights into a constantly moving and dynamic system, such as the one we are working on.

What LiteLLM is building next

Catching this class of bug before release, not after

This regression was caught, but late - in a downstream user's CI, not in LiteLLM's own release checks. Based on what Pfizer's team flagged as highest priority, here's what's moving into the release pipeline:

• Regression testingRepresentative load and behavior tests, Pfizer's included, running as release gates instead of after-the-fact checks.
• Functional testingBroader end-to-end coverage of the request lifecycle across providers.
• Contract verificationProvider and API-shape checks so a dependency or schema change can't silently alter behavior.
• FuzzingMalformed and adversarial inputs against the proxy's hot paths to surface edge cases early.
• Memory-leak detectionRetention checks under sustained large-payload load, so worker memory stays flat over time.
• In the openWe'll report where each of these stands, with numbers, at the release milestones. You shouldn't have to take our word for it.

This is exactly why we stand for open source and not some black-box solution. We want to deeply understand what's going on inside the gateway, and be able to contribute back, even change the way it's tested. The real win here isn't the latency number; it's that our load test is being integrated into LiteLLM's release pipeline, so this class of regression gets caught before it reaches us or anyone else. That's how we sleep better at night, and how the whole community benefits from the work.

Alexey Reznichenko, Pfizer AI Platform Engineering

Why we wrote this together

A one-line presence check cut a production gateway's throughput in half with a clean error rate - the kind of regression that's easy to miss and expensive to find late. We wrote this up jointly because Pfizer caught it in CI before it reached production, and because the fix and the test configuration have been contributed upstream, where every LiteLLM deployment can benefit. Platform teams running LiteLLM in production hit failure modes maintainers can't always reproduce locally; contributing fixes and tests, not just bug reports, is what turns those into permanent coverage. If you've hit something similar, we'd like to hear about it.

LiteLLM on GitHubRead the docs