3 posts tagged with "redis"

Date: September 7 to September 9, 2026
Affected versions: v1.100.0
Severity: Medium (production outage for one customer; low blast radius, required an unreachable Redis)
Status: Resolved in v1.100.1

Note: If you are running v1.100.0, upgrade to v1.100.1 or later. v1.99.1 and earlier are not affected. The v1.100.0 dev and rc pre-releases, v1.101.0-dev.1, and v1.101.0-dev.2 carry the same defect; v1.101.0-rc.1 and later are not affected.

Summary​

Between September 7 and September 9, 2026, an enterprise customer running litellm-database:v1.100.0 behind nginx experienced two waves of kernel OOM kills across their proxy fleet: dozens of kills by the customer's count, one worker process reaching 61 GB RSS, and thousands of HTTP 502s and tens of thousands of HTTP 499s while the platform was down.

This needed two things to be true at once: the provider on the customer's traffic kept failing, so the router kept retrying, and Redis was unreachable at the same time, so every completing request logged the growing result of those retries in full. The retries grew unbounded because of a one-line change in PR #38133 (August 24), which ran retry breadcrumbs, the record the router keeps of each failed LLM attempt, through an existing credential-masking helper before storing them. The helper was correct for its original inputs but assumed it was copying a plain tree; the breadcrumb was not a tree, it contained a pointer back to the same list it was about to be appended to. Copying it without preserving that shared reference turned a structure that had cost nothing in memory since December 2023 into one that roughly doubled in size with every retry, until a cost-tracking error handler turned it into a log string on every request for as long as the Redis outage lasted, and a single allocation reached 27 GB.

We fixed the underlying growth in PR #39491, merged September 3 and included in v1.101.0-rc.1. It was not backported to v1.100.0 before that version was tagged stable two days later. We backported it to stable/1.100.x on September 9 (PR #40455) and published v1.100.1.

We own this outcome entirely. The defect existed in a form we could have caught, a ticket filed three days before the stable release correctly identified a growth problem in the same code path, and our own fix PR turned an untested hedge in that ticket into a stated fact. The rest of this post explains how each of those things happened and what we're changing so that a defect like this can't reach a stable release the same way again.

A joint debugging story with LiteLLM, and the release testing that comes next.

A LiteLLM version bump exposed a long-standing Redis configuration bug that cut Pfizer's gateway throughput by ~48%, with zero HTTP errors in the application logs. Here's how the team isolated it through configuration bisection, and the testing infrastructure both teams are building to catch this class of regression before it ships.

Last Updated: April 2026

Enterprise AI Gateway deployments put Redis in the hot path for nearly every request: rate limiting, cache lookups, spend tracking. When Redis is healthy, the latency contribution is single-digit milliseconds, invisible to end users. When it degrades, a production AI Gateway needs to stay up regardless.

Running LiteLLM at scale across 100+ pods means designing for failure modes before they appear. The easy case is Redis going fully down: fail fast, fall through to the database, continue serving requests. The hard case, the one that takes down gateways, is a slow Redis: still accepting connections, still responding, but timing out after 20-30 seconds per operation.