One post tagged with "memory"

Date: September 7 to September 9, 2026
Affected versions: v1.100.0
Severity: Medium (production outage for one customer; low blast radius, required an unreachable Redis)
Status: Resolved in v1.100.1

Note: If you are running v1.100.0, upgrade to v1.100.1 or later. v1.98.x and earlier are not affected. v1.101.0-rc.1 and later are not affected.

Summary​

Between September 7 and September 9, 2026, an enterprise customer running litellm-database:v1.100.0 behind nginx experienced two waves of kernel OOM kills across their proxy fleet: 15 kills total, one worker process reaching 61 GB RSS, and thousands of HTTP 502s and tens of thousands of HTTP 499s while the platform was down.

This needed two things to be true at once: the provider on the customer's traffic kept failing, so the router kept retrying, and Redis was unreachable at the same time, so every completing request logged the growing result of those retries in full. The retries grew unbounded because of a one-line change in PR #38133 (August 24), which ran retry breadcrumbs, the record the router keeps of each failed LLM attempt, through an existing credential-masking helper before storing them. The helper was correct for its original inputs but assumed it was copying a plain tree; the breadcrumb was not a tree, it contained a pointer back to the same list it was about to be appended to. Copying it without preserving that shared reference turned a structure that had cost nothing in memory since December 2023 into one that roughly doubled in size with every retry, until a cost-tracking error handler turned it into a log string on every request for as long as the Redis outage lasted, and a single allocation reached 27 GB.

We fixed the underlying growth in PR #39491, merged September 3 and included in v1.101.0-rc.1. It was not backported to v1.100.0 before that version was tagged stable two days later. We backported it to stable/1.100.x on September 9 (PR #40455) and published v1.100.1.

We own this outcome entirely. The defect existed in a form we could have caught, a ticket filed three days before the stable release correctly identified a growth problem in the same code path, and our own fix PR turned an untested hedge in that ticket into a stated fact. The rest of this post explains how each of those things happened and what we're changing so that a defect like this can't reach a stable release the same way again.