Security Townhall Updates
Thank you to everyone who joined our town hall.
We wanted to use that time to walk through what we know, what we've done so far, and how we're improving LiteLLM's release and security processes going forward. This post is a written version of that update. Slides available here
What happened​
On March 24, 2026 at 10:39 UTC, LiteLLM v1.82.7 was pushed to PyPI. Version v1.82.8 was published soon after. Those packages were live for about 40 minutes before being quarantined by PyPI. By 16:00 UTC, the LiteLLM team had worked with PyPI to delete the affected packages.
At this point, our understanding is that this was a supply-chain incident affecting those two published versions.
Q. Were older packages impacted?​
Our current findings show no indicators of compromise in the last 20 versions of LiteLLM. This was manually verified by our team and independently reviewed by Veria Labs.
We have also published the verified versions for users to use. Check Security Blog for release verification.
How did this happen?​
Our understanding is that the issue came from the compromised Trivy security scanner dependency in our CI/CD pipeline.
There were three major contributing factors:
1. Shared CI/CD environment​
At the time, everything was running on CircleCI, and all steps shared a common environment. That increased blast radius: if one component was compromised, it could potentially access credentials or context intended for other parts of the pipeline.
2. Static credentials in environment variables​
Release credentials, including credentials for PyPI, GHCR, and Docker publishing, were available as static secrets in the environment. That meant a compromised step could access long-lived release credentials.
3. Unpinned Trivy dependency​
In our security scanning component, we had an unpinned Trivy dependency. Our present understanding is that a compromised Trivy package ran during the scan, had access to environment variables, and enabled attackers to obtain those credentials.
In summary: a compromised package in CI had access to secrets it should not have had, and those secrets were then used in the release path.
What we've already done​
In the last 3 days, we've taken the following steps:
1. Minimize Scope of Impact​
Prevented further key abuse​
We deleted or rotated all impacted or adjacent secret keys, including PyPI, GitHub, Docker, and related credentials. Out of an abundance of caution, we've also rotated LiteLLM maintainer accounts.
Prevent branch attacks​
We removed roughly 6,000 open branches and added an auto-deletion policy for branches merged into main. This reduces the surface area for branch-based abuse.
Pinned CI/CD dependencies​
We've pinned all Github Actions, and are working on pinning all CircleCI dependencies as well.
Paused releases​
We've paused new releases until we've confirmed codebase security and put stronger release controls in place.
2. Secured LiteLLM​
Forensic analysis​
We are working with Google's Mandiant cybersecurity team to confirm the source of the attack and verify the security of the codebase. We also confirmed that no malicious code was pushed to main.
Confirm Application Security​
In parallel, we are working with whitehat hackers at Veria Labs to verify application security and review improvements to our CI/CD process.
We have also confirmed that the last 20 LiteLLM releases contain no indicators of compromise, and that no unauthenticated attacks can be made against LiteLLM Proxy based on our current investigation. Check Security Blog for release verification.
Created a security working group​
We created a new security working group inside LiteLLM focused on:
- Building threat models
- Auditing the build process and dependencies
If you're interested in joining the security working group, please file an issue here.
3. Improved CI/CD​
We've already begun making structural changes to how releases are built and published. These align with our goals (covered in the next section) around isolated environments, ephemeral credentials, and release auditing.
Roadmap​
We plan on following 4 guiding principles for our new CI/CD pipeline:
- Limit what each package can access
- Reduce the number of sensitive environment variables
- Avoid compromised packages
- Prevent release tampering
Isolated environments​
We are breaking our CI/CD into 4 semantic concepts:
- Unit tests
- Integration tests
- Security scans
- Release publishing
And will be running each of these in isolated environments.
This will limit the damage that any single compromised component can cause.
Ephemeral credentials​
We plan to move to ephemeral credentials for PyPI (Trusted Publisher) and GHCR (Token-based authentication) releases. This will reduce the risk of credentials being leaked or compromised.
We have already begun doing this:
Release auditing​
Our goal is to allow users to independently verify that a release came from us and prevent silent modifications of releases after they are published.
This will ensure, your releases are safe, even when:
- Stolen PyPI/GHCR credentials are used to publish malicious releases
- Tampered registry artifacts are published
- Tag mutations are made after the release is published
We believe that Cosign is a good fit for this, and have already begun working on it PR.
Avoid Compromised Packages​
- Move to pinned, verified SHAs for packages and actions used in CI/CD, avoiding
latestwherever possible. - Add a cooldown period before upgrading to a new version of a package - allows more time to investigate and verify the new version.
We've added zizmor to help us catch issues such as unpinned dependencies and credential leakage. commit.
Questions & Support​
If you believe your systems may be affected, contact us immediately:
- Security: security@berri.ai
- Support: support@berri.ai
- Slack: Reach out to the LiteLLM team directly here
Hiring​
We are currently hiring for:
- DevOps Engineer - to keep ci/cd secure and running smoothly
- Security Engineer - to keep the application secure
If you're interest in joining, please apply here