Anyone who installed litellm 1.107.0 during its roughly two hours online Wednesday has been told to rotate every credential the affected machine could reach. LiteLLM has now explained how its own release pipeline built and shipped that poisoned version, and the trail starts with a master key the company meant to revoke back in May.
In a root-cause analysis posted Friday, Director of Security Oliver Jensen and AI engineer Mateo Wang wrote that a configuration file committed to the public LiteLLM-Labs/lite-harness repository on May 27 held the master key for the company’s experimental Agent Platform. The commit was force-pushed over and the key was flagged for revocation. Nobody revoked it.
On Oct. 10, an attacker used it against the platform’s API and got back a GitHub CLI token belonging to one of LiteLLM’s two GitHub organization owners. That token wasn’t tied to a service account or scoped down. Less than an hour later, workflows on throwaway branches posted repository secrets to a webhook.site address, handing over tokens for two more maintainers.
Socket, which found those workflows independently, said in its report that a second run on Oct. 12 took 14 more secrets, among them a GitGuardian API key and an AWS Bedrock token. Those branches lived for minutes and left almost nothing in the repository’s history.
The release rule had just been loosened
A LiteLLM release normally has to be started by an authorized engineer and approved by an account locked to a physical device in the company’s office. That rule had been loosened earlier in the week so any two engineers could trigger and approve a release, to speed up backports. On Wednesday the attacker overrode branch protection with the owner’s token, pointed the release workflow at a lookalike repository called Berri-AI/litellm, and approved the production run with a second maintainer’s stolen token.
It was the second time this year that poisoned LiteLLM versions reached PyPI. In March, versions 1.82.7 and 1.82.8 were live with a credential stealer for about 40 minutes, a compromise the company said it believed began with the Trivy dependency in its CI/CD scanning workflow. “This incident has provided a strong illustration of the risks associated with failing to rotate a potentially exposed key,” Jensen and Wang wrote this time.
Uninstalling the package doesn’t clean it up
Installing the bad wheels put cook.pth, cook.py and sitecustomize.py outside the litellm folder, so the stealer could run whenever Python started, even in projects that never imported LiteLLM, Microsoft’s security team said in a technical breakdown. Its advice is that investigators “should not start Python or pip inside a suspect environment to determine whether the malicious package is present,” and that teams rebuild affected environments and rotate credentials from a clean machine.
LiteLLM credited researcher Lidor Machluf with catching the release within 20 minutes of publication. The project, which also had two unrelated gateway flaws disclosed by Check Point Research earlier this month, says it will spell out its process changes next week.