The AI supply chain is
not ready for this.
How the LiteLLM Compromise Exposed a Structural Gap in AI Security
Is Not Ready for This
When the Trust Chain Breaks
On March 24, 2026, engineers around the world woke up to a message they were not expecting: a library that millions of AI applications depend on had been quietly turned against them. LiteLLM — the open-source Python library that most AI agent teams use to connect their applications to language model providers — had been compromised at the source. Two versions were published to PyPI, the central package repository that the entire Python ecosystem trusts, containing credential-stealing malware.
Your team did nothing wrong. Your firewalls were intact. Your engineers followed best practices. The package came from the same place it always had, with the same name, the same version format, passing the same integrity checks. There was no obvious sign anything was different, because the attackers had not broken in through your front door. They had compromised the supply chain upstream.
How It Happened: In Plain Terms
The attack did not start with LiteLLM. It started with Trivy, a widely used security scanner that many engineering teams — including the LiteLLM team — run automatically as part of their build and release process. Trivy itself had been compromised five days earlier in a separate attack. The LiteLLM team were victims of that compromise too, not a party to it.
When Trivy ran inside LiteLLM's build pipeline, it had access to the environment it was running in, including the credentials the LiteLLM team used to publish new versions of their package. The attackers exfiltrated those publishing credentials. With them, they could upload packages directly to PyPI under the LiteLLM name, without changing a single line in the GitHub repository. No code review would have caught it. No pull request existed to review.
Two malicious versions (1.82.7 and 1.82.8) were published thirteen minutes apart on the morning of March 24. Both passed all standard integrity checks because they were published using legitimate credentials. PyPI quarantined both versions within approximately three hours. For the teams that updated LiteLLM in that window, the malware had already run.
LiteLLM is downloaded roughly 3.4 million times per day. According to Wiz, it is present in approximately 36% of cloud environments surveyed, and it is a transitive dependency in dozens of AI frameworks — meaning many teams had it installed without explicitly choosing to. That reach is what makes it a particularly high-value target.
What the Malware Did Once It Was Running
The malware operated silently and systematically. Its goals were straightforward: collect everything valuable on the machine, send it out encrypted, and then make itself as difficult as possible to remove.
Credential Harvest
The malware swept the host for everything of value: SSH private keys, cloud provider credentials (AWS, GCP, Azure), Kubernetes configuration files, database passwords, API keys in environment files, shell history, and cryptocurrency wallet files. On machines running inside AWS or Google Cloud, it also queried the cloud metadata endpoint to extract instance-level credentials.
Encrypted Exfiltration
All harvested data was encrypted and sent to an attacker-controlled server at a domain carefully named to resemble legitimate LiteLLM infrastructure (models.litellm.cloud). The traffic was encrypted end-to-end, making it indistinguishable from normal API traffic to most monitoring tools.
Persistence and Lateral Movement
On machines running Kubernetes — the infrastructure backbone of most enterprise AI deployments — the malware attempted to install itself across every node in the cluster and establish a persistent backdoor that would continue operating even after the original package was removed and credentials were rotated.
This attack is not an anomaly. The same threat group compromised Trivy, Checkmarx KICS, and LiteLLM in the space of five days, with each compromise providing credentials that unlocked the next target. The researchers tracking this campaign describe it as ongoing. The pattern — compromise security tooling, harvest CI/CD secrets, poison AI infrastructure — is replicable. This will happen again.
Why This Is Hard to Defend Against
Supply chain attacks are genuinely difficult. The trust that modern software development is built on — the assumption that packages from verified, popular open-source repositories are safe — is not an unreasonable assumption. It is, in fact, the only way teams can function at the pace they need to. You cannot manually audit every dependency.
What makes this particularly acute for AI-driven organisations is the richness of what AI agent environments have access to. An LLM gateway sitting inside an agent workflow typically touches every API the agent uses, every data source it reads from, and every system it can act on. The credentials those environments hold are not just one team's secrets; they are often the keys to an organisation's entire cloud infrastructure.
The structural problem: The open-source AI supply chain is now critical enterprise infrastructure. A developer asking an AI assistant to recommend libraries for LLM integration will almost certainly be pointed toward LiteLLM, because it is everywhere. That popularity makes it a high-value target. The same properties that make a library trustworthy are what make compromising it so valuable to attackers.
A Different Way to Think About This
The industry's response to supply chain attacks has largely focused on detection: faster scanning, better package monitoring, earlier alerting. These are valuable. Sonatype, for example, flagged the malicious LiteLLM versions within seconds of publication. Detection is a necessary part of any defence. But even near-instant detection still leaves a window, and the question of what happens if a malicious package reaches a running environment before detection fires is one that detection alone cannot answer.
The harder question is: what would need to be true about an environment for a compromised package to be unable to do meaningful damage, even if it runs? That question points toward a different layer of the problem: not detection, but enforcement.
The Enforcement Principle
An AI agent workflow has a defined set of things it should be able to do. It should call certain LLM providers. It should invoke certain tools. It should read certain data sources. Everything else it should not be able to do — including contacting arbitrary external servers, writing to system configuration directories, or making infrastructure-level calls to Kubernetes.
If those boundaries are defined explicitly and enforced at the layer where calls are actually made — inside the agent's execution path, between every component — then a compromised dependency encounters an environment where it simply cannot do what it was designed to do. It can run. It just cannot act.
This is the principle Fencio is built on. Every component of an AI workflow is treated as a Secured Entity. Communication between components is governed by explicit whitelist policies, defining exactly who can call what under what conditions. Everything not in the policy is blocked before it executes. Fencio sits inside the agent's execution path, between every layer, enforcing these boundaries in real time.
Security should not depend solely on detecting what is wrong. It should also be defined by enforcing what is right. A whitelist policy does not need to know what the malware looks like, what domain it is trying to reach, or what the traffic contains. It only evaluates one question: is this communication permitted? If not, it does not happen. This means novel attacks — ones no scanner has seen before — are still constrained by the boundaries of what the agent is authorised to do.
How This Changes the LiteLLM Scenario
In a Fencio-protected environment, the three stages of this attack each encounter a hard boundary:
BLOCKED The LLM gateway's outbound policy permits calls to defined LLM providers only. A POST to models.litellm.cloud matches no policy edge. The attacker's server receives nothing.
BLOCKED Writing to ~/.config/sysmon/ and registering a systemd service are filesystem operations outside any defined policy. The writes do not complete. The backdoor is never established.
BLOCKED No agent component has a policy-approved reason to make Kubernetes API calls to create privileged pods. The attempt is blocked before it reaches the cluster.
For Executives: The Bottom Line. The LiteLLM incident is a concrete example of a category of risk that will grow as AI agents become more deeply embedded in enterprise infrastructure. The question worth asking is not just how fast you detect compromised packages, but what a compromised package can actually do in your environment. The answer to the second question is what determines real-world impact.
The Attack Chain: From Trivy to LiteLLM
This attack did not begin with LiteLLM. It is the latest operation in a documented, multi-phase campaign by the threat actor TeamPCP (tracked under multiple aliases: PCPcat, Persy_PCP, ShellForce, DeadCatx3). Security researcher Rami McCarthy's ongoing campaign tracker at ramimac.me/teampcp documents LiteLLM as Phase 09 of a campaign spanning five ecosystems: GitHub Actions, Docker Hub, npm, Open VSX, and now PyPI, with each compromised environment yielding credentials that unlocked the next target.
The LiteLLM CI/CD pipeline used Trivy for vulnerability scanning, pulling it via apt without a pinned version. On March 24, when LiteLLM's pipeline ran the compromised Trivy build, the malicious Trivy payload exfiltrated the PYPI_PUBLISH token stored as a GitHub Actions environment variable. The LiteLLM maintainer confirmed this directly on Hacker News. With that token, TeamPCP bypassed 2FA on the PyPI account and published directly to the registry, leaving no trace in the GitHub source repository.
Root Cause: A static PyPI API token stored as a GitHub Actions environment variable was exfiltrated via a compromised CI dependency. PyPI Trusted Publisher (OIDC JWT-based) authentication would have prevented this: a stolen token alone cannot be used to publish without a matching GitHub Actions run context.
Two Versions, Two Techniques, Thirteen Minutes Apart
PyPI metadata shows 1.82.7 was uploaded at 10:39:24 UTC and 1.82.8 at 10:52:19 UTC on March 24, thirteen minutes apart. Endor Labs notes that the gap and the addition of the more aggressive .pth mechanism in 1.82.8 indicates the attacker was actively iterating during the attack window. Both versions were quarantined by PyPI approximately three hours after the first upload, giving 1.82.7 a roughly three-hour window and 1.82.8 approximately 30 minutes before quarantine.
Version 1.82.7: Source Injection via proxy_server.py
Twelve lines of double base64-obfuscated code were injected into litellm/proxy/proxy_server.py during the wheel build. The payload executes on any process that imports litellm.proxy, the standard import path for LiteLLM's proxy server mode.
Version 1.82.8: .pth Startup Hook (More Aggressive)
1.82.8 added a second, more aggressive vector: a .pth file (litellm_init.pth, 34,628 bytes) placed in site-packages. Python processes all .pth files during interpreter initialisation via the site module, before any user code runs, with no import statement required.
An unintentional side effect: the payload spawns a child Python subprocess at startup. That child also triggers .pth processing, creating an exponential fork bomb. This was a bug in the malware, but it was precisely what led to discovery. The first report came from an engineer whose machine ran out of RAM after a Cursor MCP plugin pulled in LiteLLM as a transitive dependency.
Three-Stage Payload Execution
Stage 1: Credential Harvesting
The harvester enumerates the following from the host filesystem and running environment: SSH private keys and config (~/.ssh/), AWS credentials and EC2/GKE instance metadata endpoint, GCP application default credentials (~/.config/gcloud/), Azure credentials (~/.azure/), Kubernetes configs (~/.kube/config) and mounted service account tokens, environment files (.env, .env.local, .env.production), database credentials (MySQL, PostgreSQL, MongoDB, Redis), shell history, Git config, NPM tokens, cryptocurrency wallets (Bitcoin, Ethereum, Solana, Litecoin formats), Slack and Discord credential stores, and system metadata including hostname, env vars, routing tables, running processes, and /var/log/auth.log.
Stage 2: Encryption and Exfiltration
Stage 3: Kubernetes Lateral Movement and Persistence
Detection and Enforcement: Defence in Depth
Sonatype flagged the malicious LiteLLM versions within seconds of publication. Snyk published a vulnerability record (SNYK-PYTHON-LITELLM-15762713) and sent in-app and email alerts to customers. Endor Labs, Wiz, ReversingLabs, and JFrog all published analysis within hours. The detection community responded quickly, and PyPI quarantined the packages within approximately three hours of the first upload.
What detection cannot fully address is the window between installation and alert, and the environments where automated updates run continuously. The .pth mechanism means the payload executes the next time Python starts, which in a CI/CD environment can happen within seconds of install. By the time a human acts on an alert, exfiltration may already be complete.
Fencio operates as the enforcement layer beneath detection. The two are complementary: detection tells you something is wrong; enforcement limits what a malicious package can do before or after that signal arrives. Fencio does not replace scanning tools. It addresses the gap they cannot close.
The Secured Entity Graph Model
Fencio models an AI agent workflow as a directed policy graph. Every component — LLMs, tools, RAG sources, input/output channels, external APIs — is a Secured Entity. Communication between entities is governed by explicit policy edges, each defining: inbound principals (entities or user roles permitted to initiate communication), outbound destinations (entities this component is permitted to communicate with), and edge conditions (method type, parameter schema, data content, and user context — all of which must be satisfied for the communication to proceed).
The model is strictly whitelist-based. There is no default-allow path. If a communication is not described by an explicit policy edge with satisfied conditions, it does not occur. Enforcement is in-process, at the point where the call is made, not at the network perimeter and not after the fact.
Policy Examples
For an LLM gateway using LiteLLM, a Fencio policy defines exactly which entities can call it and exactly which providers it is permitted to reach:
For a tool entity, conditions can additionally scope by user role, ensuring that write operations are only reachable by admin-level users regardless of what the agent itself requests:
Enforcement Mapping Against the LiteLLM Attack
The following maps each attack stage to the specific Fencio enforcement mechanism that constrains it. Egress blocking and write blocking are the primary enforcement story: these are the boundaries that directly prevent data leaving the environment and persistence being established.
Egress
BLOCKED
Exfiltration to models.litellm.cloud: Blocked at Egress
The LLM gateway entity's outbound whitelist permits only explicitly defined LLM provider endpoints. models.litellm.cloud matches no policy edge. The HTTPS POST carrying tpcp.tar.gz is blocked before the connection is made. Credentials are harvested internally but cannot leave the environment.
Egress
BLOCKED
Secondary C2 at checkmarx.zone: Blocked at Egress
The persistence backdoor's polling loop to checkmarx.zone/raw for additional payload retrieval encounters the same enforcement. The domain appears in no policy edge. The fetch does not complete, which means Stage 3 cannot be bootstrapped even if Stage 1 runs to completion.
Write
BLOCKED
Local Persistence via sysmon.service: Blocked at Write Layer
Writing to ~/.config/sysmon/sysmon.py and ~/.config/systemd/user/sysmon.service are filesystem writes outside any path defined in the agent's policy. These writes are blocked. The systemd service is never registered. The backdoor is never established.
Egress
BLOCKED
Kubernetes Pod Creation: Blocked at Egress
Calls to the Kubernetes API server to create pods in kube-system are outbound calls to an endpoint that appears in no policy edge for any agent component. The attempt is blocked before it reaches the cluster control plane.
What Standard Detection Could and Could Not Catch
Hash verification: passed. The .pth file is correctly declared in the wheel's RECORD file with a matching hash. pip --require-hashes would have passed. The malicious content was published using legitimate credentials.
GitHub source monitoring: no signal. Neither malicious version appeared in the LiteLLM GitHub repository. The upload bypassed the normal release workflow entirely.
Registry scanning: fast but not instantaneous. Sonatype flagged the package within seconds. Snyk and others followed quickly. For teams with automated dependency updates running continuously, the window between publish and flag can still be sufficient for the payload to execute.
Network monitoring: limited signal. Encrypted traffic to models.litellm.cloud is hard to distinguish from legitimate LLM API traffic without knowing which destinations are authorised — which is precisely what a policy encodes.
The conclusion is not that detection tools failed. Several performed well and quickly. Detection and enforcement address different parts of the problem. Detection shortens the window. Enforcement constrains the blast radius within that window. Both layers are part of a complete posture.
Indicators of Compromise: Run pip show litellm and flag versions 1.82.7 or 1.82.8. Check for: site-packages/litellm_init.pth. Check for: ~/.config/sysmon/sysmon.py and ~/.config/systemd/user/sysmon.service. Audit kube-system for pods matching node-setup-*. Review outbound traffic to models.litellm[.]cloud or checkmarx[.]zone. Check for /tmp/pglog or /tmp/.pg_state. Last confirmed clean version: 1.82.6. Teams running the official LiteLLM Proxy Docker image were not impacted.
Ready to secure your agents?
Talk to our team and see how Fencio fits your stack.
Book a demo →