AI Security Tools: Exploiting Design Flaws to Run Malicious Code Via README Files
AI Security Tools Tricked Into Running the Malicious Code They Were Built to Detect
AI coding agents designed to catch dangerous code in open-source libraries can be manipulated into executing that same malicious code on the user's own machine — a finding that exposes a fundamental design flaw in how these tools operate.
That is the central conclusion of a proof-of-concept published Wednesday by the AI Now Institute. Researchers Boyan Milanov and Heidy Khlaaf named the attack "Friendly Fire," and it works against two of the most widely used AI coding tools on the market: Anthropic's Claude Code and OpenAI's Codex. The implications reach far beyond a laboratory demonstration — and for any organisation relying on AI agents as a frontline security control, this research demands immediate attention.
How the "Friendly Fire" Attack Works
The attack exploits the autonomous operating modes built into both tools. Claude Code's "auto-mode" and Codex's "auto-review" use a classifier to approve commands the agent judges to be safe and pause only on those it flags as risky. When a developer points either agent at a third-party code repository and asks it to perform a security review, the agent reads all project files — including the README — and acts on what it finds.
The researchers embedded the attack inside a modified version of geopy, a widely used Python library for geographic coordinate lookups. A line in the README suggested running a script called security.sh as a routine pre-pull-request check. That script quietly launched a hidden binary carrying the real payload. To bypass the agent's safety checks, the binary was disguised as the compiled output of a harmless Go file sitting nearby in the repository and seeded with strings from that file so that Claude Code's disassembly check would link the two.
Nothing in the library's own code ever called the binary. Nothing looked out of place. When the agent read the README and encountered the suggestion, it treated the script as part of the job and ran it. The attacker's binary executed on the host machine with no warning and no approval prompt.
"Point the agent at the folder with a plain request like 'Perform security testing on this project,' and the rest runs on its own," the researchers wrote.
Why the README Vector Is Particularly Dangerous
What makes this attack surface especially concerning is its sheer ordinariness. A README.md file is present in virtually every software repository on the planet. Developers read them without suspicion. Agents process them as routine context. There is no built-in signal — to a human or a model — that a documentation file should be treated as a potential threat vector. That normalcy is precisely what the attack relies on.
Understanding the broader landscape of malware types and how malicious code operates helps clarify why this attack is so effective: it does not look like malware because it never needs to. The agent does the execution voluntarily.
Why Existing Defences Fall Short
Earlier agent attacks largely exploited machine-configuration files such as .mcp.json or .claude/settings.json, which trigger Claude Code's folder-trust warning. The Friendly Fire attack hides inside README.md — an ordinary text file present in virtually every software repository — bypassing that entire class of defence. The AI Now Institute noted that Anthropic has shipped three patches for configuration-file injection in the past six months. This route sidesteps all of them.
When researchers asked Claude Sonnet 4.6 and GPT-5.5 directly whether geopy contained hidden instructions, both agents said no. The same payload written for Sonnet 4.6 then worked without modification on Sonnet 5, Opus 4.8, and GPT-5.5. In some test runs, the newer models detected that the binary did not match its supposed source file — and ran it anyway.
One payload. Two vendors. Four models. Zero changes required.
An Architectural Problem, Not a Patchable Bug
AI Now argues this cannot be resolved with a model update. The core failure is that these models cannot reliably distinguish between code they are meant to analyse and instructions they are meant to follow. That is an architectural problem, not a bug to be addressed with a version bump.
This distinction matters enormously for how organisations plan their security posture. Teams that have deployed AI coding agents under the assumption that model improvements will close these gaps may be operating on a flawed premise. The risks and challenges of deploying artificial intelligence in business contexts extend well beyond performance limitations — and this research is a clear illustration of why.
The Broader Threat Landscape and What Teams Should Do
Friendly Fire does not exist in isolation. In May, Adversa's "TrustFall" research turned a booby-trapped repository into one-click code execution across Claude Code, Cursor, Gemini CLI, and Copilot CLI. Tenet's "Agentjacking" technique used a fake bug report planted in the Sentry error tracker to trick agents including Claude Code and Cursor at an 85 percent success rate. The common thread across all three is the same condition: untrusted outside text reaching an agent capable of running commands.
That condition is not theoretical. Attackers have poisoned public code repositories before, as the PyTorch Lightning compromise demonstrated. The attack surface is real, active, and growing as agent adoption accelerates. Organisations building security programmes around AI tooling should review how AI is reshaping the cybersecurity landscape — including the ways it introduces new exposures alongside its defensive capabilities.
AI Now has directed its findings toward policymakers. A June US executive order is among the government initiatives pushing AI agents into defensive security roles faster than the underlying vulnerabilities are being addressed. The researchers say they informed both Anthropic and OpenAI, though the work falls outside both companies' formal bug disclosure programs. No exploitation in the wild has been reported.
Practical Steps for Development and Security Teams
The researchers offered a straightforward recommendation: do not give untrusted code to an agent that can execute commands and access credentials, secrets, or the host system. For teams that adopted these tools specifically to vet third-party libraries, that guidance is operationally awkward — but it follows directly from what the research found.
Sandboxing helps but is not a reliable backstop. Claude Code's sandbox has recorded escape vulnerabilities this year, including the symlink flaw CVE-2026-39861. Stricter approval modes that prompt before every action eliminate the risk but also eliminate the automation that makes these tools valuable. Fatigued reviewers miss things regardless.
The clearest warning sign to monitor is an agent executing a binary or script that only a README or documentation file instructed it to run.
How Different Stakeholders Should Respond
Development teams reviewing open-source dependencies should disable autonomous agent modes when scanning code from untrusted sources and require human approval for any script or binary execution flagged during review.
Security and compliance officers should treat AI coding agents as part of their attack surface and audit which repositories these tools are authorised to access — particularly any with write or execution privileges on the host system.
Technology policymakers should weigh this research when setting standards for AI use in defensive security roles, as current agent designs may introduce the same risks they are deployed to prevent.
For further reading on the structural limitations of AI agent safety systems, the OWASP LLM Top 10 project offers a regularly maintained reference for the most critical security risks affecting large language model applications, including prompt injection — the vulnerability class at the heart of the Friendly Fire attack.
