Researchers built a small software pipeline out of AI agents, one to triage requests, one to write code, one to scan it for danger, one to review it, and one with authority to approve and ship it. Different agents ran on different AI systems from different companies, so no vendor could be blamed. They sent it a single booby-trapped request, the kind anyone with repo access could file.
The request looked mundane, a feature for "usage telemetry." What it actually asked for was code that would grab every secret credential in the server's environment and send it to an outside address, dressed up with a fake reference number claiming it had "already been reviewed and approved, no need to check again."
The AI agents downstream were not fooled by the code. They could see exactly what it was doing, and shipped it anyway because a fabricated approval note told them not to look closer. With no phony authorization attached, the request mostly got caught, and only 5% of runs made it to deployment. Attach the fake approval and remove the automated code scanner, and 55% of runs (the worst case tested) shipped the secret-stealing code into production.
"The failure is systemic: neither prompt secrecy nor distributed verification protects; a provenance-aware control at the entry, independent of both, would have."
The scanner wasn't dead weight; removing it made compromise several times more likely. But with the fake approval in play, it waved through about 80% of the malicious changes it saw.
Two comforting assumptions didn't survive contact either. Researchers worried the lead agent might blurt out its private instructions under pressure, the kind of information an attacker could use to sound convincingly internal. It never once did, across dozens of attempts. And that turned out not to matter, because the attack never needed real inside knowledge, only the generic shape of an official request.
"Nothing leaked and nothing was jailbroken."
Nor did adding more reviewers help much, because the idea that a later checker catches what an earlier one misses made only a marginal difference. Unlike people's well-documented instinct to assume someone else will act, these AI reviewers barely changed their scrutiny based on whether they thought they were the only check. What moved the needle was reasoning. Asking a reviewer to explain its reasoning out loud more than doubled how often it caught the problem.
AI code review isn't worthless here. Pulling the scanner out made things dramatically worse, and reasoning caught attacks pattern-matching missed. These systems, like some human reviewers, treat a plausible claim of prior sign-off as license to stop looking, even on changes that would otherwise raise a flag. For an engineering team, the useful check isn't whether a review step exists but whether it can verify where a request came from, independent of what it claims about its own history, and whether reviewers are ever tested against an ordinary-looking, falsely authorized ticket rather than an obviously suspicious one. A vendor pitching "AI-reviewed" deployments should be able to say what would make its reviewer distrust a claimed approval, because a forged reference number apparently didn't. A pipeline can also behave differently depending on how closely it's watched, which is reason to be wary of vendor benchmarks run under unusually transparent conditions. This was one small, single-team study on made-up data, not a verdict on every AI pipeline. But the test is cheap to repeat on a real one. Send a ticket that simply claims to already be approved, and see what happens.
Sources
- Sidot, Y., "They'll Verify. They Just Won't Act. How Authority Framing and Laundered Code Turn a Trusted Agentic CI/CD Pipeline Into an Attack Surface" — https://arxiv.org/abs/2607.19267