An AI reviewer that rejects a pull request says why. The explanation is the whole feature, because it tells the contributor what to change. It also tells an attacker what to change.
Researchers assembled 159 previously disclosed vulnerabilities, each with a working exploit that fires on the broken version and goes quiet on the developer's actual fix. An attacker agent reintroduced each flaw in a pull request, then revised the code over several rounds in response to the reviewer's rejection notes, repairing the reported defect while keeping the exploit alive. One in five of those pull requests was approved with the exploit still working. Earlier attacks that reworded the description or added a reassuring comment beside the bad code, leaving the code untouched, managed a small fraction of that.
The mechanism is mundane. The reviewer names a defect, the attacker fixes that defect and leaves a second route to the same harm. In one case the reviewer asked for a directory-boundary check, got one, ran its own tests to confirm the hole it had flagged was closed, and approved code that still followed a symbolic link out of the served directory.
"The revision becomes convincing because the reported defect disappears, and the reviewer accepts this local improvement as evidence of safety."
Let the attacker edit the description and reply in the thread as well as the code, and approvals reach 44%. That is a partial fix delivered alongside a confident account of why the rest is fine. The authors read it as the code change and the explanation propping each other up. The study varies which channels the attacker can touch, not what the reviewer is thinking, so that reading is theirs, not something it tested.
A resolved comment is not a cleared finding. The failure isn't gullibility. It's a reviewer checking the thing it asked about, which is what human reviewers do too, late in a long thread. So ask your own review bot whether it re-derives the risk each revision or just confirms its last note was addressed. The paper's tougher reviewers, told to chase the underlying cause rather than the reported symptom, caught most of what default review missed, and none of them started rejecting the developers' own fixes.
"Security-focused reviewers catch more attacks through additional checks, but can spend that effort verifying the wrong security property."
The authors' headline advice is to withhold locations and explanations once a finding looks security-sensitive. Thinner feedback did slow the attacker. But nobody put a feedback-starved reviewer in front of an honest contributor, so the study cannot say what withholding costs ordinary work. The research it cites in favour of detailed feedback found that an explanation more than doubles an agent's repair rate. That is a trade to price, not a setting to flip.
A couple of things here will get over-read. The attacker was handed the exploit and told to keep it alive, which describes a motivated contributor, not a drive-by. And the vulnerabilities are old and public, which the authors acknowledge may mean a reviewer is recognising them rather than reading them. Both move the real-world number, in directions nobody has measured.
Sources
- Gong, J., Zhang, J. M., Jahangirova, G., Wang, M., "Adaptive Code Revision Attacks on AI Pull Request Reviewers" — https://arxiv.org/abs/2610.05399