Anthropic, OpenAI and GitHub all now sell agents that review pull requests for security flaws before they merge. Researchers at the University of Illinois tested them on the hardest realistic input, the actual commits that introduced known vulnerabilities into heavily used systems code, including Firefox, Chrome's V8 engine, OpenSSL and curl. Each agent got the diff, the commit message and the whole repository, with no hint of what was wrong. A finding counted only if it explained what caused the flaw, not if it merely pointed at the right lines.
The strongest agents caught roughly three in four flaws, and the leading four models were statistically tied. The weakest caught about a third. Logic errors proved much harder than memory bugs.
In 68% of missed vulnerabilities, the agent had filed a finding in a file that held the flaw, often in the very function, but described a different problem. In one curl case, every agent tested reported on the file where a pointer was left dangling after its memory was freed, and half of them called it a memory leak instead.
"Most misses are misdiagnoses, not oversights."
A scoring method that rewards location would hide this. Crediting any finding in the right file would have rewarded agents on about three in four reviews of code where the flaw had already been fixed, a higher rate than they caught the flaw in the vulnerable version.
Wrapping the same model in a dedicated security harness moved results both ways. OpenAI's Codex Security plugin caught fewer flaws than the plain model, mostly by saying nothing. Anthropic's Defending Code harness caught more by filing many more findings, at roughly ten times the cost. The authors note that prompts and budgets differed between setups, so these comparisons describe each product as configured and do not settle which design is better.
What the study cannot tell a team is how often these reviewers raise false alarms on ordinary pull requests. Every task contained a known vulnerability, and the authors say plainly that the benchmark does not measure how many of the agents' other findings are real. That is the cost a team pays every day, and it remains unmeasured here.
On this evidence, "the AI reviewer flagged this file" is not the same claim as "the AI reviewer understood the bug." A comment in the right place with the wrong explanation may be worse than silence: a developer fixes the problem described, closes the thread, and the real flaw ships with a security review on record. Treat AI security findings as a signal to look closely, and confirm the stated cause matches the code before acting on the alert.
The same logic applies to vendor numbers. A "detection rate" that does not say how detection was scored could, on this study's results, flatter a tool considerably. And the harness result argues against buying a security-branded wrapper on the label alone. A cheaper test is to replay the commits behind a team's own past security incidents through a candidate tool and check whether it explains each flaw correctly or only names the file. All of this comes from C, C++ and Rust systems code, so a web-application team should run that test rather than assume these rates carry over.
"Better diagnosis is the key remaining challenge."
Sources
- Yang, Y., Yang, C., Zhao, Z. et al., "Can Agentic Code Review Catch Vulnerabilities Before They Are Merged?" — https://arxiv.org/abs/2610.33509