Every coding-agent vendor sells roughly the same pitch, a tireless second reviewer that reads every pull request, day or night. A new study puts that pitch to a real test. It tracked 54,791 review comments left by three widely used agents, GitHub Copilot, Cursor, and Codex, across hundreds of active, real-world GitHub projects. (Devin and Claude showed up too rarely in the data to include.)
Developers don't treat these tools as interchangeable. Developers resolve 73% of the comments Copilot leaves, compared with 67% of Cursor's and 55% of Codex's, a wide gap for tools doing the same job on the same platform. Whatever explains the difference, it isn't the codebase.
"Finally, our analysis reveals that the presence of an inline code suggestion is the strongest predictor of comment resolution, while lengthy and complex comments are less likely to be acted upon."
Across every agent, the single biggest lever on whether a comment gets acted on is whether it arrives as a ready-to-apply fix rather than a paragraph explaining the problem. A comment with a drop-in fix attached gets acted on far more often than one that just explains what's wrong, even a well-written, accurate explanation. For a team choosing or configuring one of these tools, that's a concrete lesson. Pushing a vendor toward "always attach a suggested edit" does more for adoption than pushing them toward better prose.
The other half of the paper is a caution for anyone measuring these tools by comment count. Researchers dug into hundreds of stalled review threads to see why comments went nowhere. The single most common reason was that the code was already doing something deliberate the agent had no way of knowing about, such as a convention or a trade-off already settled. Being flatly incorrect came second. One developer's reply, preserved verbatim and now the source of the paper's title, captures the mood.
"No it isn't, go home copilot you're drunk."
That reaction points to a gap no benchmark captures. These tools pattern-match against code that looks risky, not against the intent behind a change, and developers can usually tell the difference even when the agent can't. That's a warning for anyone reading a vendor's "comments generated" number. It costs nothing to inflate; a resolved comment reflects something a human actually checked and trusted. It also means part of an agent's apparent error rate is really a context gap, not a flaw to tolerate forever. Better project information fed into the agent could close some of it.
Before rolling this out broadly, one more pattern deserves a check. Senior contributors handle most of the resolved comments, especially anything touching design, while newer contributors are comparatively more likely to be closing out straightforward bug fixes. A team adopting agentic review to lighten senior engineers' load should check whether that's actually happening. It's just as plausible the tool relocates work instead, handing routine fixes to junior developers while still routing every "was this intentional?" call back to the people with the least spare time.
Sources
- Cynthia, S. T., Widyasari, R., Roy, B. et al., "'Go Home Copilot, You're Drunk': Understanding Developer Responses to Agent-Generated Code Review Comments" — https://arxiv.org/abs/2607.21997