Every AI coding benchmark asks the same question: does the generated code pass a test? None of them ask what happens after a developer hits accept. A team at Carnegie Mellon went looking for that answer, capturing over a thousand developers' edits to AI code completions inside their own editors, rather than relying on the tidy final version that ends up in a Git commit.
The pattern cuts against how most people picture "collaborating" with an AI assistant. It isn't a back-and-forth polish. It's a verdict. Code tends to get kept almost entirely intact, or thrown out almost entirely, with very little middle ground where a developer nudges a suggestion into shape over several small edits. When it's rejected, it's rejected fast: 31% of accepted completions are eventually deleted outright, and most of that deciding happens in the first fifteen minutes.
What triggers the deletion is more telling than the deletion itself. It isn't broken code. Developers who started by tweaking a completion, renaming a variable, adjusting a value, were more likely to abandon it than developers who started by fixing an outright error. Code that's obviously wrong gets fixed. Code that's quietly slightly wrong gets thrown away instead, a harder failure mode to design against since it never shows up in an error log or a test suite.
"This suggests that AI completions that subtly do not align with a developer's intent or programming context are difficult for developers to adapt."
That's a direct challenge to acceptance rate, the metric most tools report: it measures the wrong moment. A completion can be "accepted" and still be dead within the hour. A retention rate, checked a day or a week later, would say more about whether the tooling is working than a click at the moment of suggestion. It would also give a team something more useful than a vanity number to watch: a tool whose completions keep getting quietly binned is failing at understanding intent, even if its acceptance rate looks healthy on a dashboard.
There's a clear order to how developers work through a completion, too. They decide whether to keep it first, then fix anything broken, then adapt it to fit, then extend it with new functionality, roughly in that sequence and mostly in the first hour. That has a practical implication for anyone reviewing AI-assisted pull requests: the earliest changes tell you whether a developer is fighting a suggestion, not polishing it, and are worth reading differently than the edits that come later in the same file.
The researchers also tried teaching AI models to predict which fate a completion would meet: kept, edited, or deleted. Leading commercial models were barely better than a coin flip at it, which is itself telling, since it means these systems currently have no sense of when they're about to be wrong in a way that matters to the developer using them. Training a much smaller, open model directly on this dataset of real edits closed most of that gap, without making the model any worse at writing code in the first place. Exposure to real editing behaviour, not a bigger model, was what fixed it.
"AI-generated code is either kept fully intact or completely discarded, suggesting that an important consideration is AI-generated code's editability (i.e., how easily the code can be adapted, even when it is not immediately usable)."
One caveat: the study only captures completions a developer accepted at least once, so suggestions rejected outright never enter the picture.
Sources
- Liang, J. T., Bairathi, M., Chi, W. et al., "Learning from 53.6K Real-World Developer Edits of AI-Generated Code" — https://arxiv.org/abs/2607.25130