Plenty now rests on search over a codebase. A coding agent hunting for prior art and a retrieval pipeline assembling context before generation both lean on embedding search, and both carry the same buried assumption. Code that's about the right thing is the right code.
Researchers at Kennesaw State built a benchmark to test that head-on. Each task pairs a working Python function with up to four near-twins that fail their tests, every one differing by a single targeted edit, such as a flipped comparison or an argument swapped with the one beside it. The median change is a single character. Correctness came from running the tests, not from labelling. They then ran more than twenty embedding systems plus a keyword baseline over the pool. The strongest system returned a working implementation as its top result about a third of the time. Widen the window to ten results and the working version was almost always in there somewhere.
The misses aren't random. When the top result was wrong, it was one of the broken twins of that same function more than nine times in ten, not some unrelated snippet that happened to rank well.
"Failure of embedding retrieval at rank 1 is therefore not an arbitrary lookup error; it is a specific, structural inability to separate a canonical from its single-mutation buggy near-clones."
On the leading systems, the correct version scored below at least one of its own broken twins on between two-thirds and four-fifths of queries, and the margin either way was vanishingly small. Working and broken sit almost on top of each other in the model's view, with the ordering tipping slightly the wrong way, which suggests no threshold or score cutoff rescues this, because there's no signal to threshold on. A bigger model won't help either. Within one model family, the larger version was no better than the smaller.
Nor does this need a crowded pool. Strip the near-twins out and top-result accuracy is close to perfect; put back a single one and the best system drops to around two-thirds.
Which raises the question the study leaves alone. How often does a real codebase hold a broken near-copy of the code you want? The authors state plainly that they never measured it, and that's where the finding will get over-read, since this is a synthetic corpus of self-contained functions with the counterfactuals deliberately planted. On this evidence, though, the conditions look less exotic the more AI a team uses. Agent loops keep failed attempts lying around, commit histories get indexed alongside their own pre-fix versions, and monorepos fill with copy-pasted helpers that drifted apart years ago. A team whose agents try several implementations and retain the wreckage is manufacturing the exact pool this paper says breaks retrieval.
So stop reading code-search benchmark scores as evidence about correctness. They aren't measuring it, and the paper is precise about why.
"None of them pair every canonical with controlled single-edit buggy variants in the search pool, so a perfect topical retriever and a perfect functional retriever score identically on them, even though the two notions disagree on the cases that matter for downstream code use."
A vendor quoting a code-search accuracy figure has almost certainly measured whether the result was on-topic. Ask which one they meant. The same check runs in-house. Sample what retrieval hands back at position one and ask whether it works, not whether it reads as relevant. The design conclusion is to stop handing a single retrieved snippet to anything, model or human, as though the ranking had already done the filtering. If the working version is reliably in the top handful but rarely first, something later in the pipeline is doing the correctness work, and on most teams nobody built that stage on purpose.
Sources
- Kapoor, A., Khan, M. A. A. H., "ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval" — https://arxiv.org/abs/2609.01865