Before a coding assistant writes anything, it gathers context: the files it thinks are relevant, similar functions elsewhere in the repository, the definitions the code imports. That step is tuned almost entirely to produce code that works. Whether it also surfaces the code that keeps the project safe, such as the permission check in another module or the validation routine every sibling parser calls, had not been tested in a controlled way.
Researchers from Tencent, Nanyang Technological University, Zhejiang University and the Chinese Academy of Sciences built a benchmark to find out. They drew 145 tasks from real, publicly disclosed vulnerabilities in open-source projects. For each, they rolled the project back to just before the fix, blanked out the vulnerable code and asked models to rewrite it from a plain description of what it should do. The description never mentioned security. Each answer was built, run against functional tests, then attacked with the real exploit. Six models were tried with more than a dozen ways of assembling context, and three coding agent setups were let loose on the full repository.
Context that made code work more often barely made it safer. The best of the standard retrieval methods raised the share of working code noticeably, but none of the methods reliably raised the share of secure code above showing the model the target file alone. Between four and five in every ten answers worked but could still be exploited. One method that produced the most working code owed most of that gain to code that passed the tests and still fell to the exploit.
The agents did no better. They read many times as much of the repository as the retrieval methods supplied, yet the agents saw the relevant safeguard in only about a third of their runs, while on tasks that had one, they came across an unfixed, equally vulnerable sibling more often than that. Their searches point to why, since fewer than one in ten queries named a security concept. Most went looking for the file format, the function, the feature.
"Functionality-oriented methods build their queries from the target code and the requirement, which describe what the code does rather than which policy constrains it."
The controlled experiments make the mechanism concrete. Removing a single safeguard from an otherwise ideal context cut secure output, and inserting one unfixed sibling raised exploitable output even with the safeguard still present. Padding the context with the same amount of irrelevant code had almost no effect.
"What a context shows thus matters far more than how much it shows."
Seeing the safeguard was not enough on its own. In 41% of agent runs that did encounter it, the agent neither called it nor copied it. In one medical-records case, some answers written with the exact permission rule in front of the model still checked it only after the request had already been processed.
The authors suggest retrieving a project's safeguards deliberately and checking retrieved examples before they serve as templates. They are candid that their agent analysis is observational. It shows that runs which saw the safeguard were more often secure, not that forcing an agent to look for one would make it so. That fix remains untested here.
For an engineering leader, the failure is worryingly ordinary. Insecure examples in retrieved code can undermine a model’s use of a safeguard. Most codebases carry legacy code written before a team tightened its rules, and an agent that searches by similarity can retrieve that older code. A larger context window did not fix this in the study, and the study's longest context, a slice of the repository many times larger than any other, produced code no more secure than the target file alone.
A vendor claim that an agent "understands your whole codebase" says what it reads and nothing about which parts it treats as binding. Security policy that lives in one central module, with no visible trace at the point of use, is exactly the policy these tools miss, which argues for making checks obvious where the sensitive operation happens. And a team measuring its agent by how often the pull request passes CI is measuring the half of the result that improved. The figure most likely to be over-read is the benchmark's low secure rate. These tasks were chosen because a human developer got them wrong first, so they are a hard, security-loaded sample and say little about how often everyday agent output is exploitable. The direction of the gap, working code improving faster than safe code, is the part worth taking into a review checklist.
Sources
- Yu, J., Lian, K., Wang, J. et al., "A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code" — https://arxiv.org/abs/2611.06294