424 deliberately security-sensitive Python tasks (shelling out to the operating system, unpacking untrusted data, handling passwords), put to a large commercial model and a small open-weight one. Five system prompts, escalating from a bare instruction, through formatting and style rules, to explicit demands for secure code and finally an instruction to assume hostile users. Everything that came back went through two security scanners.
The first result is about compliance. Under the bare prompt the commercial model refused outright, declining to produce code for around four out of five tasks. Adding structure fixed that, in the form of a persona, an output format and style rules, with not one word about security. The largest single effect of prompt engineering here had nothing to do with security. It was getting the model to answer at all. That moves the goalposts under any before-and-after comparison.
"Comparing counts of security weaknesses without conditioning on successful code generation can therefore misattribute improvements to security rather than compliance."
Counting only the answers that were actually code, the picture flattens. The share of files carrying a flagged weakness barely moved, at roughly three-quarters, with or without the security language. The mix shifted. The most serious findings shrank from about a fifth of the total to about an eighth, and the mildest took up the slack. The two commonest flaws, command injection and unsafe deserialization, held firm, and command injection got more common rather than less.
Then the part that belongs in a design review. Reading a sample of the code by hand, the researchers looked for cases where the model had swapped out an unsafe construct the task explicitly asked for. Under a plain structured prompt it did this in one task in five; once the prompt asked for security, in two out of three.
"In many of these cases, unsafe constructs central to the task specification are replaced or removed entirely, despite being explicitly required."
The authors call this a trade-off between security and fidelity to the specification, and are careful to say the outputs still addressed the task. A security preamble is partly a spec-editing instruction, and the code got safer in part because the brief changed underneath it. Fine, when the requested construct was careless. A problem when it was the point, like a legacy wire format your system has to speak. Nothing announces the substitution; it arrives as ordinary code that scans clean.
So the number to distrust is a drop in scanner findings after a prompt change. On this evidence it can mean fewer refusals, milder flaws, or a requirement the model rewrote on its own, and none of those is a safer system. Ask a vendor claiming fewer vulnerabilities from secure prompting what the denominator was, and whether severity was reported apart from volume. The cheap internal habit is to review AI-written code against the specification as well as the scanner, which is where this failure lives.
The study can't settle its own central worry. It never executed a single generated program, because the tasks are unsafe by construction, so it can't say whether the rewritten versions still did the job. The authors name this as the obvious next experiment. The drift audit is a small sample, the tasks are synthetic, and each configuration was generated once, so none of the numbers here is precise.
This will be over-read as an argument against asking for secure code. It isn't one, because the serious findings did fall. The narrower claim is that prompt guidance shapes what a model builds, and shaping is not the same as securing.
Sources
- Urmi, M. D., Pourleyli, J., Santos, F. et al., "Prompt Structure Redistributes, Not Reduces: An Empirical Analysis of Security-Weaknesses in LLM-Generated Python Code" — https://arxiv.org/abs/2608.24857