Six web applications, each built twice by the same coding assistant on the same model version, in one shot with no follow-up turns. The two prompts were identical down to the byte, except one had a generic paragraph of security requirements stapled on the end. Same paragraph every time, with no hint as to what that particular application was likely to get wrong.
The plain-spec builds came back with 51 confirmed security problems; the ones with the paragraph came back with 24, fewer in every single application. Severity is where it gets pointed. Every critical and high-severity problem the study confirmed sat on the plain-spec side, and the other builds had none at all.
"Descriptively, security-aware prompting was associated with fewer confirmed findings in all six applications and with the removal of every Critical and High confirmed issue, while the detection methods proved strongly complementary and the most severe true finding was uncovered only by manual testing."
None of what the plain builds got wrong was subtle. Authentication tokens were signed with a secret hard-coded into the source, so anyone who could read the repository could mint a valid token for any user account. Passwords in plaintext in half the applications. Debug mode left switched on in others. The model knows how to do all of this properly, and did the moment someone asked, which suggests a defaults problem rather than a capability one. Defaults belong to whoever owns the prompt scaffold. If a team's shared instruction file for its coding agents says nothing about authentication or secrets, this is a cheap argument for fixing that.
The second result deserves to outlive the first. Every application faced three kinds of check, scanners reading the source, a scanner probing the running instance, and a person testing by hand. Barely one in six of the confirmed problems was caught by more than one of them. The worst issue in the study, that forgeable token, was found only by hand, and no scanner flagged it as exploitable. The loudest automated alert turned out to be a false alarm.
"Relying on any single method, and especially on a single automated scanner, would have materially misrepresented the security posture of these applications."
A team acting on the first finding and ignoring the second ends up worse off than before. Add the paragraph, the obvious failures disappear, and a scanner comes back clean while the hard problems sit untouched. Those need somebody who understands what the application is supposed to mean, such as who may view this order, whether a price can be tampered with, and what happens when the last administrator locks themselves out. On this evidence, the question to put to a vendor claiming secure generation is not its pass rate but which methods produced it. A figure from source scanning alone measures about half the surface.
None of which establishes that the paragraph caused anything. Each version was generated exactly once, and these models don't produce the same thing twice; the author names this as the design's most serious limitation and the reason no causal claim is made. The study is also open that functional equivalence within each pair was only smoke-tested, so it can't rule out that the security-aware builds simply did less. A program that implements less has less to get wrong.
Sources
- Andročec, D., "Vibe Coding and Web Application Security: A Twin-Prompt Study" — https://arxiv.org/abs/2608.20963