Teams putting a coding agent near a payments system tend to write it a rules file covering how the repository is laid out, which conventions to follow and how the team works. A pre-registered study tested something different. Its preamble, 267 words and fixed for every task, says nothing about the house and only states what has to be true of the answer. Money never sits in the kind of number that quietly rounds. Timestamps carry a time zone. A request that gets retried does not charge twice. Passwords are stored as slow, salted hashes.
Backend tasks drawn from finance, healthcare and insurance work went to five frontier models from five different vendors, each task run twice, first as the ticket on its own and then as the same ticket behind the preamble. Nothing was regenerated, so every answer was a model's first and only attempt, and the scoring code was frozen and published before the first call went out. The bare prompts produced 148 defects of the named kinds. The prompts carrying the specification produced 23.
"In no model and in no domain did the frame arm record more findings than the bare arm."
Money arithmetic was the worst class in the unprompted runs and improved the most. Retry safety, whether a payment webhook delivered twice charges the customer twice, came close behind. Retry safety has no security-scanner rule behind it. The general-purpose scanner run over the same code as an independent check agreed with the result in every model, but would have caught almost none of that particular gain. Which means a team grading its AI-written code by scanner output alone is blind to the class where the preamble did its heaviest work. Checks for money, time and retries are cheap to write and belong beside the scanner.
Two tasks named their own time-zone requirement in the ticket text, and those produced no defects at all, with or without the preamble, from any model. The preamble is covering for tickets that leave the constraint unsaid, rather than making models smarter. The teams with most to gain are the ones whose tickets say least.
The largest thing the study didn't do is check whether any of the code worked.
"It does not show that code generated with the frame is correct: the checkers measure nine named classes, and a module can pass all nine and still be wrong."
That deserves more than a footnote. Work the author cites found that some methods for making generated code pass security checks partly succeed by deleting the risky lines. A short list of constraints seems unlikely to do that, but nothing here rules it out, and a team adopting the practice should watch its test suite as closely as its defect counts.
The messiest models improved most and the tidiest least. The author reads that gradient as the preamble supplying discipline a model lacks rather than topping up discipline it has. He flags it as an unplanned observation, and with a roster this small the study has no way to test the explanation. He also offers a reconciliation with the largest trial of rules files so far, which found they don't help.
"A typical instruction file tells the model how the team works. The frame tells the model what must be true of the output."
That is a hypothesis for follow-on work, not a result. Nothing here was delivered through a rules file, and nothing ran inside an agent. On this evidence the two artefacts do different jobs and should be judged separately, which matters because the tempting move, folding a constraint list into the rules file and assuming the effect travels, is exactly what nobody has tested. The author designed the preamble, designed most of the checks that scored it, wrote a book about the method, and discloses all of it. What makes the result stand anyway is the frozen predictions, the published raw outputs and an outside instrument that could have disagreed. Hold the next vendor claiming its prompt layer makes code safer to that same standard.
Sources
- Dhuri, S., "Specification Before Generation: A Pre-Registered, Five-Model Paired Evaluation of a Specification Frame for LLM-Generated Code in Money, Time, Idempotency, and Access Tasks" — https://arxiv.org/abs/2609.23270