A model is asked to write a refund function for an online shop. The code credits the customer, marks the order refunded and records the refund. It is short, readable and passes every test. Then the server restarts halfway through, after the credit but before the order is marked, and an operator re-runs the failed job. Whether the customer gets refunded once or twice depends on a setting the code never mentions.

That is the opening example of a study from Bangladesh University of Engineering and Technology. The researchers had ten models write solutions to two dozen everyday backend tasks: placing orders, transferring money, sending queued emails, importing customers. They then crashed every solution just before each database write, file change or email, re-ran it the way an operator would, and checked what was left. They did all of it twice, under two settings of Python's standard SQLite library. In the default mode the library quietly opens a transaction before the first write, so a crash rolls everything back. In autocommit mode, which Django uses by default, every statement is saved the instant it runs.

"A crash test run in the default mode would pass this code. The pass is real, but it belongs to the library, not to the code."

Across 9,100 generated solutions, more than half (54%) of the database code that survived every crash in the default mode was left half-updated in autocommit mode. Every model did it. Even the strongest, Claude Sonnet 5, went from leaving 7% of its code half-updated to 31% when the hidden safety net was removed. None of that code stopped working under autocommit; it only broke when it crashed.

The failures were hard to see. Most of the database code had no transaction handling at all. Another slice wrapped its writes in Python's with conn: block, which reads like a transaction but opens none in autocommit mode, and nearly three quarters of that code was safe only by accident. A reviewer skimming a pull request would probably approve it.

The researchers then tried two cheap fixes. Telling the model in the prompt that the connection was in autocommit mode made the hosted models write much safer code. The smaller local code models barely changed, and a dummy note naming the default mode had much the same lack of effect on them.

"These models keep writing code that relies on a transaction they have been told they will not get."

Asking for crash safety outright helped every model. Given unsafe code and a report that it had failed a crash test, the strong hosted models fixed nearly all of it. Left to themselves, though, they mostly wrote safe code only when asked for it.

The study cannot say how often any of this happens in production. The researchers wrote the tasks themselves, and each solution was a single answer with no tests or tools, so an agent that runs its own tests might do better, though the authors note those tests would run in the default mode too. Nor does it compare models with people, whose code may lean on the same default.

The authors expect, but did not measure, the same effect wherever a default quietly supplies a guarantee, such as ORMs that wrap each request in a transaction. On this evidence, a test suite that runs in only one configuration may be grading the environment as well as the code. AI-generated code probably widens that gap, because the model writing a function cannot see how its caller sets up the connection.

For an engineering team the practical response is narrow. Code whose writes must succeed or fail together, especially anything that moves money, should open its own transaction, and review should treat a bare with conn: as a prompt to ask who opens it. Anything that sends email needs a record or an idempotency key so a re-run can skip it. The more useful change is probably to the test harness: run crash and retry tests in at least one less forgiving configuration than the one the code was developed in, because that second run is the only thing in this study that told borrowed safety from real safety. It also suggests a question for any vendor citing pass rates on generated backend code: passed under which settings?

The prompt result points the same way. A well-written note about the environment helped some models and was ignored by others, so context files and prompt notes do not replace checking the code under the condition the note describes.

Sources

  1. Mahmud, A., Rahman, M. A., Shahriyar, R., "Safe by Accident: Crash-Testing LLM-Written Code" — https://arxiv.org/abs/2611.10585