Studies of AI-written code keep reporting how many security problems the machine left behind. Almost none report how many a person leaves behind on the same job, with the same scanners pointed at them. Without that, a number like "eight findings per resource" is unreadable, because it could mean the model is careless, or just that it wrote more infrastructure than a human would have.

A new benchmark supplies the missing half. A hundred cloud deployment scenarios went to every frontier model worth testing, with the output scanned by three independent policy engines. The new part is that a corpus of human-written infrastructure templates went through the identical toolchain.

"The practical question facing an engineering team is not whether generated IaC contains findings—all IaC does—but whether adopting a model changes their risk relative to writing it themselves."

The baseline exposes a flaw in how this normally gets measured. Flagged problems per resource drop sharply as templates get bigger, for people and machines alike, and models write much larger templates than humans do. Any comparison that doesn't correct for size is measuring verbosity, not security.

Corrected for it, the result is unusually tidy. Every configuration tested landed at roughly three and a half times the human rate of findings per resource. Open weights and closed, every vendor, nobody meaningfully better. The scenarios said what to build and never mentioned security, so this measures defaults, not whether a model can follow a security instruction.

Then the finding that should rearrange a process. The gap was widest on the simplest tasks, nearly five times on single-resource templates, shrinking to almost nothing on the largest architectures. That inverts the usual instinct, which is to scrutinise the big generated thing and wave the small one through.

"Review effort is therefore misallocated by default: it scales with artifact size, while relative risk scales inversely."

The move is to stop letting diff size decide what gets a human. A five-line bucket definition reads as obviously correct and is exactly where the model sits furthest from a competent engineer. Make the review trigger the resource type instead, flagging anything touching networking or access permissions, where most of the findings clustered.

The consistency across vendors matters for procurement, because it removes an option people assume they have. The author's reading is that the gap sits upstream of any vendor's safety work, in training data full of public repositories with permissive defaults. That is a reasonable hypothesis and the study can't test it. What the numbers support is narrower and enough.

"This argues against a model-selection remedy: choosing a different frontier model does not meaningfully change the security profile of the generated infrastructure."

Two smaller findings belong in the next vendor conversation. Turning on a model's built-in reasoning mode cut findings by about a tenth, which is measurable and nowhere near enough. Appending "think step by step" to the prompt did nothing measurable at all. And the model with the fewest raw findings was the worst per resource; it had simply failed to emit much valid infrastructure. A low absolute vulnerability count can as easily mean a model produces little usable output as that it produces safe output, which is reason enough to ask what the denominator was.

This will get over-read, because the findings are policy violations, not demonstrated attacks, and nobody established that any of them is exploitable. The study is also candid that its human corpus was never a matched control, since those templates were written for other purposes, many of them curated reference examples rather than production infrastructure, and the author says plainly they can't tell which way that biases the baseline. A comparison nobody had bothered to make now exists, and it points one way consistently. That makes it a starting figure, not a settled one.

Sources

  1. Shaw, A., "Compared to What? A Human-Anchored Security Benchmark for LLM-Generated Infrastructure-as-Code" — https://arxiv.org/abs/2608.28021