The standard recipe for getting an AI to write safe infrastructure code is to put a checker in the loop. The model writes some Terraform, a scanner grades it against a published security benchmark, the failures go back to the model, it tries again. Repeat until it passes or the budget runs out. Every published result on this approach says the same thing, that compliance goes up with each round.
Two researchers at Colorado State went back through thousands of these repair runs and looked at something nobody had measured. They asked not whether the failing checks got fixed, but whether the passing ones stayed passing. In about one run in seven, a security control that passed at one step was failing at the next. Tighten the definition to count only the cases with no measurement ambiguity whatsoever and it falls to 3.3%, which the authors adopt as their own headline rather than the larger number they were entitled to keep.
Either figure is invisible in the way these loops normally report themselves. The score they publish is the best result achieved so far, which by construction can only go up.
"What the data does support robustly is the qualitative claim: iterative IaC repair sometimes degrades a previously-passing security check, the effect is not negligible, and cumulative-best reporting hides it entirely."
The mechanism matters more than the rate, because it isn't really about Terraform. Nearly four in five of the breakages were classified as resource restructuring, where the model, asked to fix one finding, rewrote whole blocks of code instead of editing the offending line. That split is the output of an automated diff classifier the authors never checked against human labels, so it describes the shape of the problem rather than measuring it precisely. The signal that tracked regressions most closely was the churn in the check results themselves, how many controls flipped state between one iteration and the next. Diff size is the weaker of the two indicators, and much the easier one to watch. The loop gives the model no reason to be careful, because it never mentions what was already working.
"Just as security went unexamined during Mozilla's bug fixes, current feedback loops never tell the model which checks its previous iteration was already passing."
Three things follow for anyone running agents against a scorecard. Any loop reported on a best-so-far basis is describing achievements, not state. The defect is in the reporting rather than in Terraform, so it applies to test pass rates, coverage and lint scores too, though how often those regress and whether wholesale rewriting is the reason nobody has measured. Ask a vendor for the per-attempt trajectory rather than the headline. Diff size is the cheap version of the warning. If an agent rewrote a file to clear one finding, that's the change that needs a human, whatever the scanner now says. And iterations aren't free. Improvement here flattened by the third attempt, after which the loop was mostly buying churn. That is an aggregate stopping point for one benchmark, something to test against your own workload rather than adopt from this one.
"Tools should not blindly iterate to convergence."
The finding will get over-read, though. This is two models from late 2024, AWS Terraform only, one run per setting, with a static scan standing in for real security. The 3.3% isn't a rate to quote for anyone else's stack. What travels is the instrumentation. Record what passes at each step, put the previously passing checks back in front of the model, and read the trajectory instead of the high-water mark. An agent that rewrites rather than edits will break things nobody was watching. About a third of those breakages repaired themselves a step later, which is a good reason not to halt on the first one, and a poor reason to relax about the rest.
Sources
- Agyekum, B., Santos, F., "Does Fixing Break Security? An Empirical Study of Security Degradation in Iterative LLM-Driven Infrastructure-as-Code Repair" — https://arxiv.org/abs/2608.13404