Most teams that run a security scanner in their pipeline read it two ways. A new alert blocks the merge. An alert that disappears after a change gets read as a fix. Automated fix assistants are scored the same way, counting a job done when the re-scan is clean and the tests pass. A new study from North Carolina State University tests that second assumption and finds it often wrong.

The researchers took more than a hundred real vulnerabilities in npm packages that CodeQL, the scanner behind GitHub code scanning, correctly flags. Most were path traversal and command injection. They gave an AI model only what the alert shows a developer (where the untrusted data comes in, where it does damage, and the route between) and asked it to rewrite one function so the flaw survived but the change looked like ordinary tidying. Two authors checked every claimed success by hand to confirm the vulnerability was still live.

With three attempts per flaw, the rewrites made CodeQL stop reporting about a quarter of the vulnerabilities it had caught, with the flaw intact in every case. The changes were small, usually a few lines. A typical one assembled a shell command from a list joined with a pipe character instead of gluing strings together. It produces the same command at runtime, but the scanner only follows that pattern when the separator is a space. Each silenced alert cost about 51 cents in model fees.

A control run rules out the obvious objection that any edit might do this. The same model, asked to refactor the same functions for readability with no mention of the alert, made larger changes and silenced nothing.

"The same generator, asked only to refactor the same functions with no knowledge of the alert, silences none of them."

The blind spots were not in obscure libraries but in common ones the scanner already claims to understand. The single biggest gap was the standard way of parsing a URL in modern JavaScript: the scanner tracks untrusted input into a URL's query string but not into its path, so reading a file path off a parsed URL drops the trail. That one gap accounted for more than half of the successful rewrites. Every gap the authors found is still present in the current CodeQL release. Nor does adding scanners rescue it. About half the silencing rewrites that two other analysers could check slipped past all three tools combined.

"A disappearing SAST alert is therefore weaker evidence than development workflows assume."

The authors argue this outcome needs no attacker. Any workflow that stops when the re-scan goes clean, whether a fix assistant or a developer trying variations until the warning stops, will sometimes land on one of these gaps. That is a reasonable argument, but it is not what the study measured. The model was told to keep the flaw. No real fix assistant trying in good faith to remove a vulnerability was tested, so how often one hides a flaw instead of fixing it remains open. The paper also flags that its own authors judged whether rewrites looked natural, knowing which ones had worked, and that nobody built working exploits. The flows were confirmed by tracing, not attack.

On this evidence, the practical change is to stop counting a vanished alert as a closed ticket. If a dashboard reports "vulnerabilities fixed" by counting alerts that went away, a share of that number is probably scanner blind spots, and the share is likely to grow as more fixes come from AI assistants that iterate against the scanner by design. For any change that touches a flagged code path, a clean re-scan is better treated as a prompt for a check than as the check itself: a test that tries the actual exploit, or a reviewer who follows the untrusted input to where it lands. The same scepticism belongs on vendor claims about autofix success rates. If "success" means the alert cleared and the tests passed, it is measuring the scanner's silence, which this study shows is cheap to produce.

The finding is narrower than it may sound. It covers JavaScript, mostly two kinds of flaw, and one scanner's models in depth. The deliberate case, a contributor who wants a flaw to survive review, is not where the study started. The authors point out that such a person sees exactly what the model saw, and needs nothing more.

Sources

  1. Cao, Z., Ding, W., Xu, B., "Preserve the Vulnerability, Break the Alert: Testing the Robustness of CodeQL Taint Tracking with Natural Rewrites" — https://arxiv.org/abs/2611.17123