Thank you for opening up the latest article in 'The Intelligent Commit' newsletter, my thoughts, research and analysis on Artificial Intelligence and how AI is reshaping the software development lifecycle. I hope you find it insightful, subscribe, and engage.
This is the first in a three-part series about the findings from my master's dissertation on the security of LLM-generated code.
For my Master's research dissertation, I chose a subject that sits directly between my studies and my work in engineering leadership: what happens to software security when more of the code is generated by AI?
I later turned the research into a talk, The Complexity Paradox, which I presented at the Northern DevOps & Platform Conference in Leeds back in May. The title came from the finding I least expected when I began the study, and one I called out in my paper as demanding further research: simple tasks produced the least secure code.
Testing three ways of working with AI
The research compared three simulated development conditions. Each used the same LLM and the same baseline prompts, and each worked through the same set of 100 tasks. Those tasks built 18 products and end-user applications, drawn from scouring public tutorials and videos on product ideas, apps and the like.
The first condition represented naive "vibe coding": simple baseline prompts that described the desired functionality in plain English, and no more.
The second built on each baseline prompt by appending context-relevant, security-aware requirements following OWASP Top 10 and MITRE CWE guidance.
The third fed the original baseline English prompt - with no appended security or technical guidance - into an agentic workflow with separate supervisor, coder and security-reviewer roles.
Eighteen applications in each condition produced 54 applications overall. I scanned the resulting code with Bandit, Bearer, SonarQube and Grype, then reviewed and suppressed false positives before analysing and classifying the vulnerabilities.
There were several useful results, but the relationship between complexity and security stood out. I expected simpler work to generate fewer vulnerabilities. I was mistaken.
Under naive vibe coding, the simple products carried just over nine times the vulnerability density of the complex ones.
Under the naive vibe-coding condition, simple products contained 19.22 vulnerabilities per thousand lines of code. Complex products contained 2.13. The vulnerability density of the simple work was just over nine times higher.
The pattern held under security-aware prompting, where simple work carried 6.3 times the vulnerability density of complex work.
Interestingly, it disappeared in the agentic condition - more on that later in the series.
Why might simple work be less secure?
The study established the pattern, not its cause. The possible explanations are questions for further research, so treat what follows as hypotheses rather than findings.
The simplest answer may be that it is just statistics: complex codebases contain more lines of code, which in turn drives down the density of vulnerabilities. Plausible, but not one I subscribe to on its own. Here are a few more candidates, and how they might translate into the real world.
Attention is one. Difficult work announces itself. Developers slow down, supply more context and examine the results more carefully. A task that sounds routine may receive a short prompt and a quick approval.
Prompt detail may create a similar effect. A request for a complex feature tends to describe its data, behaviour and constraints. "Build a login form" sounds self-contained, yet leaves important decisions about authentication, validation, sessions and error handling unstated.
Training data could also matter. Complex implementation examples may be accompanied by fuller discussion of design and security trade-offs, giving the model a stronger signal than a short example of an apparently basic feature. Those trivial implementations may have been lifted from demo apps and getting-started guides that never went through hardening.
These are plausible mechanisms, not conclusions. The study used one model, one programming language and a moderate sample of products. Replication across models and languages is needed before treating the size of the effect as general.
The practical lesson
Engineering teams naturally concentrate review effort around visibly difficult changes. We bring senior people into complicated designs, write detailed acceptance criteria and expect several rounds of review.
Small changes often travel through a faster path. AI coding tools make that path faster still.
My findings suggest that task simplicity is a poor proxy for security risk. A change can be easy to describe while still touching authentication, data handling or access control. The smallest prompt may contain the largest set of unstated assumptions.
That has implications for code review, for platform defaults and for the way we introduce AI-assisted development. Security checks should follow the risk of the behaviour being implemented, rather than our impression of how difficult the implementation appears.
The tasks we trust most may be the ones we inspect least.
In the second article in this series I explore what happened when security knowledge was added to the prompt. The third examines whether automated reviewer agents can apply that knowledge more consistently.
Thank you for reading my latest article. Please share your experiences and engage with the discussion - I am not always right, but I try to be insightful and thought-provoking.
- Where does your path to production treat "small" as a synonym for "low risk"?
- If a change touches authentication or data handling, does your review process notice, or does it only notice how large the diff is?
- What would it take to route review effort by the behaviour being changed rather than by how difficult the work looked?
Please reach out if you'd like to continue the discussion.