Writing an AGENTS.md or CLAUDE.md is now standard practice, and teams put hours into them: conventions, architecture notes, the house rules a new hire gets told in week one. A new study ran the controlled version of the question. Two frontier coding agents took on real merged pull requests from three Python repositories, each task attempted three ways: with no context file, with the file forced into every turn, and with the same material split into a wiki the agent could look up. Every attempt was scored against the project's own tests, which the agent never saw.
Just over half the tasks passed, and that held under all three conditions on both agents. The strategy that gave the agent nothing did as well as the strategy that shoved a curated guide into every single turn.
The author then went through the near-misses, the runs that failed by only a test or two, on the theory that those were likeliest to flip if some missing fact were supplied. One agent built an entire optimisation pass correctly and then introduced a bug in it. Another knew the rule perfectly well, having read it in the code, and wired the check up wrong. None of these were failures of knowledge about the repository. They were failures of judgement, and a document can't supply judgement.
"None of these failures isolate a knowable-fact gap that an AGENTS.md could fill."
The obvious objection is that the context files were simply bad ones. The author graded them against a rubric, one of them rating Excellent, then re-ran the two closest near-misses under every condition on both agents. The real, unmodified context file never once turned a failure into a pass.
One thing did work, and it points at what a context file is actually for. The file in one repository warned that the full test suite takes over 20 minutes. Given that warning, the agent stopped blindly running the whole suite and ran targeted tests instead, finishing about a quarter faster.
"Stripped of the AGENTS.md warning, the agent repeatedly runs the slow full suite; given the warning, it runs targeted tests instead."
That is what a useful line looks like, and most files are not full of them. Codebase layout, naming conventions, the patterns already in use: the agent gets all that by reading, and it does read. The lines that earn their place are the operational facts that cost something to discover: what's slow, what's flaky, which of two obvious paths wastes an afternoon. A file of those is short. A file restating what the linter already enforces is long, satisfying to write, and inert.
There is a related warning for anyone judging vendor claims. Earlier studies disagreed, one reporting that context files made agents cheaper to run, another reporting the opposite. Both numbers, this study argues, are bookkeeping about how text gets stuffed into a prompt rather than evidence the agent got better. When a tool markets a memory or context feature, ask whether the improvement claimed is in outcomes or in token accounting.
The result should not be over-read. Three Python repositories and seventeen tasks can rule out a large effect, not a small one, and what was tested was general-purpose house-style writing, not context built for a specific task. Nobody should delete their AGENTS.md on the strength of it. One more finding teams will feel directly, though: difficulty is agent-specific, and for roughly four tasks in ten the agent that would reveal an effect wasn't the same agent. A team running its own before-and-after on a context change is probably measuring tasks already too easy or too hard for the model in front of it, and will see no difference for mechanical reasons.
Sources
- Khatri, P., "Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories" — https://arxiv.org/abs/2607.27250