Researchers at Purdue, Microsoft and Chicago built six ways of handing a coding agent the same abilities, then set it loose on real GitHub issues, 11,700 runs across three models. One agent got a plain shell and nothing else. One got a small set of named operations to view a file, replace a string or create a file. One got a search box that takes questions in English. One wrote Python instead of calling tools. Two got somewhere to jot down their thinking. None of it let the agent do anything it couldn't already do from a shell prompt.
The share of issues fixed came out much the same across all six. Then everything a team actually cares about moved.
"Even when two tool setups support similar tasks, they can still lead the model to behave differently because they present similar underlying capabilities through different interfaces."
Letting the agent write Python instead of firing tool calls one at a time got the same results with about 40% fewer steps and roughly half the input tokens. The saving came from fewer round trips, not cheaper ones. Every step re-sends the whole conversation so far, so the way to make an agent cheap is to make it stop asking. What the halving is worth in money is a separate question the study doesn't answer, since that depends on a provider's prices, and on how much of the repeated context gets billed at the cached rate.
The small set of named operations was the only setup that made repeated attempts at the same task more consistent for every model, mostly by cutting the sloppy stuff like broken edits and malformed commands. The weakest model gained most. The same tidy interface cost the two stronger ones about a fifth more in tokens, because a capable model chains several operations into one shell command and neat little tools chop that back into separate turns.
"This pattern suggests that the efficiency effect of tool architecture depends not only on what operations the interface makes available, but also on whether its granularity matches the actor's natural interaction style."
The two note-taking tools barely registered. They were deliberately bare, somewhere to write and nothing more, and the models used them to restate what they were going to say anyway.
"In other words, the scratchpad frequently acts as an explicit channel for restating existing reasoning, rather than as a tool that elicits substantially different intermediate thoughts or planning structure."
That leaves a measurement problem. All six setups scored the same on the metric most teams judge them by, which is whether it fixed the issue. A team A/B-testing its harness on pass rate will conclude the harness doesn't matter, while the two things engineers actually complain about are the two that moved. Nobody experiences an average success rate. They experience whether it worked when they asked, and what it cost. An agent that succeeds intermittently gets abandoned long before anyone files a ticket.
Then there's the finding a team should argue about. The right interface depends on the model behind it and on which property you're buying. The named operations bought consistency for every model and charged the strong ones tokens for it. Compound Python interaction, meanwhile, was the efficiency winner for every model in the study, the strongest included. So the split doesn't run between cheap models wanting guard rails and strong ones wanting the shell. Consistency and cost pull against each other, at an exchange rate that moves with the model. A harness tuned against a flagship still doesn't transfer downward, so a team switching to a cheaper tier to control spend may land on exactly the wrong tooling for it. "Our tool design improves agent performance" needs two follow-ups. On which model? And on which of the three things a team has to weigh together? Did it work, does it keep working, and what did it cost?
Two ways this will get over-read. It isn't evidence that semantic code search is snake oil. The search box here had no index behind it, just grep with a friendlier front door, and it still got the agent reading more of the right code, so some of the credit given to expensive retrieval infrastructure belongs to the interface. And the broader reading barely changed the patches the agent produced. Exploring more is not a result. All six were research interfaces built to hold capability constant, which is not the same thing as an evaluation of the commercial coding tools they resemble.
Sources
- Xu, X., Saghir, H., Wu, Q. et al., "The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior" — https://arxiv.org/abs/2608.11386