MCP vs. CLI: Does an AI Agent’s Tool Interface Still Matter?

We ran a controlled 1-to-1 comparison of MCP vs. CLI interfaces on the same backend servers and tools across 50 long-horizon agentic tasks. What we found:

• CLI is not a better tool interface than MCP by default. When the underlying tools are identical, whether CLI outperforms MCP largely depends on the model and the task: CLI helps less capable models on retrieval-heavy tasks, but is neutral or worse on lighter, more targeted tasks. Also, giving the model both interfaces doesn't automatically make it route between them.
• Interface choice matters much less for the latest frontier models. Earlier models were sensitive to it, but Opus 4.8 and GPT-5.5 perform similarly across all of our interface setups.
• CLI helps mainly by compensating for tool-use errors. CLI reduces tool-use errors for less capable models; more capable models don't make those errors, so the bottleneck shifts to planning and coverage, which a better interface can't fix.
• Interface choice still affects cost, even when performance converges. While task success rate for latest frontier models converges across interface setups, the differences that remain are trajectory length, latency, and dollar cost.

Motivation

We set out to answer one question: when the backend operations are exactly the same, does the tool interface still change how well an agent performs?

The Model Context Protocol (MCP) has quickly become a common way to expose structured tools to agents, especially in enterprise-style environments where actions need to be typed, auditable, and permissioned. At the same time, recent work (e.g., Terminal Agents Suffice for Enterprise Automation and MCP is up to 32× more expensive than CLI) has suggested that terminal-style agents can be highly effective, matching or outperforming agents built around structured MCP interfaces.

But this comparison leaves an important question unresolved. In many systems, the terminal agent is not only using a different interface; it may also be operating against a different tool surface, with different API coverage, different documentation, and different opportunities for programmatic composition.

At the same time, other work has shown that MCP performance can be limited by the way tools are exposed: incomplete or ambiguous tool descriptions, large tool inventories, prompt bloat, and difficult tool selection. In other words, when an MCP agent underperforms, it is not obvious whether the problem is MCP as an interface, the specific MCP server implementation, or the size and quality of the tool surface.

Our main finding is that the interface can affect agent performance, but that sensitivity is shrinking quickly as model capability improves.

Experimental Setup: Same Tools, Different Interfaces

We built a total of 50 long-horizon agentic tasks across two realistic, enterprise-style environments, each containing thousands of artifacts and server entities designed to mimic a different type of real-world business.

Environment 1 is lighter, containing 8 servers and 113 tools over a modest data volume of ~2,000 artifacts, where tasks focus on precise lookups and targeted updates across different applications.

Environment 2 is high-cardinality and data-heavy, containing over 30,000 data artifacts across 8 servers and 140 tools, where tasks require broad retrieval, repeated inspection, and gathering evidence across many records.

The agent can use tools to inspect or modify the environment. Our setup allows each tool to be exposed in three different ways:

1. As an MCP tool: the agent sees the operation as a structured MCP tool with a tool name, description, and parameter schema.
2. As a CLI command: the agent sees the operation as a command-line interface.
3. As both: the agent sees both the MCP tool and the CLI command for the same operation, and can choose which interface to use.

The key control is that the CLI command is generated from the MCP tool schemas. Every MCP tool has a corresponding CLI command with the same underlying backend capability from the server. Therefore, the agent is not getting a better tool in one form factor than another.

To keep the CLI realistic, we exposed it the way an agent would encounter one in practice: a lightweight skill pointing to the binary and its --help for discovery, rather than a full embedded command schema. 

Tasks from both environments are intentionally hard for current frontier agents. Successful completion of a task usually requires 50 to 100 tool calls, multi-source retrieval, keeping track of intermediate state, long-horizon reasoning and information synthesis, and producing the correct final state rather than just a plausible text response.

Each of the 50 tasks comes with 11 to 49 (median 24) human-created rubric criteria. The final trajectory is evaluated against those criteria using an LLM judge.

Throughout, we use "less capable" and "more capable" to describe the agent along a single axis in two ways: model generation (Opus 4.6 vs 4.8, GPT-5.4 vs 5.5) and reasoning effort (GPT-5.4 with reasoning off vs reasoning on). We used Claude Code harness for Opus 4.6 and Opus 4.8, and OpenAI Agents SDK and the Responses API for GPT-5.4 and GPT-5.5. The reasoning effort is set to “medium” for all four models, except for the setup when we turned off reasoning on GPT-5.4.

With each model, we tested four interface conditions with at least 4 runs per task, per interface condition:

• MCP-only: the agent only sees MCP tools.
• CLI-only: the agent only sees CLI commands.
• Both: the agent can access every tool in both interfaces; CLI and MCP interface expose the same backend operations
• Hybrid: broad search/list operations are routed to CLI, while targeted reads, writes, and state-changing operations remain on MCP.

We found the main result to be conditional.

Less Capable Models Are More Sensitive to Interfaces

The clearest interface effects appear when the model is less capable: With Opus 4.6, interface choice had a clear conditional effect. On the data-heavy Environment 2 tasks, CLI improved task success over MCP: roughly from 8.5% to 17.7%. On the lighter Environment 1 tasks, the direction reversed: MCP was stronger than CLI, roughly from 7.3% to 11.8%. CLI was not uniformly better. It helped most where the task required broad retrieval, repeated inspection, and gathering evidence across many records.

We see the same pattern in GPT-5.4 when reasoning is turned off. With reasoning off, task success is close to 0, so the binary task-level pass metric is no longer a reliable differentiator, therefore we look at the percentage of rubric criteria that were passed for each task instead. CLI shows substantially more progress than MCP: roughly passing 9% more rubric criteria overall and 14% more on Environment 2 tasks alone.

The split between the two environments suggests that CLI is not inherently superior to MCP on all tasks. For less capable models, CLI helps especially on data-heavy tasks that require broader tool calls and retrieval. We think it is because CLI encourages the model to work more iteratively by trying scoped commands, inspecting outputs, and refining the search over multiple steps.

More Capable Models Flatten the Interface Gap

As model capability improves, the performance gap between MCP and CLI interfaces becomes much smaller.

With Opus 4.8, MCP-only, CLI-only, Both, and Hybrid all perform similarly on the full task set. GPT-5.5 shows a similar pattern: MCP, CLI, Both, and Hybrid all performed closely, with a modest edge to CLI and Hybrid. For GPT-5.4 with reasoning turned on, task success rises sharply and the pure MCP-vs-CLI gap collapses to almost zero, although the both interface setup still performed significantly worse than deliberate hybrid routing.

With the latest released models from both Opus and GPT family, agent performed a lot more similarly across different interface setups on our tasks, suggesting that more capable models are increasingly able to adapt to the interface they are given.

Why the Interface Gap Shrinks

To better understand why tool interface sensitivity fades as models get more capable, we label each failed rubric criterion as an outcome, tool-selection, query-construction, or hallucination error.

With reasoning turned off, GPT-5.4 makes far more process errors, and switching it from MCP to CLI reduces failures across the board, most sharply on query construction (36% to 23%).

When reasoning is turned on, those process errors largely disappear. Holding MCP as the tool interface, tool-selection errors drop from 20% to 7% and query-construction errors drop from 36% to 16%, leaving much less for CLI to fix. That’s why the broad CLI advantage shrinks with model capability: CLI helps less capable models by reducing tool-use process errors, while more capable models make far fewer of those errors.

CLI Runs Longer, But Not Always Costlier

CLI can help agents explore more broadly, but at the cost of longer trajectories: CLI-only runs take more turns than MCP-only, with extra steps on command discovery, retries, and output parsing. CLI is also 20% to 40% slower on wall clock time compared to MCP-only. Part of that extra length is command discovery, which is a direct consequence of how we expose CLI to the model, as described in the setup. Follow-up experiments with pre-documented CLI commands, even though less realistic, would help isolate the interface effect.

The token and dollar cost picture is less straightforward, and it doesn't track turn count. For Opus 4.8, MCP is cheaper than CLI on both per-run and per-solved-task cost. For GPT-5.5, the cost flips, with CLI being the cheaper interface, despite using more turns. The difference appears to come from harness behavior rather than the interfaces themselves. We ran Opus with the Claude Code harness, which spills large MCP outputs to files, so CLI’s longer trajectories dominate cost. We ran GPT-5.5 with the OpenAI Agents SDK, which re-sends large MCP outputs inline across calls, so MCP becomes more expensive despite fewer turns.

Although not included in our experiments, a more coding-agent-like harness could in theory reduce GPT-5.5’s MCP-side cost because it could save large MCP outputs to files and allow the model to inspect those outputs selectively.

* Prices ($/M tokens): Opus 4.8 base input $5 / cache-read $0.50 / cache-write $6.25 / output $25; GPT-5.5 input $5 / cached $0.50 / output $30. $/run is the mean over measured per-run token usage; cache splits are read exactly from the trajectories.

What This Means in Practice

Our findings suggest that the practical guidance to choose between tool interfaces for AI agents is not “one is always better than the other.” This is especially true for the latest frontier models. The choice is often more nuanced, and depends on the operation and model capabilities.

Don’t assume CLI is the better default. The common assumption is that terminal-style agents are simply better for long-horizon work. We didn’t find that to be a universal rule. CLI's advantage is conditional: it helped less capable models on data-heavy tasks, where Opus 4.6 roughly doubled its pass rate on tasks from our data-heavy environment, but it was flat or worse on lighter, more targeted tasks. MCP remained the better starting point for precise reads and writes. Treat CLI as something you add where broad search and iteration actually need it, not as a complete replacement of MCP.

Don't expose both interfaces and expect the model to choose. In our experiments, giving the agent both MCP and CLI did not produce smart routing between the two interfaces. Models almost always fell back to MCP and barely touched the CLI. A deliberate split that routes bulk search and listing through CLI and keeps targeted reads and writes on MCP was the most reliable configuration for a handful of models in our runs, but the model did not figure that out. If you want the benefits of both, encode the interface split yourself instead of leaving it to the model.

Decide not only on accuracy, but on cost, and measure it per solved task. For the latest frontier models, the interfaces land in roughly the same place on success rate, so the deciding factor becomes efficiency, and there they differ. CLI trajectories ran 20 to 40% longer on wall clock time and used more turns than MCP. Dollar cost, though, didn't follow turns and even reversed by harness: Opus 4.8 on Claude Code was cheaper on MCP, but GPT-5.5 on OpenAI Agent SDK was cheaper on CLI. So compare cost per solved task on your own stack rather than assuming one interface is leaner, and keep in mind that the harness can matter as much as the interface.

Don't mistake clean tool calls for quality, and don't expect the interface to supply reasoning. Raw tool-error counts are not a good proxy for agent quality in long-horizon tasks. Many CLI errors were recoverable: GPT-5.5 made more raw command errors than Opus and still finished stronger in pure CLI. With more capable models, the failures that remained were rarely about calling tools, but about coverage and reasoning. A better interface can scaffold a less capable model through that work, but it can't help with the planning and state-tracking a hard task demands.