Why Your AI Agent's Harness Matters More Than the Model for Cost
A benchmark shows agent harness choice, not model choice, cuts token costs by up to 75% versus Claude managed agents on identical tasks.
Direct answer: what actually drives agent costs?
A recent benchmark from True Foundry found that the software wrapped around a model, the harness, has more influence on agent token costs than the model itself. Running the same 14 enterprise tasks on the same model (Claude Opus), a self-managed harness called True Forge used 3.8 million tokens per run on average versus 10 million tokens for Claude’s managed agents, at matching accuracy. Swapping in a cheaper open model on top of the efficient harness cut total cost by 75% compared to the original Claude managed agent baseline.
TL;DR
- Harness efficiency beat model choice in True Foundry’s benchmark: the same Opus model used 62% fewer tokens (3.8M vs 10M) on True Forge compared to Claude managed agents, at equal accuracy.
- Stacking a cheaper model on an efficient harness compounds the savings, pushing total cost down 75% when GLM 5.2 replaced Opus inside True Forge.
- The benchmark used 14 tasks from DevRev’s enterprise bench, requiring agents to join data across a CRM, an issue tracker, and a document store through multiple MCP servers, scored by three trials and a blind LLM judge.
- Token waste usually comes from tool bloat and context management, not raw model capability: how many tool calls a harness makes, in what order, and what stays in the context window drives the bill.
- Managed agents from Claude, Gemini, and OpenAI lock you into their models and infrastructure, whereas open-source harnesses like True Forge let you swap in any model, including local ones.
- Data residency and zero-day retention requirements can rule out hosted managed agents entirely, pushing enterprise teams toward self-hosted alternatives regardless of raw performance.
- True Forge is MIT-licensed and installable with one command, running locally on SQLite with optional Postgres/Redis/Docker setups for hosted deployments.
What is an agent harness, and why does it matter?
An agent harness is the layer of software that sits between a model and the outside world: it manages tool calls, session state, sandboxing, approvals, streaming, and how much of the conversation history gets fed back into context on each turn. The model generates text. The harness decides what that text can act on, how many round trips it takes to finish a task, and what gets kept or dropped from the context window between steps.
There are roughly three categories in use today. Coding harnesses (Claude Code, Codex CLI, OpenCode, and similar tools) wrap a model with a file system, shell access, and a small toolset built specifically for writing and editing code. Framework-based harnesses (LangGraph, Deep Agents, various agent SDKs) hand you the building blocks and expect you to wire up state persistence, approval flows, and the execution loop yourself. Hosted managed agents (Claude’s managed agents, Gemini’s managed agents, OpenAI’s agent APIs) run that entire loop for you, but almost always tie you to that provider’s models and infrastructure.
True Forge, an open-source project from True Foundry, is positioned as a fourth option: a self-hosted version of the managed agent category. It handles the same plumbing (sessions, tool servers, sandboxes, approvals, a UI) but runs on your own infrastructure with any model you choose.
How much does harness choice actually save?
True Foundry’s team ran a controlled comparison to isolate the harness variable from the model variable. They used 14 tasks pulled from DevRev’s enterprise benchmark, each requiring an agent to pull and reconcile information across a CRM, an issue tracker, and a document store via multiple MCP servers, all using identical prompts. Each task ran three trials, and results were scored by a blind LLM judge to keep evaluation consistent.
The critical design choice: they didn’t switch models to make the harness look good. They ran Claude Opus through Claude’s own managed agent system and through True Forge, holding the model constant. Claude managed agents burned through about 10 million tokens per run on average. True Forge, running the identical model on the identical tasks, used about 3.8 million tokens, roughly a third as many, while matching accuracy. That’s a 30% cost reduction from harness efficiency alone, with zero change to the model.
Then they layered in a model swap. Replacing Opus with GLM 5.2 inside True Forge dropped the total cost by 75% relative to the original Claude managed agent run. That number combines two independent effects: a leaner harness and a cheaper model. But the headline finding is the first one. Even without touching the model, harness design alone accounted for roughly a third of the total token bill.
Why do inefficient harnesses burn so many tokens?
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Every token in a request costs money, and what actually fills the context window is largely a harness decision, not a model decision. A harness controls how many tool calls an agent makes to complete a task, the order those calls happen in, and how much of each tool’s output gets retained versus summarized or discarded before the next step.
Poorly designed harnesses tend to accumulate what’s sometimes called tool bloat: long tool definitions kept in context even when irrelevant to the current step, full outputs from every tool call retained rather than progressively disclosed, and redundant round trips because the agent isn’t given enough structure to plan efficiently. None of that is the model’s fault. A more capable model can sometimes compensate by being more efficient with fewer calls, but the ceiling on efficiency is set by the harness’s context management, not the model’s raw intelligence.
This is why the same model, run through two different harnesses, can produce wildly different token bills for identical accuracy. The model is answering questions. The harness is deciding how many questions it has to ask, and how much scaffolding surrounds each answer.
Is vendor lock-in a bigger problem than cost?
For many teams building inside larger organizations, yes. Hosted managed agent systems keep your agent’s entire definition, including the system prompt, tool wiring, approval logic, and session history, on the provider’s infrastructure. That’s workable until you want to switch models, change sandboxing behavior, or move to a different configuration, at which point you discover how much of your setup is tied to that specific vendor.
Two practical constraints push teams away from hosted managed agents regardless of quality. Data retention policies: if an organization requires zero-day data retention, some managed agent offerings simply don’t support that, which rules them out on compliance grounds alone. And infrastructure mandates: many enterprises require agent systems to run on-premises or inside their own cloud account, which a hosted managed agent product cannot satisfy no matter how capable it is.
Open-weight models compound this shift. As open models close the gap with proprietary ones, more providers are hosting the same weights behind compatible API endpoints, which means teams facing legal or geographic restrictions on data movement now have real alternatives instead of being stuck writing an entire agent loop from scratch just to stay compliant.
How does True Forge work in practice?
True Forge is an MIT-licensed TypeScript repository that installs with a single command and runs locally by default, using SQLite so data stays on your machine. It also supports hosted deployment modes using Postgres, Redis, and Docker for teams that want to run it as shared infrastructure rather than a local tool.
Configuration happens across four areas: models (Claude, Gemini, and other off-the-shelf APIs, plus fully custom endpoints for open or local models such as those served through Ollama or LM Studio), connectors (prebuilt integrations like Tavily for search or DeepWiki, plus the ability to wire up any custom MCP server), skills (importable skill definitions via a skill.md file), and sandbox providers (an area the project is still building out).
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
The built-in chat interface is a test bench, not the product. It’s there to verify a model, connector, and skill combination works before deploying it. The actual runtime is exposed as an HTTP API and a TypeScript SDK, meaning the finished agent can be embedded into Slack, a mobile app, or any internal tool, letting non-technical team members run predefined agent workflows without touching configuration.
Frequently Asked Questions
What is the difference between a managed agent and an agent harness?
A managed agent is a hosted service (like Claude’s or Gemini’s managed agents) where the provider runs the entire execution loop, tool calls, sessions, and sandboxing, for you, typically restricted to their own models. An agent harness is the general term for that execution layer; it can be hosted by a vendor or self-run, and open-source harnesses let you plug in any model.
Does a more expensive model always produce better agent results?
Not necessarily, and not in a way that’s separable from the harness. True Foundry’s benchmark held the model constant (Claude Opus) and still found a 62% token reduction from switching harnesses alone, at matching accuracy, which indicates the harness was the primary driver of cost in that comparison, not model capability.
Why would a team choose a self-hosted harness over a managed agent service?
The two main reasons are compliance and control. Some organizations require zero-day data retention or on-premises/private cloud deployment, which hosted managed agent services don’t support. Self-hosted, open-source harnesses also avoid lock-in: the agent’s prompts, tool wiring, and session history stay on infrastructure the team controls rather than a vendor’s platform.
What does “tool bloat” mean in the context of agent costs?
It refers to unnecessary tokens consumed by tool definitions, full tool outputs, or redundant tool calls that stay in an agent’s context window longer than needed. Since context window content is what gets billed, a harness that doesn’t manage this well (no progressive disclosure, no summarization) drives up token usage regardless of which model is generating responses.
Can open-source agent harnesses use local models?
Yes. Harnesses built for model flexibility, such as True Forge, support connecting to local model servers like Ollama or LM Studio in addition to hosted APIs from providers like Anthropic, Google, or open-model hosts, which matters for teams wanting to keep inference entirely on their own hardware.
