What Is an AI Agent Harness? The Scaffolding Explained
Agent harnesses turn raw LLMs into capable agents. Here's how the scaffolding evolved from GPT-2's simple loop to self-improving systems.
What is an AI agent harness?
An agent harness is the code and structure wrapped around a language model that turns raw text prediction into goal-directed behavior. It handles what goes into the model’s context, what tools it can call, how many turns it gets, and what happens to its output afterward. The model weights stay fixed. The harness is where almost all the practical intelligence gains have come from over the last few years, sometimes producing bigger jumps in task performance than swapping to a better model.
TL;DR
- A harness is the scaffolding around an LLM: the loop, the tool list, the memory rules, and the turn limits that decide what the model sees and what it’s allowed to do.
- On the ARC-AGI benchmark, the same underlying model went from roughly 30% to 95% or higher purely by changing the harness around it, not the weights.
- Harness history runs from GPT-2’s bare generate-until-end-of-sequence loop in 2019 through few-shot prompting, chain of thought, tool calling, memory, and multi-agent reflection.
- Tools like ReAct, Reflexion, and Voyager introduced the idea of agents checking their own work, storing reusable skills, and spawning sub-agents to handle bigger problems.
- The current frontier is self-improving harnesses, where the system rewrites its own prompts or structure instead of a human doing it, as seen in frameworks like DSPy.
- Static harnesses (fixed scaffolding, human-tuned) dominated for years; the newest systems are meta-harnesses that optimize themselves through something closer to genetic search than backpropagation.
- Researchers increasingly treat harness design as a serious lever for capability, not “just prompt engineering,” because it exposes how well a model adapts to new problems on the fly rather than how much it memorized in training.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
How did the first agent harnesses work?
The earliest version was almost nothing. When GPT-2 shipped in February 2019, its “harness” was just a loop: sample tokens with top-p sampling until you hit an end-of-sequence token, then stop. No tool calling, no memory, no chain of thought. Feed it a math word problem with a system prompt like “you are a math teacher,” and it would just emit an answer directly after a formatting marker, with no intermediate reasoning steps. Correctness was scored as a simple pass or fail.
Almost every advance since then has been about adding functionality to that loop rather than changing the underlying model architecture. Few-shot prompting (from the “Language Models are Few-Shot Learners” paper in 2020) let the harness stuff a few solved examples into the context so the model could infer the pattern before answering. Chain-of-thought prompting stretched the model’s reasoning across many tokens instead of demanding a direct answer, giving it room to work through logic instead of guessing.
What did tool use and memory add to the harness?
The next wave gave models the ability to act outside their own context window. WebGPT and Toolformer introduced the idea of exposing tools as callable functions: instead of trying to compute “5 minus 2” purely inside the weights, the model could call a subtraction function and get a reliable answer back. Tools got listed in the system prompt, and the harness handled dispatching calls and returning results.
MemGPT pushed further by giving models create, read, update, and delete access to a persistent memory store, rather than only ever appending to a growing context. That distinction matters: before this, a model could only pile more text onto what it already had. With editable memory, a harness could maintain a working notebook that persisted and got refined over time.
Voyager, built around agents playing Minecraft, introduced skills: reusable procedures that a harness could chain together, name, store in something like a “skills.md” file, and search when a similar task came up again. That pattern, an agent distilling a successful sequence of actions into something searchable and reusable, is close to what most people mean by “skills” in agent frameworks today. Intercode extended this by having agents output executable code directly, blurring the line between a predefined tool and an on-the-fly generated one.
How do agents check and improve their own work?
A separate line of development focused on self-correction. ReAct, Self-Refine, and Reflexion introduced the idea of an agent evaluating its own output before finalizing it. In a simple example, an agent solving a math problem might flip two numbers by mistake, have an internal evaluator flag the error, and get sent back to try again, either looping internally or pulling a reward signal from the actual environment.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
This evolved into multi-agent structures, where one agent could spawn sub-agents to handle pieces of a larger task, maintain a list of active sub-agents, and interact with them as persistent, running processes. Recursive versions of this pattern let an orchestrating agent call out to further agents as needed, effectively building a tree of specialized workers under one coordinator. This structure, an agent spec defining system prompt, turn limits, tool list, skill list, and sub-agent list, all wrapped in a loop, is what a mature “static” harness (sometimes called harness v1) looks like: capable, but not modifying itself.
Why does the harness matter more than model size?
Because the same model, given a better harness, can perform dramatically better on tasks that require adapting to something new rather than recalling something memorized. ARC-AGI is built specifically to test this: each puzzle is designed to require fluid reasoning on an unfamiliar pattern, not knowledge the model already has baked in from training. On ARC-AGI’s private holdout set, a frontier model with a fairly bare harness scored around 30%. With significantly more elaborate scaffolding, later systems (built independently by teams referred to as Prime Agent and an NVIDIA-affiliated effort called AVO) pushed that same class of underlying capability to 95% and beyond, reportedly reaching 100% on parts of the benchmark.
That gap, tens of percentage points on identical model weights, is why serious researchers now treat harness design as a real technical problem rather than dismissible “prompt engineering.” Model scaling improves something like general reasoning ability. Harness design improves something closer to test-time adaptability: how well a system can use its existing intelligence against a brand-new problem it hasn’t seen the shape of before. Those are different axes, and right now the second one is where a lot of the cheap, fast gains live.
What are self-improving harnesses?
The newest category flips the direction of control: instead of a human iterating on the harness by hand, the system iterates on itself. DSPy (short for “demonstrate, search, predict”) is a well-known example. Rather than a researcher hand-tuning a system prompt, DSPy takes a small set of training examples and searches for better prompts automatically, generating candidates, merging them, evaluating results, and repeating. Since you can’t backpropagate through this kind of discrete search the way you train a neural network, the process looks more like genetic programming: mutate, evaluate, keep what works, discard what doesn’t.
This is a meaningfully different idea from earlier harnesses. Static harnesses (tool calling, memory, reflection, sub-agents) all rely on a human designing the structure once and letting the model operate inside it. Self-improving harnesses treat the scaffolding itself as something to optimize, sometimes described informally as “Darwin machines” for the way they evolve their own prompts and structures over successive generations rather than staying fixed.
Frequently Asked Questions
What’s the difference between a harness and a model?
The model is the fixed set of weights that predicts text. The harness is everything around it: the system prompt, the tool definitions, the memory rules, the loop that decides when to stop, and any sub-agents it can call. You can swap harnesses on the exact same model and get very different results.
Is harness design considered legitimate research?
It’s debated. Some researchers have publicly dismissed it as “just prompt engineering” unworthy of serious academic attention. But results on benchmarks like ARC-AGI, where harness changes alone moved scores from roughly 30% to 95%+ on the same model, have pushed more people to treat it as a real technical problem, particularly around test-time adaptation.
What was the first agent harness?
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
The simplest identifiable harness is the loop used with GPT-2 in 2019: sample tokens until an end-of-sequence marker, then stop. It had no tools, no memory, and no reasoning steps, just direct generation. Everything since (few-shot examples, chain of thought, tool calling, memory, skills, reflection, sub-agents) has been added functionality layered onto that basic loop.
What makes a harness “self-improving”?
A self-improving harness modifies its own prompts or structure based on performance, rather than relying on a human to hand-tune it. DSPy is a concrete example: it searches over candidate system prompts using a genetic-programming-style process (generate, merge, evaluate, repeat) instead of manual iteration.
Why does ARC-AGI matter for evaluating harnesses?
ARC-AGI is designed so each puzzle requires reasoning about a new, unfamiliar pattern rather than recalling memorized knowledge. That makes it a clean test of how well a harness helps a model adapt on the fly, which is precisely the capability that scaffolding improvements target.