Skip to main content
MindStudio
Pricing
BlogAbout
My Workspace
AI agent evalshow to evaluate AI agentsLLM as a judge

How to Build Evals for Your AI Agent (So It Doesn't Break in Prod)

A practical framework for testing AI agents: code checks, LLM judges, and error analysis to find failures before users do.

Edited by Luis Chavez-Mattos, Director of Product RSS
How to Build Evals for Your AI Agent (So It Doesn't Break in Prod)

Why testing an AI agent isn’t like testing normal code

An AI agent eval is a repeatable scoring system you run against your agent to measure whether it actually does its job, not just whether it sounds right on a handful of test questions. Regular software testing assumes a function returns the same output every time, so a test either passes or fails. Agents break that assumption twice over: the model samples its output, so the same question can produce three different answers on three different runs, and there’s usually no single correct string to match against. A real evaluation framework has to check properties of an answer (is the number right, did it follow business rules, did it make something up) rather than exact text, and it has to run multiple times per question to catch the cases where an agent is right most of the time but wrong often enough to matter.

TL;DR

  • Agents need property-based testing, not string matching, because sampling means the same question can produce different correct-looking answers on different runs, so you score properties (correctness, rule compliance, no hallucination) instead of exact text.
  • Public benchmarks and product evals solve different problems: benchmarks like MMLU Pro or Bird tell you if a model is generally good at a task, but only your own test questions can check whether it follows your specific business rules, like a fiscal year that starts in February.
  • Grading should combine three tools in sequence: start with a human reading real transcripts to understand failure modes, add code checks for anything with an exact answer, and reserve an LLM judge for the ambiguous cases left over.
  • Error analysis beats guessing, because reading a batch of real agent conversations and tagging what went wrong first often surfaces failure clusters (like date handling) that nobody thought to test for in advance, while supposed risks (like hallucination) sometimes don’t show up at all.
  • Your grader can be the bug, not the agent. A code check that compares the wrong field, or a judge prompt that’s too vague, will produce false failures and false passes, so you have to audit the grader against human labels before trusting any score.
  • A naive LLM judge barely catches anything. A simple “is this correct and helpful” prompt can miss the vast majority of real failures because a confident, well-formatted wrong answer still looks correct to a model skimming it.
  • Fixing one failure can create another, so evals need to run on the full question set after every change, not just the subset you were trying to fix, and the whole process has to repeat as usage and data drift over time.
Build enterprise applications with AI that are
Secure.Role: Adjuster✕ Export PHI
Regulation-compliant.✓ HIPAA✓ SOC 2
Auditable.09:14M. Chen approved claim #4471
Scalable.12,480 users online

What’s the difference between model evals and product evals?

A model eval is a public benchmark: a fixed set of questions every lab runs so models can be compared against each other. MMLU Pro is one example for general knowledge, and Bird is a benchmark specifically for how well a model turns natural language into SQL. These are useful for picking a model that’s generally strong at a task category, whether that’s SQL generation, agentic coding, or something else.

The problem is that a benchmark only knows about its own test database. It has no idea that your company’s fiscal year starts in February, that three customer accounts in your database are internal test accounts that should never be counted, or that a product got renamed last year. A product eval fills that gap: it’s a set of questions your actual users would ask, graded against your actual business rules. The practical split is to use benchmarks to choose a capable model, then build product evals to confirm the agent applies that capability the way your business needs.

How do you grade an agent’s answers?

There are three types of graders, and in practice most real evaluation setups use all three.

Code is a script that checks an answer in a fixed, deterministic way. For an agent that writes SQL, that means running the query it generated and comparing the resulting rows against a query you know is correct. Code is fast, cheap, and consistent, but it only works for things that can be checked exactly. It can’t tell you whether the agent stated its assumptions or used the right definition of a business term in a written explanation.

LLM as a judge fills that gap. Another model reads the agent’s answer and scores it against a rubric written in plain English. The catch is that a judge is still a language model, with its own biases, including a documented tendency to favor whichever answer it sees first, particularly in smaller models.

Human review is the slowest option but the most trustworthy one, and it’s what you use to check the other two. A developer or someone who knows the business reads answers and labels them pass or fail. Those labels become the ground truth you use to validate both your code checks and your judge prompt.

The order matters. Writing a judge rubric before you’ve read any real failures means you’re guessing at what to check for. The better sequence is: read real answers as a human first, write code checks for anything exact, then build a judge for whatever’s left that’s too ambiguous for code.

How do you find out what’s actually going wrong?

The process is called error analysis: read a batch of real agent conversations, write down the first thing that went wrong in each one, then group and count those notes to see which problems show up most often. This tells you where to spend your limited testing time instead of guessing.

Every team builds with AI.
Nothing gets lost.

In a demonstrated example with a data agent answering questions against a coffee wholesaler’s database, this surfaced two dominant failure clusters. The biggest was fiscal calendar handling: whenever a question used phrases like “last quarter” or “year-to-date,” the agent defaulted to calendar dates because nothing told it the business’s fiscal year starts in February. The second cluster was revenue definition: the agent reported sales before refunds were subtracted, instead of net revenue.

What’s notable is what didn’t show up. A set of questions designed to test for hallucination, asking about data the company simply doesn’t track, like profit margins, consistently got an honest “I don’t have that data” response rather than a fabricated number. The failure mode everyone worries about most wasn’t actually the problem in this case. Teams that write their eval rubric before reading real outputs risk spending their time testing for the wrong failure entirely.

Why might your grading code be wrong instead of your agent?

Because the grader is also something you wrote, and it can have bugs just like the agent it’s scoring. In the same demonstrated example, a code check for “which region made the most revenue” was marking correct answers as wrong because it also tried to match an exact revenue figure that the question never asked for. Fixing that single grading bug eliminated about a quarter of the “failures” in that benchmark.

The same applies to LLM judges. A naive judge prompt that just asks “is this answer correct and helpful” can catch only a small fraction of real failures, because a wrong number embedded in a clean, confident table still reads as correct and helpful to a model skimming it. A sharper version of the judge, built directly from the clusters found in error analysis (fiscal calendar, revenue definition, test account exclusion, missing-data handling), catches far more real failures, but can initially flag some correct answers as wrong too. Resolving the disagreements between that judge and human labels, case by case, improves both: in a worked example, several of the disagreements turned out to be mislabeled ground truth, not judge error, including a case where two internal test accounts had incorrectly been counted as real customers.

How do you fix an agent once you know what’s broken?

Once error analysis identifies the gap, the fix is often simple. In the SQL agent example, the agent didn’t know the finance team’s unwritten rules, so those rules got added directly into the system prompt. Rerunning the full eval set afterward showed the agent’s accuracy on business-rule questions roughly double, while performance on simpler benchmark-style questions stayed flat, because the new instructions fixed one case and broke another elsewhere (one question started failing entirely because the agent used up its step budget on extra tool calls).

That last detail is the reason the full eval set has to rerun after every change, not just the subset tied to the fix. A change that solves one problem can quietly introduce a new one somewhere else in the system.

Is this a one-time setup or an ongoing process?

Developers have their tools. Enterprises have Remy.
Developers · AI coding tools
Claude Code
Codex
Cursor
GitHub Copilot
Enterprises · built into Remy
⌂Self-hosted
✓SOC 2
✚HIPAA
☰Audit log

It’s ongoing. Evaluation isn’t a static test set you write once and reuse forever. Real usage drifts: new types of questions appear, data changes, business rules get updated. A judge or grader that was accurate last month can quietly go stale. There are also structural limitations worth watching for, such as using the same model to judge answers that it also generates, which can make a judge too lenient toward its own writing style (using a different model as judge helps avoid this). Error analysis itself also needs a reasonably large sample of real conversations to produce reliable failure clusters rather than noise.

Frequently Asked Questions

What is an AI agent eval?

It’s a structured, repeatable test suite that scores an agent’s outputs on specific properties (correctness, rule compliance, honesty about missing data) rather than matching exact text, run across multiple attempts per question since agent outputs vary between runs.

What’s the difference between a benchmark and a product eval?

A benchmark is a public, fixed test set used to compare models in general, like MMLU Pro for knowledge or Bird for text-to-SQL. A product eval is a custom set of questions and rules specific to your own users and business, which benchmarks can’t capture.

Can an LLM judge replace human grading entirely?

No. A judge is still a language model and can be biased or wrong, including favoring whichever answer it reads first. Human-labeled answers are needed to check the judge’s accuracy and refine its rubric before trusting it at scale.

How many conversations should I read for error analysis?

Error analysis practitioners generally recommend reading a substantial sample, often cited as around 100 real conversations, to reliably surface and count recurring failure patterns rather than reacting to one-off issues.

Why would a code-based grader give a false failure?

Because the grading script itself is code you wrote, and it can check the wrong thing, such as matching an exact figure nobody asked for. Auditing grader logic against human-reviewed answers catches these bugs before they distort your scores.

Editorial standards

Presented by MindStudio

No spam. Unsubscribe anytime.