AI Evals Are Not Software Tests

A software test compares deterministic output to an exact expected result, so it either passes or fails. An AI eval samples a system that can answer differently each time and reports a rate, such as how often the output is acceptable across a set of real cases. You need both, and a green test suite says almost nothing about whether the AI part of your product is any good.

That sounds obvious when you write it down. In practice it is the most common blind spot we see in teams shipping their first AI feature. The CI pipeline is green, the demo went well, and nobody can say how often the thing is actually right. The other half of the blind spot is the leaderboard: "the model scored 90 on the benchmark" gets treated as if it were a measurement of your workflow. It is not, and two papers from this year show why.

What is the difference between AI evals and software tests?

Splunk's September 2026 explainer on evaluating AI systems puts the old half of this neatly: "Software tests compare deterministic outputs with exact expected results." (Splunk sells observability products, so read it as a vendor's framing, but the line is right.)

A unit test asks a yes or no question. Given this input, does the function return exactly this output? Run it a thousand times and you get the same answer a thousand times. That is what makes tests useful: a red test means something broke.

An AI feature does not work that way. Ask a model to summarize the same intake form twice and you can get two different summaries, both reasonable, or one reasonable and one that drops the allergy. There is no single expected string to compare against. So the question changes from "did it pass" to three different questions:

• How often is the output acceptable?
• Acceptable on which cases? Mine, or someone else's?
• Is that rate moving when we change something?

An eval is the machinery that answers those three questions. It is a set of cases, a way to grade each output, and a number you track over time.

Why does a passing test suite not mean the AI works?

Because tests check the plumbing, and the plumbing is not where AI features fail.

Your tests can confirm that the model call is made, the response parses as JSON, the retry fires on a timeout, the field lands in the database, and the UI renders it. All of that matters. None of it tells you whether the answer inside the JSON was correct.

Here is a scene we see in some form on most projects. Picture a typical intake-form summarizer. The team writes good tests: schema validation, a timeout test, a snapshot of the rendered summary card. Every test passes. Then someone changes the prompt to make summaries shorter, and the summaries start dropping the one line a reviewer actually needs. The tests still pass, because the JSON is still valid and the card still renders. Quality moved, and nothing in CI noticed.

The same thing happens without anyone touching your code. A model version changes, a tool returns a slightly different format, the documents you retrieve shift. Any of those can move the rate of good answers up or down without breaking a single test. If you only have tests, you only find out from users.

Can you trust a public benchmark score instead?

Not as a stand-in for your workflow. A public benchmark is useful for one thing: deciding which models are worth trying. It cannot tell you whether a model handles your documents, your edge cases, and your definition of "right." And this year's research gives two concrete reasons to hold benchmark numbers loosely even for the shortlist.

Contamination. If a model has seen a benchmark's questions during training, its score partly measures memory, not ability. A systematic review by Nourbakhsh and colleagues, presented at the GEM workshop in July 2026 and named an Outstanding Paper, surveyed contamination detection methods and reports "inflation estimates spanning roughly 6%–40% under benchmark- and setting-dependent assumptions" (ACL Anthology, fetched 2026-10-09). Read that carefully: these are estimates, and they depend on the benchmark and the setting. It does not mean every score is 40 percent too high. It means you cannot assume a public number is clean.

Wrong answer keys. A benchmark is only as good as its answers. A January 2026 study of text-to-SQL benchmarks found, through expert analysis, that "BIRD Mini-Dev and Spider 2.0-Snow have error rates of 52.8% and 62.8%, respectively" in their annotations (arXiv 2601.08778v3, fetched 2026-10-09). That finding is about those two text-to-SQL benchmarks, not all benchmarks. But it is a sharp reminder that a leaderboard can reward a model for matching a wrong answer.

The takeaway is not that benchmarks are useless. It is that a leaderboard tells you which model is interesting, and only your own cases tell you whether it works for you.

How do you build an eval that is yours?

You do not need a platform to start. You need cases, a grader, and a bar.

Start with 30 to 50 real cases. Pull them from your own work: real documents, real tickets, real questions your users asked. Include the ugly ones, the scanned PDF, the email thread with three forwarded replies, the request that is half in another language. Invented examples are too clean and will flatter the model. Keep the set private, out of prompts and out of anything a vendor could train on, so it stays a fair test.

Grade with code wherever you can. A surprising amount of AI output can be checked deterministically. Did the extraction find the right date? Is the dollar amount the one on the invoice? Did the classifier pick one of the allowed labels? Those are string and number comparisons, and they are cheap and exact.

Use a model as grader only for the rest. For open-ended output, like a summary or a reply, a second model with a written rubric can grade at scale. Use a different model from the one that generated the output, check the grader against human judgment on a sample, and watch for its biases. We wrote up how in LLM as a judge.

Anthropic's own eval guidance pushes the same direction. It advises "Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals," and to "Structure questions to allow for automated grading" (Claude docs, fetched 2026-10-09). Fifty automatically graded cases you can rerun in two minutes beat ten perfect hand-graded ones you rerun once a quarter.

Write down a pass bar before you look at results. Decide in advance what rate is good enough to ship, and which failures are never acceptable no matter the rate. If you set the bar after seeing the number, the bar will always land just under the number.

One example from our own portfolio, kept deliberately general: Smile PreVue runs a second AI model on every simulation that checks the preview against the patient's photo and the requested treatment, and shows the clinic team the result. That is an eval running in production on every output, not a test that ran once in CI.

When should you rerun your evals?

Every time something that can move quality changes:

• The model or model version.
• The prompt or system instructions.
• The tool list or a tool's output format.
• The data the system retrieves.

Also rerun on a schedule even when nothing on your side changed, because hosted models and upstream data can change underneath you.

Then treat the result as a trend, not a verdict. One run tells you little; the rate across runs tells you whether you are getting better or worse. And every time a user finds a failure the eval missed, add that case to the set. A year in, the set is the most valuable artifact in the project, because it encodes everything you have learned about how your system fails. For how that fits a full agent harness, see eval-driven agent development. Evals check whether the system does its job; to find the inputs that make it break the rules, add AI red teaming to the same set.

Evals also only work if the cases reflect reality, which is a data problem before it is a model problem. If your source documents are a mess, start with data readiness for AI.

This is a large part of how we figure out, build and ship AI work: the eval set gets written in the first week, before the feature is good, so we know when it gets good. It is also the first thing we build in custom software for AI workflows.

FAQ

Do AI evals replace unit tests? No. Keep both. Tests guard the plumbing; evals guard the judgment. A feature with evals and no tests breaks in boring ways, and a feature with tests and no evals breaks in expensive ones.

How many eval cases do you need to start? Dozens of real ones beat hundreds of invented ones. Start with 30 to 50 and grow the set from every failure you find.

Are vendor benchmark numbers useless? No. They are a shortlist filter. Use them to pick which two or three models to try, then run your own cases before you commit.

I am learning Claude. Do I need any of this? The same idea works at personal scale. Keep five real examples of the task you use Claude for, and rerun them whenever you change your prompt. If you want help setting that up, you can learn to use Claude one on one.

If you are shipping an AI feature and cannot say how often it is right, that is the first thing to fix. Talk to us about building an eval set for your workflow.