Offline vs Online Evals: Why AI Needs Both Halves

Offline evals score a fixed dataset with known good answers before a change ships. Online evals score live production traces after it ships, where there is no known answer. Offline catches regressions on the cases you already wrote down; online is the only place the cases nobody imagined show up, and the useful system is the loop that turns one into the other.

We have written a fair amount lately about the offline half: gates that block a merge, judges that grade a fixed set, scoring retrieval and answers separately. This post is about the other half, and about why neither half is worth much on its own.

What is the difference between offline and online evals?

The cleanest definition we have seen is in the LangSmith docs on evaluation concepts (fetched 2026-10-28). Offline evaluations "target examples from datasets: curated test cases with reference outputs." Online evaluations "target runs and threads from tracing: real production traces without reference outputs."

That one difference, whether a reference answer exists, shapes everything else.

Langfuse makes the same split in its guide to evals (published 2025-11-12, updated July 2026, fetched 2026-10-28), and puts the reason plainly: effective evaluation "blends offline and online methods, because each catches errors the other misses."

Why does an offline pass rate overstate quality?

Because the dataset only contains cases someone thought to write down. Your offline pass rate is a score on the questions you already knew to ask.

That is not a reason to skip offline evals. A fixed set with reference answers is the only way to know, before you merge, that a prompt change or a model swap did not break something that used to work. That is exactly why we treat evals as a CI quality gate: the gate is the offline half, and it earns its keep every time it blocks a bad merge.

The trouble starts when a team reads a high pass rate as a statement about production. It is not. Real users phrase things differently, upload documents nobody tested, chain requests in odd orders, and hit the edge of a tool's schema in ways that never appeared in the set. A dataset built in week one describes week one's understanding of the problem. Six weeks later, the people using the system have taught it new failure modes, and the offline set has no way to know.

So the honest reading of an offline score is narrow: "on these cases, this version is no worse than the last one." That is useful. It is just not the whole story.

What can an online eval actually check without a right answer?

Less than an offline eval, and the limits matter. LangSmith's docs are direct about it: reference-based evaluators "require reference outputs and only work for offline evaluation." Online, you are limited to checks that can judge an output on its own.

In practice that still covers a lot. These are the reference-free checks we reach for first:

• Format and schema validity. Did the output parse? Did a tool call match the tool's schema? These are cheap code checks and they should run on every trace.
• Safety and policy checks. Did the response leak something it should not, cross a policy line, or answer outside the product's scope?
• Groundedness against retrieved context. A judge cannot compare to a gold answer, but it can ask whether the answer is supported by the documents the system actually retrieved. That is the online cousin of scoring retrieval and answers separately.
• Tool and runtime errors. Failed calls, retries, timeouts, loops that hit a step limit.
• Refusals. A refusal on a request the product should handle is a failure, even if it reads politely.
• User signals. A user who immediately rephrases, retries, abandons or clicks a thumbs down is giving you a label for free.

A concrete example from our own portfolio: in Smile PreVue, every smile preview is checked by a second AI model against the patient's original photo and the requested treatment, and the clinic sees the result as Excellent, Good or Review. There is no "correct" smile to compare against, so it is a reference-free check by design. It grades whether the image stayed faithful to the photo and the request, not whether the clinical result will match, which is exactly the kind of question an online eval can answer honestly.

One practical note on cost. Code checks are nearly free, so run them everywhere. Judge calls cost money and add load, so our recommendation is to score a sample of normal traffic with the judge and route every trace that tripped a cheap code check or a user signal to the judge as well. The sample rate is a judgment call for each system; start small and raise it if the sample keeps surprising you.

How do production traces become offline test cases?

This is the part that makes the two halves a system instead of two dashboards. Both LangSmith and Langfuse describe the same loop. LangSmith says "online evaluations surface issues that become offline test cases"; Langfuse says the edge cases online evaluation catches "go back into the dataset so the next experiment catches them."

The loop, the way we run it:

1. Online flags it. A reference-free check, a judge score or a user signal marks a trace as suspect.
2. A person reviews the trace. This step stays human. Someone who understands the domain decides whether it is a real failure and what the right output would have been.
3. The trace becomes a dataset example. The input goes into the offline set with the reference answer the reviewer wrote. Now the set contains a case it never had.
4. The offline set fails until the fix lands. The new example proves the bug exists and proves the fix works, before anything ships.
5. Online confirms it held. After the fix ships, the same online checks should stop flagging that class of trace.

Two habits make this loop much more useful over time. First, label every promoted example with where it came from, so you can see what share of the offline set is real traffic versus cases someone invented at a whiteboard. A set that is mostly invented cases is a set that has not met production yet. Second, keep the review step honest. If promoted cases pile up unreviewed, the loop quietly stops working, and the offline set freezes at whatever it knew the last time someone had an afternoon free.

You cannot run any of this without traces in the first place. If a system does not record inputs, retrieved context, tool calls and outputs per request, there is nothing to score and nothing to promote, which is why we start with what to trace on an agent before we talk about online scoring at all.

When is online evaluation not worth it yet?

When a person can still read every trace. A low-traffic internal tool used by a handful of people a day does not need an automated online scoring pipeline. Read the traces by hand once a week, promote the interesting ones into the offline set, and you have the loop without the infrastructure.

Online scoring earns its cost at the point where nobody can read everything anymore. That point arrives sooner than teams expect, especially for anything customer-facing, but it is a real threshold, and building the pipeline before you reach it is effort spent on the wrong half.

It is also not worth it without tracing, as above, and not worth it if nobody owns the review step. An online eval that flags traces into a queue nobody reads is monitoring theater. It looks like quality work and changes nothing.

FAQ: offline and online evals

Can I use the same judge for both? Often, yes, but not in the same mode. Offline, a judge can compare an output to a reference answer. Online, the same judge has to grade the output on its own terms, against a rubric or the retrieved context.

Do online evals replace monitoring? No. They sit on top of it. Monitoring tells you the system is up and fast; online evals tell you whether what it said was any good.

How big should the offline set be to start? Our recommendation, not a statistic: start small with real cases you have actually seen, and let production grow it. A small set of real failures beats a large set of invented ones.

Who should review flagged traces? Someone who knows the domain well enough to write the correct answer. If nobody can, the trace is not ready to become a test case.

Where should you start?

Pick one reference-free check and run it on live traffic this week; a schema check on tool calls is the easiest place to begin. Then take one flagged trace, have someone write the right answer, and add it to your offline set. That single promoted case is the start of the loop, and it is the part of our approach that keeps an AI feature honest after launch.

If you want an AI feature shipped with both halves in place from day one, this is the kind of work we do. Tell us what you are building at yikesdude.com/contact.