Runtime Guardrails for AI: Check the Output Before Users Do

Runtime guardrails are the checks that run on every real input and output of an AI feature in production, and decide whether that output gets used, fixed, retried or handed to a person. Evals tell you the feature was good on test day. Guardrails decide what happens on the ten-thousandth real request, when the input is strange and nobody is watching.

Most teams we talk to have the first half and not the second. They ran a test set, the numbers looked fine, and they shipped. Then a real user uploaded a sideways photo or a scanned PDF with a coffee ring on it, and the feature confidently produced something wrong.

What are runtime guardrails, and how are they different from evals?

An eval is a fixed set of test cases you run before a release or on a schedule. It produces a score. A runtime guardrail runs inline, on each live request, and it produces a decision: use this output, or do something else with it.

They answer different questions. An eval asks "is this feature good enough to ship?" A guardrail asks "is this particular output good enough to use, right now, for this user?" You need both. We have written about the eval side before, including using an LLM as a judge, so this post stays on the runtime decision.

The security world already has vocabulary for why this matters. The OWASP Top 10 for LLM Applications, 2025 edition (genai.owasp.org, fetched 2026-10-19), lists Improper Output Handling and Excessive Agency as named risks. In plain operator language: do not let a model's output flow into something that matters without checking it, and do not give a model the power to act on its own where a mistake is expensive. Runtime guardrails are how you honor both.

Which checks need no model at all?

The cheapest guardrail is the one that does not call another model. A surprising amount of AI failure is format failure, and format failure is easy to catch with plain code.

The checks we reach for first:

• Schema and type checks. The output is supposed to be JSON with these fields, these types, these allowed values. If it is not, it never reaches the user.
• Required fields. A load with no destination, an invoice with no total, a record with no ID is a failed output, not a partial success.
• Range and format checks. Dates that land in the past, rates that are negative, phone numbers with the wrong number of digits.
• Length and cost limits. Cap tokens, cap retries, cap spend per request, so one bad input cannot run away with your bill.
• Checks against a source of truth you already own. If the model names a customer, does that customer exist? If it names an address, is it a real address?

That last one is where we lean hardest. In Howdy Dispatch, an AI reads a broker's rate-confirmation PDF and pre-fills the load: customer, origin, destination, rate, pickup and delivery times. Before any of that is saved, Google address validation runs on the origin and the destination. No model judges whether "1200 Industrial Blvd" is real. A deterministic check does.

These checks are boring, fast and nearly free. They should be the first layer of every AI feature you ship, and they are often the layer teams skip because the demo never needed it.

When should a second model check the first one?

Some correctness is not about format. It is about meaning or appearance: did the output stay faithful to the input? Code cannot easily answer that. A second model can.

Smile PreVue is our clearest example, and you can read how we built Smile PreVue in full. A dentist takes a photo of a patient, picks a treatment and a shade, and the product generates a preview of the patient's smile after treatment. The risk is subtle. A generated image can quietly change the patient's face, their lips or the framing, and then it is no longer a picture of their result.

So every preview is graded by a second AI model that compares it against the patient's original photo and the exact treatment requested. It checks things like whether identity and framing were preserved, whether the lips were left alone, whether only the teeth were edited, and whether the shade matches. The clinic team sees the result as Excellent, Good or Review. A Review names the specific problem, and the team can regenerate with that fix written into the instructions, or regenerate as is. Earlier attempts are archived, not overwritten.

To be precise about what that check is: it grades how faithful the image is to the photo and the treatment. It is not a clinical prediction, it is not human review, and the patient never sees it. Being honest about what a guardrail does and does not measure is part of building one.

The design lessons we would pass on, without the prompts:

• Grade against the original input, not the output alone. "Does this look good?" is a weak question. "Does this match what we started with and what we asked for?" is a strong one.
• Make the grader name a reason. A score with no reason cannot drive a fix. A named problem can.
• Keep earlier attempts. Overwriting a failed output destroys the evidence you need to improve the system.
• Decide what a failure does before you ship. If you have not designed the Review path, the grader is just a number nobody acts on.

The grader does not have to come from the same lab as the generator. We pick models per job, and a checker with different strengths than the generator often catches more.

Where does a human have to sign off?

When an action is costly or hard to undo, the right guardrail is a person with context. The software's job is to make that review fast, not to skip it.

Back to Howdy Dispatch. The AI pre-fills the load from the rate confirmation, and the dispatcher reviews it and saves it. The AI never accepts a load and never dispatches a truck. Typing goes away. Judgment stays with the dispatcher, who knows that this broker always lists the wrong dock door.

That split is deliberate. The cost of a wrong address on a load is a truck in the wrong city. A review screen that takes seconds is cheap insurance against that. We go deeper on where to put the person in human-in-the-loop agent design.

The mistake we see most is putting the human in the wrong place: approving every trivial output until they rubber-stamp everything, while the one consequential action slips through unreviewed. Put people on the decisions that carry stakes, and let deterministic checks and graders handle the rest.

What does a guardrail do when it fails?

A guardrail that can only say "no" is half built. When a check fails, there are four useful moves:

• Block. Do not show or save the output. Right for unsafe or malformed results.
• Retry. Run the generation again, ideally with a different seed or a tighter instruction. Cap the retries.
• Repair. Feed the specific failure back in and ask for a corrected output. This is what "Regenerate with fix" does in Smile PreVue.
• Escalate. Send it to a person with the reason attached.

Most teams only build block. Users then see a generic error and the feature feels broken, even when a single repair pass would have fixed it.

Two more things matter in practice. First, log every guardrail decision with its reason. Real failures caught in production are the best eval cases you will ever get, so turn them into tomorrow's test set. Second, think about latency. Heavy checks can run in the background when the user can keep working, which is how Smile PreVue grades a preview after it is already on screen. Checks that gate a save or an action have to run inline.

Do you need a guardrails platform, or can you build this yourself?

There is a real market here. Galileo, for example, markets evaluation and observability for AI apps and agents, and its homepage (galileo.ai, fetched 2026-10-19) describes a setup "where offline evals become production guardrails." Platforms like that bundle evals, monitoring and guardrails behind one dashboard.

Our honest take: start with the three layers in your own code. Deterministic checks, a second-model grader where meaning matters, and a human gate where stakes are high. That gets most teams most of the value, and it forces you to understand your own failure modes. A platform starts paying for itself when you have enough traffic, enough features and enough people that a shared view of every decision is worth buying.

This is the same discipline we put on code an agent writes, which we covered in guardrails for AI-generated code. The difference is when it runs: that post is about checks before code merges, and this one is about checks on every live request after the feature ships. It is also a core part of our agent orchestration methodology.

Runtime guardrails FAQ

Do guardrails slow the app down? Only the ones that run inline. Background grading adds no wait for the user. Keep inline checks deterministic and fast, and push model-based grading to the background where you can.

Can the same model check its own output? It can, but a separate check against the original input catches more. Self-review tends to agree with itself.

Are guardrails the same as content moderation? Moderation is one kind of guardrail. Most business guardrails are about correctness: right fields, right address, faithful to the input.

How do guardrails relate to evals? Evals measure the feature before launch. Guardrails act on each request after launch. Every failure a guardrail catches should become a new eval case.

If you are shipping an AI feature and only have the eval half, we can help design the runtime half with you, as part of our custom software for AI workflows. Start a conversation, or if you just want to learn to run Claude this way yourself, see the 1:1 sessions.