AI Red Teaming for Business: Break It Before Customers Do

AI red teaming is deliberately attacking your own AI system to find the ways it can be pushed into wrong, harmful or unauthorized behavior before someone else finds them. NIST describes it as a structured testing exercise to probe an AI system for flaws and vulnerabilities, often in collaboration with the people who built it. For a business, it is the step that answers a question evals do not: what happens when someone tries to make your AI do the wrong job?

Many teams shipping an AI feature run evals and assume that covers safety. It does not, and the gap is where the embarrassing incidents live: the support bot that reads out its own instructions, the agent that follows a sentence hidden in a customer's PDF, the feature that quietly burns through an API budget overnight.

The good news is that you do not need a security firm to start. You need a short list of attacks matched to what your AI is allowed to touch, run before launch and again after every change.

What is AI red teaming?

The cleanest definition comes from NIST. Its Generative AI Profile, NIST AI 600-1 (published July 2024, read October 14, 2026), defines AI red-teaming as "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlled environment and in collaboration with system developers."

Three words in that definition matter for a business.

Structured. Red teaming is not a few people poking at a chatbot on a Friday afternoon. It is a written list of things you are trying to make go wrong, run deliberately, with the results recorded.

Probe. The goal is to find failures, not to confirm the system works.

In collaboration with system developers. The people who built the system know where the seams are, and they have to fix what gets found.

NIST also lists the kinds of red teaming that fit different situations: general public testers, domain experts, a combination of the two, and what it calls human and AI red teaming, where a model generates attacks alongside people. On that last one, the document notes that "GAI-led red-teaming can be more cost effective than human red-teamers alone." That matters for a small team, and we come back to it below.

How is red teaming different from AI evals?

Evals measure whether the system does its intended job on representative inputs. Red teaming hunts for the inputs that break the rules.

An eval for a support assistant asks: across a set of real customer questions, how often is the answer correct, on policy and useful? A red team for the same assistant asks: can I get it to reveal its instructions, promise a refund it cannot give, leak another customer's details, or answer questions it should refuse?

Both produce failure cases, and both belong in the same workflow. We wrote about the first half in AI evals vs software tests: an eval reports a rate across real cases, not a pass or fail. Red teaming adds the adversarial cases to that set. If you grade outputs with a second model, the same cautions from our post on LLM as a judge apply here too: a judge has to be checked against cases where you already know the right answer.

Run both before launch and after every model, prompt or tool change. A prompt edit that makes answers friendlier can also make the system easier to talk out of its rules.

Which attacks should a business try first?

Pick by what the AI can touch. The best reference lists are free. OWASP maintains a Top 10 for LLM Applications (the 2025 edition, listed March 12, 2025 on genai.owasp.org, read October 14, 2026), and in December 2025 published a separate Top 10 for Agentic Applications.

If your AI answers questions in a chat window, start with three entries from the LLM list:

• LLM01 Prompt Injection. Inputs that override the system's instructions. Try the obvious ones first, then the polite, roleplay and multi-step versions.
• LLM07 System Prompt Leakage. Can a user get it to print its instructions? Assume anything in a system prompt will be read eventually, and never put secrets there.
• LLM02 Sensitive Information Disclosure. Can it be coaxed into revealing data it has access to but should not share, including details from other users or internal documents?

If every call costs money, add LLM10 Unbounded Consumption. Find out what happens when someone sends very long inputs, loops the system, or calls it thousands of times. The failure here is not a bad answer. It is an invoice.

If your AI reads documents, emails or web pages, test indirect injection: instructions hidden inside content the system processes rather than typed by the user. A line of text in a PDF that says to ignore previous instructions is the classic version. This is the attack most teams forget, because the attacker never touches the chat box.

Why do AI agents raise the stakes?

Because an agent that can send, write, book or spend turns a bad answer into a bad action.

OWASP's agentic list starts with three entries that map directly onto that risk. ASI01 Agent Goal Hijack is an attacker redirecting what the agent is trying to do. ASI02 Tool Misuse is the agent using a legitimate tool in a damaging way. ASI03 Identity and Privilege Abuse is the agent operating with more access than its job needs, often through credentials it should not have had. The rest of the list covers supply chain, unexpected code execution, memory, inter-agent communication, cascading failures, human trust and rogue agents.

The LLM list has a related entry, LLM06 Excessive Agency, and it is worth reading as a design problem as much as a testing one. If an agent can delete records, the red team will eventually find a way to make it delete records. The fix is usually not a cleverer prompt. It is giving the agent fewer permissions, narrower tools and a human checkpoint before anything irreversible. We covered how to write those limits in scope authorization for AI agents.

For an agent, the red team questions become concrete:

• Can a document or web page the agent reads change what it decides to do?
• Can it be talked into calling a tool outside its stated task?
• If it hits an error, does it retry forever, or escalate to wider permissions?
• Does every irreversible action stop for a person, every time?

How does a small team run its first red team?

Keep it small and keep it written down. The version we use inside our own builds looks like this.

Write the attack list per risk. One page. For each OWASP entry that applies to your system, write three to five specific attempts in plain language. Skip the entries that do not apply. A chatbot with no tools does not need tool-misuse tests.

Keep a known-bad input set with the code. Every attack that works, and every one that nearly works, becomes a saved case that runs again later. This is the single habit that matters most. Without it, a fix from March quietly breaks in June and nobody notices.

Use a model to generate variations, and a person to read the findings. This is where NIST's point about cost helps. A model can produce hundreds of rephrasings of a prompt injection in minutes, far more than a person would write. But a person should review what got through, because deciding whether an output is actually harmful in your business is a judgment call.

Log results and re-test fixes. A finding is not closed until the same attack fails against the fixed system.

Repeat on change. New model, new prompt, new tool, new data source: rerun the set before it ships.

A worked example from our own work: one of the products we built, Smile PreVue, runs a second AI model that checks each generated preview against the patient's original photo. Before we trusted that checker's verdicts, we fed it deliberately wrong inputs, including previews that should obviously fail, to see whether it would notice. That is red teaming the guardrail itself, and it is the step people skip most often. A safety check you have never tried to fool is an assumption, not a check.

None of this replaces a professional security assessment for systems that handle money, health data or anything regulated. It is the floor, not the ceiling.

Where does red teaming fit in a build?

At the start, not the end. In the way we figure out, build and ship AI work, the first week produces two lists: the eval cases that define good, and the attack cases that define unacceptable. Writing the attack list early changes the design: a team that sees in week one that the agent's broad database access is the riskiest item scopes it down before it is built.

If you want help working out which attacks matter for your system, you can book a consultation and we will go through it with you.

FAQ

Do small businesses need AI red teaming? If your AI talks to customers, reads outside content or can take actions, yes. The effort scales with what the system can touch. A small internal tool needs an afternoon; a customer-facing agent with tools needs more.

How often should we red team? Before launch, then again after every model, prompt, tool or data source change. Keep the saved attack set running so old fixes stay fixed.

Can red teaming be automated? Partly. A model can generate attack variations at scale, and NIST notes model-assisted red teaming can be more cost effective than people alone. A person should still review findings and decide what counts as a real failure.

Is AI red teaming the same as a penetration test? No. A penetration test looks at your infrastructure and applications. AI red teaming looks at the model's behavior: what it says, what it reveals and what it does when someone tries to misuse it. Many systems need both.

Does red teaming make an AI system secure? No test does. It finds failures you can fix and gives you a record of what you checked. Treat it as ongoing work, not a certificate.

If you are about to ship an AI feature or agent and have not tried to break it yet, talk to us about testing it or building it with the attack list in from day one.