How We Think About AI Red Teaming: A Field Framework for Enterprise Deployments

AI red teaming is the practice of deliberately attacking an AI agent, its model, its guardrails, and the tools it can reach, to find failures before an adversary does. This post dives into what that looks like once you’re doing it with a customer, on a real deployment, with a deadline.

I run these engagements for a living. The same question comes up in most kickoff calls I sit in. They ask “have we tested our model for jailbreaks?” That question is too narrow and is worth reframing.

Map the attack surface first

A model card tells you almost nothing about deployed risk. Risk lives at the system level: the model, the guardrails wrapped around it, the tools and data it can reach, and the interface a user touches. I’ve tested models that were well-behaved in isolation and then watched them fail badly the moment they were wired into a customer database or a second agent.

So the first thing I do on any engagement is to map what the system can do. Can it call tools? Does it hold memory across sessions? Is it multimodal, and is testing covering images and audio the same way it covers text? Is it one model, or a chain of models talking to each other through something like Model Context Protocol (MCP)? This effort scopes the engagement, and in my experience, most teams under scope it by a wide margin before we start.

Test like an adversary

The second habit I try to break is treating red teaming like functional quality assurance (QA): run a known list of bad prompts, patch what breaks, then move on. Real adversaries don’t work off a fixed list. They adapt, chaining small failures together and using techniques that didn’t exist when the last test cycle ran.

On the engagements I run, we don’t stop at known jailbreak patterns. We probe the failure modes specific to how the system is used in production: prompt injection through a document a user uploads, tool-calling abuse inside an agentic workflow, a guardrail that catches an obvious one-shot attack but misses a slow, multi-turn one. The goal isn’t confirming the system passes a known test. It’s finding the failure nobody thought to test for, because that’s the one that shows up in production.

Make it repeatable

A red team report that gets filed away is worth little. Models get updated, guardrails get retuned, and new tools get connected. Every one of those changes can reopen a gap that was closed earlier.

The customers who get the most value out of this treat it as a standing practice with consistent scoring and consistent failure categories. Without a repeatable baseline, every engagement starts from zero, and nobody, including the security team footing the bill, can tell whether the system is getting safer over time or just getting tested more often.

Tie findings to something a compliance team can act on

The gap between finding a problem and fixing it is organizational: nobody owns it, it’s unclear which policy it violates, or leadership doesn’t see why it matters yet.

The engagements that move fastest are the ones where findings map directly to frameworks the customer already has to answer to: NIST’s AI Risk Management Framework, the OWASP Top 10 for LLM Applications, MITRE ATLAS, or increasingly the EU AI Act. A finding that reads “this maps to a high-risk obligation under the EU AI Act” moves a lot faster through an organization than one that just says “the model said something concerning.”

Run it across the lifecycle

The best version of this work doesn’t sit at the end of development as a final check before launch. It runs at the start, when the system’s risk profile is still being defined, through development as new capabilities get added, and after launch, as the model, its connected tools, and the threat landscape keep moving. An agent that was safe at launch can stop being safe the day someone gives it a new tool.

Why this matters more now than it did a year ago

None of this was ever optional, even for a simple chatbot, but it’s gotten more urgent for a specific reason: the systems I’m testing now can act in the world. In the two months leading up to our August 2026 acquisition announcement, Enkrypt AI scanned more than 268,000 tools, the individual functions AI agents call, across 25,000 MCP servers, and found more than 143,000 vulnerabilities, affecting 73% of those servers.

That means nearly three-quarters of the MCP servers we looked at had a real, exploitable gap, as the layer teams may not be testing yet because it didn’t exist as a serious attack surface two years ago. A model that only generates text has a bounded blast radius. A model wired into tools, other agents, and live data doesn’t.

This is the gap that AI security & guardrails in the Anaconda Platform is built to close: red teaming for models, agents, and MCP connections across more than 300 attack categories before release, runtime guardrails that stay in force once the system is live, and continuous evidence mapped to the same frameworks above. All of it runs inside your own environment, so the prompts and data we test against stay inside your security boundary. If you’re deciding where to start on your own deployment, start with the tools and data your agents can already reach.

FAQ

What is AI red teaming?
AI red teaming is the practice of deliberately attacking an AI system, including its model, its guardrails, and any tools or data it can access, to find security and safety failures before real adversaries find them. Learn more.

How is AI red teaming different from regular AI testing or QA?
QA checks whether a system does what it’s supposed to do. Red teaming checks what a system does when someone is actively trying to make it fail, using adversarial techniques that adapt rather than a fixed test list.

How often should AI red teaming happen?
It should run continuously across the system’s lifecycle, not just once before launch. Any change to the model, its guardrails, or the tools it can access can reopen a previously closed vulnerability.

How common are MCP and agent tool vulnerabilities?
In Enkrypt AI’s own scanning data, 73% of MCP servers assessed had at least one exploitable vulnerability across more than 268,000 scanned tools, indicating this is a widespread gap rather than an edge case. Learn more.

What frameworks does AI red teaming map to?
Findings are commonly mapped to NIST’s AI Risk Management Framework, the OWASP Top 10 for LLM Applications, MITRE ATLAS, and, for EU-facing deployments, the EU AI Act.