Your AI Passed the Jailbreak Tests: Here’s What They Missed

A clean jailbreak report means your system resisted the attacks someone wrote down before the test ran. It doesn’t tell you what happens when an attacker adapts after a refusal, hides an instruction inside a tool response, or gets an agent to act on data it should never have trusted. A static test suite can’t tell you, because it only checks for what’s on the list.

Finding those failures takes continuous adversarial testing: inputs, or chains of inputs, built to make an AI system fail, like leaking data it shouldn’t, taking an action it shouldn’t, or producing content it should have blocked. For security teams, it matters where that testing runs as much as what it finds. Ours runs inside your own environment, so the prompts and data stay inside your security boundary.

We take known attack patterns and mutate them into thousands of variations to see which ones slip through, then hand the adaptive work to autonomous agents that keep going once a system stops failing in predictable ways: changing tactics after a refusal, the part a fixed script can’t do. Below, we walk through where a static library stops, how our agents pick up from there, why testing has to cover more than text (images, audio, and tool calls included), and how every finding gets scored against your team’s compliance frameworks.

The attack pattern library as the starting point

Every engagement starts with a library of known attack patterns. That includes prompt injection variants, jailbreak templates (roleplay framing, hypothetical wrappers, instruction-override attempts), and encoding tricks designed to slip past keyword filters. We update it constantly as new techniques show up in research, in the wild, or in past engagements where something new worked. The library is structured against MITRE ATLAS‘s catalog of adversarial AI tactics and techniques, so we can track coverage systematically instead of ad hoc.

But a static library only gets you so far. The engagements that uncover the most meaningful gaps are policy-based. The tests are tuned to a specific deployment’s risk scenarios and threat model, rather than a generic list run the same way against every target. A global consumer brand needs its chatbot tested in the languages its customers speak, not just English. A financial services agent needs different scenarios than a retail agent.

Running the static library is the floor. A system that passes every pattern in it has only proven it’s safe against attacks that existed the last time the library was updated.

Autonomous agents take over the adaptive work

This is where most of our engineering time goes, and it resembles what a skilled human red teamer does, except running continuously, in parallel, at machine scale.

An autonomous red-teaming agent doesn’t run a fixed script. It forms a working hypothesis about where a target is weak, tests it, reads the result, and updates that hypothesis before deciding what to try next. A prompt that gets a partial disclosure, a hedge instead of a clean refusal, a guardrail that trips on the third sentence of a paragraph instead of the first, all of that becomes information the agent uses to construct a sharper follow-up.

A few things make this meaningfully different from running a bigger fuzzing pass:

• Strategy switching mid-session. Say the agent asks a customer-support bot to reveal its system prompt and gets a flat refusal. A fuzzer (which generates thousands of variations on known attack prompts to see which ones slip through) would retry the same request in different words. The agent changes tactics instead. It might frame the request as a role-play, or start a multi-turn conversation with harmless questions and escalate gradually. If one approach gets a partial answer, the agent pushes further down that path. If not, it moves on.
• Session memory. The agent tracks what it’s already tried against a given target within an engagement, so it isn’t repeating failed approaches or wasting a run rediscovering the same dead end in various ways.
• Parallel exploration. Multiple agent threads pursue different hypotheses against the same target concurrently, one chasing a data-disclosure angle, another probing tool-calling behavior, another working a multi-turn manipulation, so the engagement covers several angles of attack at once instead of exhausting one strategy before starting the next.
• Self-critique before escalation. Before a result gets flagged as a genuine finding, the agent evaluates its own output: was this actually a bypass, or a response that looks concerning but doesn’t cross the line the test was checking for? This is what keeps an adaptive system from drowning a report in false positives the way raw fuzzing volume can.
• Cross-surface chaining. What an agent learns probing the model directly can inform how it probes the tools and data that model can reach, and vice versa. A weakness found in how a tool’s metadata gets parsed can shape the next prompt-level attempt, and something that works at the prompt level can get reused as a payload injected through a tool response.

None of this replaces human judgment; it runs under it. Every finding an agent surfaces still goes through scoring and human review before it reaches a customer report. What autonomous agents change is the volume and depth of genuinely novel attempts that get made before a human ever looks at the results, which is the part a fixed library, however large, was never going to cover on its own.

Testing has to cover the whole system

A text-only jailbreak test tells you nothing about how a system handles an adversarial image, an audio file with an embedded instruction, or a tool call that returns manipulated data back into the model’s context. Multimodal systems need testing built for each modality, not a text methodology applied to everything else.

In one engagement on a customer-facing hospitality chatbot, testing turned up a model that would confirm a reservation that didn’t exist, recommend a competitor’s property over its own, and quote different prices for the same hotel room depending on demographic signals in the conversation. None of those are jailbreaks in the traditional sense. They’re a system doing exactly what a user asked, in ways that create real brand, legal, and revenue exposure the team hadn’t scoped as a “security” flaw.

Agentic systems raise a related problem. Once a model can call tools, especially over something like Model Context Protocol (MCP), the test surface expands from what the model says to what it can be made to do. In practice, the failures we see cluster around a few specific patterns:

• Confused-deputy chains: a tool with broad, legitimate permissions gets invoked on behalf of an input the model shouldn’t have trusted in the first place
• Tool description poisoning: the metadata a tool exposes to the model (its name, description, parameters) is itself an injection vector, since the model reads that metadata as context
• Indirect injection via tool output: a tool call returns data from an untrusted source, and that data contains instructions the model then follows
• Over-privileged scopes: a tool works exactly as intended but was granted far more access than the task needed, turning a minor prompt-level slip into a major action-level issue

This is the layer where we’ve focused most of our agent development recently. Our agents probe tool descriptions, inject adversarial content into tool responses, and track whether a model passes untrusted data into a subsequent tool call without checking it, adapting each next move based on what the target just did. It’s the same underlying capability that found vulnerabilities in 73% of the 25,000 MCP servers we assessed across a two-month window, spanning more than 268,000 scanned tools, real evidence that this attack surface is underspecified across the ecosystem, not just in isolated systems.

Every finding gets scored

A raw list of successful attacks isn’t useful to a customer, or to an engineering team, on its own. Every finding gets scored for severity, categorized by failure type, and mapped to the compliance frameworks the customer cares about: NIST’s AI Risk Management Framework, the OWASP Top 10 for Large Language Model (LLM) Applications, and the EU AI Act.

Consistent scoring is what makes it possible to compare one engagement to the next, and to rerun the same suite against a new model version or a retuned guardrail config to check whether a previously closed gap reopened.

From lab to platform

This lab, and the autonomous agents behind it, power the AI security and guardrails in the Anaconda Platform: red teaming across more than 300 attack categories before release, adaptive agents that go past a static library, and the same scoring and framework mapping carried through into runtime guardrails once a system is live. All of it runs inside your own environment, so the prompts and data under test stay inside your security boundary.

If you’re deciding where your own testing has the biggest gap, the tool-calling layer may be the next area of focus for your team. Learn more.

FAQ

How do you build an adversarial test for an AI system?
By starting from a library of known attack patterns, then handing the adaptive work to Enkrypt AI’s autonomous red-teaming agents, which form hypotheses about a target’s weaknesses, adjust strategy based on each response, and chain what they learn into sharper follow-up attempts.

What makes autonomous red-teaming agents different from a bigger fuzzing pass?
Fuzzing mutates and runs known patterns at volume, but every attempt is independent of the last. Autonomous agents carry memory across a session, switch strategy when one framing stalls, run multiple hypotheses in parallel, and critique their own results before flagging a finding, adaptive depth a fixed pattern library can’t reach on its own. Learn more.

What makes MCP and tool-calling systems harder to test than a plain chatbot?
The attack surface shifts from what a model says to what it can be made to do. Specific patterns include confused-deputy chains, tool description poisoning, indirect injection via tool output, and over-privileged tool scopes, none of which show up in text-only jailbreak testing.

How are red-team findings scored?
Findings are scored for severity and categorized by failure type. The underlying attack techniques are structured against MITRE ATLAS’s taxonomy of adversarial AI tactics, while findings themselves are mapped to compliance frameworks like NIST AI RMF, the OWASP LLM Top 10, and the EU AI Act, so results are both technically comprehensive and directly actionable for a compliance team.