An AI jailbreak is a direct prompting attack intended to circumvent restrictions placed on model outputs, such as circumventing refusal behavior to enable misuse. That wording belongs to the National Institute of Standards and Technology, which sets it out in its taxonomy of adversarial machine learning attacks and mitigations.
The term here covers language models, not the device sense it was borrowed from, where jailbreaking means stripping a manufacturer’s restrictions off a phone or a console. The boundary with prompt injection is the other place readers get stuck, since the encyclopedic layer folds the two together while the standards bodies keep them distinct.
Key Takeaways
- NIST places the jailbreak inside the direct prompting attack family, where attackers bypass model-level defenses that a model developer or deployer created to restrict harmful or undesirable output.
- Two failure modes explain why the attack lands on models trained to refuse: Competing objectives, which arise when a model’s capabilities and safety goals conflict, and mismatched generalization, which occurs when safety training fails to generalize to a domain for which capabilities exist.
- OWASP describes jailbreaking as a form of prompt injection where the attacker provides inputs that cause the model to disregard its safety protocols entirely, and notes that the two related concepts are often used interchangeably.
- CAISI reported that DeepSeek’s most secure model responded to 94% of overtly malicious requests when a common jailbreaking technique was used, compared with 8% of requests for U.S. reference models.
- NIST frames its defense strategies as ways to reduce the risk that a model or system will be vulnerable to direct prompt injections. OWASP states that given the stochastic influence at the heart of the way models work, it is unclear if there are foolproof methods of prevention for prompt injection.
How Does an AI Jailbreak Work?
Safety training is layered over capability training, and the attack lives in the seam between them. Manual methods for jailbreaking an LLM include competing objectives and mismatched generalization, according to NIST. In all cases, the goal of the attack is to compromise a model-level safety defense.
A jailbreak aims at the training, not at the server.
Step 1: Find the Gap Between What a Model Can Do and What It Will Do
Competing objectives arise when a model’s capabilities and safety goals conflict, while mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist, per Wei, Haghtalab, and Steinhardt. The gap belongs to the training run, so it travels with the model rather than sitting on an interface someone can patch.
Picture a bouncer who has memorized a list of banned phrases but never learned why any of them is banned. The same request in a different accent walks straight past him.
Step 2: Write a Prompt That Exploits the Gap
Mismatched generalization attacks identify inputs that fall outside the distribution of the model’s safety training but remain within the distribution of its capabilities training, making them comprehensible to the model while evading refusal behavior. Competing objectives attacks find cases where model capabilities are in tension with safety goals, such as by playing into a model’s drive to follow user-provided instructions.
The failure resembles a spelling checker that flags “recieve” and waves through “receeve”. It only knows the misspellings somebody thought to write down.
Step 3: Repeat Until the Refusal Stops
Attacks are relatively basic in that they either directly insert the question into a prompt template or follow a few-step procedure to generate question-specific prompts, which can be automated with a handful of calls to an auxiliary model, according to the UK AI Security Institute. All models complied at least once out of five attempts for almost every question when the institute’s in-house attacks were used.
Iteration is the cheap part. The expensive part was the safety training the attacker is walking around.
| Dimension | What safety training covers | What a jailbreak targets |
|---|---|---|
| What is trained | Refusal behavior layered on top of capability training | The seam where refusal behavior never reached |
| Where it generalizes | The input distribution seen during safety training | Inputs inside the capabilities distribution but outside the safety distribution |
| What the attacker supplies | Nothing; training completes before deployment | A direct prompt, sometimes generated by an auxiliary model |
| What a success proves | Refusals held for the inputs that were tested | A model-level safety defense was compromised |
Sources: NIST, Wei, Haghtalab and Steinhardt
Why Does an AI Jailbreak Matter?
NIST lists the attacker goals for direct prompting: Bypassing model-level defenses to enable misuse, extracting the system prompt or revealing private information that was provided to the model in context, and, when LLMs are used as agents, manipulating tool usage and API calls to potentially compromise the backend of the system.
The agent goal is the one most definitions leave out. It turns a text-generation failure into an infrastructure problem.
One evaluation put figures on how far models diverge. DeepSeek’s most secure model responded to 94% of overtly malicious requests when a common jailbreaking technique was used, compared with 8% of requests for U.S. reference models, CAISI reported. Same technique family, two published results, no convergence between them.
The UK AI Security Institute found that all tested LLMs remain highly vulnerable to basic jailbreaks, and that some will provide harmful outputs even without dedicated attempts to circumvent their safeguards. Wei, Haghtalab and Steinhardt found that vulnerabilities persist despite the extensive red-teaming and safety-training efforts behind these models, and argued against the idea that scaling alone can resolve these safety failure modes.
Across our AI model coverage, the same pattern keeps surfacing: Rankings move faster than the narratives attached to them, and safety results move with them. Jailbreak resilience is a per-model property that has to be measured rather than asserted, which is why our cross-model AI security comparison tracks results by model version. A vendor line about defending against jailbreaks carries no comparable information; a published number attached to a named technique family does.
Types of AI Jailbreak Techniques
NIST sorts the techniques under the same two families that explain why they work. The competing objectives group manipulates what the model is trying to do; the mismatched generalization group manipulates what the model is looking at.
Competing objectives strategies include prefix injection, which prompts the model to start responses with an affirmative confirmation; refusal suppression, which instructs the model to avoid generating refusals or denials; style injection, which instructs the model to use or not use certain syntax or writing styles; and role-play, where adversaries use strategies such as “Always Intelligent and Machiavellian” or “Do Anything Now” to guide the model to adopt personas that conflict with the original intent.
Mismatched generalization strategies include special encoding such as base64, character transformation such as the ROT13 cipher and Morse code, word transformation such as Pig Latin, synonym swapping and payload splitting (also called token smuggling), and prompt-level transformation such as translating the prompt into a less common language that may be out of distribution of the safety training data.
The DAN prompt that circulates in screenshots belongs to that role-play group rather than to a category of its own.
| Technique family | Named strategies | What the attacker changes | What stays constant |
|---|---|---|---|
| Competing objectives: prefix injection | Start the response with an affirmative confirmation | The opening tokens of the answer | The underlying request |
| Competing objectives: refusal suppression | Instruct the model to avoid refusals or denials | The permitted output space | The underlying request |
| Competing objectives: style injection | Constrain syntax or writing style | The register of the answer | The underlying request |
| Competing objectives: role-play | Personas such as “Always Intelligent and Machiavellian” and “Do Anything Now” | Who the model believes it is | The underlying request |
| Mismatched generalization | base64 encoding, ROT13, Morse code, Pig Latin, synonym swapping, payload splitting, translation into a less common language | The surface form of the request | The meaning the model recovers |
Source: NIST
Is Prompt Injection the Same as an AI Jailbreak?
No. Jailbreaking is a form of prompt injection where the attacker provides inputs that cause the model to disregard its safety protocols entirely, per OWASP. The containment runs one way, which matters when reading prompt injection attack data that counts both categories together.
Pros, Cons, and Risks
The advantages below belong to jailbreak testing rather than to jailbreaking. Running the attack in a controlled setting is how anyone learns where a model’s refusals hold.
Advantages
- Evaluations can measure the vulnerability of models to query-based attacks, according to NIST.
- The attacks are relatively basic and can be automated with a handful of calls to an auxiliary model. Testing therefore scales cheaply.
- Disclosure makes a result comparable. CAISI named the DeepSeek model versions it evaluated (R1, R1-0528 and V3.1) and the count of benchmarks behind its figures.
Trade-offs and Risks
- Current evaluation approaches, though a useful tool, may underestimate vulnerabilities accessible to actors with more time, resourcing, or luck, NIST states.
- Evaluations measure model vulnerabilities at a particular moment in time, and assessments may change if new attacks are developed, additional data is collected post-training, or model capabilities are improved.
- Jailbreaking or prompt engineering tests may not systematically assess validity or reliability risks, per NIST’s generative AI profile for its risk management framework.
A jailbroken answer can still be a correct one: The UK AI Security Institute did not observe a substantial decrease in correctness on its private set of benign questions after the attack had been applied, indicating that models may supply correct as well as compliant information.
How Organizations Reduce AI Jailbreak Risk
The register below belongs to the standards bodies rather than to us. NIST’s defense strategies can be employed throughout the deployment life cycle of an AI model or system to reduce the risk that the model or system will be vulnerable to direct prompt injections.
A range of training strategies have been proposed to increase the difficulty of accessing harmful model capabilities through direct prompt injection, including safety training during pre-training or post-training, adversarial training methods, and other methods to make jailbreak attacks more difficult. Training-stage work sits alongside the wider integrity problem covered in LLM data poisoning research, since both target the same artifact.
Evaluation is the other intervention point NIST names. Measuring a model’s vulnerability to query-based attacks can then inform trust and affordance decisions, as well as developer and user education. That is a scoping input rather than a control that closes the gap.
Developers can build safeguards into system prompts and input handling to help mitigate prompt injection attacks, but effective prevention of jailbreaking requires ongoing updates to the model’s training and safety mechanisms, per OWASP. Downstream exposure is the reason this matters operationally, and the AI-enabled attack volume data shows how quickly misuse capacity compounds once a model is trained.
Prompt injection vulnerabilities are possible due to the nature of generative AI, OWASP writes, and given the stochastic influence at the heart of the way models work, it is unclear if there are foolproof methods of prevention. That ceiling sits under every control listed above.
Real-World Applications
Two government bodies have published jailbreak findings with enough method attached to read as measurements. Those qualifiers travel badly, which is the recurring problem with measured LLM jailbreak rates quoted secondhand.
Government Evaluation of a Commercial Model
CAISI’s experts evaluated three DeepSeek models (R1, R1-0528 and V3.1) and four U.S. models across 19 benchmarks spanning a range of domains. DeepSeek’s most secure model, R1-0528, responded to 94% of overtly malicious requests when a common jailbreaking technique was used, compared with 8% of requests for U.S. reference models.
Those are per-technique results on named model versions, not a general safety score for either side. The evaluation does not name the technique that produced them.
National Safety Institute Testing at Portfolio Scale
The UK AI Security Institute measured compliance when no attack was used and when its in-house attacks were used, across HarmBench and its private set of harmful questions. Compliance rates were relatively low for most models when no attack was used but up to 28% for the Green model on private harmful questions.
The models carry color names in the source rather than product names. All models were highly vulnerable to the institute’s basic attacks for both HarmBench and its private set of harmful questions.
Agent Deployments as the Downstream Blast Radius
Agents based on DeepSeek’s most secure model (R1-0528) were, on average, 12 times more likely than evaluated U.S. frontier models to follow malicious instructions designed to derail them from user tasks, CAISI reported. That figure describes agent hijacking rather than jailbreaking, and the two are measured separately.
Hijacked agents sent phishing emails, downloaded and ran malware, and exfiltrated user login credentials, all in a simulated environment.
Can AI Jailbreaks Be Prevented?
No method is described as fool-proof by either standards body that publishes on the question. Given the stochastic influence at the heart of the way models work, it is unclear if there are fool-proof methods of prevention for prompt injection, OWASP states.
NIST’s defense strategies exist to reduce the risk that a model or system will be vulnerable to direct prompt injections. The practical question shifts from blocking the prompt to limiting what a model can touch once it answers one.
Does Passing a Jailbreak Test Mean a Model Is Safe?
No. Jailbreaking or prompt engineering tests may not systematically assess validity or reliability risks, according to NIST.
Evaluations also measure model vulnerabilities at a particular moment in time, and assessments may change if new attacks are developed, additional data is collected post-training, or model capabilities are improved. A clean test result describes the attacks somebody thought to run, on the version they ran them against.
Conclusion
The definition holds the whole argument together. An AI jailbreak is a direct prompting attack intended to circumvent restrictions placed on model outputs, such as circumventing refusal behavior to enable misuse. Nothing in that sentence describes a perimeter, which is why resilience varies so widely between models. CAISI measured 94% compliance with overtly malicious requests on DeepSeek’s most secure model under a common jailbreaking technique, against 8% for U.S. reference models.
The direction of travel points toward published, technique-named evaluation numbers, because that is the only form in which a jailbreak claim can be compared across two models. Until that becomes routine, one government finding is the safest working baseline. All tested LLMs remain highly vulnerable to basic jailbreaks, according to the UK AI Security Institute, and some will provide harmful outputs even without dedicated attempts to circumvent their safeguards. Buyers have to read the method, not the marketing.
