AI red teaming is a structured testing effort, often adopting adversarial methods, to find flaws and vulnerabilities in an AI system, including unforeseen or undesirable system behaviors or potential risks associated with the misuse of the system, according to NIST.
That wording repays close reading. The object of the search is flaws and vulnerabilities, unforeseen or undesirable system behaviors, and potential risks associated with misuse of the system, per NIST’s taxonomy and terminology of adversarial machine learning attacks and mitigations. Nothing in the definition mentions a score. Military wargaming and conventional network red teaming are separate exercises that share the label, and neither is the sense used here.
Key Takeaways
- The glossary entry covers flaws and vulnerabilities in an AI system, including unforeseen or undesirable system behaviors and potential risks associated with misuse of the system, according to NIST.
- Red teams are advised to assess resilience against prompt injection, data poisoning, membership inference, model extraction, adversarial examples, and abuse that facilitates attacks on other systems, per NIST.
- Article 55(1)(a) of the EU AI Act obliges providers of general-purpose AI models with systemic risk to conduct and document adversarial testing of the model, and Annex XI names red teaming as the worked example of that testing.
- Microsoft’s AI Red Team has red-teamed over 100 generative AI products and published eight lessons drawn from that work.
- CAISI reported that DeepSeek’s most secure model responded to 94% of overtly malicious requests when a common jailbreaking technique was used, compared with 8% of requests for U.S. reference models.
- Two of the eight lessons state that AI red teaming is not safety benchmarking and that the work of securing AI systems will never be complete.
How Does AI Red Teaming Work?
The exercise runs in 3 stages, from scoping to a written record. The definition specifies a structured testing effort, according to NIST. That word is doing real work, because it excludes opportunistic probing.
1. Define the System and the Harm You Are Hunting
The first lesson is to understand what the system can do and where it is applied, per Microsoft. Scoping comes first for that reason. A summarizer wired to a public web page and a summarizer wired to an internal mailbox share a model and face different harms, so the same weights yield two different target lists.
2. Attack It, by Hand and by Machine
The human half of the work is manual probing. The machine half runs a loop. When the attacker has access to a high-quality classifier that judges whether model output is harmful, per NIST, it may be used as a reward function to train a generative model to generate jailbreaks of another generative model.
Only query access is required for each of the models, and no human intervention is required to update or refine a candidate jailbreak, according to NIST. The prompts may also be transferable from the target model to other closed-source LLMs. Transferability is stated as a possibility, so a finding on one model is a lead rather than a verdict.
Microsoft’s second lesson states that you don’t have to compute gradients to break an AI system. The skill barrier is lower than the term adversarial machine learning suggests. Plain language, placed where the model will read it, does most of the work.
3. Write Down What Happened
NIST recommends documenting the instructions given to data annotators or AI red-teamers. Instructions are the missing variable in most published results: two teams told to look for different things find different things, and both file it as red teaming.
| Dimension | Conventional red teaming | AI red teaming |
|---|---|---|
| Primary target | Networks, hosts, and application code | The model’s behavior and the system around it |
| Entry point | Exposed services, credentials, configuration | Text, files, and tool calls the model reads |
| Failure signal | Access gained where it should have been denied | Flaws, undesirable behaviors, and misuse risk |
| Skill assumed | Exploit development against a software stack | No gradient computation needed to break it |
| When it ends | Scope closes once findings are remediated | The work of securing AI systems is never complete |
Sources: NIST, Microsoft AI Red Team
A fire drill is the closest parallel. It is graded on how many exits fail to open, and nobody leaves one believing the building is fireproof.
The process version is a proofreader paid per typo found. Ten typos tell you the manuscript holds at least ten, and nothing about how many remain.
Why Does AI Red Teaming Matter?
The practice has a legal reference point. Providers of general-purpose AI models with systemic risk shall perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks, under Article 55(1)(a) of the EU AI Act.
Annex XI, Section 2, point 2 asks for a detailed description of the measures put in place for the purpose of conducting internal and/or external adversarial testing, naming red teaming as the example. The annex is what binds the named practice to the obligation.
Scope discipline matters. The obligation binds providers of general-purpose AI models with systemic risk, so it is not a universal duty on every AI system or every company shipping a chatbot.
The tension sits in the phrase “standardised protocols”. A survey of industry activity and the research literature found that prior methods and practices of AI red-teaming diverge along several axes, including the purpose of the activity, which is often vague, the artifact under evaluation, the setting in which the activity is conducted, and the resulting decisions it informs, per Feffer and co-authors at Carnegie Mellon University.
An obligation arrived before a method did. That sequence is the reader’s actual exposure, because two suppliers can both write “we red-teamed it” in a procurement answer and mean incomparable things: different purposes, different artifacts, different thresholds for calling a finding a finding. The security column in our AI model security comparison data exists to make that gap legible.
Who Performs AI Red Teaming?
The field has one near-standard answer, and it is about independence. NIST advises verifying that those conducting structured human feedback exercises are not directly involved in system development tasks for the same model, filed as suggested action MS-1.3-003. A team grading its own homework produces a document rather than a test.
The EU AI Act contemplates internal and/or external adversarial testing. Both arrangements are in scope. Microsoft’s AI Red Team has red-teamed over 100 generative AI products and works from an internal threat model ontology.
An Anthropic team ran red teaming across 3 model sizes, at 2.7 billion, 13 billion, and 52 billion parameters, and 4 model types: a plain language model, one prompted to be helpful, honest, and harmless, one with rejection sampling, and one trained using reinforcement learning from human feedback. Microsoft’s fifth lesson holds that the human element of AI red teaming is crucial. Automation sits alongside people on that reading rather than in place of them.
Is AI Red Teaming the Same as Penetration Testing?
No. The two differ by target and by failure signal. A penetration test looks for access it should not have, while an AI red team looks for behavior the system should not produce.
The suggested scope names GAI attacks such as prompt injection and ML attacks such as adversarial examples, data poisoning, membership inference, and model extraction, per NIST, none of which are network intrusions. Measured volumes for the first of those categories sit in our prompt injection attack data.
Pros, Cons, and Risks
Advantages
- Automation can help cover more of the risk landscape, per Microsoft’s fourth lesson. A small team reaches a wider surface that way.
- Anthropic’s team released its dataset of red team attacks for others to analyze and learn from, and described its instructions, processes, statistical methodologies, and uncertainty about red teaming.
- The RLHF models are increasingly difficult to red team as they scale, with a flat trend with scale for the other model types.
- Findings arrive as prose a regulator can read, which is the form Annex XI asks for.
“Increasingly difficult to red team” is a difficulty gradient. It is not a claim that a model is safe, secure, or resistant.
Trade-offs and Risks
- Gestures towards red-teaming, based on public definitions, as a panacea for every possible risk verge on security theater, per Feffer and co-authors, who allow in the same breath that red-teaming may be a valuable big-tent idea and that industry may effectively apply red-teaming and other strategies behind closed doors to safeguard AI.
- Responsible AI harms are pervasive but difficult to measure, per Microsoft’s sixth lesson.
- The work of securing AI systems will never be complete, per the eighth lesson. A closed report describes a moment rather than a state.
What a clean report does not establish: AI red teaming is not safety benchmarking, per Microsoft’s third lesson. A red team that finds nothing has established that this team, with this scope, in this window, found nothing. Treating that as evidence of safety inverts what the exercise measures.
Types and Boundaries of AI Red Teaming
NIST groups the suggested scope into three families: Abuse to facilitate attacks on other systems, such as malicious code generation and enhanced phishing content; GAI attacks such as prompt injection; and ML attacks such as adversarial examples and prompts, data poisoning, membership inference, model extraction, and sponge examples.
The first family turns a model into an instrument against third parties, and machine-assisted intrusion volumes sit in our AI-enabled attack volume data.
| Attack category | What the red team tries | What a finding tells you | What it does not tell you |
|---|---|---|---|
| Abuse against other systems | Malicious code generation, enhanced phishing content | The model will produce usable attack material | How often that happens in production |
| Prompt injection | Planted text that redirects the model’s instructions | One route into the instruction channel is open | That every other route is closed |
| Adversarial examples and prompts | Inputs crafted to force a wrong output | The decision boundary bends under pressure | The size of the reachable error space |
| Data poisoning | Corrupted training or fine-tuning data | The pipeline accepted tainted input | Whether shipped weights are clean |
| Membership inference, model extraction | Queries that recover training records or behavior | Private detail leaks through the interface | The full extent of what can be recovered |
Sources: NIST, Microsoft AI Red Team
Two of those rows have their own measurement literature. Jailbreak success rates across public models are collected in our LLM jailbreak measurement data.
Poisoning sits further upstream, in the training pipeline rather than the prompt, and its published evidence base is gathered in our LLM data poisoning research.
A benchmark reports a score against a fixed set of items, while a red team reports what it managed to break. Neither result certifies the other.
Real-World Applications
Government Evaluation of Commercial Models
A U.S. standards body has published named results against named commercial models. CAISI found that DeepSeek’s most secure model, R1-0528, responded to 94% of overtly malicious requests when a common jailbreaking technique was used, compared with 8% of requests for U.S. reference models.
Agents based on that same model were, on average, 12 times more likely than evaluated U.S. frontier models to follow malicious instructions designed to derail them from user tasks, and hijacked agents sent phishing emails, downloaded and ran malware, and exfiltrated user login credentials, all in a simulated environment.
Read those figures narrowly. They are per-technique results on named models under a stated method, never a general safety score. CAISI describes the work in terms of jailbreaking techniques and agent hijacking attacks rather than red teaming, so treating it as an example of the practice is an editorial reading. It earns the place because it shows what a published adversarial result looks like when the body running it names the model, the technique, and the comparison set.
Vendor Red Teams at Product Scale
Microsoft’s AI Red Team reports red teaming over 100 generative AI products, and presents an internal threat model ontology alongside eight main lessons learned. The instructive part is the shape of the disclosure, since the published account prints its limits next to its methods.
Published Attack Corpora
Anthropic’s team released a dataset of 38,961 red team attacks for others to analyze and learn from, and exhaustively described its instructions, processes, statistical methodologies, and uncertainty about red teaming. Releasing the raw attacks lets a third party recount the findings instead of trusting a summary, which is the exception.
Does Red Teaming Make an AI Model Safe?
No. AI red teaming is not safety benchmarking, and the work of securing AI systems will never be complete, per two of the eight lessons published by Microsoft’s AI Red Team. A red-team report is evidence of the flaws found, and it says nothing about flaws nobody looked for.
The comparability problem compounds that limit. Practices diverge along the purpose of the activity, which is often vague, the artifact under evaluation, the setting, and the resulting decisions the activity informs. Red teaming helps reduce risk by surfacing weaknesses early. It does not certify, secure, or protect the model it was run against.
Can AI Red Teaming Be Automated?
Partly. Automated model-based red teaming employs an attacker model, a target model, and a judge, and only query access is required for each of the models, with no human intervention required to update or refine a candidate jailbreak, per NIST. That loop runs at a volume no human team matches.
Automation can help cover more of the risk landscape, per Microsoft’s fourth lesson, while its fifth holds that the human element of AI red teaming is crucial. The honest reading is coverage: machines widen the search, and people still decide what counts as harm.
Conclusion
CAISI found that DeepSeek’s most secure model responded to 94% of overtly malicious requests under a common jailbreaking technique, against 8% for U.S. reference models. Numbers that specific exist because one body published the model, the technique, and the comparison set together. Most results reach buyers as a sentence in a questionnaire.
The direction of travel is set by the statute. Article 55(1)(a) asks for model evaluation in accordance with standardised protocols and tools reflecting the state of the art. The literature currently describes practices that diverge along purpose, artifact, setting, and the decisions they inform. Until those converge, a red-team result stays what NIST’s definition makes it: a structured search for flaws, useful for reducing risk, and silent on everything nobody thought to test.
