Red Teaming AI Red Teaming

Summary

Red teaming has evolved from its origins in military applications to become a widely adopted methodology in cybersecurity and AI. In this paper, we take a critical look at the practice of AI red teaming. We argue that despite its current popularity in AI governance, there exists a significant gap between red teaming's original intent as a critical thinking exercise and its narrow focus on discovering model-level flaws in the context of generative AI. Current AI red teaming efforts focus predominantly on individual model vulnerabilities while overlooking the broader sociotechnical systems and emergent behaviors that arise from complex interactions between models, users, and environments. To address this deficiency, we propose a comprehensive framework operationalizing red teaming in AI systems at two levels: macro-level system red teaming spanning the entire AI development lifecycle, and micro-level model red teaming.

Plain-language summary

Red teaming has evolved from its origins in military applications to become a widely adopted methodology in cybersecurity and AI. In this paper, we take a critical look at the practice of AI red teaming. We argue that despite its current popularity in AI governance, there exists a significant gap between red teaming's original intent as a critical thinking exercise and its narrow focus on discovering model-level flaws in the context of generative AI. Current AI red teaming efforts focus predominantly on individual model vulnerabilities while overlooking the broader sociotechnical systems and emergent behaviors that arise from complex interactions between models, users, and environments. To address this deficiency, we propose a comprehensive framework operationalizing red teaming in AI systems at two levels: macro-level system red teaming spanning the entire AI development lifecycle, and micro-level model red teaming.

Research problem

We argue that despite its current popularity in AI governance, there exists a significant gap between red teaming's original intent as a critical thinking exercise and its narrow focus on discovering model-level flaws in the context of generative AI. We argue that despite its current popularity in AI governance, there exists a significant gap between red teaming’s original intent as a critical thinking exercise and its narrow focus on discovering model-level flaws in the context of generative AI. While that is not necessarily a bad thing—because red teaming is definitely a part of solving the AI security and safety problem—it is only a small part of a more complete set of tools that should be used.

Methodology

Red teaming has evolved from its origins in military applications to become a widely adopted methodology in cybersecurity and AI. We argue that despite its current popularity in AI governance, there exists a significant gap between red teaming's original intent as a critical thinking exercise and its narrow focus on discovering model-level flaws in the context of generative AI. Current AI red teaming efforts focus predominantly on individual model vulnerabilities while overlooking the broader sociotechnical systems and emergent behaviors that arise from complex interactions between models, users, and environments. To address this deficiency, we propose a comprehensive framework operationalizing red teaming in AI systems at two levels: macro-level system red teaming spanning the entire AI development lifecycle, and micro-level model red teaming. Rudd Abstract Red teaming has evolved from its origins in military applications to become a widely adopted methodology in cybersecurity and AI.

Main findings

We argue that despite its current popularity in AI governance, there exists a significant gap between red teaming's original intent as a critical thinking exercise and its narrow focus on discovering model-level flaws in the context of generative AI. We argue that despite its current popularity in AI governance, there exists a significant gap between red teaming’s original intent as a critical thinking exercise and its narrow focus on discovering model-level flaws in the context of generative AI. Introduction In recent past, the term red teaming has gained significant attention in diverse conversations around AI as a potential solution to find and address security, safety, and reliability concerns in generative AI (genAI) systems. This missing link negates the tremendous benefits that are not realized in bridging communities of practice in AI and cybersecurity, which could ultimately lead us to finding ways to achieve robust and effective AI governance and AI solutions that adhere more closely to the espoused principles of responsible AI. It should bring together multi- functional and cross-functional teams around the common goal of ultimately improving a product or user experience.

Key contributions

Limitations

• As the field matures, researchers increasingly recognize that red teaming alone cannot solve all the challenges in AI risk assessment (Ahmad et al., 2025).
• Critically, they should help everyone be on the same page to answer the question “What does good look like?” To this end, stakeholders must articulate not just what they want the system to do, but what they absolutely cannot allow it to do.
• Data distribution shifts over time can adversely impact system performance in ways that may not be immediately apparent.
• Secondly, the integration of cybersecurity red teaming practices with AI-specific concerns assumes transferability that may not hold, given AI systems’ probabilistic behav- iors and sociotechnical complexities.

Future work

• Future work should propose frameworks that can map cross-component interactions, track vulnerabilities across system boundaries, and automatically identify emergent risks arising from component interactions.

Evaluation metrics

Keywords