AI red teaming vs penetration testing: what is the difference?

Penetration testing checks a system for known technical weaknesses — the digital equivalent of walking around a building and checking that every door and window is locked. Red teaming gives a team of testers the goal of a real attacker — steal something, trick the system, get in — and the freedom to try any route, including routes nobody thought to lock. For AI systems, this difference has become decisive, and ISO/IEC 27090, the international guidance on security threats to AI systems, describes red teaming as a practice increasingly applied to test them.

If you work with management systems — as an implementer, consultant or auditor — you do not need to be a technical specialist to understand why. You need one idea, and it fits in a sentence: most attacks on AI systems do not break the technology; they persuade it.

What is penetration testing?

Penetration testing (or “pen testing”) examines a system against known weaknesses: software that was not updated, services left exposed, settings configured insecurely, passwords that are easy to break. It usually works from a defined checklist, within a defined perimeter, and it answers a clear question: are the known doors locked? It is established practice, it is necessary, and nothing in this article suggests dropping it.

What is AI red teaming?

Red teaming is older than AI — it comes from military exercises where a “red team” plays the enemy. Applied to AI systems, it means giving testers an objective a real adversary would have, and letting them pursue it the way a real adversary would: through the technology, the processes around it, and the people.

Typical objectives read like a wish list of what could go wrong with your organization’s AI: make the system give wrong answers on purpose; make an assistant reveal confidential information; trick the system into approving something it should refuse; copy the organization’s model by simply using it cleverly; discover whether a particular person’s data was used to train it.

Notice what these objectives have in common: none of them requires “hacking” in the traditional sense. The attacker uses the system through its normal interface, the way any user would — just with intent.

Why is ordinary security testing not enough for AI?

Think of a conventional IT system as a building: security means locks, alarms, access cards. A penetration test checks all of them. Now think of an AI system as a building with a helpful employee standing in the lobby — one who answers questions, fetches documents and performs tasks for whoever asks. You can have perfect locks and alarms, and still have a problem if that employee can be talked into handing over the wrong file.

That, in essence, is the AI security gap. The model — the “employee” — behaves based on what it learned and what it is told. It can be misled through ordinary inputs: a cleverly phrased question, an instruction hidden inside a document it was asked to read. A penetration test will inspect the servers, the connections and the passwords around that employee, find everything in order, and issue a clean report — for a system that can still be persuaded to misbehave.

Those weaknesses do not appear on any scan, because there is nothing technically broken. They appear only when someone sits down and tries to make the system misbehave. That is exactly what AI red teaming does.

Penetration testing vs red teaming at a glance

A simple example

Imagine an internal assistant that answers employees’ questions from the company’s document store. The security analysis identified three worries: someone could hide instructions inside a document and the assistant might follow them; someone could coax the assistant into showing a document they are not allowed to see; someone could get the assistant to reveal its own internal instructions.

A red team takes those three worries and turns them into assignments. They plant a document containing hidden instructions and watch whether the assistant obeys. They ask carefully designed questions aimed at a confidential document. They try the known tricks for making an assistant recite its instructions. Then they report: what they achieved, how, and how far they got.

If the design said “each user can only retrieve documents they are authorized for,” and the red team obtained a confidential document anyway — that is the finding that matters. A control that was supposed to exist is not working. No scanner would ever have said so.

What does good AI red teaming look like?

Whether you are commissioning an exercise or auditing an organization that ran one, six characteristics separate a serious exercise from a checkbox:

1. A defined scope and rules, agreed in advance — which systems, which objectives, what is off-limits. And for AI, the scope must name the behavior of the model explicitly; otherwise you have bought a conventional penetration test with an AI label.
2. More than one attacker scenario. Attackers differ in what they know — an outsider who can only use the public service, someone who knows which widely available model the system is built on, an insider with full access. Testing under more than one assumption tells you far more than a single run.
3. Testing your deployment, not just the model. If the AI comes from a large provider, the provider tested the base model. They did not test your configuration, your data connections, or the tasks your system can perform. Those belong to you — and they are usually where the findings are.
4. Independence. The people attacking the system should not be the people who built it; builders share their system’s blind spots. A separate internal team or an external specialist both work.
5. Repetition. A red team exercise is a photograph, not a guarantee. A new model version, a new data source, a new capability — each changes the picture, and attack techniques evolve on their own.
6. Findings treated as risk, not as a grade. The report is not pass-or-fail. It is a list of things an adversary could do — input for the risk assessment, where each finding is fixed, mitigated or knowingly accepted, with the decision recorded.

For auditors, that last point is the one to underline. The record — scope, findings, decisions — is strong objective evidence that AI-specific threats were genuinely examined rather than assumed away. Its absence invites a very reasonable audit question: how does the organization know its AI systems resist attacks that are now public knowledge?

One vocabulary warning

In the AI world, “red teaming” is also used for something else: testing whether a model produces harmful or offensive content. That is a worthwhile exercise, but it is a different one from security red teaming, which is what ISO/IEC 27090 means. The two overlap, but when you commission or audit an AI red team, be clear which question you are paying to have answered — “can the model say bad things?” or “can an attacker steal, leak or manipulate through it?”

Frequently asked questions

Is AI red teaming required by ISO/IEC 27090 or ISO/IEC 42001?
ISO/IEC 27090 is guidance, so it requires nothing — it describes red teaming as an increasingly applied practice for testing AI security. ISO/IEC 42001 requires AI risks to be assessed and treated; red teaming is one of the strongest ways to show that treatment rests on evidence rather than assumption.

Does red teaming replace penetration testing?
No. The infrastructure around an AI system still needs conventional testing. Red teaming adds the layer that penetration testing cannot see: the behavior of the model under adversarial use.

Our AI comes from a major provider. Isn’t their testing enough?
Their testing covers their base model. Your configuration, your data connections and the actions your system can take were never in their scope — and that is where most real-world findings come from.

Do smaller organizations need this?
In proportion. A small organization using AI assistants mainly needs assurance about its deployment and its suppliers; a full adversarial exercise matters most where AI makes consequential decisions or handles sensitive data. The principle scales; the budget should match the risk.

Want to go deeper? Our courses on ISO/IEC 27001 and ISO/IEC 42001 cover the management systems this testing feeds into. And if you want to prove your knowledge, the ISO/IEC 42001 AIMS Practitioner and AIMS Auditor certifications, as well as the Responsible AI Specialist certification, are smart options.