Blog
How to run an annual AI-incident tabletop (and why your policy may now require one)
AI governance policies increasingly call for annual incident exercises. What an AI-incident tabletop covers, how to run one, and why it is not adversarial or pen testing.
Many security policies still talk about "incident response" as if every incident is ransomware or a leaked database. AI systems change the shape of failure: model misuse, data leakage through prompts, third-party model outages, poisoned training or retrieval sources, and unsafe outputs that create legal or safety exposure. If your AI governance policy now requires an annual AI-incident exercise, a tabletop is a practical way to produce evidence without pretending you ran a red-team engagement.
Why policies are starting to require this
Boards and customers are asking how you would respond when an AI feature causes harm or when a model provider is unavailable. Internal AI policies often borrow the structure of classic IR: define roles, detection paths, escalation, customer communication, and a cadence of testing. Annual exercises are a common mandate because a written AI policy that nobody practices is hard to defend.
That mandate is about readiness practice and a record of practice. It is not a requirement that a vendor grade your AI program, and it is not a substitute for technical evaluations your risk team may separately require.
What an AI-incident scenario should cover
A useful AI-incident tabletop is grounded in your real stack: which models you call, which data can enter prompts or retrieval, who owns the feature, and who talks to customers if something goes wrong. Scenario families that tend to exercise the plan include:
- Suspected leakage of sensitive customer data through an AI assistant or support bot
- Harmful or non-compliant model output that reached production users
- Provider outage or degraded model behavior that breaks a customer-facing workflow
- Abuse of an AI feature for fraud, phishing assistance, or policy evasion
- Uncertainty about whether training, fine-tuning, or logging retained regulated data
The injects should force the same decisions a real event would: who freezes the feature, who reviews logs, who notifies counsel, who talks to the model vendor, and what you tell customers before you have perfect forensics.
How to run the session
Invite the roles your AI and security policies name: product owner for the AI surface, security/IR lead, engineering for the integration, privacy or counsel if data exposure is in scope, and customer-facing owners if notification clocks exist. State the objective up front: exercise the plan and capture gaps, not win the scenario.
Use timed injects so the room cannot skip uncomfortable steps. Record attendance, decisions, and plan-says versus room-did differences. If the plan is silent on AI-specific steps (for example, disabling a feature flag versus rotating a provider key), that silence is a finding worth writing down.
What this is not
An AI-incident tabletop is not adversarial testing of the model. It is not a penetration test of your prompt filters. It is not a red team engagement, a jailbreak contest, or a claim that the model is "safe." Those activities, if you run them, are separate work with separate evidence.
It is also not a live chaos test of production AI traffic unless you deliberately design such a test under change control. A tabletop is a facilitated decision exercise. Keep that boundary clear in the evidence packet so nobody later misreads it as technical proof.
Evidence you can keep
When the session ends, you should be able to produce a record that shows the scenario, who participated, what decisions were made under time pressure, and which plan gaps you will fix. That is the artifact many policies are looking for when they say "annual AI incident exercise." Whether it satisfies a particular auditor, customer questionnaire, or framework control remains a judgment for those parties, not for the exercise vendor.
Bottom line: if your policy now requires annual AI-incident practice, run a grounded tabletop, keep an honest record, and do not confuse a decision exercise with adversarial or technical testing. Practice the response; leave the verdict to the people who own assurance.