Annex-K: AI Threat Modeling Template
Provides a structured template for documenting AI system threats, risks, and mitigations across model, infrastructure, data, and operational layers.
What this file does
Provides a structured template for documenting AI system threats, risks, and mitigations across model, infrastructure, data, and operational layers.
When to use it
- Starting a security review for a new AI system
- Compliance audit requires formal threat modeling documentation
- Onboarding a third-party AI component into your infrastructure
- Preparing for a penetration test or vulnerability assessment
Annex-K: AI Threat Modeling Template
1. System Overview
- AI System Name:
- Version:
- Business Purpose:
- Risk Classification: Low / Medium / High
- Data Types Processed:
- Deployment Environment:
- Integration Points:
2. System Architecture
- Model Type/Framework:
- API Dependencies:
- Data Storage:
- Authentication Mechanisms:
Include architecture diagram here
3. Trust Boundaries
| Boundary | Description | Protection Mechanism | Assessed Strength |
|---|---|---|---|
| External-Internal | Low/Medium/High | ||
| Model-Infrastructure | Low/Medium/High | ||
| Data Processing Zones | Low/Medium/High | ||
| User-System | Low/Medium/High | ||
| Third-Party Components | Low/Medium/High |
4. Threat Categories & Risk Assessment
4.1 Model Security Threats
| Threat | Likelihood | Impact | Risk Rating | Controls |
|---|---|---|---|---|
| Prompt injection | ||||
| Model extraction | ||||
| Training data poisoning | ||||
| Adversarial examples | ||||
| Model inversion attacks | ||||
| Membership inference |
4.2 Infrastructure Threats
| Threat | Likelihood | Impact | Risk Rating | Controls |
|---|---|---|---|---|
| API abuse | ||||
| DDoS attacks | ||||
| Supply chain attacks | ||||
| Data exfiltration | ||||
| Authentication bypass |
4.3 Data Security Threats
| Threat | Likelihood | Impact | Risk Rating | Controls |
|---|---|---|---|---|
| Training data leakage | ||||
| Sensitive info in outputs | ||||
| Data poisoning | ||||
| Unauthorized access |
4.4 Operational Threats
| Threat | Likelihood | Impact | Risk Rating | Controls |
|---|---|---|---|---|
| Availability disruption | ||||
| Performance degradation | ||||
| Dependency failures |
5. Attack Vectors & Scenarios
| ID | Attack Vector | Attack Scenario | Impact | Mitigation |
|---|---|---|---|---|
| AV-1 | ||||
| AV-2 | ||||
| AV-3 |
6. Security Controls Matrix
| Control Category | Control | Implementation Status | Testing Method | Testing Frequency |
|---|---|---|---|---|
| Input Validation | ||||
| Authentication | ||||
| Authorization | ||||
| Data Protection | ||||
| Monitoring & Detection | ||||
| Response & Recovery |
7. Vulnerability Testing Plan
| Test Type | Scope | Methodology | Schedule | Responsible Team |
|---|---|---|---|---|
| Prompt injection testing | ||||
| Fuzzing | ||||
| Adversarial testing | ||||
| Penetration testing | ||||
| Supply chain review |
8. Risk Mitigation Plan
| Risk | Mitigation Action | Owner | Timeline | Success Criteria | Status |
|---|---|---|---|---|---|
9. Security Monitoring Requirements
- Logging Requirements:
- Alert Thresholds:
- Incident Response Integration:
10. Approvals & Reviews
| Role | Name | Signature | Date |
|---|---|---|---|
| AI Security Architect | |||
| IT Security Officer | |||
| DPO | |||
| AI Ethics Board Rep |
Threat Model Version: ______
Review Date: //20__
Next Review Due: //20__
What's inside
10 sections: system overview, architecture, trust boundaries, 4 threat categories, attack vectors, controls matrix, test plan, risk plan, monitoring, approvals
Change this for your project
- Replace
sjefvanleeuwen/generative-ai-ethics-policywith your own repository or project name - Fill all empty fields in tables (e.g.,
| | | | |) with your system-specific data - Update
Annex-K: AI Threat Modeling Templatetitle to match your document naming convention
Where it goes
Keep it in your repository where the agent or team that needs it will read it.
Worth borrowing
- Trust boundaries table with assessed strength column forces explicit security posture per zone
- Threat categories separated by layer (model, infra, data, ops) makes coverage gaps visible
Related Documents
Overview
Summarises a completed C4 audit of Coinbase's Smart Wallet, listing 1 high and 2 medium severity vulnerabilities with findings, mitigations, and warden credits.
Table of Contents
Outlines governance, risk management, and legal considerations for building an information security strategy.
Week 6
Summarises risk management concepts from an information security textbook, including identification, assessment, control strategies, and cost-benefit analysis.
ch-17
Covers risk management, assessment, disaster recovery, and privacy concepts for the CompTIA Security+ exam.