SecureRAG Incident Response Playbook

Owner: Security Team
Last Updated: 2025-12-10
Review Cycle: Quarterly

---

Incident Classification

Severity	Definition	Response Time	Examples
P1 - Critical	Active data breach, service down	15 minutes	Confirmed exfiltration, complete outage
P2 - High	Security bypass detected, partial outage	1 hour	Injection bypass, DoS in progress
P3 - Medium	Attempted attack blocked, degraded service	4 hours	Blocked exfiltration attempts, latency spike
P4 - Low	Minor security event, no impact	24 hours	Failed auth attempts, policy edge cases

---

Incident Response Phases

Phase 1: Detection & Triage (0-15 min)

Goals: Confirm incident, assess severity, notify stakeholders

Checklist:

• Alert received and acknowledged
• Initial severity assessment (P1-P4)
• Incident channel created (#incident-YYYY-MM-DD-brief)
• On-call responders notified
• Initial timeline started

Key questions:

1. Is this a real incident or false alarm?
2. What is the scope (users affected, data involved)?
3. Is the attack ongoing?
4. What evidence do we have?

---

Phase 2: Containment (15-60 min)

Goals: Stop bleeding, preserve evidence, limit blast radius

Actions by incident type:

Data Exfiltration

1. Identify affected user accounts
2. Revoke sessions: POST /admin/revoke-sessions?user_id=<id>
3. Disable affected accounts if necessary
4. Preserve logs (do not delete)
5. Snapshot affected systems

Injection Bypass

1. Document the attack payload
2. Add payload to blocklist (immediate)
3. Review similar patterns in logs
4. Assess what was accessed/output
5. Consider temporary defensive mode

DoS Attack

1. Enable aggressive rate limiting
2. Block attacking IPs at edge
3. Scale up if legitimate traffic affected
4. Engage CDN/DDoS protection
5. Monitor for attack pattern changes

Account Compromise

1. Disable compromised account
2. Revoke all sessions and API keys
3. Notify account owner (if not attacker)
4. Audit account activity (last 30 days)
5. Check for lateral movement

---

Phase 3: Eradication (1-4 hours)

Goals: Remove attacker access, fix vulnerability

Checklist:

• Vulnerability identified
• Fix developed and tested
• Fix deployed to production
• Verify fix effectiveness
• Check for persistence mechanisms

For security control bypass:

# Example: Adding new pattern to input validator
INPUT_VALIDATOR_PATTERNS.append({
    "name": "incident_2025_12_payload",
    "pattern": r"<specific regex>",
    "severity": "high",
    "added_date": "2025-12-XX",
    "incident_ref": "INC-2025-XXX"
})

---

Phase 4: Recovery (4-24 hours)

Goals: Restore normal operations, verify security

Checklist:

• All containment measures reviewed
• Affected accounts restored (if appropriate)
• Monitoring enhanced for recurrence
• User notifications sent (if required)
• External reporting (if required by compliance)

User notification template (data breach):

Subject: Security Notice - Action Required

We detected unauthorized access to your account on [DATE].

What happened: [Brief description]

What we did: [Actions taken]

What you should do:
1. Reset your password
2. Review your recent activity
3. Contact us if you notice anything unusual

We apologize for this incident and have taken steps to prevent recurrence.

---

Phase 5: Post-Incident (24-72 hours)

Goals: Document lessons learned, improve defenses

Postmortem template:

# Incident Postmortem: [Brief Title]

**Date:** YYYY-MM-DD
**Duration:** X hours
**Severity:** P1/P2/P3/P4
**Author:** [Name]

## Summary
[2-3 sentence description]

## Timeline
| Time | Event |
|------|-------|
| HH:MM | ... |

## Root Cause
[Detailed explanation]

## Impact
- Users affected: X
- Data exposed: Y
- Duration: Z hours

## What Went Well
- ...

## What Went Poorly
- ...

## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| ... | ... | ... | ... |

## Lessons Learned
- ...

---

Runbooks by Incident Type

Runbook: Prompt Injection Bypass

Trigger: Output validator logs show blocked content that reveals injection success

Steps:

1. Pull logs for the session

   python -m src.admin.logs --session-id <id> --full
   

2. Extract attack payload

   python -m src.security.extract_payload --log-file <file>
   

3. Test payload against current defenses

   python -m src.security.test_payload --payload "<payload>"
   

4. Add to blocklist

   python -m src.security.add_blocklist --payload "<payload>" --incident <id>
   

5. Assess impact

   • What did the LLM output?
   • Did any sensitive data leak?
   • Was output validator effective?

6. Update input validator ML model (if pattern is novel)

   python -m src.security.retrain_classifier --add-sample <payload>
   

---

Runbook: Suspected Exfiltration

Trigger: Unusual access patterns, high volume of document retrievals, blocked exfil attempts

Steps:

1. Identify suspicious user(s)

   python -m src.admin.anomaly_report --last 24h
   

2. Review their query history

   python -m src.admin.user_queries --user-id <id> --last 7d
   

3. Check what documents were retrieved

   python -m src.admin.user_retrievals --user-id <id> --last 7d
   

4. Look for exfiltration patterns:

   • Systematic document enumeration
   • Encoding in queries (base64, etc.)
   • Unusual query-to-retrieval ratios
   • Access outside normal hours

5. If confirmed malicious:

   • Revoke access immediately
   • Preserve all logs
   • Assess what was exposed
   • Determine if notification required

---

Runbook: System Prompt Extraction

Trigger: Output contains fragments of system prompt

Severity: P2 (system prompt is not highly sensitive, but indicates bypass)

Steps:

1. Document the attack that extracted the prompt

2. Review system prompt content

   • Does it contain sensitive info? (Usually no)
   • Does exposure enable further attacks?

3. Update output validator

   # Add system prompt fragments to blocklist
   OUTPUT_BLOCKLIST.append(system_prompt[:100])
   OUTPUT_BLOCKLIST.append(system_prompt[-100:])
   

4. Consider rotating system prompt structure (not content)

   • Add decoy instructions
   • Use different delimiters

5. Update input validator for the extraction technique

---

Escalation Matrix

Severity	Primary	Secondary	Executive
P1	Security Lead	Engineering Lead	CTO
P2	Security On-Call	Security Lead	—
P3	Security On-Call	—	—
P4	Async review	—	—

Contact methods:

• Slack: #security-oncall
• PagerDuty: security-critical escalation policy
• Email: security@example.com (not for P1/P2)

---

Communication Templates

Internal Status Update

🔴 INCIDENT UPDATE - [TITLE]
Status: [Investigating/Contained/Resolved]
Severity: P[X]
Impact: [Brief description]
Current actions: [What we're doing now]
Next update: [Time]

External Status Page

[Investigating/Identified/Monitoring/Resolved] - [Title]

We are currently investigating reports of [brief description].

Updates will be posted as they become available.

Posted: [Time] UTC

---

Evidence Preservation

What to preserve:

• Application logs (API, security, audit)
• Database query logs
• Network flow logs
• System snapshots (if relevant)
• User session data

How to preserve:

# Snapshot logs to incident bucket
python -m src.admin.preserve_logs \
  --incident-id INC-YYYY-XXX \
  --start "YYYY-MM-DD HH:MM" \
  --end "YYYY-MM-DD HH:MM" \
  --user-id <affected_user>

Chain of custody:

1. Document who accessed evidence
2. Use read-only access after preservation
3. Hash all preserved files
4. Store in tamper-evident location

---

Post-Incident Checklist

• Postmortem document created
• Action items assigned with due dates
• Security controls updated
• Monitoring/alerting improved
• Team debriefed
• Documentation updated
• Compliance notifications sent (if required)
• User notifications sent (if required)
• Incident closed in tracking system

---

Playbook version 1.1 | Next review: 2026-03-01