Incident Response & SLA Management: Alerting Stack, Escalation, and Service Level Agreements

# Incident Response & SLA Management: Alerting Stack, Escalation, and Service Level Agreements

## Overview

Enterprise platforms require sophisticated incident response infrastructure:

- **Alerting Stack**: Monitoring, detection, and notification
- **Escalation Policies**: On-call rotations, escalation chains
- **Runbooks**: Automated and manual remediation
- **SLA Management**: Uptime guarantees, response times
- **Post-Incident**: Root cause analysis, prevention

This guide covers complete incident response systems for platforms like Manus.

## 1. Alerting Stack Architecture

### 1.1 Recommended Stack

```typescript
/**
 * Complete alerting stack for enterprise platform
 * 
 * Stack:
 * 1. Prometheus/Grafana - Metrics collection and visualization
 * 2. AlertManager - Alert routing and grouping
 * 3. PagerDuty - Incident management and on-call
 * 4. Opsgenie - Backup alerting and escalation
 * 5. Custom webhooks - Integration with internal systems
 */

interface AlertingStackConfig {
  metricsCollection: {
    tool: 'Prometheus';
    scrapeInterval: 15; // seconds
    retention: 15; // days
    replicas: 3;
  };
  visualization: {
    tool: 'Grafana';
    dashboards: number;
    alertRules: number;
  };
  alertRouting: {
    tool: 'AlertManager';
    grouping: 'by-labels';
    groupWait: 10; // seconds
    groupInterval: 10; // seconds
    repeatInterval: 4; // hours
  };
  incidentManagement: {
    primary: 'PagerDuty';
    backup: 'Opsgenie';
    features: [
      'On-call scheduling',
      'Escalation policies',
      'Incident tracking',
      'Post-mortem management'
    ];
  };
  customIntegration: {
    webhooks: true;
    slack: true;
    email: true;
    sms: true;
    phone: true;
  };
}

const recommendedAlertingStack: AlertingStackConfig = {
  metricsCollection: {
    tool: 'Prometheus',
    scrapeInterval: 15,
    retention: 15,
    replicas: 3
  },
  visualization: {
    tool: 'Grafana',
    dashboards: 50,
    alertRules: 200
  },
  alertRouting: {
    tool: 'AlertManager',
    grouping: 'by-labels',
    groupWait: 10,
    groupInterval: 10,
    repeatInterval: 4
  },
  incidentManagement: {
    primary: 'PagerDuty',
    backup: 'Opsgenie',
    features: [
      'On-call scheduling',
      'Escalation policies',
      'Incident tracking',
      'Post-mortem management'
    ]
  },
  customIntegration: {
    webhooks: true,
    slack: true,
    email: true,
    sms: true,
    phone: true
  }
};
```

## 2. Alert Rules and Thresholds

### 2.1 Comprehensive Alert Rules

```yaml
# Prometheus alert rules for Manus-like platform

groups:
  - name: platform_alerts
    interval: 15s
    rules:
      # API Performance Alerts
      - alert: HighAPILatency
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
          component: api
        annotations:
          summary: "High API latency detected"
          description: "P99 latency is {{ $value }}s"
          runbook: "https://wiki.company.com/runbooks/high-api-latency"

      - alert: CriticalAPILatency
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 5
        for: 2m
        labels:
          severity: critical
          component: api
        annotations:
          summary: "Critical API latency"
          description: "P99 latency is {{ $value }}s"
          runbook: "https://wiki.company.com/runbooks/critical-api-latency"

      # Error Rate Alerts
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
          component: api
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }}"
          runbook: "https://wiki.company.com/runbooks/high-error-rate"

      - alert: CriticalErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
        for: 2m
        labels:
          severity: critical
          component: api
        annotations:
          summary: "Critical error rate"
          description: "Error rate is {{ $value | humanizePercentage }}"

      # Database Alerts
      - alert: HighDatabaseLatency
        expr: histogram_quantile(0.99, rate(db_query_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
          component: database
        annotations:
          summary: "High database latency"
          description: "P99 query latency is {{ $value }}s"

      - alert: DatabaseConnectionPoolExhausted
        expr: db_connection_pool_active / db_connection_pool_max > 0.9
        for: 2m
        labels:
          severity: critical
          component: database
        annotations:
          summary: "Database connection pool exhausted"
          description: "Pool utilization is {{ $value | humanizePercentage }}"

      - alert: ReplicationLagHigh
        expr: pg_replication_lag_seconds > 5
        for: 5m
        labels:
          severity: warning
          component: database
        annotations:
          summary: "High database replication lag"
          description: "Replication lag is {{ $value }}s"

      # Kubernetes Alerts
      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[1h]) > 0.1
        for: 5m
        labels:
          severity: warning
          component: kubernetes
        annotations:
          summary: "Pod crash looping"
          description: "Pod {{ $labels.pod }} is crash looping"

      - alert: NodeNotReady
        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
        for: 5m
        labels:
          severity: critical
          component: kubernetes
        annotations:
          summary: "Kubernetes node not ready"
          description: "Node {{ $labels.node }} is not ready"

      # LLM Service Alerts
      - alert: LLMProviderDown
        expr: up{job="llm-provider"} == 0
        for: 2m
        labels:
          severity: critical
          component: llm
        annotations:
          summary: "LLM provider down"
          description: "Provider {{ $labels.provider }} is down"

      - alert: HighTokenCost
        expr: rate(llm_tokens_used_total[1h]) * llm_cost_per_token > 1000
        for: 10m
        labels:
          severity: warning
          component: llm
        annotations:
          summary: "High LLM token cost"
          description: "Hourly cost is ${{ $value }}"

      # Infrastructure Alerts
      - alert: HighCPUUsage
        expr: (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) > 0.9
        for: 5m
        labels:
          severity: warning
          component: infrastructure
        annotations:
          summary: "High CPU usage"
          description: "CPU usage is {{ $value | humanizePercentage }}"

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.9
        for: 5m
        labels:
          severity: warning
          component: infrastructure
        annotations:
          summary: "High memory usage"
          description: "Memory usage is {{ $value | humanizePercentage }}"

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.1
        for: 5m
        labels:
          severity: warning
          component: infrastructure
        annotations:
          summary: "Low disk space"
          description: "Available disk space is {{ $value | humanizePercentage }}"

      # SLA Alerts
      - alert: SLAViolationWarning
        expr: (1 - (increase(requests_success_total[1h]) / increase(requests_total[1h]))) > 0.005
        for: 10m
        labels:
          severity: warning
          component: sla
        annotations:
          summary: "SLA violation warning"
          description: "Error rate is {{ $value | humanizePercentage }}"

      - alert: SLAViolationCritical
        expr: (1 - (increase(requests_success_total[1h]) / increase(requests_total[1h]))) > 0.01
        for: 5m
        labels:
          severity: critical
          component: sla
        annotations:
          summary: "Critical SLA violation"
          description: "Error rate is {{ $value | humanizePercentage }}"
```

## 3. Escalation Policies

### 3.1 On-Call Escalation

```typescript
/**
 * On-call escalation policies for incident management
 */

interface EscalationPolicy {
  name: string;
  levels: EscalationLevel[];
  handoffTimeout: number; // seconds
  repeatCycle: boolean;
}

interface EscalationLevel {
  level: number;
  delay: number; // seconds from previous level
  targets: OnCallTarget[];
  notificationChannels: NotificationChannel[];
}

interface OnCallTarget {
  type: 'user' | 'schedule' | 'team';
  id: string;
  name: string;
}

interface NotificationChannel {
  type: 'email' | 'sms' | 'phone' | 'slack' | 'webhook';
  priority: 'low' | 'medium' | 'high' | 'urgent';
}

const escalationPolicies: Record<string, EscalationPolicy> = {
  critical: {
    name: 'Critical Incident Escalation',
    levels: [
      {
        level: 1,
        delay: 0,
        targets: [
          { type: 'schedule', id: 'primary-oncall', name: 'Primary On-Call' }
        ],
        notificationChannels: [
          { type: 'sms', priority: 'urgent' },
          { type: 'phone', priority: 'urgent' },
          { type: 'email', priority: 'high' },
          { type: 'slack', priority: 'high' }
        ]
      },
      {
        level: 2,
        delay: 300, // 5 minutes
        targets: [
          { type: 'schedule', id: 'secondary-oncall', name: 'Secondary On-Call' }
        ],
        notificationChannels: [
          { type: 'sms', priority: 'urgent' },
          { type: 'phone', priority: 'urgent' },
          { type: 'email', priority: 'high' }
        ]
      },
      {
        level: 3,
        delay: 600, // 10 minutes
        targets: [
          { type: 'team', id: 'platform-team', name: 'Platform Team Lead' }
        ],
        notificationChannels: [
          { type: 'sms', priority: 'urgent' },
          { type: 'phone', priority: 'urgent' },
          { type: 'email', priority: 'high' }
        ]
      },
      {
        level: 4,
        delay: 900, // 15 minutes
        targets: [
          { type: 'user', id: 'cto', name: 'CTO' }
        ],
        notificationChannels: [
          { type: 'phone', priority: 'urgent' },
          { type: 'email', priority: 'high' }
        ]
      }
    ],
    handoffTimeout: 3600, // 1 hour
    repeatCycle: true
  },

  high: {
    name: 'High Priority Escalation',
    levels: [
      {
        level: 1,
        delay: 0,
        targets: [
          { type: 'schedule', id: 'primary-oncall', name: 'Primary On-Call' }
        ],
        notificationChannels: [
          { type: 'email', priority: 'high' },
          { type: 'slack', priority: 'high' }
        ]
      },
      {
        level: 2,
        delay: 900, // 15 minutes
        targets: [
          { type: 'schedule', id: 'secondary-oncall', name: 'Secondary On-Call' }
        ],
        notificationChannels: [
          { type: 'email', priority: 'high' },
          { type: 'slack', priority: 'high' }
        ]
      }
    ],
    handoffTimeout: 7200, // 2 hours
    repeatCycle: true
  },

  medium: {
    name: 'Medium Priority Escalation',
    levels: [
      {
        level: 1,
        delay: 0,
        targets: [
          { type: 'schedule', id: 'primary-oncall', name: 'Primary On-Call' }
        ],
        notificationChannels: [
          { type: 'email', priority: 'medium' },
          { type: 'slack', priority: 'medium' }
        ]
      }
    ],
    handoffTimeout: 14400, // 4 hours
    repeatCycle: false
  }
};

class EscalationManager {
  /**
   * Execute escalation policy
   */
  async executeEscalation(
    incident: Incident,
    policy: EscalationPolicy
  ): Promise<void> {
    for (const level of policy.levels) {
      // Wait for delay
      await this.sleep(level.delay * 1000);

      // Check if incident is resolved
      const resolved = await this.isIncidentResolved(incident.id);
      if (resolved) {
        console.log(`Incident ${incident.id} resolved, stopping escalation`);
        break;
      }

      // Notify targets
      for (const target of level.targets) {
        await this.notifyTarget(incident, target, level.notificationChannels);
      }

      // Wait for handoff timeout
      const acknowledged = await this.waitForAcknowledgment(
        incident.id,
        policy.handoffTimeout
      );

      if (acknowledged) {
        console.log(`Incident ${incident.id} acknowledged at level ${level.level}`);
        break;
      }

      // If not acknowledged and not last level, continue to next
      if (level.level < policy.levels.length) {
        console.log(`No acknowledgment at level ${level.level}, escalating...`);
      }
    }

    // If repeat cycle enabled and still not resolved, restart
    if (policy.repeatCycle) {
      const resolved = await this.isIncidentResolved(incident.id);
      if (!resolved) {
        await this.executeEscalation(incident, policy);
      }
    }
  }

  private async notifyTarget(
    incident: Incident,
    target: OnCallTarget,
    channels: NotificationChannel[]
  ): Promise<void> {
    // Implementation: Send notifications via all channels
  }

  private async waitForAcknowledgment(
    incidentId: string,
    timeout: number
  ): Promise<boolean> {
    // Implementation: Wait for acknowledgment
    return false;
  }

  private async isIncidentResolved(incidentId: string): Promise<boolean> {
    // Implementation: Check incident status
    return false;
  }

  private sleep(ms: number): Promise<void> {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

interface Incident {
  id: string;
  severity: 'critical' | 'high' | 'medium' | 'low';
  title: string;
  description: string;
  createdAt: Date;
  resolvedAt?: Date;
  assignee?: string;
}
```

## 4. SLA Management

### 4.1 SLA Definitions and Tracking

```typescript
/**
 * SLA definitions and tracking for enterprise platform
 */

interface SLADefinition {
  name: string;
  availability: number; // 99.9, 99.99, etc.
  responseTime: number; // ms
  resolutionTime: number; // minutes
  maintenanceWindow: number; // hours per month
}

interface SLAMetrics {
  uptime: number;
  downtime: number;
  incidents: number;
  mttr: number; // Mean Time To Resolution
  mtbf: number; // Mean Time Between Failures
  errorBudget: number;
}

const slaDefinitions: Record<string, SLADefinition> = {
  enterprise: {
    name: 'Enterprise SLA',
    availability: 99.99, // 4 nines
    responseTime: 100, // 100ms p99
    resolutionTime: 15, // 15 minutes for critical
    maintenanceWindow: 4 // 4 hours per month
  },

  premium: {
    name: 'Premium SLA',
    availability: 99.9, // 3 nines
    responseTime: 200, // 200ms p99
    resolutionTime: 30, // 30 minutes for critical
    maintenanceWindow: 8 // 8 hours per month
  },

  standard: {
    name: 'Standard SLA',
    availability: 99.5, // 2.5 nines
    responseTime: 500, // 500ms p99
    resolutionTime: 60, // 60 minutes for critical
    maintenanceWindow: 16 // 16 hours per month
  }
};

class SLATracker {
  /**
   * Calculate uptime percentage
   */
  calculateUptime(
    totalTime: number, // seconds
    downtime: number // seconds
  ): number {
    return ((totalTime - downtime) / totalTime) * 100;
  }

  /**
   * Calculate error budget
   */
  calculateErrorBudget(sla: SLADefinition, period: 'month' | 'year'): number {
    const periodSeconds = period === 'month' ? 30 * 24 * 60 * 60 : 365 * 24 * 60 * 60;
    const allowedDowntime = periodSeconds * ((100 - sla.availability) / 100);
    return allowedDowntime;
  }

  /**
   * Check SLA violation
   */
  checkSLAViolation(
    metrics: SLAMetrics,
    sla: SLADefinition,
    period: 'month' | 'year'
  ): SLAViolationReport {
    const errorBudget = this.calculateErrorBudget(sla, period);
    const violated = metrics.downtime > errorBudget;

    return {
      violated,
      uptime: metrics.uptime,
      target: sla.availability,
      downtime: metrics.downtime,
      allowedDowntime: errorBudget,
      violationAmount: Math.max(0, metrics.downtime - errorBudget),
      creditPercentage: this.calculateCredit(metrics.uptime, sla.availability)
    };
  }

  /**
   * Calculate service credit
   */
  private calculateCredit(uptime: number, target: number): number {
    if (uptime >= target) {
      return 0; // No credit
    }

    if (uptime >= target - 0.1) {
      return 10; // 10% credit
    } else if (uptime >= target - 0.5) {
      return 25; // 25% credit
    } else if (uptime >= target - 1.0) {
      return 50; // 50% credit
    } else {
      return 100; // 100% credit
    }
  }

  /**
   * Generate SLA report
   */
  generateSLAReport(
    metrics: SLAMetrics,
    sla: SLADefinition,
    period: 'month' | 'year'
  ): SLAReport {
    const violation = this.checkSLAViolation(metrics, sla, period);

    return {
      period,
      sla: sla.name,
      metrics,
      violation,
      summary: {
        uptime: `${metrics.uptime.toFixed(3)}%`,
        target: `${sla.availability}%`,
        status: violation.violated ? 'VIOLATED' : 'MET',
        creditEarned: `${violation.creditPercentage}%`
      }
    };
  }
}

interface SLAViolationReport {
  violated: boolean;
  uptime: number;
  target: number;
  downtime: number;
  allowedDowntime: number;
  violationAmount: number;
  creditPercentage: number;
}

interface SLAReport {
  period: string;
  sla: string;
  metrics: SLAMetrics;
  violation: SLAViolationReport;
  summary: {
    uptime: string;
    target: string;
    status: string;
    creditEarned: string;
  };
}
```

## 5. Runbooks and Automation

### 5.1 Automated Remediation

```typescript
/**
 * Automated remediation runbooks
 */

interface Runbook {
  id: string;
  name: string;
  trigger: AlertTrigger;
  steps: RunbookStep[];
  rollback: RunbookStep[];
  timeout: number; // seconds
}

interface RunbookStep {
  name: string;
  type: 'check' | 'action' | 'notify' | 'wait';
  condition?: string;
  action?: string;
  timeout?: number;
}

interface AlertTrigger {
  alertName: string;
  severity: string;
  condition?: string;
}

const runbooks: Record<string, Runbook> = {
  highAPILatency: {
    id: 'high-api-latency',
    name: 'High API Latency Remediation',
    trigger: {
      alertName: 'HighAPILatency',
      severity: 'warning'
    },
    steps: [
      {
        name: 'Check API pod status',
        type: 'check',
        action: 'kubectl get pods -n api'
      },
      {
        name: 'Check resource usage',
        type: 'check',
        action: 'kubectl top pods -n api'
      },
      {
        name: 'Scale up API deployment',
        type: 'action',
        action: 'kubectl scale deployment api --replicas=20 -n api',
        condition: 'pod_cpu_usage > 80%'
      },
      {
        name: 'Wait for new pods',
        type: 'wait',
        timeout: 60
      },
      {
        name: 'Verify latency improved',
        type: 'check',
        action: 'check_metric("http_request_duration_seconds_bucket", "< 1s")'
      },
      {
        name: 'Notify on-call',
        type: 'notify',
        action: 'notify_slack("API latency remediated")'
      }
    ],
    rollback: [
      {
        name: 'Scale down API deployment',
        type: 'action',
        action: 'kubectl scale deployment api --replicas=10 -n api'
      }
    ],
    timeout: 300
  },

  databaseConnectionPoolExhausted: {
    id: 'db-pool-exhausted',
    name: 'Database Connection Pool Exhaustion',
    trigger: {
      alertName: 'DatabaseConnectionPoolExhausted',
      severity: 'critical'
    },
    steps: [
      {
        name: 'Check active connections',
        type: 'check',
        action: 'SELECT count(*) FROM pg_stat_activity'
      },
      {
        name: 'Identify long-running queries',
        type: 'check',
        action: 'SELECT * FROM pg_stat_activity WHERE state = "active" AND query_start < now() - interval "5 minutes"'
      },
      {
        name: 'Terminate idle connections',
        type: 'action',
        action: 'SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = "idle" AND query_start < now() - interval "10 minutes"'
      },
      {
        name: 'Increase connection pool size',
        type: 'action',
        action: 'pgbouncer_update_config("default_pool_size=50")'
      },
      {
        name: 'Notify database team',
        type: 'notify',
        action: 'page_on_call("database")'
      }
    ],
    rollback: [
      {
        name: 'Restore connection pool size',
        type: 'action',
        action: 'pgbouncer_update_config("default_pool_size=25")'
      }
    ],
    timeout: 600
  },

  highErrorRate: {
    id: 'high-error-rate',
    name: 'High Error Rate Remediation',
    trigger: {
      alertName: 'HighErrorRate',
      severity: 'warning'
    },
    steps: [
      {
        name: 'Check error logs',
        type: 'check',
        action: 'kubectl logs -n api -l app=api --tail=100 | grep ERROR'
      },
      {
        name: 'Identify error pattern',
        type: 'check',
        action: 'analyze_error_logs()'
      },
      {
        name: 'Restart affected pods',
        type: 'action',
        action: 'kubectl rollout restart deployment/api -n api',
        condition: 'error_pattern == "crash_loop"'
      },
      {
        name: 'Rollback recent deployment',
        type: 'action',
        action: 'kubectl rollout undo deployment/api -n api',
        condition: 'error_pattern == "regression"'
      },
      {
        name: 'Verify error rate decreased',
        type: 'check',
        action: 'check_metric("http_requests_total{status=~\"5..\"}", "< 5%")'
      }
    ],
    rollback: [],
    timeout: 900
  }
};

class RunbookExecutor {
  /**
   * Execute runbook
   */
  async executeRunbook(runbook: Runbook, incident: Incident): Promise<RunbookResult> {
    const result: RunbookResult = {
      runbookId: runbook.id,
      incidentId: incident.id,
      startTime: new Date(),
      steps: [],
      success: false,
      error: null
    };

    try {
      for (const step of runbook.steps) {
        const stepResult = await this.executeStep(step);

        result.steps.push({
          name: step.name,
          status: stepResult.success ? 'success' : 'failed',
          duration: stepResult.duration,
          output: stepResult.output
        });

        if (!stepResult.success) {
          throw new Error(`Step failed: ${step.name}`);
        }
      }

      result.success = true;
    } catch (error) {
      result.error = (error as Error).message;

      // Execute rollback
      for (const step of runbook.rollback) {
        await this.executeStep(step);
      }
    }

    result.endTime = new Date();
    return result;
  }

  private async executeStep(step: RunbookStep): Promise<StepResult> {
    const startTime = Date.now();

    try {
      let output = '';

      switch (step.type) {
        case 'check':
          output = await this.executeCheck(step.action!);
          break;
        case 'action':
          output = await this.executeAction(step.action!);
          break;
        case 'notify':
          output = await this.executeNotification(step.action!);
          break;
        case 'wait':
          await this.sleep((step.timeout || 60) * 1000);
          output = 'Waited ' + (step.timeout || 60) + 's';
          break;
      }

      return {
        success: true,
        duration: Date.now() - startTime,
        output
      };
    } catch (error) {
      return {
        success: false,
        duration: Date.now() - startTime,
        output: (error as Error).message
      };
    }
  }

  private async executeCheck(action: string): Promise<string> {
    // Implementation
    return '';
  }

  private async executeAction(action: string): Promise<string> {
    // Implementation
    return '';
  }

  private async executeNotification(action: string): Promise<string> {
    // Implementation
    return '';
  }

  private sleep(ms: number): Promise<void> {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

interface RunbookResult {
  runbookId: string;
  incidentId: string;
  startTime: Date;
  endTime?: Date;
  steps: StepResult[];
  success: boolean;
  error: string | null;
}

interface StepResult {
  name?: string;
  status?: string;
  success: boolean;
  duration: number;
  output: string;
}
```

## 6. Post-Incident Management

### 6.1 Post-Mortem Process

```typescript
/**
 * Post-mortem and root cause analysis
 */

interface PostMortem {
  incidentId: string;
  title: string;
  severity: string;
  startTime: Date;
  endTime: Date;
  duration: number; // minutes
  impact: IncidentImpact;
  timeline: TimelineEvent[];
  rootCauses: RootCause[];
  actionItems: ActionItem[];
  lessons: string[];
}

interface IncidentImpact {
  affectedUsers: number;
  affectedServices: string[];
  downtime: number; // seconds
  dataLoss: boolean;
  revenue: number; // estimated loss
}

interface TimelineEvent {
  timestamp: Date;
  event: string;
  actor: string;
}

interface RootCause {
  cause: string;
  category: string;
  preventable: boolean;
}

interface ActionItem {
  title: string;
  owner: string;
  dueDate: Date;
  priority: 'high' | 'medium' | 'low';
  status: 'open' | 'in-progress' | 'done';
}

class PostMortemManager {
  /**
   * Generate post-mortem report
   */
  generatePostMortem(incident: Incident, events: TimelineEvent[]): PostMortem {
    const duration = (incident.resolvedAt!.getTime() - incident.createdAt.getTime()) / 60000;

    const postMortem: PostMortem = {
      incidentId: incident.id,
      title: incident.title,
      severity: incident.severity,
      startTime: incident.createdAt,
      endTime: incident.resolvedAt!,
      duration,
      impact: this.calculateImpact(incident),
      timeline: events,
      rootCauses: [],
      actionItems: [],
      lessons: []
    };

    return postMortem;
  }

  /**
   * Identify root causes
   */
  identifyRootCauses(postMortem: PostMortem): RootCause[] {
    const causes: RootCause[] = [];

    // Analyze timeline for patterns
    for (let i = 0; i < postMortem.timeline.length; i++) {
      const event = postMortem.timeline[i];

      // Look for contributing factors
      if (event.event.includes('deployment')) {
        causes.push({
          cause: 'Recent deployment introduced regression',
          category: 'deployment',
          preventable: true
        });
      }

      if (event.event.includes('database')) {
        causes.push({
          cause: 'Database performance degradation',
          category: 'infrastructure',
          preventable: true
        });
      }
    }

    return causes;
  }

  /**
   * Generate action items
   */
  generateActionItems(postMortem: PostMortem): ActionItem[] {
    const items: ActionItem[] = [];

    for (const cause of postMortem.rootCauses) {
      if (cause.preventable) {
        items.push({
          title: `Implement prevention for: ${cause.cause}`,
          owner: 'TBD',
          dueDate: new Date(Date.now() + 7 * 24 * 60 * 60 * 1000), // 1 week
          priority: 'high',
          status: 'open'
        });
      }
    }

    return items;
  }

  private calculateImpact(incident: Incident): IncidentImpact {
    // Implementation: Calculate actual impact
    return {
      affectedUsers: 0,
      affectedServices: [],
      downtime: 0,
      dataLoss: false,
      revenue: 0
    };
  }
}
```

## 7. Recommended SLA Tiers

### 7.1 SLA Tier Comparison

```typescript
const recommendedSLATiers = {
  enterprise: {
    availability: '99.99%', // 52 minutes downtime per year
    responseTime: '100ms (p99)',
    criticalResponseTime: '5 minutes',
    criticalResolutionTime: '15 minutes',
    monthlyPrice: 10000,
    features: [
      'Dedicated support team',
      '24/7 on-call support',
      'Priority incident response',
      'Guaranteed SLA credits',
      'Quarterly business reviews',
      'Custom integrations'
    ]
  },

  premium: {
    availability: '99.9%', // 8.7 hours downtime per year
    responseTime: '200ms (p99)',
    criticalResponseTime: '15 minutes',
    criticalResolutionTime: '30 minutes',
    monthlyPrice: 5000,
    features: [
      'Business hours support',
      'On-call support',
      'Standard incident response',
      'SLA credits',
      'Monthly reports',
      'API access'
    ]
  },

  standard: {
    availability: '99.5%', // 21.6 hours downtime per year
    responseTime: '500ms (p99)',
    criticalResponseTime: '30 minutes',
    criticalResolutionTime: '60 minutes',
    monthlyPrice: 2000,
    features: [
      'Business hours support',
      'Email support',
      'Standard incident response',
      'Monthly reports',
      'Community forum'
    ]
  }
};
```

## 8. Implementation Checklist

- [ ] Set up Prometheus + AlertManager
- [ ] Configure Grafana dashboards
- [ ] Integrate with PagerDuty
- [ ] Set up Opsgenie backup
- [ ] Create alert rules
- [ ] Define escalation policies
- [ ] Write runbooks
- [ ] Set up automated remediation
- [ ] Configure SLA tracking
- [ ] Create post-mortem process
- [ ] Train on-call team
- [ ] Document procedures

## 9. Recommendations for Manus-like Platform

**Alerting Stack:**
- Primary: PagerDuty (incident management)
- Backup: Opsgenie (redundancy)
- Metrics: Prometheus (collection)
- Visualization: Grafana (dashboards)
- Custom: Webhooks to internal systems

**SLA Tiers:**
- **Enterprise**: 99.99% availability, $10k/month
- **Premium**: 99.9% availability, $5k/month
- **Standard**: 99.5% availability, $2k/month

**On-Call Rotation:**
- Primary: 24/7 coverage
- Secondary: Backup coverage
- Escalation: Team lead after 5 minutes
- Executive: After 10-15 minutes

This comprehensive incident response system ensures enterprise-grade reliability and customer satisfaction!