Aquaculture Platform - SLO/SLI Definitions

# Aquaculture Platform - SLO/SLI Definitions

> **Owner:** Platform Team
> **Effective Date:** 2026-03-14
> **Review Cadence:** Quarterly

## Overview

This document defines the Service Level Objectives (SLOs) and Service Level Indicators (SLIs)
for the Aquaculture Platform. These targets establish the reliability contract between the
platform team and its users. All SLOs are enforced via Prometheus alert rules defined in
`infrastructure/monitoring/prometheus/alerts/slo-alerts.yml`.

---

## SLI / SLO Definitions

| # | SLI | Measurement | SLO Target | Window | Severity on Breach |
|---|-----|-------------|------------|--------|--------------------|
| 1 | Gateway availability | `up{app="gateway-api"}` ratio | >= 99.9% | 30 days (rolling) | critical |
| 2 | API latency p95 | `histogram_quantile(0.95, http_request_duration_seconds_bucket)` | < 500 ms | 30 days (rolling) | warning |
| 3 | API latency p99 | `histogram_quantile(0.99, http_request_duration_seconds_bucket)` | < 2000 ms | 30 days (rolling) | critical |
| 4 | Error rate (5xx) | `rate(http_requests_total{status_code=~"5.."}[...]) / rate(http_requests_total[...])` | < 0.1% | 30 days (rolling) | critical |
| 5 | Sensor data freshness | `time() - max(sensor_reading_timestamp)` | < 60 s lag | 30 days (rolling) | warning |
| 6 | Login success rate | `rate(auth_login_success_total) / rate(auth_login_attempts_total)` | >= 99.5% | 7 days (rolling) | warning |
| 7 | Webhook processing latency | `histogram_quantile(0.95, webhook_processing_duration_seconds_bucket)` | < 5 s | 30 days (rolling) | warning |

---

## Error Budget

### What is an error budget?

An error budget is the maximum amount of unreliability the platform can tolerate within a
given window before corrective action is required.

### Budget calculations (30-day window)

| SLO | Error Budget (per 30 days) | Equivalent |
|-----|---------------------------|------------|
| 99.9% availability | 0.1% of 43,200 min = **43.2 minutes** downtime | ~2.6 seconds/hour |
| 99.5% login success (7d) | 0.5% of all login attempts over 7 days | Varies with traffic |
| < 0.1% error rate | Up to 0.1% of all requests may be 5xx | Varies with traffic |

### Burn rate alerts

Burn rate measures how fast the error budget is being consumed relative to the budget window.
If the current error rate is constant and would exhaust the budget faster than expected,
we trigger alerts at two thresholds:

| Alert | Burn Rate | Window | Meaning |
|-------|-----------|--------|---------|
| `SloErrorBudgetFastBurn` | 14.4x | 1 hour (short) + 5 min (fast) | Budget will exhaust in ~2 days at current rate. **Page immediately.** |
| `SloErrorBudgetSlowBurn` | 6x | 6 hours (short) + 30 min (fast) | Budget will exhaust in ~5 days at current rate. **Create ticket.** |

The multi-window approach (long + short window) reduces false positives by requiring
sustained elevated error rates in both windows before firing.

---

## Metric Dependencies

The following custom metrics **must** be exported by the respective services for these SLOs
to be measurable:

| Metric | Type | Exported By | Labels |
|--------|------|-------------|--------|
| `http_request_duration_seconds` | Histogram | All HTTP services (via NestJS prom-client) | `method`, `route`, `status` |
| `http_requests_total` | Counter | All HTTP services | `method`, `route`, `status_code` |
| `sensor_reading_timestamp` | Gauge | sensor-service | `sensor_type` |
| `auth_login_attempts_total` | Counter | auth-service | `result` (success/failure) |
| `auth_login_success_total` | Counter | auth-service | - |
| `webhook_processing_duration_seconds` | Histogram | gateway-api / alert-service | `webhook_type` |

> **Note:** If a metric is not yet instrumented, the corresponding SLO alert will not fire
> (PromQL returns empty on missing series). Implementing these metrics is tracked in the
> platform backlog.

---

## Escalation Policy

| Severity | Action | Response Time | Example |
|----------|--------|---------------|---------|
| `critical` | Page on-call engineer (PagerDuty) | < 15 min acknowledgement | Gateway down, budget fast-burn |
| `warning` | Create ticket (Jira/Linear) | < 4 hours triage | p95 latency breach, slow-burn |
| `info` | Dashboard visibility only | Next business day | Approaching threshold |

---

## Review Process

1. **Weekly:** Review error budget consumption in Grafana SLO dashboard.
2. **Monthly:** Assess if SLO targets are appropriate given observed traffic patterns.
3. **Quarterly:** Formal SLO review meeting. Adjust targets if needed. Update this document.
4. **Post-incident:** If an SLO is breached, conduct a blameless postmortem and determine
   if the SLO target or alerting thresholds need adjustment.

---

## Related Resources

- Alert rules: `infrastructure/monitoring/prometheus/alerts/slo-alerts.yml`
- Existing operational alerts: `infrastructure/monitoring/prometheus/aquaculture-rules.yaml`
- Prometheus values: `infrastructure/monitoring/prometheus/prometheus-values.yaml`
- Alertmanager routing: See `alertmanager.config.route` in prometheus-values.yaml