Phase 2B.1: Monitoring & Alerting Blueprint

Purpose: Real-time system health monitoring and automated incident response
Scope: Testnet E2E through mainnet production
Update Frequency: Every 60 seconds

---

🎯 Monitoring Hierarchy

Tier 1: Critical System Invariants (HALT on violation)

These must NEVER be violated. Any violation halts the relayer immediately.

Critical Invariant Violations (Triggers Relayer HALT)
├─ SUPPLY_MISMATCH
│  ├─ Trigger: locked_db != chain_locked (difference > 0.001 SIGIL)
│  ├─ Action: Stop all operations, alert on-call
│  └─ Log Level: CRITICAL
│
├─ DUPLICATE_BATCH
│  ├─ Trigger: UNIQUE(merkle_root) constraint violation
│  ├─ Action: Rollback operation, investigate indexer
│  └─ Log Level: CRITICAL
│
├─ PROOF_VERIFICATION_FAIL
│  ├─ Trigger: merkle proof rejects valid withdrawal
│  ├─ Action: Investigate merkle logic, halt settlement
│  └─ Log Level: CRITICAL
│
└─ NONCE_DESYNC
   ├─ Trigger: expected_nonce != actual_nonce on chain
   ├─ Action: Rescan chain, realign nonce state
   └─ Log Level: CRITICAL

Tier 2: Operational Health (Alert but don't halt)

These indicate problems that need attention but don't require immediate stop.

Operational Health Alerts (Alert, don't halt)
├─ RPC_TIMEOUT
│  ├─ Trigger: 3+ consecutive RPC call failures
│  ├─ Action: Switch RPC endpoint, retry with backoff
│  └─ Log Level: ERROR
│
├─ DB_CONNECTION_LOST
│  ├─ Trigger: 5+ database connection failures
│  ├─ Action: Check DB health, restart connection pool
│  └─ Log Level: ERROR
│
├─ HIGH_LATENCY
│  ├─ Trigger: Cycle time >2s (target <1s)
│  ├─ Action: Investigate bottleneck, adjust batch size
│  └─ Log Level: WARNING
│
├─ GAS_ESTIMATE_MISS
│  ├─ Trigger: Actual gas > estimate * 1.5
│  ├─ Action: Increase gas limit buffer, log for analysis
│  └─ Log Level: WARNING
│
└─ QUEUE_BUILDUP
   ├─ Trigger: Pending operations > 10
   ├─ Action: Increase batch frequency or size
   └─ Log Level: WARNING

Tier 3: Operational Metrics (Track but don't alert)

These are tracked for trends and long-term optimization.

Operational Metrics (Tracked, no alert threshold)
├─ Settlement Latency
│  ├─ Metric: Time from batch creation to anchor confirmation
│  └─ Target: <5 minutes
│
├─ Proof Verification Success Rate
│  ├─ Metric: successful_verifications / total_attempts
│  └─ Target: >99.9%
│
├─ Duplicate Event Rate
│  ├─ Metric: duplicate_events / total_events
│  └─ Target: 0%
│
├─ RPC Success Rate
│  ├─ Metric: successful_calls / total_calls
│  └─ Target: >99%
│
└─ Database Performance
   ├─ Metric: Query time percentiles (p50, p95, p99)
   └─ Target: p95 <100ms

---

📊 Dashboard Specification

Dashboard 1: System Health (Main)

Refresh: Every 60 seconds
Audience: Operations team, monitoring

┌─────────────────────────────────────────────────────────┐
│ SIGILBRIDGE RELAYER - SYSTEM STATUS                    │
├─────────────────────────────────────────────────────────┤
│                                                         │
│ ┌────────────────────────┐  ┌──────────────────────┐  │
│ │ SUPPLY INVARIANT       │  │ RELAYER STATUS       │  │
│ ├────────────────────────┤  ├──────────────────────┤  │
│ │ DB Total:   1000.00 ✅ │  │ Status: RUNNING ✅   │  │
│ │ Chain Total: 1000.00 ✅│  │ Uptime: 48h 23m      │  │
│ │ Delta:         0.00 ✅ │  │ Cycles: 2,847 ✅     │  │
│ │ Status: HEALTHY        │  │ Errors: 0 ✅         │  │
│ └────────────────────────┘  └──────────────────────┘  │
│                                                         │
│ ┌────────────────────────┐  ┌──────────────────────┐  │
│ │ SETTLEMENT PROGRESS    │  │ NETWORK STATUS       │  │
│ ├────────────────────────┤  ├──────────────────────┤  │
│ │ Deposits Indexed: 1,234│  │ Execution Chain: OK  │  │
│ │ Batches Created:   847 │  │ Cronos RPC: OK       │  │
│ │ Batches Anchored:  847 │  │ DB Connection: OK    │  │
│ │ Success Rate:    99.8% │  │ All Systems: SYNCED  │  │
│ └────────────────────────┘  └──────────────────────┘  │
│                                                         │
│ Last Update: 2026-02-09 14:32:15 UTC                   │
└─────────────────────────────────────────────────────────┘

Dashboard 2: Detailed Metrics

┌─────────────────────────────────────────────────────────┐
│ DETAILED PERFORMANCE METRICS                           │
├─────────────────────────────────────────────────────────┤
│                                                         │
│ INDEX CYCLE (target: <100ms)                           │
│ ████████░░ 89ms                                    ✅  │
│                                                         │
│ BATCH CREATION (target: <50ms)                         │
│ ████░░░░░░ 38ms                                    ✅  │
│                                                         │
│ EXECUTION LATENCY (target: <500ms)                     │
│ ████████░░ 425ms                                   ✅  │
│                                                         │
│ CONFIRMATION DEPTH (target: ≥10 blocks)                │
│ ████████████ 12 blocks                             ✅  │
│                                                         │
│ PROOF VERIFICATION (success rate)                      │
│ ██████████ 100.0%                                  ✅  │
│                                                         │
│ RPC SUCCESS RATE (target: >99%)                        │
│ ██████████ 99.7%                                   ✅  │
│                                                         │
└─────────────────────────────────────────────────────────┘

Dashboard 3: Alerts & Events

┌─────────────────────────────────────────────────────────┐
│ ACTIVE ALERTS & RECENT EVENTS                          │
├─────────────────────────────────────────────────────────┤
│                                                         │
│ CRITICAL ALERTS: 0                                    ✅│
│ ERROR ALERTS: 0                                       ✅│
│ WARNING ALERTS: 0                                     ✅│
│                                                         │
│ Recent Events:                                         │
│ ─────────────────────────────────────────────         │
│ 14:30:02  ✅ Batch #847 anchored (root: 0xa3f...)    │
│ 14:29:45  ✅ Invariant check passed (delta: 0.00)    │
│ 14:28:32  ✅ 12 deposits indexed, added to batches   │
│ 14:27:15  ✅ Proof verification successful (batch 846)│
│ 14:15:00  🔄 RPC retry #1 on execution chain        │
│                                                         │
└─────────────────────────────────────────────────────────┘

---

🚨 Alert Configuration

Alert 1: Supply Mismatch (CRITICAL)

Alert: SUPPLY_MISMATCH
Trigger: |
  locked_db_total != chain_locked_total
  AND abs(difference) > 0.001 SIGIL
Severity: CRITICAL
Action:
  - Halt relayer immediately
  - Page on-call engineer
  - Create incident ticket
  - Log full balance breakdown
  - Save DB snapshot for audit
Notification:
  - Email to ops@sigilbridge.dev
  - Slack to #incidents
  - PagerDuty trigger (critical)
  - SMS to on-call
Recovery:
  1. Investigate root cause
  2. Verify chain state (Web3 call)
  3. Check DB logs for corruption
  4. Manually correct data if needed
  5. Re-validate invariant
  6. Resume relayer

Alert 2: Duplicate Batch (CRITICAL)

Alert: DUPLICATE_BATCH_DETECTED
Trigger: |
  UNIQUE(merkle_root) constraint violation
  OR duplicate batch number in DB
Severity: CRITICAL
Action:
  - Halt batch creation immediately
  - Page on-call engineer
  - Investigate indexer logic
  - Check for event replay attack
  - Audit event processing logs
Notification:
  - Email to ops@sigilbridge.dev
  - Slack to #incidents (with logs)
  - PagerDuty critical
Recovery:
  1. Stop indexer immediately
  2. Review last 100 events
  3. Identify duplicate source
  4. Manually remove duplicate from DB (if safe)
  5. Restart indexer with manual event replay
  6. Verify no further duplicates

Alert 3: RPC Timeout (ERROR)

Alert: RPC_TIMEOUT
Trigger: |
  3 consecutive RPC call failures
  OR any single call timeout >10s
Severity: ERROR (not critical)
Action:
  - Log error with retry count
  - Switch to fallback RPC endpoint
  - Retry with exponential backoff
  - Alert ops team if persists >5min
Notification:
  - Log at ERROR level
  - Slack notification if >10 failures
  - Email if >1 hour of issues
Backoff Strategy:
  - Attempt 1: immediate
  - Attempt 2: wait 1s
  - Attempt 3: wait 2s
  - Attempt 4: wait 5s
  - Attempt 5: switch RPC + restart

Alert 4: Database Connection Lost (ERROR)

Alert: DB_CONNECTION_LOST
Trigger: |
  5+ consecutive database connection failures
Severity: ERROR
Action:
  - Halt all DB operations
  - Attempt connection pool restart
  - Alert ops team
  - Check PostgreSQL logs on server
Notification:
  - Log at ERROR level
  - Email to ops@sigilbridge.dev
  - Slack to #ops
Recovery:
  1. Check database server health
  2. Verify network connectivity
  3. Check PostgreSQL logs
  4. Restart connection pool
  5. Verify connectivity
  6. Resume relayer

Alert 5: High Latency (WARNING)

Alert: HIGH_LATENCY
Trigger: |
  Cycle time > 2 seconds
  (target: <1 second)
Severity: WARNING
Action:
  - Continue operations (don't halt)
  - Log warning with latency measurements
  - Alert ops for investigation
  - Check for bottlenecks
Notification:
  - Log at WARNING level
  - Slack notification if >5min sustained
Investigation:
  1. Check RPC latency (should be <500ms)
  2. Check database query times
  3. Check batch size (might be too large)
  4. Check confirmation depth setting
  5. Analyze merkle tree generation time

---

📈 Monitoring Infrastructure

Logging Level Configuration

Logging Levels:
  
  CRITICAL: System invariants violated
    - Supply mismatch detected
    - Duplicate batch created
    - Nonce desynchronized
    - Proof verification failed
    
  ERROR: Operational failures
    - RPC connection lost
    - Database error
    - Transaction reverted
    - Proof generation failed
    
  WARNING: Degraded performance
    - High latency
    - Gas estimate miss
    - Queue buildup
    - Low success rate
    
  INFO: Normal operations
    - Batch created (batch_id, deposits_count)
    - Deposit indexed (deposit_id, amount)
    - Proof verified (withdrawal_id, valid=True)
    - Cycle completed (duration)
    
  DEBUG: Detailed traces
    - RPC call details (method, args, result)
    - DB query execution (sql, duration)
    - Merkle tree construction steps
    - Nonce management state

Log Storage & Retention

Log Destinations:
├─ Console: All levels (for development)
├─ File: /var/log/sigilbridge/relayer.log (rotate daily)
│  ├─ Retention: 30 days
│  └─ Size: Max 500MB per file
├─ Database: CRITICAL/ERROR only (for audit)
│  └─ Retention: 90 days
└─ Centralized Logging: ELK or similar
   ├─ All levels
   ├─ Real-time indexing
   └─ 1-year retention for compliance

Metrics Collection

# Metrics to track (export to Prometheus/CloudWatch)

metrics = {
    # Counter: Total operations by type
    "relayer_deposits_indexed_total": 1234,
    "relayer_batches_created_total": 847,
    "relayer_batches_anchored_total": 847,
    "relayer_withdrawals_claimed_total": 234,
    
    # Gauge: Current state
    "relayer_pending_deposits": 12,
    "relayer_pending_batches": 0,
    "relayer_db_locked_sigil": 1000.00,
    "relayer_chain_locked_sigil": 1000.00,
    "relayer_invariant_delta": 0.00,
    
    # Histogram: Latency measurements
    "relayer_index_cycle_seconds": [0.045, 0.089, 0.052, ...],
    "relayer_batch_creation_seconds": [0.038, 0.041, 0.039, ...],
    "relayer_settlement_latency_seconds": [0.425, 0.389, 0.412, ...],
    "relayer_proof_verification_seconds": [0.012, 0.011, 0.013, ...],
    
    # Rate: Operations per minute
    "relayer_index_rate": 234.5,  # deposits per minute
    "relayer_batch_rate": 12.3,   # batches per minute
    "relayer_settlement_rate": 12.1,  # settlements per minute
}

---

🚑 Incident Response Runbook

Incident: Supply Mismatch Detected

Severity: CRITICAL
Alert: Supply invariant violated (db_total != chain_total)
On-Call: Page immediately

Timeline:

• T+0m: Alert fires, relayer halts
• T+1m: On-call acknowledges
• T+2m: Begin investigation
• T+5m: Root cause identified
• T+10m: Decision to rollback or fix forward
• T+20m: Issue mitigated
• T+30m: Post-mortem meeting scheduled

Investigation Steps:

1. Verify Alert (T+1m)

   SELECT SUM(amount) FROM deposit_confirmed = 1000.00
   SELECT SUM(amount) FROM withdrawal_confirmed = 234.50
   # Calculate expected: locked_db = 1000.00 - 234.50 = 765.50
   
   # Query chain
   SIGIL.balanceOf(cronos_lockbox) = ?
   ExecutionSigilBridged.totalSupply() = ?
   # Calculate actual: locked_chain = cronos_lockbox + execution_bridged
   

2. Determine Direction (T+3m)

   If locked_db > locked_chain:
     SIGIL was LOST (worse scenario)
     Action: Audit settlement transactions
     
   If locked_db < locked_chain:
     SIGIL was CREATED (inflation)
     Action: Check minting logic
   

3. Audit Transactions (T+5m)

   -- Check recent batch anchoring
   SELECT * FROM settlement_batch 
   WHERE status='anchored' 
   ORDER BY created_at DESC LIMIT 10;
   
   -- Check proof executions
   SELECT * FROM proof_executions 
   ORDER BY created_at DESC LIMIT 10;
   
   -- Check where SIGIL went
   SELECT * FROM relayer_operations 
   ORDER BY created_at DESC LIMIT 20;
   

4. Fix Forward or Rollback (T+10m)

   • Option A (Fix Forward): If issue identified in code, deploy fix
   • Option B (Rollback): If transaction has bad effects, manually reverse

5. Validate (T+20m)

   ./scripts/validate_invariant.py
   # Should show: INVARIANT VALID
   

6. Resume (T+25m)

   relayer resume --from-last-checkpoint
   

---

📞 Escalation Policy

Severity	Response Time	Escalation
CRITICAL	<5 min	Page on-call immediately
ERROR	<15 min	Email + Slack
WARNING	<60 min	Slack notification
INFO	<24h	Daily report

---

✅ Monitoring Readiness Checklist

• All alerts configured and tested
• Dashboard deployed and accessible
• Logging pipeline operational
• Metrics exported to monitoring system
• Runbook accessible to on-call
• Team trained on incident response
• PagerDuty/alerting tool configured
• Backup RPC endpoints configured
• Database backups automated
• 24/7 monitoring active

---

Document Created: February 9, 2026
System Status: Ready for production monitoring
Next Phase: Deploy and test monitoring on testnet