Sprint 13: Security Hardening & Production Readiness

Previous Sprint: SPRINT12.md - Operator Fix & Production Readiness

---

🚨 CRITICAL: Pre-commit E2E Tests Blocking - Fix Before Any Other Work

Context

The pre-commit hook now runs FULL E2E tests on every commit (no flags, no skipping). This is by design - security-critical code should not be committed without full validation. However, the E2E tests are currently failing, which blocks all commits.

Current State:

• Pre-commit runs: formatting ✅, clippy ✅, gitleaks ✅, E2E tests ❌
• E2E infrastructure starts correctly (Anvil, LocalTerra, PostgreSQL)
• EVM contract deployment works
• Terra contract deployment works
• Basic connectivity tests pass

Failing Tests (must fix to unblock commits)

1. Terra Bridge Configuration Query Fails

[FAIL] TEST: Terra Bridge Configuration
[WARN] Could not query Terra bridge config

• The config {} query to the Terra bridge contract returns empty
• Need to verify the contract was instantiated correctly
• May be a CosmWasm query format issue

2. EVM → Terra Deposit Transaction Reverts

[INFO] Step 3: Executing deposit on router...
[ERROR] Deposit transaction failed

• Token approval succeeds
• Deposit call to router reverts
• Likely causes:
  • Token not registered on bridge
  • Chain not configured
  • LockUnlock adapter not set up correctly

Files Changed (Uncommitted)

These changes are staged but can't be committed until E2E passes:

File	Change
.githooks/pre-commit	Runs FULL E2E tests with make e2e-test (no flags)
scripts/e2e-test.sh	Fixed SCRIPTS_DIR path issue for operator-ctl.sh/canceler-ctl.sh
scripts/e2e-helpers/common.sh	Fixed balance parsing (strip [1e12] formatting from cast output)

How to Debug

# Start infrastructure manually
./scripts/e2e-setup.sh

# Source environment
source .env.e2e

# Test Terra bridge config query
docker exec -it cl8y-bridge-monorepo-terrad-cli-1 terrad query wasm contract-state smart \
  $TERRA_BRIDGE_ADDRESS '{"config":{}}' --node http://localterra:26657

# Test EVM deposit manually
cast send $EVM_ROUTER_ADDRESS "deposit(address,uint256,bytes32,bytes32)" \
  $TEST_TOKEN 1000000 $TERRA_CHAIN_KEY $TERRA_DEST_ACCOUNT \
  --rpc-url $EVM_RPC_URL --private-key $EVM_PRIVATE_KEY

# Check if token is registered
cast call $TOKEN_REGISTRY_ADDRESS "isTokenSupported(address)" $TEST_TOKEN \
  --rpc-url $EVM_RPC_URL

# Teardown when done
./scripts/e2e-teardown.sh

Priority

This is the #1 priority. All other Sprint 13 work is blocked until E2E tests pass.

The security enforcement is working correctly - it's blocking commits because tests fail. Fix the underlying issues, don't bypass the checks.

---

Sprint 12 Retrospective

What Went Right

1. E2E Canceler Integration Tests Implemented:

   • test_canceler_fraudulent_detection() now creates actual fraudulent approvals
   • test_canceler_cancel_flow() submits real cancel transactions
   • test_canceler_withdrawal_fails() verifies ApprovalCancelled revert
   • Tests use environment variables for fraud details between test stages

2. CI/CD Pipeline Established:

   • .github/workflows/e2e.yml runs full E2E suite on PRs and pushes to main
   • .github/workflows/test.yml runs unit tests for all packages (EVM, Operator, Canceler, Terra)
   • Workflow supports manual trigger with Terra tests option
   • Artifacts uploaded on failure for debugging

3. Security-First Test Design:

   • All security tests run by default (no opt-in required)
   • --no-* flags show explicit warnings about reduced coverage
   • Operator and canceler started automatically in E2E

4. Operator Working:

   • Tendermint/cosmrs version compatibility fixed
   • Operator connects to both EVM and Terra
   • Block processing working on both chains

What Went Wrong

1. E2E Tests Are Still "Best Effort":

   • Tests pass even when OPERATOR_ROLE or CANCELER_ROLE not granted
   • The test verifies the mechanism works but doesn't verify actual canceler daemon
   • Manual cast send calls replace automated canceler detection

2. CI Workflow May Have Issues:

   • dtolnay/rust-action@stable might not be the correct action name
   • Should verify workflow runs successfully before merge

3. Canceler Not Actually Started in E2E:

   • The E2E tests manually submit cancel transactions
   • The canceler daemon itself is not started and verified to detect/cancel autonomously
   • This means we're testing the contract mechanism, not the canceler service

4. No Integration With Running Canceler:

   • ./scripts/canceler-ctl.sh start is mentioned but never called in E2E
   • Canceler logs not verified for fraud detection messages

---

Security Gap Analysis

Critical Security Gaps (Bridges Are High-Value Targets)

Gap	Severity	Risk	Mitigation Required
Canceler not tested as running service	🔴 Critical	Canceler may have bugs that aren't caught	Add E2E test that starts canceler and verifies detection
Single canceler instance	🔴 Critical	SPOF for fraud detection	Deploy multiple independent cancelers
No canceler health monitoring	🔴 Critical	Canceler failure goes unnoticed	Add /health endpoint and monitoring
Terra VerifyDeposit query not tested	🟡 High	Terra→EVM cancellation may not work	Add Terra source verification test
No rate limiting on approvals	🟡 High	Operator could spam approvals	Add per-block approval limits
Private keys in env vars	🟡 High	Key exposure in logs/env	Use KMS/HSM for production

Testing Gaps

Gap	Type	Impact
Canceler daemon E2E	Integration	Canceler service bugs undetected
Multi-chain verification	Integration	BSC/opBNB specific issues missed
Reeneable flow	Functional	Admin recovery path untested
Rate limiting guards	Security	TokenRateLimit bypass possible
RPC failure handling	Resilience	Network outages could cause missed cancellations
Concurrent approvals	Load	Race conditions untested
Hash computation fuzzing	Security	Edge cases in hash computation
Frontend E2E	Integration	User-facing flow untested

Operational Gaps

Gap	Risk	Priority
No canceler health endpoint	Undetected failures	🔴 Critical
No time-to-cancel metrics	Unknown detection latency	🟡 High
No testnet deployment	Production issues discovered too late	🟡 High
Canceler failure runbook missing	Slow incident response	🟡 High
No backup canceler nodes	Single point of failure	🔴 Critical

---

Sprint 13 Objectives

Priority 0: Verify CI/CD Works

Before proceeding, verify the CI pipeline actually runs.

# 1. Create a test branch
git checkout -b test/verify-ci

# 2. Push and check GitHub Actions
git push origin test/verify-ci

# 3. Monitor workflow runs
# - e2e.yml should trigger
# - test.yml should trigger
# - Both should pass

# 4. Fix any issues (likely rust-action name)

Acceptance Criteria:

• CI workflow triggers on push
• All jobs pass (contracts-evm, operator, canceler, contracts-terra, e2e-evm)
• Failures block merge (branch protection configured)

Priority 1: Real Canceler E2E Test

The current E2E tests verify cancel works, but don't verify the canceler service detects fraud.

1.1 Add Canceler Daemon E2E Test

# New test function: test_canceler_autonomous_detection()
# 1. Start canceler daemon (./scripts/canceler-ctl.sh start)
# 2. Create fraudulent approval
# 3. Wait for canceler to detect (poll logs)
# 4. Verify cancel transaction submitted automatically
# 5. Verify approval marked as cancelled
# 6. Stop canceler

1.2 Add Canceler Health Check

The canceler should expose a /health endpoint for monitoring.

// packages/canceler/src/api.rs (new file)
async fn health_check() -> impl Responder {
    let stats = watcher.get_stats();
    HttpResponse::Ok().json({
        "status": "healthy",
        "verified_count": stats.verified,
        "cancelled_count": stats.cancelled,
        "last_block_evm": stats.last_evm_block,
        "last_block_terra": stats.last_terra_block,
    })
}

Acceptance Criteria:

• Canceler daemon starts and is monitored in E2E
• E2E test creates fraud, canceler detects within 30s
• E2E test verifies cancel tx submitted by canceler (not manual)
• Canceler exposes /health endpoint

Priority 2: Multi-Canceler Network

For production security, need multiple independent cancelers.

2.1 Document Multi-Canceler Setup

# Minimum Viable Canceler Network

| Environment | Cancelers Required | Locations |
|-------------|-------------------|-----------|
| Local/Dev   | 1                 | Same machine |
| Testnet     | 2                 | Different regions |
| Mainnet     | 3+                | Different operators |

## Independence Requirements

Each canceler should have:
- Independent RPC endpoints (not same provider)
- Independent hosting (not same cloud region)
- Independent operator (not same organization)

2.2 Add Canceler Instance ID

The canceler should identify itself in logs for multi-instance debugging.

// packages/canceler/src/config.rs
pub canceler_id: String, // e.g., "canceler-us-east-1"

Acceptance Criteria:

• Canceler has configurable instance ID
• Multiple cancelers can run simultaneously
• Documentation for multi-canceler deployment

Priority 3: Resilience Testing

Test failure scenarios that could allow fraud.

3.1 RPC Failure Test

# Test: What happens when Terra LCD is down during verification?
# Expected: Approval stays Pending, verified on retry
# Not: Approval passes as Valid

3.2 Delayed Detection Test

# Test: Create approval, delay canceler start by N seconds
# Expected: Canceler still cancels within remaining delay window
# Risk: If delay < detection time, fraud succeeds

3.3 Concurrent Approval Test

# Test: Create 10 fraudulent approvals rapidly
# Expected: All 10 cancelled
# Risk: Race conditions, gas exhaustion

Acceptance Criteria:

• RPC failure doesn't cause false Valid
• Delayed start still cancels if within window
• Multiple approvals all cancelled

Priority 4: Production Preparation

4.1 Testnet Deployment

Deploy to BSC testnet for real-world validation.

# 1. Deploy EVM contracts to BSC testnet
./scripts/deploy-evm-testnet.sh bsc

# 2. Deploy Terra contracts to pisco testnet
./scripts/deploy-terra-testnet.sh

# 3. Run operator against testnet
# 4. Run canceler against testnet
# 5. Execute test transfer

4.2 Monitoring Stack

Add Prometheus metrics for:

• canceler_approvals_verified_total (counter)
• canceler_approvals_cancelled_total (counter)
• canceler_verification_latency_seconds (histogram)
• canceler_last_block_processed (gauge per chain)

4.3 Alerting Rules

# Example Prometheus alert
- alert: CancelerNotProcessingBlocks
  expr: rate(canceler_last_block_processed[5m]) == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Canceler not processing blocks"

Acceptance Criteria:

• Testnet deployment working
• Prometheus metrics exposed
• Alert rules defined for critical failures

---

Technical Debt

Code Quality Issues

Issue	Location	Priority
Unused code warnings in operator	cargo build	Low
Unused code warnings in canceler	cargo build	Low
TODO comments in test files	Various	Low
Missing error handling in some paths	verifier.rs	Medium

Architecture Improvements

Improvement	Benefit	Effort
Operator as Docker container	Better lifecycle management	Medium
Canceler as Docker container	Better lifecycle management	Medium
Shared Rust library for hash computation	Single source of truth	High
GraphQL API for bridge status	Better frontend integration	High

---

Risk Assessment

Risk	Likelihood	Impact	Mitigation
CI workflow doesn't work	High	Medium	Verify before merge
Canceler has undiscovered bugs	Medium	Critical	More E2E coverage
RPC outages cause missed cancellation	Medium	Critical	Multi-provider, retries
Single canceler compromised	Low	Critical	Multi-canceler network
Testnet differs from mainnet	Low	Medium	Use mainnet forks
Gas price spikes block cancellation	Medium	High	Gas buffering, priority

---

Questions for Next Agent

1. Should the CI workflow use the correct action name? The dtolnay/rust-action@stable may not exist. Check if it should be dtolnay/rust-toolchain@stable or actions-rust-lang/setup-rust-toolchain.

2. What's the minimum viable canceler network for mainnet launch? Currently single canceler. Need to define decentralization requirements.

3. Should canceler run as Docker service in docker-compose? Currently runs as background process. Docker would provide better lifecycle management.

4. How should canceler keys be managed in production? Currently env vars. Need HSM/KMS for real security.

5. What's the acceptable detection latency? Current target is "within delay window" (~60s local, ~300s prod). Need SLA definition.

---

Quick Start for Sprint 13

Step 1: Verify CI Works

# Push a small change and watch GitHub Actions
git push origin main

# Check: https://github.com/[repo]/actions
# Both e2e.yml and test.yml should run

Step 2: Fix CI If Needed

# If dtolnay/rust-action doesn't exist, change to:
- uses: dtolnay/rust-toolchain@stable
  with:
    toolchain: stable
    components: rustfmt, clippy

Step 3: Add Canceler Daemon E2E

# Update e2e-test.sh to include:
test_canceler_autonomous_detection() {
    log_step "=== TEST: Canceler Autonomous Detection ==="
    
    # Start canceler
    ./scripts/canceler-ctl.sh start 1
    sleep 5
    
    # Create fraudulent approval
    # ... (already exists in test_canceler_fraudulent_detection)
    
    # Wait for canceler to detect (poll logs)
    for i in {1..30}; do
        if grep -qi "cancellation submitted" .canceler.1.log; then
            log_pass "Canceler detected and cancelled fraud"
            break
        fi
        sleep 2
    done
    
    # Verify approval cancelled
    # ...
    
    # Stop canceler
    ./scripts/canceler-ctl.sh stop 1
}

Step 4: Add Canceler Health Endpoint

// packages/canceler/src/main.rs
// Add axum server for health endpoint

---

Definition of Done for Sprint 13

CI/CD Verification

• e2e.yml workflow runs and passes
• test.yml workflow runs and passes
• Branch protection requires passing CI

Canceler E2E

• Canceler daemon started in E2E tests
• Fraudulent approval detected by daemon (not manual cancel)
• Canceler /health endpoint implemented
• Health endpoint tested in E2E

Resilience

• RPC failure doesn't cause false Valid
• Multiple concurrent approvals all cancelled
• Delayed canceler start still works

Production Prep

• Testnet deployment documented
• Monitoring metrics implemented
• Alert rules defined

---

Appendix: File Changes in Sprint 12

Modified Files

File	Changes
scripts/e2e-test.sh	+302 lines - Added real canceler E2E tests
.github/workflows/e2e.yml	Rewrote - Full E2E pipeline with operator/canceler
.github/workflows/test.yml	+158 lines - Added Rust package tests
SPRINT12.md	Updated completion markers

Files Created

None (all modifications to existing files)

Files That Should Be Created in Sprint 13

File	Purpose
packages/canceler/src/api.rs	Health endpoint
docs/runbook-canceler-operations.md	Ops runbook
monitoring/alerts.yml	Prometheus alert rules
docker-compose.canceler.yml	Multi-canceler local setup

---

Created: 2026-02-03 Previous Sprint: SPRINT12.md - Operator Fix & Production Readiness