Stage and Readiness

Current stage

• Status: Production-ready MVP with Conditional GO (see release_assessment.md).
• Done: Estimator, lead intake with captcha/referrals/export, slot/booking creation with deposit policy, Stripe webhook, email reminders/resend, admin metrics CSV, retention cleanup endpoints, worker portal with checklists/time tracking, photo uploads + admin review/feedback with signed-download redirects (R2/CF Images), SaaS auth + billing plans, rate limiting and CORS controls, structured logging with PII/token redaction and per-request request/org/user context.
• Done: Estimator, lead intake with captcha/referrals/export, slot/booking creation with deposit policy, Stripe webhook, email reminders/resend, admin metrics CSV, retention cleanup endpoints, worker portal with checklists/time tracking, photo uploads + admin review/feedback with signed-download redirects (R2/CF Images), SaaS auth + billing plans, rate limiting and CORS controls, outbox-based delivery with retries/DLQ and admin replay.
• Blocked/Risks: Operators must wire schedulers for cleanup/email/export/retention, configure Stripe/email/export credentials, and set CORS/proxy trust lists in production.
• Sprint 4–6: DONE – logging redacts PII and emits unhandled_exception with request IDs and identity context while clearing job/outbox log context; canonical storage keys use orders/{org}/{booking} prefixes across code/tests; X-Test-Org honored only in testing/dev; FastAPI lifespan attaches AppServices container with clean shutdown.
• Sprint A (Admin safety): DONE – admin/iam surfaces enforce IP allowlisting, an incident-mode read-only gate returning Problem+JSON responses, and org-scoped break-glass tokens (reason + TTL) with audit trails for emergency writes.
• Operator Productivity Pack (Sprints 20-22 + Release Hardening): DONE – work queues for pending photos, overdue invoices, unassigned bookings, and dead letter queue (outbox + export); enhanced global search v2 with weighted/ranked results (exact/prefix/contains), worker coverage, and context-aware quick actions (email/call/timeline links); unified timeline view for bookings/invoices aggregating audit logs, outbox events, payments, photo reviews, NPS responses, and support tickets with before/after diffs for critical state changes; all queues org-scoped with pagination and quick action buttons for common operator workflows. RELEASE-GRADE HARDENED: strict RBAC enforced (dispatch for photos/assignments, finance for invoices, admin for DLQ, viewer for timeline with PII masking); full org-scoping on all joins (Booking/Worker/Lead/Team); DLQ pagination at SQL level using UNION ALL for scalability; timeline queries use safe patterns (no LIKE %id%) with limits and structured prefix patterns; Worker field correctness fixed; PII masking utility for viewer role (emails/phones/sensitive text); comprehensive semantic tests for filters/counts/pagination/cross-org/RBAC/PII (35+ tests total including hardening suite).
• Updates: Operator hardening tests fixed for collection (Payment import) and tightened timeline action masking to avoid recipient leakage.
• Updates: Added a merge revision to reconcile parallel 0049 migrations (invoice tax snapshots + Stripe event metadata) so Alembic has a single head.
• Sprint A (Admin safety): DONE – admin/iam surfaces enforce IP allowlisting and an incident-mode read-only gate returning Problem+JSON responses.
• Sprint B (Admin safety - idempotency): DONE – dangerous admin actions (payments, resends/replays, bulk updates, IAM resets) enforce org-scoped Idempotency-Key headers with persistent dedupe records, per-action rate limits, and audit logging for replayed responses.
• Sprint D (Admin safety - MFA): DONE – SaaS OWNER/ADMIN users can enroll/verify/disable RFC6238 TOTP (/v1/auth/2fa/*), sessions carry mfa_verified, and /v1/admin/* + /v1/iam/* reject SaaS admin tokens without MFA when ADMIN_MFA_REQUIRED=true. Legacy Basic Auth remains available but is not MFA-capable.
• Finance Recon A: DONE – read-only invoice reconciliation views enumerate mismatches between invoice status and succeeded payments for FINANCE roles, scoped per org.
• Finance Recon D: SHIPPED – tax snapshots (taxable subtotal, tax cents, effective rate) are persisted on invoices, GST/P&L reports read the stored snapshots, and migrations backfill existing invoices for historical correctness.
• Finance Recon C: DONE – finance-only reconcile action recalculates succeeded payments/outstanding balances, safely reopens unfunded paid invoices, audits before/after status + cents, and is idempotent across repeated calls.
• Finance Recon B: DONE – FINANCE-only Stripe event reconciliation view listing org-scoped webhook activity with pagination and metadata (event type, invoice/booking references, status, error reason).
• P1-BILL: SHIPPED – SaaS billing/usage reporting returns per-org plan limits, computed usage, overage flags, and drift detection between stored counters and ground truth; access limited to finance/admin/owner roles.
• Next milestones: Expand DLQ self-healing (after replay endpoint) and wire dashboarding for job error counters/storage janitor retries. Admin productivity Sprints 11–15 shipped (global search, scheduling controls, time tracking surface area, messaging previews/resend, safe CSV + bulk actions). Sprints 16–19 deliver client self-service (bookings/invoices/photos), subscription pause/resume with reasons, NPS ticket filters, guarded feature-flag/config viewers, and unified Problem+JSON error handling with consistent request IDs.
• Lifecycle/DI: FastAPI lifespan now wires a typed AppServices container (storage, email, Stripe, rate limiting, metrics) onto app.state.services with shutdown cleanup to reduce global-state bugs; deprecated @app.on_event hooks removed.
• Sprint 1 (Security baseline): DONE – admin middleware reordered to isolate /v1/admin/*, org-scoped finance/report/export/payment endpoints, and regression tests for cross-org leakage.
• P1 (Postgres RLS guardrail): DONE – org isolation enforced via Postgres RLS on core tables using app.current_org_id; keep existing org_id filters and follow OPERATIONS RLS verification steps during rollout.

Production readiness gates (must stay green)

• ✅ Tests and migrations: make test, pytest -m "migrations", and Alembic head matches /readyz.
• ✅ Migration hygiene: CI fails fast if alembic heads returns more than one revision—add a merge migration before merging.
• ✅ Migrations applied and alembic_version matches alembic/versions head.
• ✅ Org isolation regressions: automated suite seeds multiple orgs and blocks cross-org finance/payments/export/iam/photo access; metrics path labels stay templated to avoid cardinality blow-ups.
• ✅ Backups: Postgres backup + restore drill validated for tenant data (org_id scoped).
• ✅ Config secrets: non-default secrets for auth tokens, portal secrets, metrics token, storage signing, Stripe/email keys; at least one admin Basic Auth pair configured.
• ✅ Job heartbeat: /readyz shows recent heartbeat when JOB_HEARTBEAT_REQUIRED=true.
• ✅ Alerts/metrics: Prometheus alerting wired (ops/prometheus/alerts.yml), HTTP metrics enabled, error rates monitored.
• ✅ Storage: delete retries running via storage_janitor job; upload size/MIME limits enforced.
• ✅ CORS/proxy: STRICT_CORS=true with explicit origins; trusted proxy IPs/CIDRs set if behind proxy.
• ✅ Email/export delivery: admin scans and jobs resolve adapters from app.state/services, send immediately while still queuing outbox events for retries and DLQ replay.

Known risks and mitigations

• Scheduler gaps: If cleanup/reminder/export jobs are not scheduled, stale bookings/emails accumulate; mitigate by wiring cron/Scheduler and monitoring job heartbeat.
• Stripe outage/circuit trips: Deposit creation or billing may fail; retries exist but preserve DB consistency—surface errors to clients and alert on repeated failures.
• Export webhook failures: Dead letters accumulate; operators must review GET /v1/admin/export-dead-letter and use POST /v1/admin/export-dead-letter/{id}/replay after fixing the target.
• Storage limits: Entitlements enforce per-plan bytes; ensure storage_janitor runs and order_photo_max_bytes tuned.
• Org context missing under RLS: Postgres requires app.current_org_id per transaction; missing context yields empty results and blocked inserts. Verify API/cron tasks set org context (see OPERATIONS.md) when enabling RLS.

Release checklist (copy/paste)

• Set environment: APP_ENV=prod, STRICT_CORS=true, CORS_ORIGINS=[...], proxy trust lists configured.
• Configure secrets: auth/portal/photo/metrics secrets, admin credentials, Stripe keys + webhook secret, email/SMTP or SendGrid keys.
• Run DB: apply migrations via make migrate; verify alembic_version matches alembic heads.
• Seed pricing config and verify estimator response.
• Start jobs: schedule cleanup/email/retention/export tasks and outbox DLQ handling (/v1/admin/outbox/dead-letter or the outbox-delivery loop); enable job_heartbeat_required if monitoring heartbeats.
• Verify health: /healthz returns ok; /readyz shows DB+migrations OK and job heartbeat fresh.
• Exercise smokes: lead intake with captcha (if enabled), booking with deposit to Stripe webhook, admin metrics CSV, worker photo upload + signed URL access.
• Run scripts/smoke_prod.sh (see docs/runbook_smoke.md) against the target environment.
• Confirm backups/restore runbook executed.