Toasty — AI Triage & Responsible Disclosure Assistant (2026 — 350 hours)

# Toasty — AI Triage & Responsible Disclosure Assistant (2026 — 350 hours)

## development program Plan (4 × 4 weeks, 350 hours)

### Phase 1 (Weeks 1–4) — Foundations and Event Ingestion
**Goal:** Stand up Toasty as a reliable service wired to BLT/GitHub events; deliver safe, useful summaries early.

**Deliverables**
- Service + GitHub App/webhook ingestion for issue/PR create/edit and attachments.
- Summarizer v1: title + 3–5 bullets; label suggester (severity/category hints).
- Information-completeness checklists (missing steps, expected vs actual, environment).
- Initial PII/secret detectors (tokens, keys, JWTs, emails, URLs with secrets) with safe previews.
- Basic offline evaluation harness (golden set + quick metrics) and contributor docs.

**Acceptance**
- New issues/PRs receive: concise summary, suggested labels, missing info list, and any detected PII flagged.
- Latency/error rates stay within agreed targets.

---

### Phase 2 (Weeks 5–8) — Search, Semantic Dedup, and Repro Extraction
**Goal:** Reduce duplicate noise and extract actionable repro steps automatically.

**Deliverables**
- Embeddings pipeline + vector store; semantic search APIs.
- Cross-repo duplicate detection (issues + PRs) with confidence + brief rationale; “link potential duplicate” action.
- Reproduction-step extractor (steps, expected/actual, environment hints) and “Request more info” auto-prompts.
- Slack/CLI helpers:
  - on-demand safe summaries (`/toasty check <issue_url>`)
  - top-N duplicate candidates
- Robust job orchestration (batching, retries, basic back-pressure) for large repos.

**Acceptance**
- On a test corpus, top-3 duplicate suggestions meet target precision.
- Repro extractor yields structured steps reviewers can paste/apply.
- Slack/CLI responds within SLOs.

---

### Phase 3 (Weeks 9–12) — Reviewer Assist, Quality Tuning, and Safe Workflows
**Goal:** Make reviewers faster with calibrated, explainable suggestions and safe “apply” flows.

**Deliverables**
- Reviewer-assist snippets: risk synopsis, CWE/category hints, “ask-for-info” templates.
- Explainability: “why this label/duplicate” snippets referencing text/lines.
- Configurable org rules (labels/severity defaults, prompt tone, supported languages).
- Quality tuning: prompt iterations, context windowing, throttling/rate limits, guardrails and timeouts.
- Optional export-only API for signals so Idea E (or others) can display them without Toasty owning readiness or dashboard UX.

**Acceptance**
- Side-by-side evaluation shows improved plan quality and fewer false positives.
- Reviewers can apply suggestions in one click with audit logs.
- Export-only API delivers signals without owning readiness.

---

### Phase 4 (Weeks 13–16) — Insights, Hardening, Docs, and Pilot Release
**Goal:** Prove measurable impact, harden for production, and pilot with real orgs.

**Deliverables**
- Lightweight “Toasty Insights” report (not a dashboard):
  - duplicate rate, estimated time saved, common gaps, top CWE/category
  - downloadable CSV/JSON
- Safety & robustness: adversarial inputs, size/time caps, abuse rate-limits; fallback paths when models fail.
- Offline evaluation suite:
  - dupe precision/recall
  - PII precision
  - summary quality rubric
  - release gates
- Documentation: reviewer/admin guides, runbooks, contribution guide; feature flags for staged rollout.
- Pilot with 1–2 orgs; post-pilot fixes; v1.0 tag and enablement checklist.

**Acceptance**
- Pilot shows ≥30% reviewer time reduction (median) and ≥0.80 dupe top-3 precision with zero critical incidents.
- Reports generated successfully; v1.0 signed off.

---

## Benefits
- **Faster triage with less noise:** Summaries, info-completeness checks, repro extraction, and semantic dedup reduce reviewer time and duplicate churn.
- **Safer collaboration:** PII/secret detectors prevent accidental leakage in summaries and Slack, while guardrails/rate limits keep the assistant reliable.
- **Complementary to Idea E:** Toasty focuses on content assistance + optional signal export; Idea E retains ownership of readiness states, CI/SAST aggregation, queues, and dashboard UX.
- **Measurable value:** An evaluation suite + pilot KPIs (time saved, dupe precision) make impact clear to mentors and orgs.

## Reuse patterns from other ideas (swap-ins)
Toasty keeps its scope; the following are **swap-ins** that improve the assistant without expanding it:
- **Idea L2 (Preflight):** When Preflight context exists (e.g. contributor declared intent, issue risk class), use it to enrich summaries and label suggestions (e.g. “This PR touches auth; consider reviewing [link]”). Toasty does not implement pre-contribution flows; it only consumes context when available.
- **Idea N (RAG over BLT docs):** Reuse the RAG-over-BLT/OWASP-docs stack to suggest relevant documentation links in “missing info” hints and reviewer-assist snippets, instead of hard-coding all guidance.
- **Idea E (PR Readiness / Risk):** Consume readiness or risk signals from Idea E when available to enrich Toasty summaries; Toasty continues to export signals to E and does not own dashboard or readiness UX.

## Non-overlap with Idea E (Extended) and Idea H
- **Idea E (Extended)** owns PR Readiness & Security Dashboard (readiness states, CI/SAST/secret/dependency aggregation, check annotations, maintainer queues, and dashboard UX). **Toasty does not** compute readiness states, aggregate CI, emit check annotations, or build/own a dashboard.
- **Idea H** focuses on contributor growth (Sizzle time tracking, “what to work on next,” mentoring on PR merge). Toasty may optionally expose read-only signals that H can consume later, but **does not** implement time tracking or mentoring features.