Market Sentinel — Improvements & Enhancement Tracker

# Market Sentinel — Improvements & Enhancement Tracker

> Planned improvements beyond the core 8-week build, prioritised by impact and feasibility. Each improvement references the relevant PRD section and includes implementation notes.

---

## Priority 1: High Impact, Near-Term

### 1.1 Automated Social Distribution (Twitter/X)

**PRD reference**: §11 Future Roadmap, item 1
**Phase**: Post-launch (after 7-day hands-off test passes)

**What**: n8n workflow that takes the daily brief and automatically posts a tweet thread summarising the key insights. Claude generates platform-appropriate copy — concise, hashtag-optimised, with key data points.

**Implementation**:
- New n8n workflow: `distribute-twitter` — triggers after `generate-brief` completes
- Claude Haiku call to reformat the brief into 3–5 tweets (280 chars each)
- HTTP Request node → Twitter API v2 (free tier: 1,500 tweets/month — more than enough for daily threads)
- Include key metrics: top divergence, BTC price, confidence score, Fear & Greed
- Schedule: 08:00 AM, 25 minutes after the brief is ready

**Cost**: Free (Twitter API v2 free tier). Claude Haiku for reformatting: ~$0.01/day.

**Risk**: Twitter API access and rate limits change frequently. Build with graceful degradation — if posting fails, log the error and continue. The brief still exists on the dashboard regardless.

### 1.2 Email Newsletter

**PRD reference**: §11 Future Roadmap, item 2
**Phase**: Post-launch

**What**: Automated daily or weekly email digest sent to subscribers with the daily brief, top divergences, and market summary.

**Implementation**:
- New n8n workflow: `distribute-email` — triggers after `generate-brief`
- HTML email template with dark theme matching the dashboard aesthetic
- Transactional email service: SendGrid free tier (100 emails/day) or Resend
- Subscriber management: simple SQLite table (`email_subscribers`) or a lightweight form
- Unsubscribe link in every email (CAN-SPAM compliance)
- Weekly digest option: n8n Cron trigger on Sundays, summarising the week's briefs

**Cost**: Free at low volume (SendGrid free tier). Scales to $15–20/month at 1,000+ subscribers.

### 1.3 Semantic Search on Dashboard

**PRD reference**: §8 Database Architecture (Qdrant collections)
**Phase**: Phase 2 (Weeks 5–6 in Roadmap)

**What**: Search bar on the dashboard that finds semantically similar briefs, articles, or divergences. Users type natural language queries like "when did ETH have a similar bearish pattern?" and get relevant historical results.

**Implementation**:
- Node.js endpoint: `/sentinel/api/search?q=...`
- Generate embedding for query text (same model used for storing embeddings)
- Query Qdrant with vector similarity search, filtered by optional params (date range, coin, confidence threshold)
- Display results as a list of matching briefs/articles with similarity scores
- UI: search bar in the header, results overlay or dedicated search page

**Cost**: Zero incremental (Qdrant self-hosted, embedding model already running for daily pipeline).

---

## Priority 2: Medium Impact, Phase 2–3

### 2.1 Cross-Source Price Validation

**PRD reference**: §6 Data Sources (CoinCap + Binance)
**Architecture reference**: Decision 5

**What**: Compare prices from CoinCap and Binance for every tracked coin. Flag when variance exceeds 1%. This catches API errors, stale data, or exchange-specific price anomalies.

**Implementation**:
- Already designed into `collect-prices` workflow
- n8n Code node compares CoinCap price vs Binance price per coin
- If variance > 1%: set `price_validation_status = 'flagged'` in `market_data` table
- If variance > 5%: trigger Telegram alert (likely data error, not real market movement)
- Dashboard System Health page shows validation status per coin

**Impact**: Prevents the daily brief from citing incorrect prices. A 5% data error in BTC price could generate a false divergence signal.

### 2.2 Weighted Confidence Formula (v2)

**PRD reference**: §4.3 Confidence Scoring
**Phase**: Phase 2

**What**: Replace the simple High/Medium/Low labels with a weighted numerical formula: 30% data volume + 25% source diversity + 25% historical accuracy + 20% recency.

**Implementation**:
- n8n Code node in `generate-brief` workflow calculates the weighted score
- Data volume: count of data points collected today (normalised 0–100)
- Source diversity: number of distinct source categories (price, sentiment, news, on-chain, Fear & Greed) out of 5
- Historical accuracy: rolling accuracy of past divergence predictions (requires tracking — see §2.3)
- Recency: age of oldest data point used (< 6h = 100, < 12h = 75, < 24h = 50, > 24h = 25)
- Output: integer 0–100 stored in `daily_briefs.confidence_score`

**Impact**: More trustworthy confidence scores. Users can calibrate their decision-making based on a transparent, reproducible formula rather than a subjective label.

### 2.3 Historical Accuracy Tracking

**PRD reference**: §11 Future Roadmap, item 5
**Phase**: Phase 2–3 (requires 30+ days of data)

**What**: Compare past divergence signals against actual price outcomes. Did a "bullish divergence" signal on BTC actually precede a price increase?

**Implementation**:
- New SQLite table: `divergence_outcomes`
- Daily job: for each divergence detected 3, 7, and 14 days ago, check what actually happened to the price
- Score: correct direction = +1, wrong direction = -1, inconclusive (< 2% move) = 0
- Rolling accuracy percentage displayed on System Health page
- Feed accuracy data back into confidence formula (§2.2)

**Impact**: This is the difference between "AI-generated market noise" and "AI that gets better over time." Even a 55% accuracy rate on divergence direction would be meaningful. Tracking it publicly (on the dashboard) builds credibility.

### 2.4 Expanded Coin Coverage

**PRD reference**: §4.1 Daily Intelligence Brief
**Roadmap reference**: Phase 2, Week 6

**What**: Expand from 6 tracked coins (BTC, ETH, SOL, AVAX, ADA, DOT) to 10–15, including trending altcoins.

**Implementation**:
- Update `collect-prices` workflow to fetch more coins from CoinCap (single API call covers all)
- Update `scrape-reddit` to monitor additional subreddits (r/solana, r/algorand, etc.)
- Update `scrape-news` cheerio selectors to detect mentions of new coins
- Add a config table in SQLite (`tracked_coins`) so coin list can be updated without editing workflows
- Update dashboard market snapshot table to handle variable coin count (scrollable)

**Cost**: Zero — CoinCap returns all coins in a single API call. More Reddit subreddits add marginal scraping time (< 30s per subreddit).

---

## Priority 3: Lower Impact, Future

### 3.1 Multi-Coin Watchlists (User-Configurable)

**PRD reference**: §11 Future Roadmap, item 3

**What**: Let users configure which coins they want to track rather than a fixed list.

**Implementation**: Requires user accounts or at minimum a cookie/localStorage-based preference system. The backend already tracks multiple coins — this is primarily a frontend feature (dropdown selector, save preferences, filter dashboard to selected coins).

**Dependency**: Low priority until there are multiple users. For a single-user system, editing the `tracked_coins` table directly is sufficient.

### 3.2 Intra-Day Alerts

**PRD reference**: §11 Future Roadmap, item 4

**What**: Move from daily cadence to event-driven alerts when significant divergences are detected mid-day.

**Implementation**: Run a lightweight version of the pipeline every 4–6 hours. Only the `collect-prices`, `scrape-reddit`, and `detect-divergences` workflows need to run (skip news, skip full brief generation). If a high-confidence divergence is detected, send Telegram alert immediately.

**Trade-off**: More API calls (~3x daily cost), more complexity (partial pipeline runs), and more noise (intra-day signals are noisier than daily aggregates). Worth building only after the daily system proves reliable for 30+ days.

### 3.3 Weekly and Monthly Digests

**PRD reference**: §11 Future Roadmap, item 7

**What**: Automated summaries spanning longer time windows. A weekly digest would summarise 7 daily briefs into key trends, recurring divergences, and accuracy of past signals.

**Implementation**:
- New n8n workflow: `generate-weekly-digest` — Cron Sunday 08:00 AM
- SQLite query: last 7 daily briefs + all divergences from the week
- Claude Sonnet call: synthesise into a 300-word weekly summary
- Store in `weekly_summaries` table (already defined in schema)
- Display on History page with a "Weekly" filter option

**Cost**: One additional Sonnet call per week (~$0.05/week = $0.20/month).

### 3.4 API Access for Third-Party Integration

**PRD reference**: §11 Future Roadmap, item 8

**What**: Expose Market Sentinel data via REST endpoints for integration with Notion, Slack, Telegram bots, or other tools.

**Implementation**: The JSON API endpoints already exist (`/sentinel/api/brief/:date`, `/sentinel/api/divergences`). This improvement adds API key authentication, rate limiting, and documentation (OpenAPI/Swagger spec).

**Dependency**: Only useful if there's demand from external tools. Low priority for a single-user system.

### 3.5 Automated LinkedIn / Other Social Platforms

**PRD reference**: §11 Future Roadmap, item 1 (extension)

**What**: Extend the social distribution pipeline beyond Twitter to LinkedIn, Telegram channels, Discord bots, and other communities.

**Implementation**: Each platform gets its own n8n workflow (or a branch within the `distribute-brief` workflow). Claude adapts the brief content for each platform's format and audience expectations — LinkedIn posts are longer and more professional; Telegram messages are shorter and more direct; Discord embeds support rich formatting.

**Trade-off**: Each platform adds maintenance overhead (API changes, auth tokens, format requirements). Add one at a time based on where the audience actually is.

### 3.6 Multi-Agent Architecture

**PRD reference**: §11 Future Roadmap, item 9
**Architecture reference**: Decision cut (What Was Cut)

**What**: Replace the single-prompt pipeline with specialised analyst agents — a sentiment agent, a technical analysis agent, a risk agent — each generating their own analysis, then a synthesis agent combining them into the final brief.

**Why deferred**: The current single-prompt approach (Haiku for classification, Sonnet for synthesis) produces equivalent output at much lower cost and complexity. Multi-agent adds value only when the daily brief needs deeper, multi-perspective analysis — e.g., if the product expands to cover equities, forex, or commodities alongside crypto.

**Revisit when**: The daily brief quality plateaus and users request deeper analysis, or when the product expands beyond crypto markets.

---

## Improvement Tracking

| ID | Improvement | Priority | Phase | Status | Depends On |
|---|---|---|---|---|---|
| 1.1 | Twitter/X auto-posting | P1 | Post-launch | Planned | Twitter API v2 access |
| 1.2 | Email newsletter | P1 | Post-launch | Planned | SendGrid/Resend account |
| 1.3 | Semantic search (Qdrant) | P1 | Phase 2 | Planned | Qdrant deployment (Week 6) |
| 2.1 | Cross-source price validation | P2 | Phase 1 | Designed | Built into collect-prices workflow |
| 2.2 | Weighted confidence formula | P2 | Phase 2 | Planned | Historical accuracy data (2.3) |
| 2.3 | Historical accuracy tracking | P2 | Phase 2–3 | Planned | 30+ days of divergence data |
| 2.4 | Expanded coin coverage | P2 | Phase 2 | Planned | None |
| 3.1 | Multi-coin watchlists | P3 | Future | Deferred | User accounts or preference system |
| 3.2 | Intra-day alerts | P3 | Future | Deferred | 30+ days of reliable daily operation |
| 3.3 | Weekly/monthly digests | P3 | Future | Planned | weekly_summaries table (schema ready) |
| 3.4 | API access | P3 | Future | Deferred | External demand |
| 3.5 | LinkedIn / other social | P3 | Future | Deferred | Twitter distribution working first |
| 3.6 | Multi-agent architecture | P3 | Future | Deferred | Quality plateau on single-prompt approach |