Development Roadmap

# Development Roadmap

## Overview

Mango News is a news aggregation platform for Turks and Caicos Islands (~45,000 residents). It scrapes local news sources, generates AI summaries/translations in 3 languages (EN/ES/HT), and produces weekly audio digests. The platform serves English, Spanish (Dominican community), and Haitian Creole speakers.

## Completed Phases

### Phase 1: Quick Wins (Completed)
- Removed hardcoded credentials, rotated DB password, scrubbed git history
- Added is_blocked filter to public articles endpoint
- Added 4 performance indexes + pg_trgm full-text search (applied to production)
- Added React ErrorBoundary wrapping all page components
- Fixed API URL inconsistency (consolidated to PUBLIC_API_URL)
- Fixed nav-items.ts hardcoded Render URLs (now relative paths)
- Fixed RSS feed hardcoded URLs (now uses SITE_URL env var)
- Removed duplicate useTranslations hook from Header.tsx
- Added loading="lazy" to all feed/article images

### Phase 2: Foundation (Completed)
- Split backend index.js (1,858->203 lines) into 8 route modules + middleware
- Database-based cron job locking (cronLock.js)
- Consolidated S3 upload into shared s3Service.js
- Consolidated schema.sql with all migrations
- Split SettingsPage (2,069 lines) into 5 lazy-loaded sub-components (chunk: 522KB->39KB)
- PWA service worker with offline caching
- Emergency/hurricane alert banner (backend + frontend)
- Updated site.webmanifest for PWA
- Backup strategy documentation

### UX/UI Refinements (Completed)
- Language switcher loading indicator
- Mobile share button truncation fix (icon-only on small screens)
- Lightbox scroll prevention
- Footer external link indicators
- Removed dead 2001 date workaround
- Dark/light mode persistence fix (no flash on load)

### UI/UX Audit — Design System & Accessibility (Completed)

Comprehensive audit covering visual design, accessibility, user journeys, and admin settings cohesiveness.

**Quick Wins (20 items):**
- Skip-navigation link (WCAG 2.4.1), aria-labels, lang attributes on language names
- Sunday Edition autoplay disabled (WCAG 1.4.2)
- Replaced 20+ hardcoded Tailwind colors with oklch design tokens (footer, NewsFeed, ArticleDetail, SundayEditionDetail, admin components)
- Added semantic color tokens: `--success`, `--warning`, `--footer-*`, `--social-whatsapp`, `--social-facebook`
- Internationalized LoginDialog, LoginButton, Header, ModeToggle, LanguageSwitcher (14 new keys in en/es/ht)
- Removed duplicate Google Fonts @import (kept `<link>` tags with preconnect)
- Removed hardcoded `#FFB88C` hex from tailwind.config.mjs (oklch version in global.css is authoritative)
- Topic badges converted from `<span onClick>` to semantic `<a>` links
- Sunday Edition cards converted from `<div onClick>` to `<a href>`

**P1 Admin Improvements (4 items):**
- Removed duplicate "Generate Sunday Edition" from Scraper Controls (now only in Sunday Editions tab)
- Moved global AI toggles from localStorage to `application_settings` DB table (auto-save on toggle)
- Added Emergency Banner management form (Switch + text input) to Scraper Controls tab
- Added article block/unblock toggle to article management (leveraging existing `is_blocked` DB column)

**P2 Features (6 items):**
- Topic navigation bar on index page (scrollable pills fetched from `/api/topics`)
- Copy Link + native Web Share API buttons on article and Sunday Edition detail pages
- Reading time estimate in article metadata bar (calculated from word count at 200 WPM)
- Search/filter in Sunday Editions admin tab
- Server-side locale redirect middleware (`middleware.ts`) using cookie + Accept-Language header (eliminates redirect flash)
- Real AI coverage metrics from actual article data (replaced "configured capacity" proxy)

**P3 Features (8 items):**
- Server-side prev/next article navigation fallback (`GET /api/articles/:id/adjacent`) for direct-link visitors
- Cross-source article search in admin Source Management tab (`GET /api/articles/admin/search`)
- Topic stats chart + article freshness KPIs (last 24h/7d/30d) + blocked count in Overview dashboard
- URL blacklist management UI with add/remove (was file-system only)
- Sunday Edition draft/publish workflow (new `status` column, public listing filters to published)
- Unified `ImageLightbox` component (focus trap, keyboard nav, focus restoration) replacing 3 separate implementations
- Article thumbnail now opens in lightbox (was not clickable)
- Server-side search in Source Articles DataTable (title, URL, or ID across all pages)
- Topic management CRUD — new "Topics" tab (6th tab) with add/edit/delete/merge, translations

---

## Phase 3: Growth (Next)

### F1. WhatsApp Channel Integration — Priority: HIGH

**Context:** WhatsApp dominates Caribbean communication. Meeting users where they are instead of requiring website visits.

**Description:** Create a WhatsApp Channel that pushes daily top-5 headlines with links.

**Implementation:**
- Manual approach (easiest): Create a WhatsApp Channel, add a daily cron job that generates a formatted message (top 5 articles by publication_date), admin copies to channel
- Automated approach: Use WhatsApp Business API (costs money) or a headless WhatsApp library

**Technical notes:**
- Add a new cron job in scraper.js or a new `whatsappDigest.js` module
- Query: `SELECT title, id FROM articles WHERE publication_date >= NOW() - INTERVAL '24 hours' ORDER BY publication_date DESC LIMIT 5`
- Format with article title + `${SITE_URL}/en/article/${id}` links
- Store generated digest in application_settings for admin to copy

**Effort:** 1 week

---

### F4. Topic-Based Email Newsletter — Priority: HIGH

**Context:** TCI diaspora and expats who want to stay connected without daily visits. Resend free tier: 3,000 emails/month -- sufficient for TCI scale.

**Description:** Weekly email combining Sunday Edition summary + top articles by topic. Users subscribe with email only (no account needed).

**Implementation:**
- New `subscribers` table: `id, email, created_at, confirmed (boolean), unsubscribe_token`
- New routes: `POST /api/newsletter/subscribe`, `GET /api/newsletter/confirm/:token`, `GET /api/newsletter/unsubscribe/:token`
- Email template: HTML email with Sunday Edition summary + top 5 articles with thumbnails
- Send via Resend API (free tier: 3,000/month)
- Integrate with Sunday Edition generation -- after edition is created, also send newsletter
- Add subscribe form to footer or as a floating component

**Dependencies:** Resend account (free), subscriber table migration

**Effort:** 2 weeks

---

### Article Offline Caching — Priority: MEDIUM

**Context:** TCI internet can be spotty. The service worker (sw.js) already caches the app shell and API responses.

**Description:** Enhance the service worker to proactively cache recent article content for offline reading.

**Implementation:**
- In `sw.js`, add a background sync that fetches the latest 50 article JSONs into `API_CACHE`
- When the app loads, populate the cache with articles from the latest feed fetch
- Add an offline indicator component that shows when the user is offline
- Cache article detail pages (HTML) when visited

**Files:** `frontend/public/sw.js`

**Effort:** 1 week

---

### G2. Bilingual Sunday Edition — Priority: MEDIUM

**Context:** Currently only English summary is generated. Spanish and Haitian Creole communities would benefit from translated editions.

**Description:** Add `summary_es` and `summary_ht` columns to sunday_editions table. After English generation, translate via existing aiService.js pipeline.

**Implementation:**
- Migration: `ALTER TABLE sunday_editions ADD COLUMN summary_es TEXT, ADD COLUMN summary_ht TEXT;`
- In `sundayEditionGenerator.js`, after creating English summary, call `aiService.translateContent()` for ES and HT
- Update `SundayEditionDetail.tsx` to display locale-appropriate summary based on `currentLocale`
- **Pitfall:** Sunday Edition summaries are ~4,000 chars which may exceed the 8,192 token translation limit in aiService.js. Split into sections or increase `AI_TRANSLATION_CONTENT` max tokens.

**Files:** `backend/src/sundayEditionGenerator.js`, `backend/src/services/aiService.js`, `frontend/src/components/SundayEditionDetail.tsx`

**Effort:** 1 week

---

### F7. Article Reaction Buttons — Priority: MEDIUM

**Context:** Lightweight engagement without comment moderation overhead.

**Description:** Emoji reaction buttons (Informative, Important, Concerning, Heartwarming) on articles. No login required -- rate limit by fingerprint/localStorage.

**Implementation:**
- New table: `CREATE TABLE article_reactions (id SERIAL PRIMARY KEY, article_id INTEGER REFERENCES articles(id) ON DELETE CASCADE, reaction_type VARCHAR(20) NOT NULL, fingerprint VARCHAR(64), created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP);`
- New endpoints: `POST /api/articles/:id/reactions` (body: `{ type, fingerprint }`), `GET /api/articles/:id/reactions` (returns counts per type)
- Frontend: Add reaction bar below article content in ArticleDetail.tsx
- Use localStorage to prevent repeat reactions from same browser

**Files:** New migration, new route handler, `frontend/src/components/ArticleDetail.tsx`

**Effort:** 3 days

---

### Topic-Based Personalization — Priority: MEDIUM

**Context:** All users see the same feed. Simple localStorage-based preferences (no backend needed).

**Description:** Let users select preferred topics. Show preferred topics first in the feed.

**Implementation:**
- Add a "Personalize" button/modal on the feed page
- Store selected topic IDs in localStorage as `preferredTopics`
- When fetching articles, if preferredTopics exist, sort preferred-topic articles first
- Show a "For You" section at the top of the feed with preferred topic articles

**Files:** `frontend/src/components/NewsFeed.tsx`, new `TopicPreferences.tsx` component

**Effort:** 1 week

---

### F5. "Island Pulse" Public Dashboard — Priority: LOW

**Context:** A public-facing dashboard showing TCI news trends. Useful for journalists, politicians, businesses.

**Description:** Public page showing: top topics this week, article volume by source, topic word cloud.

**Implementation:**
- New Astro page: `frontend/src/pages/[lang]/pulse.astro`
- Use existing `/api/stats` data (currently auth-required -- create a public version with limited data)
- New endpoint: `GET /api/pulse` (public) returning top topics, article counts, trending topics
- Reuse existing Recharts chart components from `components/charts/`

**Files:** New page, new API endpoint, reuse chart components

**Effort:** 2 weeks

---

### F8. Podcast RSS Feed for Sunday Edition — Priority: LOW

**Context:** Expose Sunday Edition audio as a podcast for Apple Podcasts, Spotify, etc.

**Description:** Dedicated RSS feed at `/api/podcast.xml` following podcast RSS spec (iTunes tags).

**Implementation:**
- New route in `routes/rss.js`: `GET /api/podcast`
- Query `sunday_editions` table for editions with `narration_url IS NOT NULL`
- Generate RSS with `<itunes:*>` tags: title, author, description, image, enclosure (MP3)
- Register with Apple Podcasts and Spotify podcast directories

**Files:** `backend/src/routes/rss.js`

**Effort:** 2 days

---

### F6. Government Gazette Integration — Priority: LOW

**Context:** TCI Government Gazette notices are critical but hard to find. Centralizing them is high-value.

**Description:** Add TCI Government Gazette as a new source using existing scraper infrastructure.

**Implementation:**
- Use source onboarding pipeline (`/api/sources/onboard`) to auto-detect selectors
- May need custom `article_link_template` for gazette notice URLs
- Tag gazette articles with a distinctive "Government Notice" topic

**Effort:** 1 week (depends on gazette website structure)

---

### Basic Test Suite — Priority: LOW

**Context:** Zero test coverage currently. Critical paths should have basic tests.

**Description:** Add tests for backend API endpoints and critical frontend components.

**Implementation:**
- Backend: Use Jest + supertest for API endpoint testing
- Test auth flow (register, login, refresh, me, logout)
- Test article CRUD (create via scraper, read, update, delete, block)
- Test rate limiting behavior
- Frontend: Use Vitest + React Testing Library for component tests
- Test ErrorBoundary render + retry
- Test NewsFeed loading/error/empty states

**Effort:** 2 weeks

---

## Phase 4: Scale (Future)

### Admin: User & Role Management — Priority: MEDIUM

**Description:** CRUD for admin users with role assignment. The `users` table already has a `role` column (default `'admin'`) but no management UI exists.

**Implementation:**
- New admin tab or section in Settings for user listing
- Create/edit/delete users with role selection (admin, editor, viewer)
- Password change from UI (currently requires DB access)
- Role-based access: editors can manage articles but not sources/settings, viewers are read-only

**Effort:** 1 week

---

### Admin: AI Cost Dashboard — Priority: MEDIUM

**Description:** Track API usage and costs across Groq (summaries/tags/translations), image generation, and Unreal Speech (audio).

**Implementation:**
- New `api_usage_log` table: `id, service (groq/image/speech), endpoint, tokens_used, cost_estimate, created_at`
- Log each AI call in aiService.js and sundayEditionGenerator.js
- Dashboard card in Overview tab showing daily/weekly/monthly spend by service
- Alert threshold when approaching budget limits

**Effort:** 2 weeks

---

### Admin: Scraper Run History & Logs — Priority: HIGH

**Description:** Persistent timeline of scraper runs with timestamps, source, articles found/added, errors. Currently only toast notifications during runs with no history.

**Implementation:**
- New `scraper_runs` table: `id, source_id, started_at, completed_at, status (success/error), articles_found, articles_added, error_message`
- Log each run in scraper.js
- New section in Overview tab or dedicated "Logs" tab showing recent runs
- Filter by source, date range, status

**Effort:** 2 weeks

---

### Admin: SettingsPage State Refactor — Priority: MEDIUM

**Description:** SettingsPage.tsx has 57 state variables in a single component. Extract to `useReducer` with typed actions or React Context to improve maintainability and prevent state bugs.

**Implementation:**
- Define a `SettingsState` type and `SettingsAction` discriminated union
- Extract state + dispatch into `useSettingsReducer()` hook or `SettingsContext`
- Each tab component receives state slice + dispatch instead of 15-38 individual props
- No UI changes — purely internal refactor

**Effort:** 2 weeks

---

### Admin: Keyboard Shortcuts — Priority: LOW

**Description:** Power-user efficiency improvements for the single admin.

**Implementation:**
- Cmd/Ctrl+S to save in any form/dialog
- Escape to close dialogs (already works for shadcn dialogs)
- Arrow keys or number keys to switch tabs
- Use a lightweight hotkey library or native `keydown` listener

**Effort:** 3 days

---

### Admin: Bulk Article Re-Translation — Priority: LOW

**Description:** Select multiple articles in the DataTable and trigger batch translation processing.

**Implementation:**
- Add "Rerun Translations" to bulk action bar in SourceArticles (alongside existing bulk delete)
- Call `POST /api/articles/:id/process-ai` with `featureType: "translations"` for each selected article
- Progress indicator showing completed/total

**Effort:** 2 days

---

### H1. Cross-Platform News Widget for Mango Jobs

**Context:** Mango Jobs (at `d:\ClaudeMCP\!projects\mango-jobs`) is a separate Next.js app in the mango.tc ecosystem.

**Description:** When Mango Jobs company pages mention a company also in the news, show a "Recent News" widget.

**Implementation:**
- Extend existing widgets in `widgets/` directory to accept `?company=` parameter
- Filter articles by company name mention in title or content
- Embed via iframe or script tag on Mango Jobs company pages

**Effort:** 2 weeks

---

### H3. Unified mango.tc Search

**Description:** Single search bar on mango.tc returning results across news, jobs, vehicles, real estate.

**Implementation:** Each platform exposes `/api/search?q=` -- thin aggregator on main mango.tc domain merges and ranks results.

**Effort:** 3 weeks

---

### C6. Backend TypeScript Migration

**Description:** Gradual migration of backend JavaScript to TypeScript for type safety.

**Implementation:**
- Add `tsconfig.json` with `allowJs: true, checkJs: true` for gradual adoption
- Convert file-by-file starting with `db.js`, `middleware/auth.js`, `cronLock.js`
- Add type definitions for database query results

**Effort:** 4 weeks (gradual)

---

### H4. Shared Authentication Across Platforms

**Description:** SSO between Mango News, Mango Jobs, and mango.tc.

**Implementation:** Either shared JWT secret across platforms or OAuth2 provider on main mango.tc domain.

**Effort:** 3 weeks

---

### H5. mango.tc Weekly Ecosystem Newsletter

**Description:** Combine Sunday Edition + "Top jobs this week" + "Featured listings" from other platforms into a unified weekly digest.

**Implementation:** Each platform provides `/api/weekly-highlights` endpoint. Newsletter service aggregates.

**Effort:** 2 weeks

---

### Web Push Notifications for Breaking News

**Context:** TCI is hurricane-prone. Breaking news alerts are high value.

**Description:** Web Push API (browser-native, free) for breaking news alerts.

**Implementation:**
- Service worker already exists (`sw.js`) -- add push event listener
- New `push_subscriptions` table
- Admin UI: "Send Push Notification" button with title + body
- Backend: Use `web-push` npm package

**Effort:** 2 weeks

---

### Advanced AI Features

**Description:** Sentiment analysis on articles, automatic story threading (grouping related articles across sources/days).

**Implementation:**
- Add `sentiment` column to articles table (positive/negative/neutral + score)
- Add `story_thread_id` column for grouping related articles
- Run sentiment analysis via Groq after summarization
- Story threading via embedding similarity (would need a vector DB or simple keyword overlap)

**Effort:** 3 weeks

---

### User Accounts with Saved Articles

**Description:** Public user accounts (not just admin) with saved articles, reading history, topic preferences.

**Implementation:**
- New `user_preferences` table, `saved_articles` table
- New auth endpoints for public registration (currently admin-only)
- Frontend: Save button on articles, reading history page, preferences page

**Effort:** 4 weeks

---

### Comment System with Moderation

**Description:** Comments on articles with admin moderation queue.

**Implementation:**
- New `comments` table (article_id, user_id, content, status: pending/approved/rejected, created_at)
- Admin moderation queue in SettingsPage
- Frontend: Comment form + display below article content
- Anti-spam: rate limiting, basic content filtering

**Effort:** 4 weeks

---

### Performance Monitoring

**Description:** Add error tracking and performance monitoring.

**Implementation:**
- Sentry free tier (5K errors/month) or Render built-in metrics
- Frontend: Sentry React SDK for error tracking
- Backend: Sentry Node SDK for API error tracking
- Custom dashboards for article count trends, scraper health, AI service performance

**Effort:** 1 week