Socrates Development Plan

1. Architecture & Infrastructure Setup

1.1 Project Structure

• Create two top-level folders: backend/ and frontend/
• Backend: Python monolith with FastAPI, organized by domain (agents/, rag/, knowledge_graph/, voice/, etc.)
• Frontend: React app with WebRTC integration
• Add Docker Compose at root to orchestrate backend, frontend, and PostgreSQL (or use Supabase for hosted DB)
• Database: Supabase PostgreSQL with pgvector extension enabled

1.2 Database Schema Design

PostgreSQL with pgvector (Supabase)

• Enable pgvector extension in Supabase
• users table: user_id, created_at, settings
• knowledge_graph table: user_id, topic, mastery_score (0-100), last_reviewed, misconceptions (JSON)
• session_logs table: user_id, session_id, timestamp, topic_touched, responses_graded
• documents table: document_id, user_id, filename, upload_date, page_count, chunks_indexed
• document_chunks table: chunk_id, document_id, page_number, section_title, content (text), embedding (vector with pgvector), bbox (JSON)
• citations table: chunk_id, page_number, bbox, content

Vector Search (pgvector in Supabase)

• Store embeddings directly in document_chunks table as vector column
• Use cosine similarity for semantic search
• Metadata co-located with vectors: document_id, page_number, chunk_index, section_title

1.3 API Contract Definition

• Define WebSocket message schema for audio streaming (Speech->Text->JSON)
• Define JSON payload structure for LLM output (text_response, audio_response, visual_data)
• Define REST endpoints: upload document, get knowledge graph, get session history
• Define vision payload: base64 image + comparison context

---

2. Phase 1: The "Blind" Tutor (Text-Only, No Voice)

2.1 Core LangGraph State Machine

• Goal: Build the conversation loop without voice latency concerns

• Design the state graph:

  • assess_state: Generate quiz question from uploaded text
  • listen_state: Accept user text input
  • grade_state: LLM grades response, extracts misconceptions
  • update_memory_state: Update knowledge_graph table
  • decide_state: Route to remediation, repetition, or advancement

• Implementation: LangGraph Node structure with conditional edges based on mastery_score

2.2 Extraction Chain (Background Task)

• After each user response, spawn an async task to:
  1. Parse the response for concepts mentioned
  2. Update misconceptions_list in knowledge_graph if errors detected
  3. Recalculate mastery_score using a simple formula (e.g., weighted average of last 5 attempts)

2.3 RAG Setup (Strict Mode)

• Ingest uploaded text file → split into semantic chunks (paragraph-level, not fixed token count)
• Use Gemini 2.5 Flash to extract section headers and concept names during chunking
• Generate embeddings for each chunk using Gemini's embedding API (or LangChain's embedding wrapper)
• Store chunks + embeddings in Supabase document_chunks table with metadata: {document_id, page_number, section, bbox_coords}
• Use pgvector cosine similarity search for retrieval
• Retriever must mark chunks as "from document" vs. "general knowledge" to prevent hallucination

2.4 Memory Initialization

• On user login, query knowledge_graph table
• If any topic has mastery_score < 70 and was last reviewed >3 days ago, flag for remediation
• Pass this context to the first assess_state to warm-start the conversation

2.5 Deliverable (Phase 1)

• Terminal-based CLI app
• User uploads a .txt file (e.g., medical notes)
• Agent quizzes the user in a loop
• Knowledge graph persists across sessions

---

3. Phase 2: Voice Integration (LiveKit Agent)

3.1 Architecture: LiveKit Voice Pipeline

• Use LiveKit Agents framework for seamless voice streaming
• Simplifies STT/TTS pipeline vs. manual WebSocket management
• Components:
  • STT (Speech-to-Text): Deepgram STT plugin (nova-2 model for low latency)
  • TTS (Text-to-Speech): Cartesia TTS plugin (sonic-english for natural voice)
  • VAD (Voice Activity Detection): Silero VAD (on-device, no API calls)
  • LLM: Gemini 2.5 Flash (streaming responses for minimal latency)

3.2 Worker Process Architecture

• worker.py: Runs LiveKit agent processes (separate from REST API)
• Listens for incoming rooms/participants from LiveKit
• Manages audio streaming to/from clients via WebRTC
• No need for custom WebSocket management—LiveKit handles it
• Scales horizontally: multiple worker processes for multiple concurrent sessions

3.3 Audio Pipeline Flow

1. User speaks → Audio captured by client via WebRTC
2. STT (Deepgram) → Transcribes to text in real-time
3. LLM (Gemini) → Generates response with streaming tokens
4. TTS (Cartesia) → Converts text to speech while LLM still generating
5. Audio played → User hears response via WebRTC

3.4 Dynamic VAD & Latency

• Silero VAD handles speech detection (no API calls, <50ms overhead)
• Can adjust silence thresholds dynamically based on question type
• Streaming tokens from Gemini reduce "thinking" latency
• Target: <800ms round-trip (audio in → response out)

3.5 Integration Points

• LiveKit server: Handles WebRTC connection, room management, auth
• Socrates backend: REST API still serves document upload, knowledge graph
• Worker nodes: Stateless agents that process voice, access DB for context
• Gemini API: Used for LLM responses (not embeddings—that's separate)

3.6 Deliverable (Phase 2)

• ✅ Voice-to-voice interaction working via LiveKit
• ✅ STT + TTS streaming with <1s latency
• ✅ Dynamic VAD with question-type awareness
• ✅ Stateless worker architecture for scaling
• ✅ Integration with knowledge graph for context

---

4. Phase 3: Multimodal Output (Whiteboard & Structured Output)

4.1 Structured LLM Output

• Modify LLM prompt to output JSON with fields:

  {
    "text_response": "...",
    "visual_data": {
      "type": "mermaid" | "latex" | "matplotlib_python",
      "content": "...",
      "description": "..."
    },
    "citation": {
      "page": 42,
      "bbox": [x, y, w, h],
      "quote": "..."
    }
  }
  

4.2 Prompt Engineering for Code Generation

• Test prompts that reliably generate valid Mermaid.js syntax (flowcharts, graphs)
• Test LaTeX generation for mathematical expressions
• Test Python code for Matplotlib graphs (with error handling)
• Include a "Self-Correction" chain:
  • If Mermaid render fails, LLM re-generates with error message in context
  • Max 2 retry attempts to avoid infinite loops

4.3 Frontend Whiteboard Component

• React component that:
  • Displays Mermaid diagrams (use mermaid.js library)
  • Renders LaTeX with react-latex
  • Executes Python/matplotlib on backend and returns SVG
  • Syncs animation timing with audio playback (visual highlight appears when audio mentions it)

4.4 Document Viewer Integration

• Show the uploaded PDF alongside the whiteboard
• When citation is returned, highlight the referenced bbox with yellow overlay
• Auto-scroll PDF to the cited page

4.5 Deliverable (Phase 3)

• Full web UI with React
• Agent speaks, draws diagrams, and cites sources simultaneously
• User sees real-time visual explanations

---

5. Phase 4: Vision & Deployment

5.1 Vision Integration

• Capture frame from user's webcam when they say "Show me" or manually trigger
• Send frame + base64 to Gemini 2.5 Flash Vision API
• Prompt template: "Analyze this handwritten solution. Compare to the correct answer. Identify the error step."
• Return structured feedback with visual markup (arrows, annotations)

5.2 Frontend Camera Component

• React component with webcam capture
• Display captured image, overlay agent feedback with annotations
• Allow user to dismiss or request another take

5.3 Deployment & Scaling

• Dockerize backend (FastAPI) and frontend (React + Node)
• Add Docker Compose for local dev, Kubernetes manifests for production
• Environment config: API keys, DB credentials, Pinecone namespace
• Set up monitoring: latency dashboards, error rates, hallucination detection

5.4 Testing & KPI Validation

• Latency benchmarks: measure voice-to-voice round-trip
• Retention tests: verify knowledge_graph recalls topics from >3 sessions ago
• Hallucination tests: create test suite of "out-of-document" queries, verify agent says "I don't know"

5.5 Deliverable (Phase 4)

• Production-ready deployment
• All 4 core pillars functional

---

6. Critical Path & Dependencies

Must Complete First:

1. Supabase project setup + pgvector schema design
2. LangGraph state machine (Phase 1)
3. FastAPI WebSocket scaffolding

Blockers to Watch:

• Deepgram VAD tuning (Phase 2) - difficult to get right, may need ML fine-tuning
• Gemini 2.5 Flash structured output reliability (Phase 3) - prompt engineering heavy
• Vision API accuracy on handwritten math (Phase 4) - test early with sample images
• pgvector cosine similarity relevance (Phase 1) - test embedding quality and similarity thresholds

Parallelizable:

• React UI development can start during Phase 2
• Docker setup can happen anytime
• Frontend and backend can be worked on concurrently once APIs are defined
• Supabase schema can be set up independently while backend infrastructure is being built

---

7. Risk Mitigation Checklist

Risk	Mitigation	Owner	Timeline
Voice latency >1s	Profile each step; use speculative tokens; test with real Deepgram/Cartesia	Backend	Phase 2
Deepgram VAD errors	Create labeled dataset of thinking vs. done; tuning threshold per question type	Audio	Phase 2
Invalid Mermaid code from LLM	Implement self-correction loop with max 2 retries; test 50+ diagram prompts	Backend	Phase 3
Hallucinations on out-of-doc facts	Enforce citation requirement; add explicit "I don't know" branching logic	RAG	Phase 1
Vision API fails on bad handwriting	Test with low-res images early; set user expectations; fallback to text input	Vision	Phase 4
WebSocket connection drops mid-session	Implement reconnection logic with session resumption; persist state to DB	Frontend	Phase 2
pgvector embedding quality	Test with multiple embedding models; tune similarity threshold; benchmark retrieval accuracy	Backend	Phase 1
Supabase latency on vector queries	Profile query times with large document sets; index optimization if needed	DB	Phase 1

---

8. Success Metrics (Testing Checklist)

• Phase 1: Text-based quiz loop works, knowledge_graph updates correctly
• Phase 2: Voice round-trip latency <1000ms on average; interrupt detection works
• Phase 3: Diagrams render correctly; citations highlight PDF correctly
• Phase 4: Vision API identifies errors in handwritten work; deployment passes load test

---

This plan is the blueprint. Ready to start Phase 1 implementation when needed.