CineChoice RAG Implementation Plan

# CineChoice RAG Implementation Plan

## 🎯 **Project Overview**
Transform CineChoice into a sophisticated RAG (Retrieval Augmented Generation) movie recommendation system using vector embeddings, semantic search, and AI-powered recommendations.

## 🏗️ **Architecture Overview**

```
User Questions → Embedding → Vector Search → Movie Matching → AI Response
     ↓              ↓           ↓              ↓              ↓
  [Frontend] → [API Routes] → [Supabase] → [HuggingFace] → [Results]
```

## 📊 **Technology Stack**

### **Core Technologies:**
- **Frontend**: Next.js 15, React 19, TypeScript, Tailwind CSS
- **Backend**: Next.js API Routes, TypeScript
- **Vector Database**: Supabase (PostgreSQL + pgvector)
- **Embeddings**: mixedbread-ai/mxbai-embed-large-v1 (1024 dimensions)
- **Text Generation**: Hugging Face Inference API
- **Text Processing**: LangChain RecursiveCharacterTextSplitter

### **Data Sources:**
- `public/content.js` - 9 structured movie objects
- `public/movies.txt` - 12 detailed movie descriptions

## 🗄️ **Database Schema Design**

### **Table: `movies`**
```sql
CREATE TABLE movies (
  id SERIAL PRIMARY KEY,
  title TEXT NOT NULL,
  release_year INTEGER,
  genre TEXT[],
  rating FLOAT,
  runtime TEXT,
  director TEXT,
  cast TEXT[],
  plot_summary TEXT,
  imdb_rating FLOAT,
  content_source TEXT, -- 'content.js' or 'movies.txt'
  raw_content TEXT,
  created_at TIMESTAMP DEFAULT NOW(),
  updated_at TIMESTAMP DEFAULT NOW()
);
```

### **Table: `movie_chunks`**
```sql
CREATE TABLE movie_chunks (
  id SERIAL PRIMARY KEY,
  movie_id INTEGER REFERENCES movies(id) ON DELETE CASCADE,
  chunk_text TEXT NOT NULL,
  chunk_index INTEGER NOT NULL,
  embedding VECTOR(1024), -- mixedbread-ai/mxbai-embed-large-v1 dimensions
  metadata JSONB,
  created_at TIMESTAMP DEFAULT NOW()
);

-- Create vector similarity index
CREATE INDEX movie_chunks_embedding_idx ON movie_chunks 
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);
```

### **Table: `user_queries`** (Optional - for analytics)
```sql
CREATE TABLE user_queries (
  id SERIAL PRIMARY KEY,
  session_id TEXT,
  questions JSONB,
  query_embedding VECTOR(1024),
  matched_movies INTEGER[],
  response_generated TEXT,
  created_at TIMESTAMP DEFAULT NOW()
);
```

## 🔧 **Implementation Phases**

### **Phase 1: Database Setup & Data Ingestion**

#### **Step 1.1: Supabase Configuration**
- [ ] Set up Supabase project
- [ ] Enable pgvector extension
- [ ] Create database schema
- [ ] Set up environment variables
- [ ] Create database connection utilities

#### **Step 1.2: Data Processing Pipeline**
- [ ] Create data ingestion script
- [ ] Parse `content.js` and `movies.txt`
- [ ] Normalize and structure movie data
- [ ] Store movies in database

#### **Step 1.3: Text Chunking**
- [ ] Install LangChain dependencies
- [ ] Implement RecursiveCharacterTextSplitter
- [ ] Chunk movie descriptions and metadata
- [ ] Store chunks with references to parent movies

#### **Step 1.4: Vector Embeddings**
- [ ] Set up Hugging Face client
- [ ] Create embedding generation function
- [ ] Generate embeddings for all chunks
- [ ] Store embeddings in vector database

### **Phase 2: Search & Matching Functions**

#### **Step 2.1: Vector Search Function**
```sql
-- Cosine similarity search function
CREATE OR REPLACE FUNCTION match_movie_chunks (
  query_embedding VECTOR(1024),
  match_threshold FLOAT DEFAULT 0.7,
  match_count INT DEFAULT 10
)
RETURNS TABLE (
  id INTEGER,
  movie_id INTEGER,
  chunk_text TEXT,
  similarity FLOAT,
  metadata JSONB
)
LANGUAGE SQL
AS $$
  SELECT 
    mc.id,
    mc.movie_id,
    mc.chunk_text,
    1 - (mc.embedding <=> query_embedding) AS similarity,
    mc.metadata
  FROM movie_chunks mc
  WHERE 1 - (mc.embedding <=> query_embedding) > match_threshold
  ORDER BY mc.embedding <=> query_embedding ASC
  LIMIT LEAST(match_count, 200);
$$;
```

#### **Step 2.2: Movie Recommendation Function**
```sql
-- Get movie recommendations with scores
CREATE OR REPLACE FUNCTION get_movie_recommendations (
  query_embedding VECTOR(1024),
  match_threshold FLOAT DEFAULT 0.7,
  movie_limit INT DEFAULT 5
)
RETURNS TABLE (
  movie_id INTEGER,
  title TEXT,
  avg_similarity FLOAT,
  matching_chunks_count INTEGER,
  movie_data JSONB
)
LANGUAGE SQL
AS $$
  SELECT 
    m.id,
    m.title,
    AVG(1 - (mc.embedding <=> query_embedding)) AS avg_similarity,
    COUNT(mc.id) AS matching_chunks_count,
    ROW_TO_JSON(m) AS movie_data
  FROM movies m
  JOIN movie_chunks mc ON m.id = mc.movie_id
  WHERE 1 - (mc.embedding <=> query_embedding) > match_threshold
  GROUP BY m.id, m.title
  ORDER BY avg_similarity DESC
  LIMIT movie_limit;
$$;
```

### **Phase 3: API Development**

#### **Step 3.1: Core API Endpoints**

**`app/api/ingest/route.ts`**
- [ ] Data ingestion endpoint
- [ ] Chunking and embedding pipeline
- [ ] Database population

**`app/api/search/route.ts`**
- [ ] Query embedding generation
- [ ] Vector similarity search
- [ ] Return matched movies

**`app/api/recommend/route.ts`**
- [ ] Process user answers
- [ ] Generate recommendations
- [ ] AI-powered response generation

#### **Step 3.2: Utility Functions**

**`lib/embeddings.ts`**
```typescript
export async function generateEmbedding(text: string): Promise<number[]> {
  const response = await hf.featureExtraction({
    model: "mixedbread-ai/mxbai-embed-large-v1",
    inputs: text,
  });
  return Array.from(response);
}
```

**`lib/chunking.ts`**
```typescript
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";

export async function chunkText(text: string): Promise<string[]> {
  const splitter = new RecursiveCharacterTextSplitter({
    chunkSize: 500,
    chunkOverlap: 50,
  });
  return await splitter.splitText(text);
}
```

**`lib/supabase.ts`**
```typescript
export async function searchSimilarMovies(
  queryEmbedding: number[],
  threshold: number = 0.7,
  limit: number = 5
) {
  // Vector search implementation
}
```

### **Phase 4: Frontend Integration**

#### **Step 4.1: Enhanced QA Component**
- [ ] Add loading states
- [ ] Real-time answer processing
- [ ] Progress indicators

#### **Step 4.2: Results Page**
- [ ] Movie recommendation display
- [ ] Similarity scores
- [ ] AI-generated explanations
- [ ] Movie details and links

#### **Step 4.3: User Flow**
```
1. User answers questions → QA Components
2. Submit answers → API processing
3. Generate embeddings → Vector search
4. Find similar movies → AI response generation
5. Display results → Results page
```

### **Phase 5: AI Response Generation**

#### **Step 5.1: Response Generation**
- [ ] Context building from matched movies
- [ ] Prompt engineering for recommendations
- [ ] AI model integration for explanations
- [ ] Response formatting and styling

#### **Step 5.2: Prompt Templates**
```typescript
const RECOMMENDATION_PROMPT = `
Based on the user's preferences: {user_answers}
And these matching movies: {matched_movies}
Generate a personalized movie recommendation explaining why these movies match their preferences.
`;
```

## 📦 **Dependencies to Install**

```bash
# Core dependencies
npm install @supabase/supabase-js
npm install @huggingface/inference
npm install langchain

# Type definitions
npm install -D @types/pg
```

## 🔧 **Environment Variables**

```env
# Supabase
NEXT_PUBLIC_SUPABASE_URL=your_supabase_url
NEXT_PUBLIC_SUPABASE_ANON_KEY=your_supabase_anon_key
SUPABASE_SERVICE_ROLE_KEY=your_service_role_key

# Hugging Face
HUGGINGFACE_API_KEY=your_hf_api_key
```

## 🚀 **Deployment & Production**

### **Performance Optimizations:**
- [ ] Embedding caching
- [ ] Query result caching
- [ ] Database connection pooling
- [ ] Vector index optimization

### **Monitoring:**
- [ ] Query performance metrics
- [ ] Embedding generation costs
- [ ] User interaction analytics
- [ ] Error tracking and logging

## 📊 **Success Metrics**

1. **Technical Metrics:**
   - Query response time < 2 seconds
   - Embedding generation < 500ms
   - 95%+ similarity search accuracy

2. **User Experience:**
   - Relevant movie recommendations
   - Intuitive interface
   - Fast, responsive interactions

3. **Business Metrics:**
   - User engagement time
   - Recommendation click-through rates
   - Return user percentage

## 🧪 **Testing Strategy**

1. **Unit Tests:**
   - Embedding generation functions
   - Vector search utilities
   - Data processing pipelines

2. **Integration Tests:**
   - API endpoint functionality
   - Database operations
   - End-to-end user flows

3. **Performance Tests:**
   - Vector search speed
   - Concurrent user handling
   - Database query optimization

## 🔄 **Next Steps**

1. **Immediate:** Set up Supabase and create schema
2. **Week 1:** Implement data ingestion and chunking
3. **Week 2:** Vector embeddings and search functions
4. **Week 3:** API development and frontend integration
5. **Week 4:** AI response generation and optimization

---

**This plan provides a comprehensive roadmap for building a production-ready RAG-powered movie recommendation system. Each phase builds upon the previous one, ensuring a solid foundation for an advanced AI-driven user experience.**