RAG System Documentation

Detailed documentation of the Retrieval-Augmented Generation (RAG) implementation in Tu2tor.

Overview

The RAG system enables AI to answer questions based on user-uploaded study materials by retrieving relevant document chunks and using them as context for generation.

Core Concepts

What is RAG?

Retrieval-Augmented Generation combines two approaches:

1. Retrieval: Search and extract relevant information from a knowledge base
2. Generation: Use retrieved information as context for AI to generate accurate answers

Benefits:

• Reduces hallucinations (AI making up information)
• Provides source citations
• Enables domain-specific knowledge without fine-tuning
• Supports multi-document queries

Tu2tor RAG Architecture

User Query
    │
    ├─► Retrieval Phase
    │   ├─► Keyword Extraction
    │   ├─► MongoDB Full-Text Search
    │   ├─► Chunk Ranking
    │   └─► Top-K Selection
    │
    └─► Generation Phase
        ├─► Context Assembly
        ├─► Prompt Construction
        ├─► Gemini API Call
        └─► Answer + Sources

Document Processing Pipeline

1. Upload and Storage

Entry Point: POST /api/knowledge-base

User uploads file (PDF/PPTX/DOCX/Image)
    │
    ├─► Multer middleware saves to disk
    │       Path: server/uploads/knowledge-base/<filename>
    │
    └─► Create KnowledgeBase document
            status: 'pending'
            processingStatus.progress: 0%

2. Text Extraction

Service: documentProcessor.js::processDocument()

PDF Extraction:

import pdfParse from 'pdf-parse';

const data = await pdfParse(buffer);
const fullText = data.text;
const pageCount = data.numpages;

PPTX Extraction:

import officeParser from 'officeparser';

const data = await officeParser.parseOfficeAsync(filePath);
// Extract text from slides

DOCX Extraction:

import mammoth from 'mammoth';

const result = await mammoth.extractRawText({ path: filePath });
const fullText = result.value;

Image OCR:

import Tesseract from 'tesseract.js';

const { data: { text } } = await Tesseract.recognize(buffer, 'eng+chi_sim');

3. Text Chunking

Algorithm: Semantic Chunking with Overlap

Chunking Strategy:

Document (full text)
    │
    ├─► Split by paragraphs (double newline)
    │
    ├─► For each paragraph:
    │   ├─► If length > MAX_CHUNK_SIZE (1000 chars)
    │   │       └─► Further split by sentences
    │   │
    │   └─► Group paragraphs into semantic chunks
    │           Target: 500-1000 characters per chunk
    │           Overlap: 100 characters between chunks
    │
    └─► Generate metadata for each chunk:
            - chunkIndex (position in document)
            - tokenCount (estimated tokens)
            - charCount (character count)
            - chunkType ('semantic' | 'paragraph' | 'section')
            - pageNumber (if available)

Implementation (ragService.js):

function chunkText(fullText, pageTexts = []) {
  const chunks = [];
  const paragraphs = fullText.split(/\n\s*\n/);
  
  let currentChunk = '';
  let chunkIndex = 0;
  
  for (const para of paragraphs) {
    if (currentChunk.length + para.length > MAX_CHUNK_SIZE) {
      // Save current chunk
      if (currentChunk.trim()) {
        chunks.push({
          content: currentChunk.trim(),
          metadata: {
            chunkIndex: chunkIndex++,
            tokenCount: estimateTokens(currentChunk),
            charCount: currentChunk.length,
            chunkType: 'semantic'
          }
        });
      }
      // Start new chunk with overlap
      const words = currentChunk.split(' ');
      currentChunk = words.slice(-20).join(' ') + ' ' + para;
    } else {
      currentChunk += '\n\n' + para;
    }
  }
  
  // Save final chunk
  if (currentChunk.trim()) {
    chunks.push({...});
  }
  
  return chunks;
}

4. Embedding Generation (Future Enhancement)

Current: No embeddings, uses full-text search

Planned:

import { GoogleGenerativeAI } from '@google/generative-ai';

async function generateEmbedding(text) {
  const model = genAI.getGenerativeModel({ 
    model: 'text-embedding-004' 
  });
  
  const result = await model.embedContent(text);
  return result.embedding.values; // Float32Array
}

5. Storage

Schema: KnowledgeChunk model

{
  knowledgeBaseId: ObjectId,     // Reference to parent document
  content: String,                // Chunk text
  embedding: [Number],            // Vector (future)
  metadata: {
    pageNumber: Number,
    chunkIndex: Number,           // Sequential index
    tokenCount: Number,
    charCount: Number,
    chunkType: String,            // 'semantic' | 'paragraph'
    summary: String               // Optional AI summary
  },
  semanticScore: Number           // Quality metric (0-1)
}

Indexes:

// Compound index for efficient querying
knowledgeChunkSchema.index({ 
  knowledgeBaseId: 1, 
  'metadata.chunkIndex': 1 
});

// Text index for full-text search (current approach)
knowledgeBaseSchema.index({
  title: 'text',
  description: 'text',
  'extractedContent.fullText': 'text'
});

Retrieval Phase

Query Processing

Entry Point: POST /api/rag/query

{
  question: "What is the deadline for the project?",
  documentIds: ["doc1", "doc2"], // User-selected documents
  subjectId: "subject123"         // Optional filter
}

Step 1: Keyword Extraction

Current Implementation: Direct MongoDB text search

const query = {
  $text: { $search: userQuestion },
  knowledgeBaseId: { $in: documentIds }
};

const chunks = await KnowledgeChunk.find(query, {
  score: { $meta: 'textScore' }
}).sort({ score: { $meta: 'textScore' } });

Future Enhancement: Use NLP for better keyword extraction

• Remove stop words
• Extract named entities
• Query expansion with synonyms

Step 2: Semantic Search (Future)

// Generate query embedding
const queryEmbedding = await generateEmbedding(question);

// Vector similarity search (cosine similarity)
const chunks = await KnowledgeChunk.aggregate([
  {
    $addFields: {
      similarity: {
        $function: {
          body: cosineSimilarity,
          args: ['$embedding', queryEmbedding]
        }
      }
    }
  },
  { $match: { similarity: { $gte: 0.7 } } },
  { $sort: { similarity: -1 } },
  { $limit: 5 }
]);

Step 3: Reranking

Sort chunks by relevance:

function rankChunks(chunks, question) {
  return chunks.map(chunk => {
    let score = chunk.score || 0; // Text search score
    
    // Boost score if chunk contains exact question words
    const questionWords = question.toLowerCase().split(/\s+/);
    const matches = questionWords.filter(word => 
      chunk.content.toLowerCase().includes(word)
    );
    score += matches.length * 0.1;
    
    // Boost semantic chunks over paragraph chunks
    if (chunk.metadata.chunkType === 'semantic') {
      score += 0.2;
    }
    
    // Penalize very short or very long chunks
    if (chunk.metadata.charCount < 100 || chunk.metadata.charCount > 2000) {
      score -= 0.1;
    }
    
    return { ...chunk, finalScore: score };
  }).sort((a, b) => b.finalScore - a.finalScore);
}

Generation Phase

Context Assembly

Select Top K Chunks (K=5):

const topChunks = rankedChunks.slice(0, 5);

const context = topChunks.map((chunk, idx) => {
  return `[Source ${idx + 1}: ${chunk.title}, Page ${chunk.pageNumber}]\n${chunk.content}`;
}).join('\n\n---\n\n');

Prompt Construction

System Prompt:

You are an AI tutor assistant. Answer the user's question based ONLY on the provided source materials. If the answer is not in the sources, say so clearly. Always cite your sources using [Source X] notation.

User Prompt:

Context from study materials:
---
[Source 1: Lecture Notes Week 3, Page 5]
The project deadline is March 15, 2025...

[Source 2: Project Specification, Page 2]
All submissions must be completed by...
---

Question: What is the deadline for the project?

Answer based on the context above, citing specific sources.

Gemini API Call

import { GoogleGenerativeAI } from '@google/generative-ai';

const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);
const model = genAI.getGenerativeModel({ model: 'gemini-2.5-flash' });

const result = await model.generateContent([
  { role: 'user', parts: [{ text: systemPrompt }] },
  { role: 'model', parts: [{ text: 'Understood.' }] },
  { role: 'user', parts: [{ text: constructedPrompt }] }
]);

const answer = result.response.text();

Source Parsing

Extract source citations from answer:

function parseSourceCitations(answer, chunks) {
  const citations = [];
  const sourceRegex = /\[Source (\d+)\]/g;
  let match;
  
  while ((match = sourceRegex.exec(answer)) !== null) {
    const sourceIndex = parseInt(match[1]) - 1;
    if (sourceIndex < chunks.length) {
      citations.push({
        docId: chunks[sourceIndex].knowledgeBaseId,
        title: chunks[sourceIndex].title,
        pageNumber: chunks[sourceIndex].metadata.pageNumber,
        chunkId: chunks[sourceIndex]._id
      });
    }
  }
  
  return citations;
}

Response Format

{
  "success": true,
  "answer": "According to [Source 1], the project deadline is March 15, 2025. This is confirmed in [Source 2] which states all submissions must be completed by this date.",
  "sources": [
    {
      "docId": "doc123",
      "title": "Lecture Notes Week 3",
      "pageNumber": 5,
      "chunkId": "chunk456",
      "similarity": 0.92
    },
    {
      "docId": "doc789",
      "title": "Project Specification",
      "pageNumber": 2,
      "chunkId": "chunk101",
      "similarity": 0.87
    }
  ],
  "chunksUsed": 5,
  "model": "gemini-2.5-flash"
}

Optimization Strategies

Current Approach

Strengths:

• Simple implementation using MongoDB built-in text search
• No additional infrastructure required
• Fast for keyword-based queries

Limitations:

• Cannot understand semantic similarity
• May miss relevant chunks with different wording
• Ranking based on keyword frequency, not meaning

Future Enhancements

1. Vector Search with MongoDB Atlas

Enable Atlas Search for vector similarity:

// Create vector search index
db.knowledgechunks.createSearchIndex({
  name: 'vector_index',
  type: 'vectorSearch',
  definition: {
    fields: [{
      type: 'vector',
      path: 'embedding',
      numDimensions: 768,
      similarity: 'cosine'
    }]
  }
});

// Vector search query
const results = await KnowledgeChunk.aggregate([
  {
    $vectorSearch: {
      index: 'vector_index',
      path: 'embedding',
      queryVector: queryEmbedding,
      numCandidates: 100,
      limit: 10
    }
  }
]);

2. Hybrid Search

Combine keyword and semantic search:

async function hybridSearch(question, documentIds) {
  // Get results from both methods
  const keywordResults = await textSearch(question, documentIds);
  const semanticResults = await vectorSearch(question, documentIds);
  
  // Merge with weighted scoring
  const combined = mergeResults(keywordResults, semanticResults, {
    keywordWeight: 0.3,
    semanticWeight: 0.7
  });
  
  return combined.slice(0, 5);
}

3. Query Expansion

Improve retrieval by expanding user queries:

async function expandQuery(question) {
  // Use AI to generate related questions
  const expanded = await gemini.generate(
    `Generate 3 related questions for: "${question}"`
  );
  
  // Combine original + expanded queries
  return [question, ...expanded];
}

4. Chunk Summarization

Pre-generate summaries for faster retrieval:

async function summarizeChunk(content) {
  const summary = await gemini.generate(
    `Summarize in one sentence: ${content}`
  );
  
  return summary;
}

// Store in chunk metadata
chunk.metadata.summary = await summarizeChunk(chunk.content);

Advanced Features

Multi-Document Queries

Query across multiple documents simultaneously:

POST /api/rag/query
{
  "question": "Compare the requirements in both specifications",
  "documentIds": ["spec1", "spec2", "spec3"]
}

RAG service:

1. Searches all specified documents
2. Groups results by document
3. Provides comparative analysis

Conversational RAG

Maintain conversation context:

{
  "question": "What about the testing requirements?",
  "documentIds": ["doc1"],
  "conversationHistory": [
    { "role": "user", "content": "What is the project about?" },
    { "role": "assistant", "content": "..." }
  ]
}

System resolves "the testing requirements" using conversation context.

Citation Granularity

Sources include:

• Document title
• Page number
• Specific chunk content
• Similarity score

Frontend displays:

References:
- Project Specification, p.5
- Lecture Notes Week 3, p.12-14
- Assignment Guidelines, p.2

Performance Metrics

Chunking Performance

Tested with:

• 50-page PDF: ~2 seconds
• 30-slide PPTX: ~1 second
• 20-page DOCX: ~1 second
• Image with text: ~5 seconds (OCR)

Chunk Characteristics:

• Average chunks per document: 30-100
• Average tokens per chunk: 200-400
• Average character count: 500-1000

Query Performance

Full-Text Search (Current):

• Query time: 50-200ms
• Scales linearly with document count
• Efficient for <10,000 chunks

Vector Search (Future):

• Query time: 10-50ms (with proper indexing)
• Sub-linear scaling with approximate nearest neighbor (ANN)
• Efficient for millions of chunks

Generation Performance

Gemini API:

• Time-to-first-token: ~500ms
• Streaming speed: ~50 tokens/second
• Total query time: 2-5 seconds

Quality Assurance

Chunk Quality Metrics

semanticScore = 
  (contentDiversity * 0.3) +
  (informationDensity * 0.4) +
  (contextCompleteness * 0.3)

• Content Diversity: Vocabulary richness
• Information Density: Key terms per 100 words
• Context Completeness: Standalone readability

Answer Quality

Evaluated by:

• Relevance: Does answer address the question?
• Accuracy: Is information from sources correct?
• Completeness: Are all aspects covered?
• Citation Quality: Are sources properly attributed?

Error Handling

Scenarios

No Documents Selected:

{
  "success": false,
  "error": "Please select at least one document",
  "code": "NO_DOCUMENTS"
}

No Relevant Chunks Found:

{
  "success": true,
  "answer": "I couldn't find relevant information in the selected documents to answer your question. Please try rephrasing or selecting different materials.",
  "sources": []
}

Processing Not Complete:

{
  "success": false,
  "error": "Document is still processing. Please wait.",
  "code": "PROCESSING_INCOMPLETE"
}

AI Generation Failed:

{
  "success": false,
  "error": "Failed to generate answer. Please try again.",
  "code": "GENERATION_FAILED",
  "details": "API quota exceeded"
}

Frontend Integration

Knowledge Base Mode

UI Component: AIChat.jsx with mode === 'kb'

// 1. User selects documents
const [selectedDocIds, setSelectedDocIds] = useState([]);

// 2. User asks question
const handleSendMessage = async () => {
  const response = await ragAPI.query({
    question: inputMessage,
    documentIds: selectedDocIds
  });
  
  // 3. Display answer with sources
  setMessages(prev => [...prev, {
    role: 'assistant',
    content: response.answer,
    sources: response.sources
  }]);
};

Chunk Visualization

UI Component: ChunkVisualization.jsx

Displays how a document was split:

• Timeline layout with vertical line
• Each chunk as a card
• Metadata (token count, char count, type)
• Expandable content
• Summary (if available)

Configuration

Environment Variables

# RAG Configuration
KB_MAX_CONCURRENT=3              # Max concurrent processing jobs
CHUNK_SIZE_MIN=300               # Minimum chunk size (chars)
CHUNK_SIZE_MAX=1000              # Maximum chunk size (chars)
CHUNK_OVERLAP=100                # Overlap between chunks (chars)
RAG_TOP_K=5                      # Number of chunks to retrieve
RAG_SIMILARITY_THRESHOLD=0.7     # Minimum similarity for vector search

Model Selection

Different models for different tasks:

const modelConfig = {
  embedding: 'text-embedding-004',      // Generate embeddings
  summarization: 'gemini-2.5-flash',    # Chunk summaries
  rag: 'gemini-2.5-flash',              # Answer generation
  thinking: 'gemini-2.5-pro'            # Deep analysis
};

Testing and Validation

Unit Tests

describe('RAG Service', () => {
  test('chunks document correctly', () => {
    const text = 'Lorem ipsum...';
    const chunks = chunkText(text);
    expect(chunks.length).toBeGreaterThan(0);
    expect(chunks[0].metadata.tokenCount).toBeLessThan(500);
  });
  
  test('retrieves relevant chunks', async () => {
    const results = await retrieveChunks('test question', ['doc1']);
    expect(results).toHaveLength(5);
    expect(results[0].score).toBeGreaterThan(0);
  });
});

Integration Tests

Test complete RAG pipeline:

test('RAG query returns answer with sources', async () => {
  const response = await request(app)
    .post('/api/rag/query')
    .send({
      question: 'What is the main topic?',
      documentIds: [testDocId]
    })
    .set('Authorization', `Bearer ${token}`);
    
  expect(response.status).toBe(200);
  expect(response.body.answer).toBeDefined();
  expect(response.body.sources).toBeInstanceOf(Array);
});

Best Practices

For Users

1. Upload quality documents: Clear, well-formatted PDFs work best
2. Use descriptive titles: Helps AI understand document context
3. Add relevant tags: Improves retrieval accuracy
4. Select appropriate documents: Choose materials related to your question
5. Ask specific questions: Better than vague or multi-part questions

For Developers

1. Chunk size balance: Too small = fragmented context, Too large = irrelevant info
2. Overlap is crucial: Prevents information loss at chunk boundaries
3. Monitor token usage: Each RAG query costs tokens (context + generation)
4. Cache embeddings: Generate once, reuse for all queries
5. Handle edge cases: Empty documents, non-text PDFs, corrupted files

Limitations and Tradeoffs

Current System

Limitations:

• Full-text search may miss semantically similar content
• No cross-document reasoning
• Limited to text content (images in PDFs ignored except via OCR)
• OCR accuracy depends on image quality

Tradeoffs:

• Simplicity vs. Accuracy: Chose simple text search for MVP
• Speed vs. Quality: Faster retrieval, potentially lower precision
• Cost vs. Features: No embedding API calls = lower cost

Scalability Limits

• MongoDB text search: Efficient up to ~10,000 chunks
• In-memory ranking: Feasible for ~1,000 results
• File storage: Disk space limited by hosting plan

Future Roadmap

1. Phase 1 (Current): Text extraction + Full-text search + Basic RAG
2. Phase 2: Vector embeddings + Semantic search + Hybrid retrieval
3. Phase 3: Query expansion + Chunk summarization + Reranking
4. Phase 4: Multi-modal RAG (images, tables, charts)
5. Phase 5: Graph-based knowledge representation

References

• RAG Paper - Original RAG research
• Yjs Documentation - CRDT library
• MongoDB Text Search - Full-text indexing
• Gemini API - Google AI documentation
• LangChain - RAG framework (inspiration)