EmergentDB Setup Guide

A comprehensive guide to get EmergentDB running locally with document ingestion and Gemini OCR.

Prerequisites

Before starting, ensure you have the following installed:

Tool	Version	Check Command
Rust	1.75+	rustc --version
Cargo	1.75+	cargo --version
Python	3.10+	python3 --version
Bun (or npm)	Latest	bun --version

Install Prerequisites

macOS (Homebrew):

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env

# Install Python 3
brew install python@3.11

# Install Bun (recommended for frontend)
curl -fsSL https://bun.sh/install | bash

Linux (Ubuntu/Debian):

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env

# Install Python 3
sudo apt update && sudo apt install python3 python3-pip python3-venv

# Install Bun
curl -fsSL https://bun.sh/install | bash

---

Quick Start (5 Minutes)

1. Clone and Enter Directory

cd emergentDB

2. Set Up Environment Variables

# Copy the example .env file (if not already present)
cp .env.example .env 2>/dev/null || true

# Ensure your .env has the Gemini API key:
# GEMINI_API_KEY=your_key_here

3. Build the Rust Backend

# Standard build
cargo build --release

# For Apple M1/M2/M3/M4 (recommended for best performance)
RUSTFLAGS="-C target-cpu=apple-m1" cargo build --release

4. Start the API Server

# In-memory mode (default, no persistence)
cargo run --release -p api-server

# With persistence (vectors survive restart)
DATA_DIR=./data cargo run --release -p api-server

# Custom settings with persistence
PORT=8080 VECTOR_DIM=768 DATA_DIR=./mydata cargo run --release -p api-server

5. Start the Frontend (New Terminal)

cd frontend
bun install
bun run dev

6. Open in Browser

• Frontend Dashboard: http://localhost:3000
• API Server: http://localhost:3001 (if using port 3001)

---

Running the Document Ingestion CLI

The ingestion CLI uses Gemini for OCR and embeddings to process documents.

1. Set Up Python Environment

cd examples/ingestion
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

2. Run the CLI

# Interactive mode
python ingest.py

# Ingest a single file
python ingest.py --file /path/to/document.pdf

# Ingest a directory
python ingest.py --dir /path/to/documents

# Query the database
python ingest.py --query "What is machine learning?"

# List all documents
python ingest.py --list

CLI Commands Reference

Command	Description
--file <path>	Ingest a single document (PDF, image, text)
--dir <path>	Ingest all documents in a directory
--query <text>	Search the vector database
--list	List all ingested documents
--delete <id>	Delete a document by ID
--clear	Clear the entire database
--stats	Show database statistics

---

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                        EmergentDB                                │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────────┐   │
│  │  Frontend    │    │  API Server  │    │  Ingestion CLI   │   │
│  │  (Next.js)   │───▶│   (Axum)     │◀───│    (Python)      │   │
│  └──────────────┘    └──────────────┘    └──────────────────┘   │
│                             │                     │              │
│                             ▼                     ▼              │
│                      ┌──────────────┐    ┌──────────────────┐   │
│                      │ Vector Core  │    │   Gemini API     │   │
│                      │ (Rust SIMD)  │    │  (OCR + Embed)   │   │
│                      └──────────────┘    └──────────────────┘   │
│                             │                                    │
│                             ▼                                    │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                    QD Engine (MAP-Elites)                 │   │
│  │  ┌────────────────┐          ┌────────────────────────┐  │   │
│  │  │   IndexQD      │          │      InsertQD          │  │   │
│  │  │ (Flat/HNSW/IVF)│          │ (SIMD Strategies)      │  │   │
│  │  └────────────────┘          └────────────────────────┘  │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

---

API Endpoints Reference

Vector Operations

Search Vectors

curl -X POST http://localhost:3000/vectors/search \
  -H "Content-Type: application/json" \
  -d '{"query": [0.1, 0.2, ...], "k": 10}'

Insert Vector

curl -X POST http://localhost:3000/vectors/insert \
  -H "Content-Type: application/json" \
  -d '{"id": 1, "vector": [0.1, 0.2, ...], "metadata": {"source": "doc1.pdf"}}'

Batch Insert

curl -X POST http://localhost:3000/vectors/batch_insert \
  -H "Content-Type: application/json" \
  -d '{"vectors": [{"id": 1, "vector": [...], "metadata": {...}}, ...]}'

Document Ingestion (New)

Ingest Document

curl -X POST http://localhost:3000/ingest \
  -H "Content-Type: multipart/form-data" \
  -F "file=@document.pdf"

Ingest with OCR

curl -X POST http://localhost:3000/ingest/ocr \
  -H "Content-Type: multipart/form-data" \
  -F "file=@scanned_document.pdf"

QD Evolution

Run Evolution

curl -X POST http://localhost:3000/qd/evolve \
  -H "Content-Type: application/json" \
  -d '{"sample_size": 1000, "generations": 10}'

System

Health Check

curl http://localhost:3000/health

---

Configuration Options

Environment Variables

Variable	Default	Description
PORT	3000	API server port
VECTOR_DIM	768	Vector dimension (768 for Gemini, 1536 for OpenAI)
DATA_DIR	-	Path for persistent storage (in-memory if not set)
GEMINI_API_KEY	-	Gemini API key for OCR and embeddings
ENVIRONMENT	development	Environment mode

Persistence Mode

When DATA_DIR is set, EmergentDB uses RocksDB for durable storage:

• Vectors survive server restarts
• Automatic recovery on startup
• No impact on search performance (still in-memory)

Rust Build Flags

# Apple Silicon optimization
RUSTFLAGS="-C target-cpu=apple-m1" cargo build --release

# Intel/AMD optimization
RUSTFLAGS="-C target-cpu=native" cargo build --release

# Debug build (slower but with symbols)
cargo build

---

Running Benchmarks

Rust Benchmarks

# Scale benchmark (1K, 10K, 50K vectors)
cargo run --release --example scale_benchmark

# Index-specific benchmarks
cargo bench -p vector-core

Python Comparison Benchmarks

cd tests
python3 -m venv venv
source venv/bin/activate
pip install numpy lancedb chromadb

python3 scale_comparison.py

Results are saved to tests/benchmark_results/.

---

Troubleshooting

Build Errors

Error: linker 'cc' not found

# macOS
xcode-select --install

# Linux
sudo apt install build-essential

Error: SIMD intrinsics not available

# Use portable SIMD fallback
RUSTFLAGS="" cargo build --release

Runtime Errors

Error: Address already in use

# Find and kill process on port 3000
lsof -i :3000
kill -9 <PID>

Error: GEMINI_API_KEY not set

# Ensure .env file exists and is sourced
source .env
# Or export directly
export GEMINI_API_KEY=your_key_here

Frontend Issues

Error: bun: command not found

# Install bun
curl -fsSL https://bun.sh/install | bash
source ~/.bashrc

Error: Module not found

cd frontend
rm -rf node_modules bun.lock
bun install

---

Development Workflow

Making Changes

1. Backend (Rust)

   cargo watch -x "run --release -p api-server"  # Auto-reload on changes
   

2. Frontend (Next.js)

   cd frontend && bun run dev  # Hot reload enabled
   

3. Ingestion CLI

   cd examples/ingestion
   python ingest.py --help
   

Running Tests

# All Rust tests
cargo test --workspace

# Specific crate tests
cargo test -p vector-core
cargo test -p qd-engine

---

Performance Tips

1. Use Release Builds: Always use --release for production
2. CPU Optimization: Use -C target-cpu=native for best SIMD performance
3. Vector Dimensions: Match embedding model dimensions exactly
4. Batch Operations: Use batch insert for large datasets
5. Let Evolution Run: Allow the QD engine to find optimal configurations

---

Getting Help

• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Next Steps

1. Try the scale benchmark to see performance
2. Use the ingestion CLI to add documents
3. Explore the frontend dashboard for visualization
4. Read about MAP-Elites algorithm for understanding the QD engine