Run Phase - Visual Product Search

Overview

The "Run" phase implements visual product search - take a picture of an item and query the Couchbase Lite database using vector similarity to find matching products.

Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Camera/Image  │────▶│  ML Model (CLIP) │────▶│  Vector Query   │
│     Input       │     │  Generate Vector │     │  Couchbase Lite │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                                         │
                                                         ▼
                                                 ┌─────────────────┐
                                                 │ Similar Products│
                                                 │   (Top N)       │
                                                 └─────────────────┘

Key Components

1. Image Embedding Generation

Convert images to vectors using CLIP or similar models that can run on-device.

Recommended Libraries:

• react-native-executorch - On-device AI inference (CLIP model support)
• react-native-fast-tflite - TensorFlow Lite for React Native with GPU acceleration
• @react-native-rag/executorch - RAG utilities with ExecuTorch integration

2. Couchbase Lite Vector Search

Use APPROX_VECTOR_DISTANCE() function to find similar vectors.

Vector Index Configuration:

const config = new VectorIndexConfiguration({
  expression: 'embedding',  // field containing the vector
  dimensions: 512,          // CLIP base model outputs 512 dimensions
  centroids: 100,           // ~sqrt(num_documents)
  metric: 'cosine',         // or 'euclidean'
});

await collection.createIndex('embedding_index', config);

Query Example:

SELECT META().id, *, APPROX_VECTOR_DISTANCE(embedding, $queryVector) AS distance
FROM catalog.vectors
WHERE APPROX_VECTOR_DISTANCE(embedding, $queryVector) < 0.5
ORDER BY APPROX_VECTOR_DISTANCE(embedding, $queryVector)
LIMIT 10

3. Hybrid Search

Combine vector search with full-text search for better results:

SELECT META().id, *
FROM catalog.products
WHERE MATCH(description, 'red shirt')
  AND APPROX_VECTOR_DISTANCE(embedding, $queryVector) < 0.5
ORDER BY APPROX_VECTOR_DISTANCE(embedding, $queryVector)
LIMIT 10

Implementation Plan

Phase 1: Camera Integration

• Install expo-camera or react-native-vision-camera
• Create camera capture screen
• Allow photo selection from gallery

Phase 2: Image Embedding

• Install react-native-executorch with CLIP model
• Load CLIP image encoder (352 MB model)
• Generate 512-dim embedding from captured image

Phase 3: Vector Query

• Create vector index on catalog.vectors collection
• Implement APPROX_VECTOR_DISTANCE() query
• Display top matching products

Phase 4: UI/UX

• Create visual search screen with camera preview
• Show search results with similarity scores
• Add product detail view

Dependencies to Add

# Camera
npx expo install expo-camera expo-image-picker

# ML/Embeddings (choose one approach)
# Option A: ExecuTorch (recommended for CLIP)
npm install react-native-executorch @react-native-rag/executorch

# Option B: TensorFlow Lite
npm install react-native-fast-tflite

Model Options

CLIP (Recommended)

• Size: ~352 MB
• Dimensions: 512
• Inference Time: ~50-70ms on modern devices
• Pros: Same embedding space for text and images
• Source: https://huggingface.co/software-mansion/react-native-executorch-clip-vit-base-patch32

MobileNet V3

• Size: ~5-15 MB
• Dimensions: 1024
• Inference Time: ~20-30ms
• Pros: Smaller model size
• Cons: Image-only embeddings

Couchbase Lite Vector Search Reference

Documentation

• CBL React Native Full-Text Search
• CBL Swift Vector Search Guide

Key Functions

• APPROX_VECTOR_DISTANCE(vector-expr, target-vector, [metric]) - Find approximate nearest neighbors
• MATCH(field, text) - Full-text search (can combine with vector search)

Vector Index Parameters

Parameter	Description
expression	Field containing the vector
dimensions	Vector size (2-4096)
centroids	Buckets for clustering (~sqrt(docs))
metric	Distance: cosine, euclidean, dot
encoding	Compression: None, SQ (8-bit), PQ

Data Flow

1. User captures image via camera or gallery
2. CLIP model processes image → 512-dim float array
3. Query Couchbase Lite using APPROX_VECTOR_DISTANCE()
4. Display matching products sorted by similarity

Considerations

Performance

• CLIP model is ~352 MB - consider lazy loading
• First inference may take 2x longer (model initialization)
• Cache model in memory for repeated searches

Accuracy

• CLIP works best with clear, well-lit images
• Training vectors should use same model as query
• Consider using cosine similarity for normalized vectors

Offline-First

• Model runs entirely on-device
• No network required for inference
• Vectors sync from Sync Gateway for catalog updates