Embedding-Based Memory Recall — Implementation Plan

Replace LIKE keyword search with semantic vector search for MemoryFS memory recall. Uses @dao-xyz/sqlite3-vec for vector storage and OpenAI embeddings API (configurable model) for embedding generation. Falls back to keyword search when no OPENAI_API_KEY is set.

Architecture Overview

Write path (memorize / write):
  content → EmbeddingClient.embed() → Float32Array
                                          ↓
  ItemsStore.insert(item) ──────→ EmbeddingStore.upsert(itemId, vector)

Query path (recall / query):
  user message → EmbeddingClient.embed() → Float32Array
                                                ↓
  EmbeddingStore.findSimilar(vector, limit) → [{itemId, distance}]
                                                ↓
  ItemsStore.getByIds(itemIds) → MemoryEntry[] (ranked by distance × salience)

Fallback: When OPENAI_API_KEY is absent, degrade to existing LIKE search.

History assembly order remains: [memory recall] → [summaries] → [recent turns] → [current message]

---

Step 1: Add @dao-xyz/sqlite3-vec dependency

File: package.json

npm install @dao-xyz/sqlite3-vec

This brings sqlite-vec (the C extension) and a Node.js wrapper that auto-loads the vec extension into better-sqlite3.

---

Step 2: Create EmbeddingClient utility

New file: src/utils/embedding-client.ts

A standalone, stateless client for generating text embeddings. Wraps the OpenAI SDK's embeddings.create() endpoint. Not an LLM provider — this is a utility.

interface EmbeddingClientConfig {
  model: string;       // e.g. 'text-embedding-3-small'
  dimensions: number;  // e.g. 1536
  apiKey?: string;     // defaults to OPENAI_API_KEY env var
  baseUrl?: string;    // defaults to https://api.openai.com/v1
}

interface EmbeddingClient {
  embed(texts: string[]): Promise<Float32Array[]>;
  readonly dimensions: number;
  readonly available: boolean;  // false when no API key
}

function createEmbeddingClient(config: EmbeddingClientConfig): EmbeddingClient

Key behaviors:

• Returns available: false when no API key set (no throw — graceful degradation)
• Returns Float32Array for direct sqlite-vec consumption
• Uses openai SDK already in dependencies — no new HTTP client
• Logs via getLogger().child({ component: 'embedding-client' })

New test file: tests/utils/embedding-client.test.ts

• Test graceful degradation when no API key
• Test Float32Array output format
• Mock OpenAI SDK for unit tests

---

Step 3: Create EmbeddingStore for sqlite-vec operations

New file: src/providers/memory/memoryfs/embedding-store.ts

A vector store backed by sqlite-vec's vec0 virtual table. Separate from ItemsStore — uses @dao-xyz/sqlite3-vec's createDatabase() for automatic extension loading.

class EmbeddingStore {
  constructor(dbPath: string, dimensions: number);

  upsert(itemId: string, embedding: Float32Array): void;
  findSimilar(query: Float32Array, limit: number, scope?: string): Array<{
    itemId: string;
    distance: number;
  }>;
  delete(itemId: string): void;
  hasEmbedding(itemId: string): boolean;
  listUnembedded(limit: number): string[];  // for backfill
  close(): void;
}

Schema:

-- vec0 virtual table for vector similarity search
CREATE VIRTUAL TABLE IF NOT EXISTS item_embeddings USING vec0(
  item_id TEXT PRIMARY KEY,
  embedding float[{dimensions}]
);

-- Mapping table to join vectors with scope for filtering
CREATE TABLE IF NOT EXISTS embedding_meta (
  item_id    TEXT PRIMARY KEY,
  scope      TEXT NOT NULL,
  created_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_emeta_scope ON embedding_meta(scope);

Separate database file (_vec.db) to avoid extension compatibility issues with the existing sync SQLite adapter used by ItemsStore.

Query pattern:

SELECT em.item_id, v.distance
FROM item_embeddings v
JOIN embedding_meta em ON em.item_id = v.item_id
WHERE em.scope = ?
  AND v.embedding MATCH ?
ORDER BY v.distance
LIMIT ?

New test file: tests/providers/memory/memoryfs/embedding-store.test.ts

• Test CRUD operations (upsert, find, delete)
• Test similarity search returns correct ordering
• Test scope filtering
• Test listUnembedded() for backfill
• Uses temp file for database

---

Step 4: Extend MemoryQuery with optional embedding field

Modified file: src/providers/memory/types.ts

export interface MemoryQuery {
  scope: string;
  query?: string;
  limit?: number;
  tags?: string[];
  agentId?: string;
  /** Pre-computed embedding vector for semantic search (optional). */
  embedding?: Float32Array;
}

Backward-compatible — existing providers ignore the field.

---

Step 5: Integrate embeddings into MemoryFS provider

Modified file: src/providers/memory/memoryfs/provider.ts

Changes to create(config):

1. Initialize EmbeddingClient from config/env vars
2. Initialize EmbeddingStore alongside ItemsStore
3. On write(): generate embedding → embeddingStore.upsert()
4. On memorize(): batch-generate embeddings for all new items
5. On query():
   • If q.embedding provided → embeddingStore.findSimilar() → join with salience
   • Else if q.query provided → existing LIKE search (fallback)
   • Rank: combine similarity = 1 / (1 + distance) with existing salienceScore()
6. On delete(): also delete from embedding store

The salience formula in salience.ts already accepts a similarity parameter — pass computed similarity instead of the hardcoded 1.0 currently used.

Modified file: src/providers/memory/memoryfs/items-store.ts

Add getByIds(ids: string[]): MemoryFSItem[] — batch lookup by ID list (needed to hydrate vector search results into full items).

---

Step 6: Update memory recall to use embeddings

Modified file: src/host/memory-recall.ts

export interface MemoryRecallConfig {
  enabled: boolean;
  limit: number;
  scope: string;
  /** Embedding client for semantic search (falls back to keywords if absent). */
  embeddingClient?: EmbeddingClient;
}

Changes to recallMemoryForMessage():

1. If embeddingClient?.available → embed user message → pass { embedding } in query
2. Else → fall back to existing extractQueryTerms() + keyword search
3. Logging: add strategy: 'embedding' | 'keyword' to recall log events

Modified file: src/host/server-completions.ts

Create EmbeddingClient instance from config and pass to memory recall:

import { createEmbeddingClient } from '../utils/embedding-client.js';

const embeddingClient = createEmbeddingClient({
  model: config.history.embedding_model,
  dimensions: config.history.embedding_dimensions,
});

const recallConfig: MemoryRecallConfig = {
  enabled: config.history.memory_recall,
  limit: config.history.memory_recall_limit,
  scope: config.history.memory_recall_scope,
  embeddingClient,
};

---

Step 7: Add config fields for embeddings

Modified file: src/config.ts

Add to the history strict object:

embedding_model: z.string().default('text-embedding-3-small'),
embedding_dimensions: z.number().int().min(64).max(4096).default(1536),

Modified file: src/types.ts

Add to Config.history:

embedding_model: string;
embedding_dimensions: number;

---

Step 8: Background backfill for existing memories

Modified file: src/providers/memory/memoryfs/provider.ts

After initialization, kick off a fire-and-forget backfill:

async function backfillEmbeddings(
  store: ItemsStore,
  embeddingStore: EmbeddingStore,
  client: EmbeddingClient,
  batchSize = 50,
) {
  if (!client.available) return;
  const unembedded = embeddingStore.listUnembedded(batchSize);
  if (unembedded.length === 0) return;

  const items = store.getByIds(unembedded);
  const vectors = await client.embed(items.map(i => i.content));
  for (let i = 0; i < items.length; i++) {
    embeddingStore.upsert(items[i].id, vectors[i]);
  }
}

• Runs once on provider creation (non-blocking)
• Processes in batches of 50
• Logs progress and errors
• Subsequent queries will have embeddings available

---

Step 9: Tests

New test files:

• tests/utils/embedding-client.test.ts — Client unit tests
• tests/providers/memory/memoryfs/embedding-store.test.ts — Vector store tests

Modified test files:

• tests/host/memory-recall.test.ts — Add embedding-based recall tests:
  • Embedding available → uses semantic search
  • Embedding unavailable → falls back to keywords
  • Mock embedding client
• tests/config-history.test.ts — Update default config assertion with new fields
• tests/providers/memory/memoryfs/ — Update provider tests:
  • Test write generates embedding when client available
  • Test query with embedding field uses vector search
  • Test graceful fallback when no embedding client

---

Step 10: Journal and lessons

Update .claude/journal/ and .claude/lessons/ per protocol before committing.

---

File Change Summary

File	Action	Description
package.json	modify	Add @dao-xyz/sqlite3-vec
src/utils/embedding-client.ts	new	Standalone OpenAI embedding client
src/providers/memory/memoryfs/embedding-store.ts	new	sqlite-vec vector store
src/providers/memory/types.ts	modify	Add optional embedding to MemoryQuery
src/providers/memory/memoryfs/provider.ts	modify	Integrate embedding on write/query
src/providers/memory/memoryfs/items-store.ts	modify	Add getByIds() batch method
src/host/memory-recall.ts	modify	Embed user message, pass to query, fallback
src/host/server-completions.ts	modify	Create embedding client, pass to recall
src/config.ts	modify	Add embedding_model, embedding_dimensions
src/types.ts	modify	Add embedding config to Config.history
tests/utils/embedding-client.test.ts	new	Embedding client unit tests
tests/providers/memory/memoryfs/embedding-store.test.ts	new	Vector store tests
tests/host/memory-recall.test.ts	modify	Add embedding recall tests
tests/config-history.test.ts	modify	Update default config assertion

---

Key Design Decisions

1. Separate EmbeddingClient utility (not extending LLMProvider) — embeddings are request/response, not streaming chat. Different API shape entirely.

2. Separate _vec.db file — avoids compatibility issues between the vec extension and the existing sync SQLite adapter. The vec extension needs to be loaded via @dao-xyz/sqlite3-vec, not the generic openDatabase().

3. MemoryQuery.embedding optional field (not new interface method) — backward compatible, all existing providers and callers continue to work unchanged.

4. FTS5/LIKE fallback — when OPENAI_API_KEY is absent, the system degrades to keyword search silently. No errors, no broken functionality.

5. Configurable model + dimensions — defaulting to text-embedding-3-small (1536 dims, $0.02/1M tokens). Can set to any OpenAI-compatible embedding endpoint via config.

6. Background backfill — existing memories get embeddings generated lazily after startup. Non-blocking, non-critical.

7. Similarity × salience scoring — vector distance maps to similarity = 1 / (1 + distance), then feeds into the existing salienceScore() formula. Combines semantic relevance with reinforcement frequency and recency.

8. EmbeddingClient created in two places — MemoryFS provider (for write-time embedding) and server-completions (for query-time embedding). Both are stateless OpenAI HTTP clients reading from the same env vars. This avoids threading the embedding client through the provider registry.