Implementation Plan: codeindex

# Implementation Plan: codeindex

## Context

We're building a local semantic index for codebases that augments Claude Code's Glob/Grep/Read tools with embedding-based search. The design is informed by OpenViking (AST skeletons, bottom-up directory summaries, score propagation), Augment Context Services (single retrieval tool for agents), sigma-ralph-grindset (`claude --print` as LLM backend), and db-harness (Bun-native dual-database skill pattern).

**Spec**: `/Users/russfugal/repo/codeindex/SPEC.md`

---

## Implementation Order (18 atomic commits)

### Phase 1: Foundation (sequential, single agent)

| # | Commit | Files | Notes |
|---|--------|-------|-------|
| 1 | Project scaffolding | `package.json`, `tsconfig.json` | Bun project, `web-tree-sitter` + `openai` + `ignore` deps. Use `web-tree-sitter` (WASM) because native `tree-sitter` N-API has known Bun gaps. |
| 2 | Type definitions | `src/search/types.ts` | All interfaces: `SearchResult`, `SearchOptions`, `CodeindexConfig`, `ScoringConfig`, `RepoRecord`, `FileRecord`, `DirectoryRecord`, `CommitRecord`, `FileCommitLink` |
| 3 | Config module | `src/config.ts` | Load `~/.config/codeindex/config.json` + `.codeindex.json`, merge with defaults. Formatter auto-detection via `Bun.file().exists()` checks in priority order. |
| 4 | PostgreSQL connection | `src/db/pg.ts` | `import { SQL } from "bun"` + `new SQL({hostname, port, database, username, max})` — exact db-harness pattern from `/Users/russfugal/ObsidianVault/tPoSO/db-harness/src/pg.ts` |
| 5 | SQLite connection | `src/db/sqlite.ts` | **Must use `import { Database } from "bun:sqlite"`** (not `SQL` from `"bun"`) because we need `db.loadExtension()` for sqlite-vec. The `SQL` class doesn't expose loadExtension. Wrap sync calls in `Promise.resolve()` for API parity with pg. Provide `.unsafe(sql, params)` matching db-harness `enl.unsafe()` pattern. |
| 6 | Schema creation | `src/db/schema.ts` | `ensurePgSchema()`: CREATE EXTENSION vector + all 5 tables with `vector(1536)` columns. `ensureSqliteSchema()`: all 5 tables + `vec0` virtual tables for embeddings (sqlite-vec stores vectors in separate virtual tables joined by ID). |

### Phase 2: Indexing pipeline (parallel after foundation)

| # | Commit | Files | Teammate | Notes |
|---|--------|-------|----------|-------|
| 7 | File walker | `src/index/walker.ts`, `.indexignore.example` | walker-agent | `ignore` npm package for .gitignore/.indexignore. `walkRepo()` async generator yields relative paths. |
| 8 | AST skeleton extraction | `src/index/skeleton.ts`, `scripts/fetch-grammars.ts` | skeleton-agent | Port OpenViking's per-language extractors (see `https://github.com/volcengine/OpenViking/tree/main/openviking/parse/parsers/code/ast`) to TS. `web-tree-sitter` WASM init via `Parser.init()`. Grammars from `tree-sitter-wasms` npm package or built via `tree-sitter build --wasm`. Extension map: `.ts`→typescript, `.tsx`→tsx, `.js`/`.jsx`→javascript, `.py`→python, `.rs`→rust, `.go`→go, `.java`→java, `.c`→c, `.cpp`/`.h`/`.hpp`→cpp, `.cs`→c_sharp. Non-code fallback: first N lines. |
| 9 | Formatter piping | `src/index/formatter.ts` | formatter-agent | `Bun.spawn(cmd.split(" "), { stdin: new Response(content) })` to pipe file through formatter in memory, then `new Bun.CryptoHasher("sha256")` for hash. Never writes to disk. |
| 10 | Embedding module | `src/index/embedder.ts` | embedder-agent | OpenAI SDK: `client.embeddings.create({ model, input, dimensions })`. Batch support (up to 2048 inputs). Retry with exponential backoff for rate limits. |
| 11 | Git commit extraction | `src/index/commits.ts` | commits-agent | `Bun.spawn(["git", "log", ...])` for per-file commit history + messages. Also: `getRepoOrigin()`, `getRepoName()`, `getChangedFiles()`. |
| 12 | Directory summaries | `src/index/directories.ts` | directories-agent | **Blocked on 7-11.** Bottom-up DAG: leaf dirs first, parents next. Two embeddings per dir: (a) concat of immediate child skeletons (non-recursive), (b) `claude --print --model haiku --output-format json --json-schema '...'` summary of concat + extracted docs + recursive child summaries. Spawn pattern from sigma-ralph-grindset lines 422-446: `Bun.spawn(["claude","--print","--model","haiku",...], { stdin: new Response(prompt), stdout: "pipe", env: {...process.env, CLAUDECODE: ""} })`. Graceful fallback if `claude` CLI unavailable (leave summary null). |

### Phase 3: Surface layer (parallel after indexing)

| # | Commit | Files | Teammate | Notes |
|---|--------|-------|----------|-------|
| 13 | Search query engine | `src/search/query.ts` | search-agent | Scoring algorithm from spec. pg uses `<=>` operator via `pg.unsafe()` with `$1::vector`. SQLite uses `vec_distance_cosine()` JOIN against vec0 virtual table. Returns all results >= minScore, optional topN cap. |
| 14 | Git hook installer | `src/hooks/post-commit.ts` | hooks-agent | Writes `.git/hooks/post-commit`, chmod +x. Appends if existing hook present. |
| 15 | pg→sqlite export | `src/db/export.ts` | export-agent | Read all rows for repoId from pg, insert into new SQLite db. Convert vector arrays to sqlite-vec format. |
| 16 | CLI entry point | `src/index.ts` | cli-agent | **Blocked on 13.** `#!/usr/bin/env bun` shebang. `process.argv` dispatch (no CLI framework). Commands: reindex, update, search, export, install-hook, config, status. Each handler orchestrates pipeline modules. Pattern: db-harness `src/index.ts` unified export object. |

---

## Agent Team Strategy

### Team 1: `codeindex-foundation` (before commit 1)

```
Create an agent team for the codeindex project foundation.
One teammate builds the project scaffolding, types, config, and database layers
sequentially (commits 1-6). Each commit is atomic. Follow the Bun SQL patterns
from /Users/russfugal/ObsidianVault/tPoSO/db-harness/src/ exactly.
The spec is at /Users/russfugal/repo/codeindex/SPEC.md.
Use Sonnet for the teammate.
```

**Coordination**: Single agent, sequential. Must complete before Team 2 starts.

### Team 2: `codeindex-indexing` (after commit 6 lands)

```
Create an agent team for the codeindex indexing pipeline. 5 parallel teammates:
- walker-agent: file tree walker with gitignore/indexignore support (commit 7)
- skeleton-agent: tree-sitter WASM AST skeleton extraction (commit 8)
- formatter-agent: format-in-memory and hash (commit 9)
- embedder-agent: OpenAI text-embedding-3-small (commit 10)
- commits-agent: git commit history extraction (commit 11)
After all 5 complete, I will build directories.ts (commit 12) which depends on all of them.
The spec is at /Users/russfugal/repo/codeindex/SPEC.md.
Use Sonnet for each teammate.
```

**Coordination**: 5-way parallel for commits 7-11. Lead builds commit 12 after all land (it imports from all 5 modules).

### Team 3: `codeindex-surface` (after commit 12 lands)

```
Create an agent team for the codeindex surface layer. 3 parallel teammates:
- search-agent: search/query.ts scoring engine with pgvector and sqlite-vec (commit 13)
- hooks-agent: git post-commit hook installer (commit 14)
- export-agent: pg-to-sqlite export (commit 15)
After search-agent completes, I will build the CLI entry point (commit 16).
The spec is at /Users/russfugal/repo/codeindex/SPEC.md.
Use Sonnet for each teammate.
```

**Coordination**: 3-way parallel for commits 13-15. Lead builds commit 16 after 13 lands.

---

## Key Patterns to Reuse

| Pattern | Source | Files |
|---------|--------|-------|
| Bun SQL PostgreSQL | db-harness | `/Users/russfugal/ObsidianVault/tPoSO/db-harness/src/pg.ts`, `config.ts` |
| Bun SQL tagged templates + `unsafe()` | db-harness | `/Users/russfugal/ObsidianVault/tPoSO/db-harness/src/articles.ts`, `neighbors.ts` |
| Unified `db` export object | db-harness | `/Users/russfugal/ObsidianVault/tPoSO/db-harness/src/index.ts` |
| `claude --print` spawn | sigma-ralph-grindset | `/Users/russfugal/repo/sigma-ralph-grindset/src/audit/cli-backend.ts:422-446` |
| JSON schema enforcement | sigma-ralph-grindset | `/Users/russfugal/repo/sigma-ralph-grindset/src/audit/schema.ts` |
| AST extraction per language | OpenViking | `https://github.com/volcengine/OpenViking/tree/main/openviking/parse/parsers/code/ast` |
| Bottom-up directory DAG | OpenViking | `https://github.com/volcengine/OpenViking/blob/main/openviking/storage/queuefs/semantic_dag.py` |

---

## Known Risks

1. **sqlite-vec + Bun**: `bun:sqlite` Database supports `loadExtension()`, but macOS system SQLite may not support extensions. May need `Database.setCustomSQLite()` pointing to Homebrew SQLite, or the `sqlite-vec` npm package may bundle its own. Test early in commit 5.
2. **web-tree-sitter WASM in Bun**: `Parser.init()` may need explicit `locateFile` callback. Test early in commit 8.
3. **pgvector `<=>` in tagged templates**: Must use `pg.unsafe()` (not tagged template) for queries with the `<=>` operator since it contains `<`/`>`. db-harness already demonstrates this working.
4. **claude CLI availability**: `directories.ts` gracefully falls back to null summary if `claude` not installed/authenticated.

---

## Verification

After all 16 commits land:

1. **Setup**: `bun install` in `/Users/russfugal/repo/codeindex/`
2. **PostgreSQL**: `createdb codeindex && psql -d codeindex -c "CREATE EXTENSION IF NOT EXISTS vector"`
3. **Full reindex**: `bun src/index.ts reindex --path /Users/russfugal/repo/codeindex` (index self)
4. **Search**: `bun src/index.ts search "AST skeleton extraction" --pretty`
   - Should return `src/index/skeleton.ts` with high score
   - Should return `src/index/directories.ts` (uses skeletons)
   - Commit results should surface commits mentioning "skeleton"
5. **Cross-repo**: `bun src/index.ts reindex --path /Users/russfugal/repo/sigma-ralph-grindset`, then `bun src/index.ts search "audit code" --scope all` — should return results from both repos with `inProject: false` for sigma-ralph-grindset
6. **Export**: `bun src/index.ts export --out test.db` — verify SQLite file created with data
7. **Hook**: `bun src/index.ts install-hook` in a test repo, make a commit, verify index updated
8. **Status**: `bun src/index.ts status` — shows file count, last indexed time