Contextinator v2.0 - Complete Usage Guide

This guide is divided into two main sections:

• v2 Core Features (Rust-Powered) - Fast filesystem tools for AI agents (zero dependencies)
• RAG Features (Optional) - Semantic code search and embeddings (requires pip install contextinator[rag])

---

v2 Core Features (Rust-Powered)

The core v2 functionality provides blazing-fast filesystem operations built in Rust. No additional dependencies required.

Installation

# Core installation (Rust-powered tools only)
pip install contextinator

# Verify installation
contextinator --help
contextinator version

CLI Usage - v2 Commands

All v2 commands use the read subcommand with different modes:

contextinator read --path <path> --mode <mode> [options]

1. Line Mode - Read Files with Line Ranges

Read specific lines from a file, with support for negative indexing.

# Read lines 10-50
contextinator read --path myfile.py --mode Line --start-line 10 --end-line 50

# Read last 10 lines (negative indexing)
contextinator read --path myfile.py --mode Line --start-line -10 --end-line -1

# Read first 20 lines
contextinator read --path myfile.py --mode Line --start-line 1 --end-line 20

# Read entire file (omit line numbers)
contextinator read --path myfile.py --mode Line

# JSON output
contextinator read --path myfile.py --mode Line --start-line 1 --end-line 50 --format json

Options:

• --path (required): File path to read
• --mode Line (required): Line reading mode
• --start-line: Starting line number (supports negative indexing)
• --end-line: Ending line number (supports negative indexing)
• --format: Output format (text or json, default: text)

Output Format (JSON):

{
  "type": "line",
  "content": "file contents here...",
  "total_lines": 100,
  "lines_returned": 41
}

---

2. Directory Mode - List Files and Folders

List directory contents with optional recursive traversal.

# Non-recursive listing (depth 0)
contextinator read --path src/ --mode Directory --depth 0

# Recursive listing (depth 2)
contextinator read --path src/ --mode Directory --depth 2

# Fully recursive
contextinator read --path . --mode Directory --depth 999

# JSON output for AI agents
contextinator read --path src/ --mode Directory --depth 1 --format json

Options:

• --path (required): Directory path to list
• --mode Directory (required): Directory listing mode
• --depth: Recursion depth (0 = non-recursive, default: 0)
• --format: Output format (text or json, default: text)

Output Format (JSON):

{
  "type": "directory",
  "entries": [
    {
      "path": "src/main.py",
      "is_dir": false,
      "size": 2048,
      "modified": 1234567890
    },
    {
      "path": "src/utils/",
      "is_dir": true,
      "size": 0,
      "modified": 1234567890
    }
  ],
  "total_count": 15
}

---

3. Search Mode - Pattern Matching with Context

Search for text patterns in files with configurable context lines.

# Find TODOs with context
contextinator read --path . --mode Search --pattern "TODO" --context-lines 2

# Find function definitions
contextinator read --path src/ --mode Search --pattern "def " --context-lines 5

# Find FIXMEs and export to JSON
contextinator read --path . --mode Search --pattern "FIXME" --format json

# Search in specific directory
contextinator read --path src/components/ --mode Search --pattern "useState" --context-lines 3

Options:

• --path (required): Directory or file path to search
• --mode Search (required): Search mode
• --pattern (required): Text pattern to search for
• --context-lines: Number of context lines before/after match (default: 2)
• --format: Output format (text or json, default: text)

Output Format (JSON):

{
  "type": "search",
  "matches": [
    {
      "file_path": "src/auth.py",
      "line_number": 42,
      "line_content": "    # TODO: implement rate limiting",
      "context_before": [
        "def authenticate_user(username, password):",
        "    \"\"\"Authenticate a user.\"\"\""
      ],
      "context_after": ["    user = get_user(username)", "    if not user:"]
    }
  ],
  "total_matches": 12
}

---

Python API - v2 Functions

Import and use the Rust-powered fs_read function directly in your code:

from contextinator import fs_read

# 1. Read file lines
result = fs_read(
    path="myfile.py",
    mode="Line",
    start_line=10,
    end_line=50
)
print(result["content"])
print(f"Lines: {result['lines_returned']}/{result['total_lines']}")

# 2. Read last 10 lines
result = fs_read(
    path="myfile.py",
    mode="Line",
    start_line=-10,
    end_line=-1
)

# 3. List directory (non-recursive)
result = fs_read(
    path="src/",
    mode="Directory",
    depth=0
)
for entry in result["entries"]:
    print(f"{'📁' if entry['is_dir'] else '📄'} {entry['path']}")

# 4. List directory (recursive, depth 2)
result = fs_read(
    path="src/",
    mode="Directory",
    depth=2
)

# 5. Search for patterns
result = fs_read(
    path=".",
    mode="Search",
    pattern="TODO",
    context_lines=2
)
for match in result["matches"]:
    print(f"{match['file_path']}:{match['line_number']}")
    print(f"  {match['line_content']}")

Function Signature

def fs_read(
    path: str,
    mode: str,  # "Line", "Directory", or "Search"
    start_line: Optional[int] = None,  # Line mode only
    end_line: Optional[int] = None,     # Line mode only
    depth: Optional[int] = None,        # Directory mode only (default: 0)
    pattern: Optional[str] = None,      # Search mode only
    context_lines: Optional[int] = None # Search mode only (default: 2)
) -> dict

---

Use Cases for v2 Core

AI Agent Integration

# Agent reads specific code sections
result = fs_read("src/auth.py", mode="Line", start_line=1, end_line=100)
agent_context = result["content"]

# Agent explores codebase structure
result = fs_read("src/", mode="Directory", depth=2)
file_tree = result["entries"]

# Agent finds TODOs for task planning
result = fs_read(".", mode="Search", pattern="TODO", context_lines=3)
tasks = result["matches"]

Batch File Operations

files_to_read = ["src/main.py", "src/auth.py", "src/api.py"]

for file_path in files_to_read:
    result = fs_read(file_path, mode="Line")
    process_file_content(result["content"])

---

RAG Features (Optional)

RAG Features (Optional)

For semantic code search, AST-powered chunking, and vector embeddings, install the RAG extras:

pip install contextinator[rag]

RAG Installation & Setup

Prerequisites:

• Docker (for ChromaDB)
• OpenAI API key (for embeddings)

Install with RAG extras:

pip install contextinator[rag]

For development:

# Clone and setup
git clone https://github.com/starthackHQ/Contextinator.git
cd Contextinator

# Create virtual environment
python -m venv .venv

# Activate environment
# Windows
.venv\Scripts\activate
# macOS/Linux
source .venv/bin/activate

# Install with RAG dependencies
pip install -e ".[rag]"

Configure environment variables:

Create a .env file:

OPENAI_API_KEY=sk-your-openai-key-here
USE_CHROMA_SERVER=true
CHROMA_SERVER_URL=http://localhost:8000

Start ChromaDB server:

docker-compose up -d

---

CLI Usage - RAG Commands

All RAG commands use the --rag flag:

contextinator --rag <command> [options]

1. Chunking

Extract semantic chunks from code using AST parsing:

contextinator --rag chunk --save --path <repo-path> --output <output-dir>
contextinator --rag chunk --save --repo-url <github-url>
contextinator --rag chunk --save-ast  # Save AST trees for debugging
contextinator --rag chunk --chunks-dir <custom-dir>  # Custom chunks directory

2. Embedding

Generate embeddings for code chunks (requires OpenAI API key):

contextinator --rag embed --save --path <repo-path> --output <output-dir>
contextinator --rag embed --save --repo-url <github-url>
contextinator --rag embed --chunks-dir <custom-dir> --embeddings-dir <custom-dir>

3. Storing in Vector Store

Note: Make sure ChromaDB server is running: docker-compose up -d

contextinator --rag store-embeddings --path <repo-path> --output <output-dir>
contextinator --rag store-embeddings --collection-name <custom-name>
contextinator --rag store-embeddings --repo-name <repo-name> --collection-name <custom-name>
contextinator --rag store-embeddings --embeddings-dir <custom-dir> --chromadb-dir <custom-dir>

4. Search Commands

Contextinator RAG provides multiple search methods for different use cases:

4.1 Semantic Search (Natural Language)

Find code using natural language queries. Uses AI embeddings for semantic similarity.

# Basic semantic search
contextinator --rag search "authentication logic" --collection MyRepo

# With filters
contextinator --rag search "error handling" -c MyRepo --language python -n 10

# Include parent chunks (classes/modules) in results
contextinator --rag search "database queries" -c MyRepo --include-parents

# Filter by file path
contextinator --rag search "API endpoints" -c MyRepo --file "src/api/"

# Filter by node type
contextinator --rag search "validation logic" -c MyRepo --type function_definition

# Export to JSON
contextinator --rag search "authentication" -c MyRepo --json results.json

# Export to TOON format (40-60% token savings for LLMs)
contextinator --rag search "authentication" -c MyRepo --toon results.json results
- `--json`: Export results to JSON file
- `--toon`: Export results to TOON file (compact format for LLMs)

---

4.2 Symbol Search (Exact Name Match)

Find specific functions or classes by name.

# Find function by name
contextinator --rag symbol authenticate_user --collection MyRepo

# Find class by name
contextinator --rag symbol UserManager -c MyRepo --type class_definition

# Search in specific file
contextinator --rag symbol get_user -c MyRepo --file "api/"

# Export results
contextinator --rag symbol main -c MyRepo --json main_functions.json

---

4.3 Pattern Search (Text/Regex)

Search for specific text patterns in code.

# Find TODOs
python -m src.contextinator.cli pattern "TODO" --collection MyRepo

# Find import statements
python -m src.contextinator.cli pattern "import requests" -c MyRepo --language python

# Find async functions
python -m src.contextinator.cli pattern "async def" -c MyRepo --file "api/"

# Find FIXMEs and export
python -m src.contextinator.cli pattern "FIXME" -c MyRepo --toon fixmes.json

**Options:**
- `-c, --collection` (required): Collection name
- `-l, --language`: Filter by programming language
- `-f, --file`: Filter by file path
- `-t, --type`: Filter by node type
- `--limit`: Maximum results (default: 50)
- `--json`: Export to JSON
- `--toon`: Export to TOON format

---

4.4 Advanced Search (Hybrid)

Combine semantic search, pattern matching, and filters for precise results.

# Semantic search with language filter
contextinator --rag search-advanced -c MyRepo \
 --semantic "authentication" --language python

# Pattern search with file filter
contextinator --rag search-advanced -c MyRepo \
 --pattern "TODO" --file "src/"

# Hybrid: semantic + pattern + type filter
contextinator --rag search-advanced -c MyRepo \
 --semantic "error handling" --pattern "try" --type function_definition

# Multiple filters with export
contextinator --rag search-advanced -c MyRepo \
 --semantic "API routes" --language python --file "api/" --toon api_routes.json

---

4.5 Read File (Reconstruct from Chunks)

Reconstruct and display a complete file from its chunks.

# Read complete file
python -m src.contextinator.cli read-file "src/auth.py" --collection MyRepo

# Show chunks separately (don't join)
python -m src.contextinator.cli read-file "src/api/routes.py" -c MyRepo --no-join

# Export to JSON
python -m src.contextinator.cli read-file "src/main.py" -c MyRepo --json main.json

**Options:**
- `-c, --collection` (required): Collection name
- `--no-join`: Show chunks separately instead of joining them
contextinator --rag read-file "src/auth.py" --collection MyRepo

# Show chunks separately (don't join)
contextinator --rag read-file "src/api/routes.py" -c MyRepo --no-join

# Export to JSON
contextinator --rag read-file "src/main.py" -c MyRepo --json main.json
python -m src.contextinator.cli search "authentication" -c MyRepo --json results.json

Output structure:
json
{
 "query": "authentication",
 "collection": "MyRepo",
 "total_results": 5,
 "results": [
   {
     "id": "chunk_0_12345",
     "content": "def authenticate_user(username, password):\n    ...",
     "metadata": {
       "file_path": "src/auth.py",
       "language": "python",
       "node_type": "function_definition",
       "node_name": "authenticate_user",
       "start_line": 10,
       "end_line": 25
     },
     "cosine_similarity": 0.89
   }
 ]
}

TOON Format (Token-Optimized)

Compact format designed for LLM prompts. Saves 40-60% tokens compared to JSON.

contextinator --rag search "authentication" -c MyRepo --toon results.json

Perfect for:

• Feeding search results to LLMs
• Building RAG (Retrieval-Augmented Generation) systems
• Minimizing token usage in AI workflows

---

6. Database Management

# Show database statistics
contextinator --rag db-info

# List all collections
contextinator --rag db-list

# Show collection details with sample documents
contextinator --rag db-show MyRepo --sample 3

# Delete a collection
contextinator --rag db-clear MyRepo

# Use custom ChromaDB location
contextinator --rag db-info --chromadb-dir <custom-dir>

# Use specific repo database
contextinator --rag db-info --repo-name MyRepo

---

📦 RAG Programmatic Usage (As a Library)

You can also import and use Contextinator RAG features directly in your Python code:

from contextinator.rag import (
    chunk_repository,
    embed_chunks,
    store_repository_embeddings,
    semantic_search,
    symbol_search,
    read_file,
)

# 1. Chunk a repository
chunks = chunk_repository(
    repo_path="./my-project",
    repo_name="MyProject",
    save=True,
    output_dir="./output"
)
print(f"Created {len(chunks)} chunks")

# 2. Generate embeddings
embeddings = embed_chunks(
    base_dir="./output",
    repo_name="MyProject",
    save=True
)

# 3. Store in vector database
stats = store_repository_embeddings(
    base_dir="./output",
    repo_name="MyProject",
    embedded_chunks=embeddings,
    collection_name="MyProject"
)
print(f"Stored {stats['stored_count']} embeddings")

# 4. Search semantically
results = semantic_search(
    collection_name="MyProject",
    query="authentication logic",
    n_results=5
)

for result in results:
    print(f"File: {result['metadata']['file_path']}")
    print(f"Code: {result['content'][:200]}...")
    print(f"Similarity: {result.get('cosine_similarity', 'N/A')}")
    print("-" * 80)

# 5. Search by symbol name
functions = symbol_search(
    collection_name="MyProject",
    symbol_name="authenticate_user"
)

# 6. Read entire file from chunks
file_content = read_file(
    collection_name="MyProject",
    file_path="src/auth.py"
)
print(file_content['content'])

---

📚 Quick Reference

Common Workflows

1. Index a GitHub repository:

contextinator chunk-embed-store-embeddings \
 --repo-url https://github.com/user/repo \
 --save \
 --collection-name MyRepo

2. Search for specific functionality:

# Natural language search
contextinator search "how is authentication handled" -c MyRepo

# Find specific function
contextinator symbol authenticate_user -c MyRepo

# Find TODOs
contextinator pattern "TODO" -c MyRepo

3. Advanced filtered search:

python -m src.contextinator.cli search-advanced -c MyRepo \
 --semantic "error handling" \
 --language python \
 --file "src/" \
 --toon error_handling.json

4. Export for LLM context:

# Get authentication-related code in token-efficient format
python -m src.contextinator.cli search "authentication and authorization" \
 -c MyRepo \
 --include-parents \
 --toon auth_context.json

All Commands

Command	Purpose	Example
chunk	Extract semantic chunks from code	chunk --repo-url <url> --save
embed	Generate embeddings for chunks	embed --repo-url <url> --save
store-embeddings	Store embeddings in ChromaDB	store-embeddings --repo-name MyRepo
chunk-embed-store-embeddings	Full pipeline (all-in-one)	chunk-embed-store-embeddings --repo-url <url> --save
search	Semantic search (natural language)	search "query" -c MyRepo
symbol	Find functions/classes by name	symbol function_name -c MyRepo
pattern	Text/regex search	pattern "TODO" -c MyRepo
search-advanced	Hybrid search with filters	search-advanced -c MyRepo --semantic "query" --language python
read-file	Reconstruct file from chunks	read-file "path/to/file.py" -c MyRepo
db-info	Show database statistics	db-info
db-list	List all collections	db-list
db-show	Show collection details	db-show MyRepo --sample 3
db-clear	Delete a collection	db-clear MyRepo