Search within specific document types

---
title: Search And Retrieval
audience: users
status: active
owner: product-education
last_reviewed: 2025-03-13
---

## Search & Retrieval Guide

> **Status**: Active  
> **Last Updated**: 2025-10-01
> **Purpose**: Complete guide to effective searching and information retrieval  
> **Audience**: Users who want to find information effectively

Master the art of finding exactly what you need with our AI-enhanced search system.
This guide covers everything from basic searches to advanced techniques that leverage
artificial intelligence to understand your intent and deliver better results.

## How Our Search Works

Our system uses multiple AI techniques working together to understand what you're looking for:

### 🧠 Intelligent Understanding

- **Semantic Search**: Understands meaning, not just keyword matching
- **HyDE Compatibility Mode**: Provides the historic HyDE entry point while delegating to the unified hybrid search pipeline
- **Smart Reranking**: Uses AI to reorder results by relevance
- **Multi-Stage Retrieval**: Searches broadly first, then refines for precision

### 🔍 What This Means For You

- **Find concepts, not just keywords**: Search for "database optimization" and find content about "performance tuning"
- **Natural language queries**: Ask questions like "how do I speed up searches?"
- **Better result quality**: Most relevant information appears first
- **Fewer irrelevant results**: AI filters out noise automatically

## Getting the Best Results

### Writing Effective Queries

**✅ Good Query Examples:**

```text
"how to optimize vector database performance"
"troubleshooting slow search responses"
"best practices for document indexing"
"web scraping rate limiting strategies"
```

**❌ Avoid These Patterns:**

```text
"database" (too vague)
"fix error" (no context)
"help" (not specific)
"documentation" (too broad)
```

### Query Writing Tips

| Technique                | Example                                      | Why It Works           |
| ------------------------ | -------------------------------------------- | ---------------------- |
| **Be specific**          | "configure OpenAI embeddings" vs "setup"     | Targets exact needs    |
| **Include context**      | "Python web scraping tutorial" vs "scraping" | Adds important details |
| **Use natural language** | "Why are my searches slow?"                  | AI understands intent  |
| **Mention the outcome**  | "reduce memory usage during indexing"        | Focuses on goals       |

### Search Strategies by Use Case

#### **📚 Research & Learning**

- Use question format: "What is the difference between dense and sparse vectors?"
- Include learning level: "beginner guide to vector databases"
- Ask for comparisons: "Redis vs DragonflyDB performance"

#### **🔧 Troubleshooting**

- Describe the problem: "search results are empty"
- Include error context: "timeout errors during large document indexing"
- Mention what you tried: "increased chunk size but still getting errors"

#### **⚙️ Configuration**

- Be specific about settings: "configure HNSW parameters for accuracy"
- Mention your environment: "production deployment configuration"
- Include constraints: "optimize for cost with minimal accuracy loss"

## Search Features Explained

### HyDE Enhancement (Compatibility Mode)

**What it does now**: The `hyde_search` MCP tool is retained for backward
compatibility, but under the hood it simply calls the consolidated hybrid search
path provided by `VectorStoreService`. No additional hypothetical documents are
generated in Phase 3.

**When to use it**: Reach for `hyde_search` when you prefer the previous API
contract or want a named tool for hybrid search. For new workflows, the default
`search_documents` or `filtered_search` tools are equivalent in capability.

**User experience**: Expect faster responses with the same query semantics—the
result list mirrors the unified hybrid search pipeline.

### Semantic Reranking (Automatic)

**What it does**: After finding potential matches, AI re-examines each result to determine true relevance to your query.

**When it helps most**:

- Eliminating false positives
- Finding nuanced matches
- Prioritizing comprehensive answers
- Filtering out tangentially related content

**User experience**: The first few results will be significantly more relevant than
traditional keyword-based searches.

### Multi-Stage Retrieval (Automatic)

**What it does**: First finds many potential matches quickly, then applies sophisticated filtering to identify the best ones.

**Benefits for users**:

- Faster initial response
- Higher precision in final results
- Balances speed with accuracy
- Handles large document collections efficiently

### Retrieval modes (Configurable)

**What it does**: Controls which embedding signals power your search.

- `dense` – Uses FastEmbed dense vectors for semantic similarity.
- `sparse` – Uses FastEmbedSparse BM25-style signals for lexical precision.
- `hybrid` (default) – Combines both and lets Qdrant fuse scores during ranking.

**How to use it**:

- REST/MCP payloads: set `search_strategy` (REST) or `retrieval_mode` (config/CLI) to `dense`, `sparse`, or `hybrid`.
- CLI: pass `--retrieval-mode` to `mcp search`, `mcp add-documents`, or `mcp add-documents-batch`.

**User experience**: Hybrid usually provides the strongest relevance, while `dense` can reduce latency for short-term experiments and `sparse` helps when chasing exact keyword matches.

## Performance Expectations

### Search Response Times

Our enhanced database connection pooling delivers significantly improved performance:

- **Simple queries**: < 250ms (50.9% faster than previous versions)
- **Complex semantic searches**: 500ms-1.5 seconds (improved from 1-3 seconds)
- **HyDE-enhanced queries**: 1-2.5 seconds (improved from 2-5 seconds)
- **Large result sets**: 1.5-3.5 seconds (improved from 3-7 seconds)
- **Peak load performance**: Up to 887.9% higher throughput during busy periods

### Result Quality Indicators

- **High relevance**: First 3-5 results directly address your query
- **Good coverage**: Results from multiple relevant documents
- **Varied perspectives**: Different approaches to the same topic
- **Recent information**: When available, newer content is prioritized
- **Consistent performance**: Intelligent connection pooling maintains fast response times even during peak usage

### Automatic Performance Optimization

The system now includes smart database optimization that:

- **Scales automatically**: Connection pools adjust based on your usage patterns
- **Monitors performance**: Tracks query speeds and optimizes accordingly
- **Prevents bottlenecks**: Intelligent load balancing prevents slowdowns
- **Recovers gracefully**: Circuit breaker patterns handle temporary issues
- **No configuration needed**: Works optimally out of the box

## Advanced Search Techniques

### Using Search Filters

```text
# Search within specific document types
query: "API documentation" filter: type=reference

# Search recent content only
query: "deployment strategies" filter: date>2024-01-01

# Search specific projects or collections
query: "configuration" filter: project=production
```

### Federated Search (Cross-Collection)

**New in this version**: Search across multiple collections simultaneously for comprehensive results.

```text
# Search across all available collections
query: "microservices architecture" collections: all

# Search specific collections with ranking
query: "API authentication" collections: documentation,examples,tutorials

# Weighted search across collections
query: "performance optimization" weights: documentation=0.5,benchmarks=0.8,examples=0.3
```

**Benefits**:

- **Comprehensive coverage**: Find information across all your data sources
- **Intelligent ranking**: Results are ranked across collections for relevance
- **Adaptive search modes**: System chooses optimal search strategy per collection
- **Result fusion**: Advanced algorithms combine results from multiple sources

### Combining Search Types

- **Broad + Specific**: Start broad ("web scraping"), then narrow ("browser automation headless mode")
- **General + Technical**: Begin with concepts ("caching"), then specifics ("DragonflyDB configuration")
- **Problem + Solution**: Search the issue first, then look for implementation approaches

### Power User Tips

#### **🎯 Precision Searching**

- Use quotes for exact phrases: `"error code 500"`
- Include specific versions: `"Python 3.13 compatibility"`
- Mention specific tools: `"Qdrant collection aliases"`

#### **🔄 Iterative Refinement**

1. Start with a broad search
2. Review top results for better terminology
3. Refine your query using discovered keywords
4. Repeat until you find exactly what you need

#### **📋 Research Workflows**

- Search for overviews first: `"system architecture overview"`
- Then dive into specifics: `"client management implementation"`
- Cross-reference with examples: `"client management code examples"`

## Troubleshooting Search Issues

### No Results Found

**Possible causes:**

- Query too specific or uses uncommon terminology
- Documents not indexed yet
- Spelling errors in technical terms

**Solutions:**

1. Try broader, more general terms
2. Check if documents are in the system: use collection listing
3. Use alternative terminology: "config" instead of "configuration"
4. Break complex queries into simpler parts

### Poor Result Quality

**Symptoms:**

- Irrelevant results in top positions
- Missing obvious matches
- Results don't match query intent

**Solutions:**

1. Rephrase query with more context
2. Use more specific terminology
3. Try question format instead of keywords
4. Add context about your use case

### Slow Search Performance

**Common causes:**

- Very broad queries requiring extensive processing
- Large result sets being processed
- Complex semantic analysis taking time

**Optimizations:**

1. Be more specific to reduce search space
2. Limit result count if you don't need many matches
3. Use simpler queries for exploratory searches
4. Consider breaking complex queries into parts

## Best Practices by User Type

### Content Researchers

- **Start broad, narrow down**: Begin with general topics, refine based on results
- **Use natural questions**: "What are the benefits of vector indexing?"
- **Follow result threads**: Use results to discover new terminology and search paths
- **Save effective queries**: Keep track of query patterns that work well

### Technical Users

- **Include technical context**: Specify versions, tools, and environments
- **Search for patterns**: Look for implementation approaches and best practices
- **Cross-reference**: Validate findings with multiple sources
- **Use troubleshooting format**: "error X when doing Y in context Z"

### Business Users

- **Focus on outcomes**: "improve search performance" vs "optimize HNSW parameters"
- **Ask impact questions**: "What's the ROI of implementing HyDE enhancement?"
- **Search for summaries**: Look for executive summaries and overview content
- **Include constraints**: "cost-effective solutions" or "minimal maintenance overhead"

## Getting Help with Search

### When Search Isn't Working

1. **Check system status**: Ensure all services are running
2. **Try basic queries**: Test with simple, known terms
3. **Review error messages**: Look for specific error guidance
4. **Check examples**: Use working queries as templates

### Improving Your Search Skills

- **Study successful queries**: Note what works well
- **Learn from results**: Use found content to improve terminology
- **Practice iteration**: Get comfortable refining searches
- **Ask specific questions**: Better queries get better answers

### Additional Resources

- **[Examples & Recipes](./examples-and-recipes.md)**: Real-world search scenarios
- **[Troubleshooting](./troubleshooting.md)**: Common issues and solutions
- **[Web Scraping](./web-scraping.md)**: Finding and extracting content from the web
- **Developer Resources**: See [../developers/index.md](../developers/index.md) for API integration

---

_🔍 Master these techniques and you'll find exactly what you need, faster and more accurately than ever before._