Introduction to Agentic Decision Tree RAG Systems

Traditional Retrieval-Augmented Generation (RAG) systems often struggle with complex queries, leading to irrelevant results or hallucinations. An agentic decision tree RAG system addresses these limitations by incorporating intelligent query routing, self-checking mechanisms, and iterative refinement processes. This approach mimics human decision-making, dynamically selecting the best retrieval strategy based on query type and context.

By leveraging tools like LangGraph for workflow orchestration and LlamaIndex for advanced indexing, you can build a system that routes queries through specialized paths—such as keyword search, semantic search, or hybrid methods—and verifies outputs for accuracy. This results in more reliable, context-aware responses, especially for enterprise knowledge bases or multi-domain applications.

In this guide, we'll walk through constructing such a system step by step. Expect to handle real-world scenarios like ambiguous queries or noisy data, with practical code examples using Python, Ollama for local LLMs, and ChromaDB for vector storage.

Key Components Explained

Intelligent Query Router

This acts as the system's brain, classifying incoming queries and directing them to appropriate retrievers. For instance:

• Simple factual queries → Keyword-based search.
• Complex analytical queries → Semantic or hybrid retrieval.
• Conversational follow-ups → Context-aware reranking.

The router uses a lightweight LLM prompt to categorize queries, reducing latency compared to full-model inference.

Self-Checking Mechanism

After retrieval, the system evaluates response quality using criteria like relevance, completeness, and factual consistency. If issues are detected, it triggers corrections without user intervention.

Iterative Refinement Loop

For suboptimal results, the agent refines queries, re-retrieves, or synthesizes from multiple sources. This loop ensures convergence on high-quality answers.

These components form a decision tree graph, where nodes represent actions (e.g., retrieve, check, refine) and edges are conditional branches.

Prerequisites and Environment Setup

Before diving in, ensure you have:

• Python 3.10+.
• Ollama installed with Llama 3.1 model: ollama pull llama3.1.
• ChromaDB for persistent vector storage.

Install dependencies via pip:

pip install langgraph llama-index llama-index-embeddings-ollama llama-index-llms-ollama llama-index-vector-stores-chroma chromadb

For the complete codebase, check out the project repository.

Set up Ollama embeddings and LLM:

import os
os.environ["OLLAMA_HOST"] = "localhost:11434"

from llama_index.core import Settings
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding

Settings.llm = Ollama(model="llama3.1", request_timeout=60.0)
Settings.embed_model = OllamaEmbedding(model_name="llama3.1")

This configuration enables local, privacy-focused inference.

Step 1: Data Ingestion and Indexing

Start by preparing your knowledge base. Load documents (PDFs, Markdown, etc.) and create a multi-index setup for flexibility.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb

# Load documents
documents = SimpleDirectoryReader("data/").load_data()

# Set up ChromaDB
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("agentic_rag")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Create index
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

This creates a persistent vector index. For production, consider metadata filtering to segment data by topic.

Add supplementary indexes:

• Keyword Index: For exact matches using BM25.
• Summary Index: For high-level overviews.

keyword_index = VectorStoreIndex.from_documents(documents, ...)  # Customize for keyword
summary_index = VectorStoreIndex.from_documents(documents, ...)

Step 2: Defining the Decision Tree Nodes

Using LangGraph, model the workflow as a stateful graph. Key nodes include:

• Router Node: Classifies query.
• Retriever Nodes: KeywordRetriever, SemanticRetriever, HybridRetriever.
• Checker Node: Validates response.
• Refiner Node: Improves weak outputs.

Define the state:

from typing import TypedDict, Annotated, Sequence
import operator
from llama_index.core.schema import NodeWithScore

class AgentState(TypedDict):
    query: str
    retrieved_nodes: Annotated[Sequence[NodeWithScore], operator.add]
    response: str
    check_score: float
    iteration: int

Implement router:

from langgraph.graph import StateGraph, END

@graph.node
def route_query(state: AgentState) -> AgentState:
    # LLM prompt to classify: "factual", "analytical", "conversational"
    category = llm.complete(classify_prompt.format(query=state["query"])).text
    return {"route": category}

Step 3: Building Retrieval Branches

Create specialized retrievers:

semantic_retriever = index.as_retriever(similarity_top_k=5)
keyword_retriever = keyword_index.as_retriever(...)  # BM25 setup
hybrid_retriever = ...  # Ensemble

In graph nodes:

def semantic_retrieve(state: AgentState) -> AgentState:
    nodes = semantic_retriever.retrieve(state["query"])
    return {"retrieved_nodes": nodes}

Similar for others, branched via router decisions.

Step 4: Implementing Self-Checking

Post-retrieval, score the synthesis:

def check_quality(state: AgentState) -> AgentState:
    response = llm.complete(check_prompt.format(
        query=state["query"], response=state["response"]
    )).text
    score = extract_score(response)  # 0-1 scale
    return {"check_score": score}

If score < 0.8, route to refiner.

Step 5: Iterative Refinement

Refinement node rewrites query or fetches more context:

def refine(state: AgentState) -> AgentState:
    if state["iteration"] > 3:
        return {"response": "Max iterations reached"}
    refined_query = llm.complete(refine_prompt.format(...)).text
    # Re-retrieve with refined_query
    return {"query": refined_query, "iteration": state["iteration"] + 1}

Step 6: Constructing the LangGraph Workflow

Compile the graph:

graph = StateGraph(AgentState)

# Add nodes
graph.add_node("router", route_query)
graph.add_node("semantic", semantic_retrieve)
# ... other nodes

# Edges
graph.add_conditional_edges("router", route_to_retriever, {"semantic": "semantic", ...})
graph.add_conditional_edges("checker", check_and_decide, {"refine": "refiner", "end": END})

graph.set_entry_point("router")
app = graph.compile()

Step 7: Querying the System

Invoke with:

result = app.invoke({"query": "What is agentic RAG?", "iteration": 0})
print(result["response"])

Example output for complex query: Routes to hybrid → Checks (score 0.92) → Final response.

Advanced Enhancements

• Multi-Agent Collaboration: Add critic agents for peer review.
• Caching: Use Redis for frequent queries.
• Evaluation: Integrate RAGAS for offline metrics.

Real-world application: Customer support bots handling technical docs, reducing escalation by 40% via precise routing.

Deployment Considerations

Containerize with Docker:

FROM python:3.10
COPY . /app
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

Expose via FastAPI for API endpoints. Monitor with LangSmith for traceability.

This system scales to production, offering explainable AI decisions. Full implementation in the GitHub repo. Experiment and adapt to your data!