notes_on_chunking

Based on the transcript, here are the key teachable insights about RAG and chunking:

## Core Chunking Concepts

• **Chunking remains essential despite long context windows**: "Even if the LM's context window size is growing, the embedding model input size tends to remain fairly constant" and "you're still paying in terms of inference time and in terms of cost per token"

• **Chunking serves multiple purposes beyond context limits**: It enables "inference efficiency," "information accuracy and the elimination of distractors," and "improving retrieval performance"

## Chunking Strategy Types

• **Heuristic approaches are widely used but brittle**: "Heuristics are typically fairly brittle and they do require a lot of sort of pre-processing and cleaning of those documents ahead of time"

• **Recursive character text splitter is most common**: "Probably the most widely used chunking strategy right now is the recursive character text splitter"

• **Semantic chunking is emerging**: Using "embedding or language models directly to find semantic boundaries of documents" to avoid "the brittleness of the heuristics by actually looking at the meaning and the content of each chunk"

## Practical Rules and Guidelines

• **Fill embedding model context windows**: "You really want to fill the embedding model context window as much as you can... try to fill it up to its maximum character limit"

• **Avoid grouping unrelated information**: "You really don't want to group unrelated information together... it's very difficult to summarize the meaning of a chunk which contains contradictory or differing information"

• **Short chunks produce poor results**: "Strategies that produce very short chunks essentially are very very noisy in terms of their retrieval accuracy"

## Critical Evaluation Principles

• **Always examine your data**: "Always always always look at your data" - this phrase appears multiple times as the most important rule

• **Default settings often fail**: "By default a lot of heuristic chunkers produce chunks that are way too short... the delimiter characters used in the hierarchical chunker are just not... in the wrong order by default"

• **Focus on recall over other metrics**: "Recall is probably the single most important metric... the models are improving at ignoring irrelevant information but they cannot do the task if you haven't given them all the relevant information"

• **Evaluate at passage level, not document level**: "You should be measuring the recall of relevant passages rather than was any passage from a document that we've labeled relevant retrieved"

## Strategy Selection and Customization

• **No universal chunking strategy exists**: "There's probably not one universal chunking strategy that is the best, there is a chunking strategy that is good for your data and your task"

• **Task-dependent approach required**: "Chunking strategy that you use actually depends pretty strongly on the tasks that you actually want to perform"

• **Retrieval systems are application-specific**: "Retrieval is not... a general system... it is dependent on your task, it is dependent on your data"

## Key Takeaway

• **Data inspection is non-negotiable**: "If you come to me with a problem and you haven't looked at your data I will laugh at you" - emphasizing that manual data examination is fundamental to successful chunking implementation.