CHUNKING.md
Document chunking strategies and splitting rules
53 documents availableCategories
Recent CHUNKING.md Documents
View allMassive JSON Kerchunking
Documents an attempt to kerchunk 23 years of daily netCDF files into a single JSON reference, hitting memory limits.
type(docs) ==> List[Document], hence used split_documents
Demonstrates four LangChain text splitters, RecursiveCharacter, Character, HTMLHeader, and RecursiveJson, with code examples and explains when to use each.
Create a list that will hold your chunks
Walks through document chunking strategies for RAG pipelines, from basic character splitting to recursive and token-based methods, with code examples in LangChain.
How to recursively split text by characters
Demonstrates LangChain's RecursiveCharacterTextSplitter for splitting text by a prioritized list of separators until chunks fit a size limit.
How to Implement Recursive Chunking
Explains recursive chunking for RAG systems, providing pure Python and LangChain implementations plus a production pipeline class.
Build a PDF Search System with Mistral OCR and Weaviate DB
Builds a pipeline to upload PDFs, extract text with Mistral OCR, embed chunks via Cohere, store in Weaviate, and expose a search webhook.
Splitters
Introduces splitters that divide large text into manageable chunks for AI processing, with configurable sizes and overlap.
elevellabs-chunking
Explains how to use the ElevenLabs Text to Dialogue API for multi-speaker audio generation with chunking for long-form content.