Splitters
Introduces splitters that divide large text into manageable chunks for AI processing, with configurable sizes and overlap.
What this file does
Introduces splitters that divide large text into manageable chunks for AI processing, with configurable sizes and overlap.
When to use it
- Chunking long documents before embedding or retrieval
- Processing PDFs with images and complex layouts
- Working with structured Node objects from parsers
- Splitting text with multiple strategies for different content types
Assumes this stack
Splitters
Splitters are pipeline components that divide large text content into smaller, manageable chunks. They help optimize content for processing, storage, and retrieval in AI applications by creating appropriately sized segments while preserving context and meaning.
Installation
All splitters are included with datapizza-ai-core and require no additional installation.
Available Splitters
Core Splitters (Included by Default)
- RecursiveSplitter - Recursively divides text using multiple splitting strategies
- TextSplitter - Basic text splitter for general-purpose chunking
- NodeSplitter - Splitter for Node objects preserving hierarchical structure
- PDFImageSplitter - Specialized splitter for PDF content with images
Common Features
- Multiple splitting strategies for different content types
- Configurable chunk sizes and overlap
- Context preservation through overlapping
- Support for structured content (nodes, PDFs, etc.)
- Metadata preservation during splitting
- Spatial layout awareness for document content
Usage Patterns
Basic Text Splitting
from datapizza.modules.splitters import RecursiveSplitter
splitter = RecursiveSplitter(chunk_size=1000, chunk_overlap=200)
chunks = splitter(long_text_content)
Document Processing Pipeline
from datapizza.modules.parsers import TextParser
from datapizza.modules.splitters import NodeSplitter
parser = TextParser()
splitter = NodeSplitter(max_char = 4000)
document = parser.parse(text_content)
structured_chunks = splitter(document)
Choosing the Right Splitter
- RecursiveSplitter: Best for general text content, articles, and most use cases
- TextSplitter: Simple splitting for basic text without complex requirements
- NodeSplitter: When working with structured Node objects from parsers
- PDFImageSplitter: Specifically for PDF content with images and complex layouts
- BBoxMerger: Utility for processing documents with spatial layout information
What's inside
1 overview, 1 installation note, 4 core splitter descriptions, 1 common features list, 2 usage patterns with code examples, 1 choosing guide
Change this for your project
- Replace
datapizza.modules.splitterswith your own package import path - Replace
datapizza.modules.parserswith your own parser module path - Replace
datapizza-ai-corewith your own core package name
Where it goes
Keep it in your repository where the agent or team that needs it will read it.
Related Documents
基于命题分块以增强RAG
Implements proposition chunking for RAG: decomposes documents into atomic facts, evaluates quality, and compares retrieval against standard chunking.
TileMap Chunk Manager
Divides a large tilemap into fixed-size chunks, loading and rendering only those visible to reduce memory and draw calls by over 99%.
Message Chunking for MCP stdio Transport
Explains macOS 64KB pipe write limit and provides a chunking layer for MCP stdio transport that splits and reassembles large messages transparently.
🤖 n8n AI Agent Mastery Course 2025
Serves as the main README for an n8n course repository, listing 10+ AI agent workflow projects with links to YouTube tutorials and Docker setup instructions.