Data & Analytics
CHUNKING.md · 14 documents
Massive JSON Kerchunking
Documents an attempt to kerchunk 23 years of daily netCDF files into a single JSON reference, hitting memory limits.
How to Implement Recursive Chunking
Explains recursive chunking for RAG systems, providing pure Python and LangChain implementations plus a production pipeline class.
BlobStore: Chunking & Verified Streaming
Explains how Snix's BlobStore uses BLAKE3 tree hashing to support verified streaming and partial reads of large blobs without embedding chunking topology into identifiers.
STREAMING_CHUNKING
Explains how to stream large datasets in chunks to avoid RAM exhaustion while keeping performance close to full-load paths.
基于命题分块以增强RAG
Implements proposition chunking for RAG: decomposes documents into atomic facts, evaluates quality, and compares retrieval against standard chunking.
Kerchunk
Explains how Kerchunk creates cloud-friendly access to netCDF/HDF5 data by extracting metadata and enabling parallel chunk reads.
chunking
Surveys heuristic and LLM-based chunking strategies for YouTube transcripts in a RAG pipeline, with an example workflow and tool recommendations.
SaltyRTC Chunking
Defines a binary chunking algorithm to split large messages into smaller chunks, originally for WebRTC data channel size limits.
Chunking Strategies
Introduces five text chunking strategies with Python implementations and shows how to pair them with cosine similarity for retrieval.
Annotation-Based Chunking
Generates ground-truth chunk boundaries from annotation positions using Voronoi midpoints, for benchmarking semantic chunking strategies.
20. Text Flushing and Chunking (Current Behavior)
Documents the message flushing and chunking pipeline for streaming model output to Discord, including buffer triggers, semantic protection, and humanizer ordering.
Chapter 18: The Art of Chunking
Explains why project files and conversation logs need different chunking strategies for vector retrieval, with code examples and parameter tables.
Marker Page Chunking Benchmark
Benchmarks Marker chunk sizes (10, 20, 30) for PDF parsing speed and GPU memory, recommending chunk size 10 for production.
Better File Chunking
Opens a discussion on chunking strategies for IPFS, aiming to reduce algorithm proliferation and improve default chunking.