# Code Span Semantic Chunking Executive Summary Executive Summary LLMC’s retrieval system must balance context relevance with token limitations, especially for large code files. Naïve fixed-size chunking is inadequate for code – it often splits functions or logical units, producing fragments that lack syntactic or semantic integrity 1 . To address this, we recommend an AST-driven chunking strategy that treats each code span (function, class, etc.) as the atomic retrieval unit, preserving natural boundaries. By leveraging tree-sitter parsers for structure, LLMC can ensure chunks align with code semantics – e.g. each chunk contains a complete function or class – and never cuts off in the middle of a code block 2 . This semantic chunking yields more meaningful context, improving retrieval precision by 15–25% over equal-length fixed chunks 3 and avoiding malformed code snippets 4 . We also propose a hierarchical chunking approach: represent code at multiple levels (file summaries, classes, methods) and use multi-stage retrieval to first identify relevant files or classes, then zoom into specific functions. This combines span-level retrieval with minimal necessary chunking, mitigating the need to arbitrarily split code. Alternative strategies beyond chunking – such as code knowledge graphs or on-the-fly AST queries – are discussed, but they add complexity. Our design favors structured span-based segmentation enriched with metadata (docstrings, imports, references) to achieve the goals. We detail algorithms to split large spans only along AST substructure (preserving logic), overlap policies to maintain context continuity, and scoring improvements using code-aware embeddings. The result is a comprehensive chunking and retrieval framework that reduces token waste ~30%, maintains high relevance (planner confidence ≥0.6), and allows the 7B model to handle the majority of queries without escalation. Key recommendations include: (1) AST-based chunk extraction for Python, JS/TS, Bash, Markdown, (2) hierarchical index schema with parent-child links and optional file summaries, (3) span-level scoring using semantic embeddings and hybrid search, and (4) targeted overlap or context augmentation only where needed (e.g. class context for method chunks). This design ensures that LLMC provides precise, context-rich code answers within strict token budgets, while remaining scalable across languages and file sizes. Chunking Algorithm Design 2.1 Semantic vs. Fixed-Size Chunking Semantic chunking uses code structure to define chunk boundaries, whereas fixed-size chunking arbitrarily splits text after N tokens/lines. For code, semantic (AST-aligned) chunks are vastly superior. Fixedsize splitting often breaks code mid-function or mid-statement, yielding unusable chunks 1 . For example, a naive splitter might separate a function signature from its body or split a loop in half, confusing the LLM. Semantic chunking, in contrast, respects logical units so each chunk is self-contained (e.g. one function, one class) 2 . This ensures syntactic validity and preserves context needed to understand the code snippet. AST-guided chunking: We parse each source file into an Abstract Syntax Tree and use it to segment code. Each function, class, or top-level block becomes a candidate chunk (a span in the database). If a span is small enough to fit the token budget, we index it whole. Empirical research shows that such AST-aware chunks improve retrieval performance consistently: Wang et al. (2025) report a +1.2–3.3 point Precision and +1.8–4.3 point Recall gain by chunking on syntactic boundaries (cAST) versus fixed-length chunks 5 . 1 In other words, structure-aligned spans make it easier for the retriever to find relevant code, and for the LLM to read it without losing context. Semantic chunks also avoid the problem of “lost in middle” tokens – LLM accuracy degrades when context is arbitrarily chopped 6 7 . Span-level vs. further splitting: We advocate treating each AST span (e.g. a full function) as the default chunk unit. Do not split a function or class unless absolutely necessary 8 . Each span naturally encapsulates a complete concept or behavior, which aligns with how developers think about code. By indexing at span-level, we maximize semantic coherence – each chunk has a clear purpose (e.g. a function’s logic or a class’s definition). This aligns with community best practices: “Don’t chunk unnecessarily. If you really need to chunk, chunk by function/class. Don’t split up a function/class” 8 . Using spans also leverages the structured and enriched data we have (symbols, docstrings, etc.), enabling “exact slice” retrieval by symbol name or AST node when possible rather than by raw text windows. When to consider fixed-size or alternative splits: In rare cases, a single span might exceed model context limits (e.g. a 1000-line function). In such cases, we avoid arbitrary slicing and instead use a recursive AST chunking strategy (see 2.3) to break the span at logical sub-boundaries. Only if AST-based splitting fails (e.g. extremely large monolithic code with no good breaks) would we fall back to line-based chunking as a last resort. Even then, we prefer summarizing or using a larger context model over coarse splitting. A hard global token limit per chunk (e.g. ~2048 tokens for Qwen-7B context) can be set, but it should guide an ASTbased splitter rather than truncate blindly. We will measure chunk sizes in tokens, not lines, for accuracy 9 – 300 lines is an approximation, but using actual token count (via tokenizer) ensures we don’t overflow the context window. In summary, semantic chunking is the default; fixed-size chunks are only used in fallback scenarios or for non-code files with no structure. 2.2 Overlap and Context Preservation Overlap between chunks involves repeating a portion of content at the boundaries so that important context isn’t lost when splitting. In text documents, a 10–20% overlap often improves retrieval recall by 15– 30% 10 , because it ensures continuity of ideas across chunks. For code, our goal is to minimize splitting within a logical unit, so ideally each chunk is independently understandable without overlaps. If we adhere to AST spans, chunks correspond to whole functions or classes, which usually have clear boundaries (one chunk ends where a function ends). In these cases, overlap is not needed – “AST-based code chunking ensures search results are complete and never cut off mid-function” 2 , eliminating the need to duplicate lines between chunks. However, when a single AST node must be split (e.g. a huge function), we implement a controlled overlap at natural breakpoints. Rather than a fixed token overlap, we consider code structure: for example, if a function is split into Chunk A (first half) and Chunk B (second half), we might repeat the function signature or a key variable definition from A at the top of B as context. Another strategy is to include a comment placeholder like # ...continued at the end of Chunk A and start of Chunk B to signal continuity (helpful for the reader/LLM but doesn’t consume many tokens). Overlap in code should be minimal – just enough to bridge context – because excessive duplication wastes tokens and can confuse the LLM if the same lines appear twice. For Markdown or documentation files, overlap may be more useful. We preserve section headings and perhaps repeat the heading at the top of each chunk in that section so the context is clear. For example, if splitting a long section into two chunks, we’d include the section title in both. This kind of document-aware 2 chunking (keeping headers, code blocks, tables intact) can improve accuracy by over 40% in structured content 11 . So for Markdown, we split by highest-level headings, then by subheadings if needed, ensuring each chunk starts with its context (title) and possibly a brief overlap of the last sentence of the previous chunk if a paragraph was split. Quantifying overlap impact: We will empirically test overlap percentages on code retrieval. Likely, because of AST alignment, 0% overlap yields high precision (no redundant lines) while small overlap (e.g. 2-3 lines) might marginally improve recall for queries that reference something near a chunk boundary. We expect diminishing returns beyond 10% overlap. Overlap also increases index size and storage. Our default for code will be no overlap, given semantic boundaries, but for any non-code or large-block splits we target ~15% overlap as a starting point 10 , tuning based on retrieval metrics. 2.3 Recursive & Hierarchical Chunking To handle large spans and maintain a hierarchy of context, we introduce a recursive chunking algorithm inspired by cAST 12 13 . The process works top-down on the AST: • Parse the AST: Get the root node for the file (e.g. Module) and its children (classes, function definitions, etc.). These top-level nodes are initial spans. • Set a size limit: Define MAX_TOKENS (e.g. ~2048 tokens for 7B model) for chunks. We measure each node’s length in tokens or characters 9 . • Chunk large nodes recursively: For each AST node: • If the node’s content <= MAX_TOKENS, make it a chunk (span) as-is. • If it exceeds MAX_TOKENS, do not cut the code arbitrarily. Instead, descend into its children: ◦ Recursively attempt to chunk its child nodes (e.g. split a large class into methods, or a large function into smaller blocks/statements). ◦ If a child itself is too large (e.g. a very large function body with no sub-functions), we then split that child by smaller AST elements (such as statements or expression blocks). This is the “split-then-merge” approach from cAST 12 : ◦ Split large node into its sub-nodes. ◦ If after splitting, some chunks are very small, merge adjacent small siblings to form a reasonably sized chunk (to avoid too many tiny pieces) 14 15 . • This yields a set of chunks whose concatenation equals the original node’s code (thus no code is lost or duplicated across chunks). • Output hierarchical chunks: The algorithm produces chunks at multiple granularities. For example, a large class may be split into function chunks; a large function may be split into logical blocks (loop, helper lambdas, etc.). Each chunk knows its parent context (more on schema in Section 3). This recursive AST-based chunking ensures we preserve as much semantic context as possible in each chunk while respecting token limits. It’s a general approach that works across languages without special heuristics 16 – we rely on the parse tree structure. Pseudocode for this algorithm is as follows: MAX_TOKENS = 2048 # or context window limit for target model def chunk_node(node): """Recursively chunk an AST node into a list of code segments.""" 3 node_size = token_count(node.code) # measure tokens in the node’s source code if node_size <= MAX_TOKENS: return [node.code] # fits in one chunk # Otherwise, chunk by children chunks = [] current_chunk = "" current_size = 0 for child in node.children: child_code = child.code # source code for this AST child child_size = token_count(child_code) if child_size > MAX_TOKENS: # Recursively split the child if it's still too large child_chunks = chunk_node(child) # Each returned chunk is guaranteed <= MAX_TOKENS for subchunk in child_chunks: # If adding the subchunk to current chunk would overflow, finalize current chunk if current_chunk and (current_size + token_count(subchunk) > MAX_TOKENS): chunks.append(current_chunk) current_chunk = "" current_size = 0 # Append subchunk (or start a new chunk if current is empty) current_chunk += subchunk current_size += token_count(subchunk) # If exactly at limit, finalize the chunk if current_size >= MAX_TOKENS: chunks.append(current_chunk) current_chunk, current_size = "", 0 else: # Child fits in a chunk; try to merge with current chunk if space allows if current_size + child_size > MAX_TOKENS: # Finish the current chunk and start a new one chunks.append(current_chunk) current_chunk, current_size = "", 0 current_chunk += child_code current_size += child_size # Append any remaining code as the last chunk if current_chunk: chunks.append(current_chunk) return chunks In practice, the above will be implemented using tree-sitter’s concrete syntax nodes (so we can extract exact code spans by line numbers). The concept is similar to Algorithm 1 in the cAST paper 17 18 : traverse the 4 AST, split large nodes, merge small ones. This yields an optimal segmentation where chunks are as large as possible but not over the limit, and aligned to syntax. The hierarchy is naturally maintained: if a large function is split into chunks, those chunks can be considered “child spans” of the function span. 2.4 Language-Specific Considerations The chunking strategy will be adapted for each language’s grammar and typical structures: • Python: Use tree-sitter Python parser. Natural chunk units: modules, classes, functions, and methods. The parser can also identify smaller blocks (e.g. if a function has inner functions or large loops, those appear as child nodes). Python’s significant whitespace means we must include indentation properly in chunks. We also ensure the first statement of a function (often a docstring) remains with the function. Docstrings are AST children of function nodes; we include them in the same chunk as the function code so that each chunk has the function’s documentation context. If a top-level file has many small functions, we might merge some into one chunk if they’re very short and related (though typically each function stands alone, merging is not required unless trying to save overhead). Imports and module-level constants will be treated as part of a special “module chunk” at the top of the file. For example, all import statements can form a chunk (or be included with the first function if under size limit). This ensures that if a retrieved function call relies on an import, the retriever can also pull in the imports chunk if needed. • JavaScript/TypeScript: Use tree-sitter for TS/JS. We chunk at function definitions (including methods inside classes) and class definitions. JS has additional forms like arrow functions and closures. An arrow function assigned to a variable can be treated like a function span (it will appear as an AST node for an “arrow_function” or similar). We capture such assignments as separate chunks if they are top-level or static fields. If an arrow function is nested inside another function, our AST algorithm will treat it as a child node; if it’s large, it may become its own chunk (with the outer function’s code split around it). Classes in TS/JS often contain many small methods – each method can be a chunk. If the class itself has a large static initialization block or a constructor with many lines, splitting within that may be needed. We will also pay attention to closures and callbacks (functions passed as arguments): these appear as AST subtrees – if they are large (e.g. a big lambda in a call), we might split them out as separate chunk text to keep each chunk focused. For JS, preserving context means ensuring that closing braces and context are included – e.g., a function chunk should ideally include its closing } . • Bash: Use tree-sitter Bash. Bash scripts may not have explicit functions (though they can). For scripts, the AST will break it into commands, pipelines, loops, etc. We treat each function definition in Bash as a chunk (if present). If the whole script is just a sequence of commands, we might chunk by grouping related commands. We can use heuristics like blank lines or comments as natural separators (since Bash often uses comments to section scripts). Tree-sitter yields nodes for each command and control structure ( if , for , etc.), which we will group. For example, an if ... fi block would be one chunk. If the script is very long, we might split after certain logical groupings (perhaps after a long series of independent commands). Note: Splitting Bash must be done carefully, because earlier commands might set variables used later. We’ll aim to keep logically related sequences together. Overlap in Bash could be useful if, say, a variable is defined in one chunk and used in the next; in such cases we might repeat the definition line in the second chunk as a reference. 5 • Markdown: Without an AST, we use a rule-based recursive splitter. We treat each top-level heading ( # Heading ) as a chunk boundary. For each section under a heading, if the content exceeds MAX_TOKENS, we split by subheadings ( ## ), and so on. Within a large section with no subheadings, we chunk by paragraphs while preserving lists and code blocks as atomic units (never split a code block across chunks – include the whole block with whichever chunk it starts in). We ensure each chunk carries context: prepend the section heading or a summary line so that even a standalone chunk is understandable. For example: if a chunk starts mid-section (because the section was long), we might repeat the section title and a note like “(continued)…”. We also keep table rows together, etc., as noted earlier. • Mixed-language files: If we encounter files that contain multiple languages (e.g. a Markdown file with embedded code, or an HTML file with

Data & Analytics

Metrics

Project Memory

Code Span Semantic Chunking Executive Summary

Text Splitting with LangChain

语义块切分 Semantic Chunking

rag-vector-chunking

语义块切分 Semantic Chunking

Chunking Strategies

Semantic Chunking

Chunking Fundamentals

Chunking & Embeddings

ADR-0006: Paragraph-Boundary Chunking

Late Chunking Strategy

With Jina Embeddings v3

Text Chunking - Anton (ChromaDB)

Text Chunking - Anton (ChromaDB)

embedding and chunking

Search and Retrieval

Semantic Chunking

Long Context Chunking

Large-scale Document Retrieval with ElasticSearch

Introduction ot Information_Retrieval

Information Retrieval - Topic Modelling Relevent Part

lib-ai-app-community-rag