Why Chunking Strategy Matters
In RAG systems, document chunking is the process of splitting long documents into smaller segments suitable for vector retrieval. This seemingly simple step directly impacts retrieval quality and the accuracy of final answers. If chunks are too large (e.g., 2000 tokens), the retrieved segments may contain too much irrelevant information, diluting the weight of key content; if too small (e.g., 100 tokens), context may be lost, leading to semantically incomplete retrieved segments. More importantly, the chunking strategy also determines whether retrieved segments can form effective semantic matches with user queries.
Many RAG projects fail not because the model is inadequate or the vector database is mischosen, but because the chunking strategy is not optimized for the specific scenario. A RAG system for legal contracts requires a completely different chunking strategy than one for technical documentation. This article will delve into the principles, pros and cons, and applicable scenarios of four mainstream chunking strategies.
Strategy 1: Fixed-Size Chunking
Fixed-size chunking is the simplest and most direct method—splitting documents by a fixed token count. For example, each chunk is 512 tokens, with a sliding window overlap of 128 tokens. The advantages are simple implementation and predictable performance; the disadvantage is that it completely ignores semantic boundaries, potentially cutting in the middle of sentences, resulting in semantically incomplete chunks.
When implementing, two key parameters need attention: chunk_size and chunk_overlap. The overlap exists to prevent key information from falling exactly on the boundary between two chunks. Generally, it is recommended that chunk_size be between 256 and 1024, and overlap be 10%-25% of chunk_size. For Chinese documents, it is recommended to use chunk_size=512 (approximately 800-1000 Chinese characters) and overlap=100.
from openai import OpenAI
client = OpenAI(api_key="your-deepseek-api-key", base_url="https://api.deepseek.com")
class FixedSizeChunker:
def __init__(self, chunk_size=512, overlap=100):
self.chunk_size = chunk_size
self.overlap = overlap
def chunk_text(self, text):
"""Simple chunking by character count"""
chunks = []
start = 0
while start < len(text):
end = min(start + self.chunk_size, len(text))
chunk = text[start:end]
chunks.append({
"text": chunk,
"index": len(chunks),
"start_char": start,
"end_char": end
})
start = end - self.overlap
return chunks
def chunk_with_metadata(self, text, source="", title=""):
"""Chunking with metadata"""
chunks = self.chunk_text(text)
for c in chunks:
c["source"] = source
c["title"] = title
c["char_count"] = len(c["text"])
return chunks
chunker = FixedSizeChunker(chunk_size=800, overlap=120)
text = "DeepSeek is a large language model developed by DeepSeek Company..." * 50
chunks = chunker.chunk_with_metadata(text, source="deepseek_intro.md", title="DeepSeek Introduction")
print(f"Generated {len(chunks)} document chunks")Strategy 2: Semantic Chunking
Semantic chunking does not split by fixed length but determines split points based on semantic boundaries—such as paragraphs, sections, or topic transitions. This approach produces chunks with more complete semantics, and retrieval quality is generally better than fixed-size chunking. However, implementation is more complex, requiring a model to judge semantic boundaries.
The core of semantic chunking is identifying "natural breakpoints" in text: paragraph separators (double newlines), heading markers (Markdown's #), topic transition sentences (e.g., "On the other hand", "Next, we discuss"), and semantic drift points detected via embedding similarity. A practical strategy is to use paragraphs or sections as the first-level split; if a paragraph still exceeds the threshold, use finer-grained methods to split further.
import numpy as np class SemanticChunker: def __init__(self, max_chunk_size=1000, similarity_threshold=0.7): self.max_chunk_size = max_chunk_size self.similarity_threshold = similarity_threshold def get_embedding(self, text): """Use DeepSeek to get text embedding""" response = client.embeddings.create( model="deepseek-chat", input=text[:8000] ) return np.array(response.data[0].embedding) def chunk_by_paragraphs(self, text): """First split by paragraphs""" paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()] return paragraphs def merge_similar(self, paragraphs): """Merge semantically similar paragraphs""" if len(paragraphs) <= 1: return paragraphs chunks = [] current = paragraphs[0] for i in range(1, len(paragraphs)): combined = current + "\n\n" + paragraphs[i] if len(combined) <= self.max_chunk_size:current = combined else: chunks.append(current) current = paragraphs[i] chunks.append(current) return chunks chunker = SemanticChunker(max_chunk_size=1000) paras = chunker.chunk_by_paragraphs(long_document) merged = chunker.merge_similar(paras) print(f"Merged from {len(paras)} paragraphs into {len(merged)} semantic chunks") re> Strategy 3: Recursive Chunking
Recursive chunking is a compromise: first try to split using coarse-grained separators (such as double newlines), and if the resulting chunks are still too large, continue splitting with finer-grained separators (such as single newlines, periods, commas). This stepwise refinement considers semantic boundaries while keeping chunk sizes within a controllable range. LangChain's RecursiveCharacterTextSplitter is a typical implementation of this strategy.
class RecursiveChunker: def __init__(self, chunk_size=512, overlap=50): self.chunk_size = chunk_size self.overlap = overlap self.separators = ["\n\n", "\n", "。", ".", "；", ";", "，", ",", " ", ""] def split(self, text): """Recursive split""" return self._recursive_split(text, self.separators) def _recursive_split(self, text, separators): if len(text) <= self.chunk_size: return [text] if text.strip() else [] if not separators: # Last resort: force split return [text[i:i+self.chunk_size] for i in range(0, len(text), self.chunk_size-self.overlap)] sep = separators[0] remaining = separators[1:] parts = text.split(sep) chunks = [] current = "" for part in parts: if len(current) + len(part) + len(sep) <= self.chunk_size: current = (current + sep + part) if current else part else: if current: chunks.extend(self._recursive_split(current, remaining)) current = part if current: chunks.extend(self._recursive_split(current, remaining)) return chunks rchunker = RecursiveChunker(chunk_size=512, overlap=50) chunks = rchunker.split(long_chinese_text) print(f"Recursive chunking result: {len(chunks)} chunks, average length {sum(len(c) for c in chunks)//len(chunks)} characters")Strategy 4: Sentence-Aware Chunking
Sentence-aware chunking ensures that each chunk's boundaries fall exactly at sentence ends, rather than truncating in the middle of a sentence. This strategy is especially important for Chinese—where sentence boundaries are less clear than in English (English uses capital letters as sentence start markers). It requires identifying sentence boundaries using punctuation such as periods, question marks, and exclamation marks. Combined with NLP tools like spaCy or jieba, more accurate sentence boundary detection can be achieved.
Sentence-aware chunking is often a complement to other strategies—on top of fixed-size or recursive chunking, it ensures that split points fall at sentence ends. The benefit is that each chunk is a combination of one or more complete sentences, making the semantics more self-contained and improving the readability of retrieval results.
Chunking Strategy Selection Guide
Technical/API documentation: Fixed-size chunking (512-1024 tokens) is recommended. Such documents have clear structure, with code and text alternating, and semantic boundaries are not obvious.Legal contracts/policy documents: Semantic chunking is recommended. There are clear semantic boundaries between clauses and paragraphs; splitting by clause ensures semantic integrity.News/blog articles: Recursive or sentence-aware chunking is recommended. Article structures vary, requiring a balance between semantic integrity and chunk size.Conversation logs: Chunking by turn is recommended. Each turn of dialogue is a chunk, preserving interactivity.Mixed document collections: Adaptive chunking is recommended—first use rules to detect document type, then choose the corresponding chunking strategy.
There is no silver bullet—the best chunking strategy must be determined through experimentation. It is recommended to evaluate retrieval quality (e.g., Recall@k, MRR) for each candidate strategy and choose the one that performs best on your dataset. Also, chunking strategies are not static; as the document collection updates and grows, periodic re-evaluation and adjustment are necessary.
Common Pitfalls in Practice
Myth 1: Bigger chunks are better. Many developers think "larger chunks contain more context, leading to higher retrieval quality." However, actual tests show that when chunk_size increases from 512 to 2048, retrieval relevance typically rises then falls, with the optimal range being 512-1024. Overly large chunks cause vector representations to become too "averaged," losing fine-grained semantic information.Myth 2: More overlap is better. Overlap does reduce boundary information loss, but excessive overlap (>30%) significantly increases storage costs and retrieval noise—the same information appears in multiple chunks, and retrieval may return multiple highly similar chunks.Myth 3: Set-and-forget chunking strategy. As the document collection changes (new document types, style changes), the original strategy may no longer be optimal. It is recommended to re-evaluate the effectiveness of the chunking strategy quarterly or after major updates to the collection.
Advanced: Content-Based Dynamic Chunking
For scenarios with diverse document types, fixed strategies are hard to cover all cases. An advanced approach is dynamic chunking—first use a small model to analyze document structure (detect heading levels, table boundaries, code block ranges, etc.), then dynamically choose chunk granularity based on local features. For example, keep code blocks intact (do not split), split long paragraphs at sentence boundaries, and keep tables whole. This method is more complex to implement but can bring significant quality improvements in scenarios with extremely high document quality requirements (e.g., legal document retrieval). It is recommended to start with fixed strategies and gradually evolve toward dynamic strategies after accumulating sufficient feedback data.
Want to orchestrate this skill chain yourself?
Open in Skill Chain →