RAG

Four Chunking Strategies for Structured Documents in RAG

By Omer DemirSep 18, 2025⏱ 6 min read👁 — reads⧉ — copies

AI disclosure: This article was moderated by AI.

Chunking can determine whether a RAG system works well at all. Split too aggressively and a passage loses the context it needs. Split too coarsely and retrieval returns a wall of text when one paragraph would answer the query. Structured documents such as contracts, manuals, and OCR output with real headings need different strategies for different queries. We put four interchangeable chunkers behind one interface.

One interface, many strategies

Every chunker implements the same BaseChunker contract. The indexing pipeline does not need to know which strategy is active:

class BaseChunker(ABC):
    @abstractmethod
    def chunk(self, content: str, metadata: dict[str, Any]) -> list[Document]:
        ...

    @property
    @abstractmethod
    def chunk_type(self) -> str:
        ...

A small factory maps a configuration string to an implementation, making the strategy a one-line change:

CHUNKER_TYPES = {
    "simple": SimpleChunker,
    "markdown": MarkdownChunker,
    "section": SectionAwareChunker,
    "semantic": SemanticChunker,
}

def get_chunker(
    chunker_type: str,
    chunk_size: int,
    chunk_overlap: int,
    breakpoint_threshold_type: str | None = None,  # semantic only
    embeddings_endpoint: str | None = None,        # semantic only
) -> BaseChunker:
    ...

Those five parameters cover everything the indexing pipeline needs. Changing "markdown" to "semantic" in the configuration re-indexes the corpus with a different splitting algorithm and requires no code changes.

The four strategies

1. Simple: character-based fallback

The simple strategy uses fixed-size character windows with overlap. It knows nothing about structure, but it is fast, deterministic, and a safe default when a document has no usable markup. It works best as a fallback rather than the preferred strategy.

2. Markdown: structure-aware with size limits

This strategy splits on Markdown structure, including headers, lists, and code blocks, while respecting a chunk_size/chunk_overlap budget. It is the main choice for documentation and other Markdown content. Chunks land on natural boundaries without exceeding the embedding model’s context window.

The implementation uses LangChain splitters in two steps: headers first, then size. It defaults to a 1000-character chunk with 200-character overlap and splits on h1 through h3:

class MarkdownChunker(BaseChunker):
    DEFAULT_CHUNK_SIZE = 1000
    DEFAULT_CHUNK_OVERLAP = 200
    DEFAULT_HEADERS = [("#", "header_1"), ("##", "header_2"), ("###", "header_3")]

    def __init__(self, chunk_size=None, chunk_overlap=None, headers_to_split_on=None):
        self._markdown_splitter = MarkdownHeaderTextSplitter(
            headers_to_split_on=self.headers_to_split_on,
            strip_headers=False,  # keep the heading in the chunk for context
        )
        self._text_splitter = RecursiveCharacterTextSplitter(
            chunk_size=self.chunk_size,
            chunk_overlap=self.chunk_overlap,
            add_start_index=True,  # record each chunk's offset in the source
        )

    def chunk(self, content, metadata):
        header_splits = self._markdown_splitter.split_text(content)   # step 1
        final_chunks = self._text_splitter.split_documents(header_splits)  # step 2
        ...

The 20% overlap (200 of 1000) carries a sentence or two of context across a boundary. A passage split mid-thought can still be retrieved from either side. With add_start_index, every chunk records its character offset in the original document, which is useful for citations.

3. Section-aware: one chunk per section

Instead of enforcing a fixed size, this chunker keeps whole sections intact, with one chunk per h1/h2. That suits high-level navigation, section-level search, and summarization. It detects headers with a regex and splits further only when a section is genuinely oversized:

class SectionAwareChunker(BaseChunker):
    DEFAULT_SECTION_PATTERN = r"^(#{1,2})\s+(.+)$"
    MAX_SECTION_SIZE = 8000  # large, to preserve sections

Because a section can exceed the model’s window, oversized sections become paragraph-based fragments. Each fragment is tagged so retrieval knows that it belongs to a larger section:

chunk_metadata={
    "section_header": header_text,
    "section_level": len(header_level),
    "is_section_fragment": True,
    "fragment_index": len(chunks),
}

4. Semantic: split at topic boundaries

The semantic option embeds the text and places breakpoints where the topic changes rather than at arbitrary character counts. Its configurable breakpoint algorithm can use percentile, standard_deviation, interquartile, or gradient, and it calls a model serving endpoint for embeddings. Use it when passages must remain topically coherent and the index-time embedding cost is acceptable.

The defaults are a 2000-character ceiling and the percentile breakpoint method, backed by a Databricks gte-large-en embedding endpoint. If embeddings are unavailable, the chunker falls back instead of failing:

class SemanticChunker(BaseChunker):
    DEFAULT_CHUNK_SIZE = 2000
    DEFAULT_CHUNK_OVERLAP = 200
    DEFAULT_BREAKPOINT_THRESHOLD_TYPE = "percentile"
    DEFAULT_EMBEDDINGS_ENDPOINT = "databricks-gte-large-en"

    def chunk(self, content, metadata):
        raw_chunks = self._split_semantic(content)      # embedding-based
        if raw_chunks is None:                          # no embeddings / import error
            raw_chunks = self._split_fallback(content)  # RecursiveCharacterTextSplitter
        final_chunks = self._enforce_max_size(raw_chunks)  # re-split anything > 2000 chars
        ...

There are two safeguards. If langchain_experimental or the embedding credentials are missing, the chunker falls back to character splitting rather than crashing the index job. Semantic splitting can also emit a chunk far larger than expected, so _enforce_max_size splits any chunk over the ceiling. The result uses semantic boundaries where possible while always enforcing the size limit.

Metadata is the real payload

Chunk text is only part of the payload. Each chunk receives metadata that persists into the vector store, giving retrieval results their provenance:

enriched = {
    **base_metadata,
    **chunk_metadata,
    "chunk_index": chunk_index,
    "chunk_id": f"{doc_id}_{chunk_index}",
    "chunk_type": self.chunk_type,
}

The chunk_type field lets you A/B different strategies in the same index and attribute retrieval quality to the chunker that produced each hit. The chunk_id provides a stable handle for deduplication and citation.

Choosing a strategy

The four strategies produce indexes with noticeably different shapes. The following illustrative but representative comparison runs the same ~40-page equipment manual, roughly 90,000 characters, through each strategy:

The table gives the counts, though the shapes are easier to remember:

Section-aware produces ~6× fewer and ~5× larger chunks than simple splitting. That works well for “which section covers X” navigation but wastes context on pinpoint fact lookup. Semantic falls between the two on size and is the only strategy that incurs an embedding cost at index time. The query determines which trade-off makes sense:

No chunker is universally best. The choice depends on the document shape and query pattern. Making the strategy configurable lets you measure retrieval quality for each option and choose based on the data without rewriting the indexing pipeline.

Takeaways

• Put every chunker behind one interface and select the implementation through configuration.
• Preserve structure, such as sections and headers, for navigational queries. Preserve topic coherence with semantic chunking for conceptual queries.
• Carry metadata such as chunk_type, chunk_id, and section headers into the index so you can debug and attribute retrieval quality.

#rag#chunking#embeddings#retrieval#python