RAG

Four Chunking Strategies for Structured Documents in RAG

By Omer DemirSep 18, 2025⏱ 6 min read👁 — reads⧉ — copies

Chunking is the quiet decision that makes or breaks a RAG system. Split too aggressively and you shred the context a passage needs to make sense; split too coarsely and retrieval returns a wall of text where one paragraph would do. For structured documents — contracts, manuals, OCR output with real headings — the right strategy depends on the query, so we built four interchangeable chunkers behind one interface.

One interface, many strategies

Every chunker implements the same BaseChunker contract, so the indexing pipeline never needs to know which strategy it’s using:

class BaseChunker(ABC):
    @abstractmethod
    def chunk(self, content: str, metadata: dict[str, Any]) -> list[Document]:
        ...

    @property
    @abstractmethod
    def chunk_type(self) -> str:
        ...

A small factory maps a config string to an implementation, so the strategy is a one-line change:

CHUNKER_TYPES = {
    "simple": SimpleChunker,
    "markdown": MarkdownChunker,
    "section": SectionAwareChunker,
    "semantic": SemanticChunker,
}

def get_chunker(
    chunker_type: str,
    chunk_size: int,
    chunk_overlap: int,
    breakpoint_threshold_type: str | None = None,  # semantic only
    embeddings_endpoint: str | None = None,        # semantic only
) -> BaseChunker:
    ...

Everything the indexing pipeline needs is five parameters. Swapping "markdown" for "semantic" in config re-indexes the corpus with a completely different splitting algorithm and no code change.

The four strategies

1. Simple — character-based fallback

Fixed-size character windows with overlap. It knows nothing about structure, but it’s fast, deterministic, and a safe default when a document has no usable markup. Use it as the fallback, not the goal.

2. Markdown — structure-aware with size limits

Splits on markdown structure (headers, lists, code blocks) while still respecting a chunk_size/chunk_overlap budget. This is the workhorse for documentation and any content that’s already markdown: chunks land on natural boundaries but never blow past the embedding model’s context window.

Under the hood it’s a two-step split — headers first, then size — built on LangChain’s splitters. Defaults are a 1000-character chunk with 200-character overlap, splitting on h1–h3:

class MarkdownChunker(BaseChunker):
    DEFAULT_CHUNK_SIZE = 1000
    DEFAULT_CHUNK_OVERLAP = 200
    DEFAULT_HEADERS = [("#", "header_1"), ("##", "header_2"), ("###", "header_3")]

    def __init__(self, chunk_size=None, chunk_overlap=None, headers_to_split_on=None):
        self._markdown_splitter = MarkdownHeaderTextSplitter(
            headers_to_split_on=self.headers_to_split_on,
            strip_headers=False,  # keep the heading in the chunk for context
        )
        self._text_splitter = RecursiveCharacterTextSplitter(
            chunk_size=self.chunk_size,
            chunk_overlap=self.chunk_overlap,
            add_start_index=True,  # record each chunk's offset in the source
        )

    def chunk(self, content, metadata):
        header_splits = self._markdown_splitter.split_text(content)   # step 1
        final_chunks = self._text_splitter.split_documents(header_splits)  # step 2
        ...

The 20% overlap (200 of 1000) is deliberate: it carries a sentence or two of context across a boundary so a passage split mid-thought is still retrievable from either side. add_start_index means every chunk knows its character offset in the original document — handy for citations.

3. Section-aware — one chunk per section

Instead of enforcing a size, this chunker keeps whole sections intact — one chunk per h1/h2 — which is ideal for high-level navigation, section-level search, and summarization. It detects headers with a regex and only falls back to splitting when a section is genuinely oversized:

class SectionAwareChunker(BaseChunker):
    DEFAULT_SECTION_PATTERN = r"^(#{1,2})\s+(.+)$"
    MAX_SECTION_SIZE = 8000  # large, to preserve sections

Because sections can exceed any model’s window, oversized sections degrade gracefully into paragraph-based fragments — each tagged so retrieval knows it’s a piece of a larger whole:

chunk_metadata={
    "section_header": header_text,
    "section_level": len(header_level),
    "is_section_fragment": True,
    "fragment_index": len(chunks),
}

4. Semantic — split at topic boundaries

The most sophisticated option embeds the text and places breakpoints where the topic actually shifts, rather than at arbitrary character counts. The breakpoint algorithm is configurable — percentile, standard_deviation, interquartile, or gradient — and it calls out to a model serving endpoint for embeddings. Use it when passages must be topically coherent and you can afford the embedding cost at index time.

Defaults are a 2000-character ceiling and the percentile breakpoint method, backed by a Databricks gte-large-en embedding endpoint. The important production detail is what happens when embeddings aren’t available — it degrades instead of failing:

class SemanticChunker(BaseChunker):
    DEFAULT_CHUNK_SIZE = 2000
    DEFAULT_CHUNK_OVERLAP = 200
    DEFAULT_BREAKPOINT_THRESHOLD_TYPE = "percentile"
    DEFAULT_EMBEDDINGS_ENDPOINT = "databricks-gte-large-en"

    def chunk(self, content, metadata):
        raw_chunks = self._split_semantic(content)      # embedding-based
        if raw_chunks is None:                          # no embeddings / import error
            raw_chunks = self._split_fallback(content)  # RecursiveCharacterTextSplitter
        final_chunks = self._enforce_max_size(raw_chunks)  # re-split anything > 2000 chars
        ...

Two safety nets matter here. First, if langchain_experimental or the embedding credentials are missing, it falls back to character splitting rather than crashing the index job. Second, semantic splitting can emit a chunk far larger than any topic boundary suggests, so _enforce_max_size re-splits any chunk over the ceiling. Semantic where it can, bounded always.

Metadata is the real payload

The chunk text is only half the story. Every chunk is enriched with metadata that survives into the vector store, so retrieval results carry provenance:

enriched = {
    **base_metadata,
    **chunk_metadata,
    "chunk_index": chunk_index,
    "chunk_id": f"{doc_id}_{chunk_index}",
    "chunk_type": self.chunk_type,
}

That chunk_type field is quietly important: it lets you A/B different strategies in the same index and attribute retrieval quality back to the chunker that produced each hit. chunk_id gives you a stable handle for deduplication and citation.

Choosing a strategy

The four strategies also differ sharply in the shape of the index they produce. Running the same ~40-page equipment manual (roughly 90,000 characters) through each — illustrative, but representative — shows the trade-off in black and white:

The table gives the counts, but I find the shape easier to remember than the numbers:

Section-aware produces ~6× fewer, ~5× larger chunks than simple splitting — great for “which section covers X” navigation, wasteful for pinpoint fact lookup. Semantic sits in between on size but is the only one that pays an embedding bill at index time. The right pick depends on the query:

There’s no universally best chunker — only the best one for a given document shape and query pattern. The win is making the strategy a configuration choice rather than a rewrite, so you can measure retrieval quality per strategy and let the data decide.

Takeaways

• Put every chunker behind one interface and select via config.
• Preserve structure (sections, headers) when the query is navigational; preserve topic coherence (semantic) when the query is conceptual.
• Carry metadata — chunk_type, chunk_id, section headers — all the way into the index so you can debug and attribute retrieval quality.

#rag#chunking#embeddings#retrieval#python