RAG Chunking Strategies for Internal Documents

RAG chunking strategies fail when one token limit is applied to every internal document. The index may contain all the words while retrieval returns a policy clause without its heading, a table row without its column labels, or text from two access levels in one result. Larger chunks then bury the answer, while more overlap creates duplicates and noisy citations. Fix the retrieval unit before tuning embeddings or adding context. Define hard boundaries for permissions and provenance, choose structural rules for each document type, connect small evidence chunks to parent context, and test each chunking version against real questions before rebuilding the production index.

Start with the retrieval failure

Chunking is the step between parsing and indexing. Parsing should recover document elements such as titles, paragraphs, lists, and tables. Chunking decides which of those elements travel together as one searchable unit. A parser can reconstruct a document correctly while a poor chunker destroys the relationships that retrieval needs.

Common failures point to different boundary problems:

• A result contains the right sentence but loses the section title that changes its meaning.
• A table row is retrieved without headers, units, or the note that defines an exception.
• A short answer sits inside a long passage whose unrelated text weakens its embedding.
• One chunk joins content from separate tenants, departments, or access-control groups.
• Overlap returns four near-identical results and crowds out independent evidence.
• A document edit changes many downstream chunk identifiers, which makes deletion and citation repair unreliable.

Do not respond to all six failures by changing chunk size. First classify the miss as structure, scope, access, context, duplication, or lifecycle. The fix should target that class.

Microsoft's document chunking guidance distinguishes fixed-length splitting from approaches that use document layout and content structure. That distinction should survive in your pipeline. Fixed-length splitting is a useful baseline for uniform text. It should not erase headings, table boundaries, or security metadata that the parser already found.

Define a versioned chunk contract

A chunk needs more than text and an embedding. Store enough information to enforce access, rebuild the unit, trace a citation, and compare two chunking versions. A practical record includes stable source identity, source revision, structural location, access scope, chunking version, parent identity, and the exact source span.

{
  "chunk_id": "policy-42:rev-7:benefits-eligibility:03",
  "source_id": "policy-42",
  "source_revision": "rev-7",
  "chunking_version": "policy-v4",
  "document_type": "policy",
  "section_path": ["Benefits", "Eligibility", "Contractors"],
  "parent_id": "policy-42:rev-7:benefits-eligibility",
  "access_scope": ["hr", "managers"],
  "source_span": {"page": 12, "start": 418, "end": 1162},
  "text": "Contractors become eligible after..."
}

Treat access_scope as a hard boundary. Never merge adjacent elements if their effective permissions differ. Apply the same rule to tenant, legal hold, region, retention class, and any other metadata that changes who may retrieve the text. Filtering after retrieval cannot repair a chunk that already combines allowed and forbidden content.

Keep the source span deterministic. A citation service should be able to open the original revision and highlight the exact evidence. If the parser emits element identifiers, retain them as well. This lets operators distinguish a parsing defect from a chunk assembly defect without guessing from rendered text.

Version the contract whenever rules change. Store old and new chunks in separate index namespaces or attach a mandatory version filter. Mixing both versions in one unfiltered search produces duplicate evidence and hides whether the new strategy improved retrieval.

Choose rules by document type

A global default is still useful, but it should be a fallback. Document families have different semantic boundaries.

Policies and handbooks

Start a new parent at each major heading. Keep a heading path with every child chunk so a clause carries its scope. Avoid crossing section boundaries when headings contain conditions such as region, employment class, product plan, or effective date. If one section exceeds the embedding limit, split on paragraphs or sentences inside that section and repeat the heading path as metadata.

Tables and forms

Keep headers, units, and nearby captions with each group of rows. A row serialized as 120 | 45 | 8 is useless without labels. For wide tables, create a normalized textual representation that repeats the row key and column names. Preserve a reference to the original table for display and citation. Do not mix table rows with surrounding narrative merely to hit a target size.

Procedures and runbooks

Keep ordered steps together when later steps depend on earlier state. Split at procedure boundaries, not every few sentences. Carry prerequisites, warning labels, and rollback instructions into the retrievable unit or its parent. A chunk that retrieves step seven without the prerequisite from step one can produce an answer that is locally accurate and operationally unsafe.

Meeting transcripts

Use speaker turns and topic shifts, then cap oversized topic blocks. Store the meeting date, participants, and topic label. Keep decisions and assigned actions as separate child chunks linked to a larger discussion parent. This reduces the chance that retrieval returns a proposal as if it were the final decision.

Mixed PDFs

Use the parser's recovered element types. Unstructured's chunking documentation describes chunking parsed document elements rather than splitting one raw text string, including a by_title strategy that preserves section boundaries. That approach is useful when PDFs mix prose, tables, captions, and page furniture. Repair reading order and element types before chunk construction. Chunking cannot restore structure that parsing discarded.

Use child retrieval with bounded parent expansion

Small chunks often retrieve precise evidence, but they can omit the context needed to interpret it. Large chunks carry context but can dilute the query signal. Parent-child retrieval separates those jobs.

Index compact child chunks for candidate retrieval. Each child points to a parent section that contains the surrounding heading, definitions, and related paragraphs. After ranking and access checks, expand only the selected children to their parents or to a bounded context window. Deduplicate parents before sending context to the model.

LlamaIndex documents sentence, semantic, hierarchical, and file-specific node parsers. Those controls show that retrieval units can follow several structures. Pick one through evaluation rather than framework defaults. A semantic splitter may help prose with weak headings, but it still needs hard limits and permission boundaries. A hierarchical parser can support parent expansion, but expansion must not bypass access filters or flood the context window.

Use overlap only where boundary loss appears in tests. Sentence-aware splitting with a small overlap can preserve a definition that spans a cut. Large blanket overlap inflates index size, creates repeated citations, and reduces evidence diversity in the top results. Record overlap in the chunking version so cost and retrieval changes can be traced.

Build a representative chunking test set

Evaluate chunking before answer generation. Otherwise model fluency can hide retrieval mistakes. Start with questions that require specific structures, not a random sample of easy lookups.

Include at least these cases:

1. A fact that sits directly under a heading and changes meaning without that heading.
2. A table lookup that requires row labels, column labels, and units.
3. A procedure question where prerequisites and rollback steps matter.
4. A question whose answer crosses two adjacent paragraphs.
5. A question with the same term in two departments but different access rules.
6. A document revision where the old answer must disappear.
7. A broad question that needs a parent section rather than one sentence.
8. A narrow question where a large parent would add distracting text.

For each question, label the source span that supports the answer. Measure whether the top retrieved units contain that span, whether the citation resolves to the correct revision, how much unrelated text is included, and whether any inaccessible chunk appears before the model call. Keep answer quality as a later test, not a substitute for retrieval evidence.

A simple comparison record can capture the decision:

question_id: benefits-17
question: When do contractors qualify for dental coverage?
required_source: policy-42
required_revision: rev-7
required_span: page-12-lines-18-31
forbidden_scopes:
  - executive-only
results:
  policy-v3:
    support_in_top_5: false
    duplicate_results: 2
  policy-v4:
    support_in_top_5: true
    duplicate_results: 0

Use production-derived questions after removing sensitive text, plus deliberately difficult fixtures for access and lifecycle failures. Compare chunking versions with the same corpus revision, embedding model, retrieval settings, and reranker. Changing several layers at once prevents a useful diagnosis.

Roll out chunk changes without mixing evidence

A chunking change is an index migration. Build the new version beside the old one. Verify document counts, source revisions, access metadata, parent links, and deletion coverage before serving traffic. Run the labeled retrieval set against both versions, then use shadow queries or a small canary if production traffic is available.

Set explicit acceptance rules. For example, the new version must improve support-span retrieval without increasing forbidden-scope retrieval, unresolved citations, duplicate parents, or context size beyond agreed limits. The exact thresholds belong to the team's own baseline. Do not copy illustrative numbers from another corpus.

Keep rollback cheap. Query routing should select one chunking version. If the canary fails, route all reads back to the old index without rewriting documents during the incident. After acceptance, stop writes to the old version, retain it for the rollback window, then delete it through the same source-revision inventory used to build the new index.

Deletion needs a negative check. Remove one source revision and prove that none of its child chunks, parents, embeddings, cache entries, or citations remain retrievable. Update testing needs the same discipline: replace a source, index the new revision, and prove the old statement cannot appear.

Diagnose misses with decision rules

When a relevant span never appears in candidates, inspect parsing and chunk construction before changing the model. If the span exists but shares a chunk with too much unrelated text, reduce the child size or use a structural split. If the child ranks well but lacks context, add parent expansion. If repeated chunks dominate the result set, reduce overlap and deduplicate by parent. If citations point to the wrong passage, repair source spans and revision identity. If access filters remove the only useful result, fix corpus permissions or source ownership rather than weakening the filter.

Semantic chunking should solve a measured boundary problem. It adds computation and another model or embedding dependency to index construction. Use it when heading structure is weak and evaluation shows that fixed or sentence-aware splitting separates concepts badly. Keep deterministic maximum sizes, stable source spans, and access boundaries around it.

Start with one high-value document family. Write its chunk contract, label 20 to 50 representative retrieval questions, and compare the current strategy with one structural alternative. Ship the new version only when it retrieves better evidence and passes the permission, citation, update, and deletion checks.

References

1. Microsoft Learn document chunking guidance supports the distinction between fixed-length and structure-aware chunking for vector search and RAG.
2. Unstructured chunking documentation supports element-aware chunk construction, size controls, and section-preserving by_title behavior.
3. LlamaIndex node parser modules supports sentence, semantic, hierarchical, and file-specific retrieval-node construction.

---