Chunking is an information boundary

Ask what a passage must carry with it

A sentence saying that a feature is enabled can become misleading when separated from a heading that limits the statement to one release. A table row can lose its units when detached from the column header. An exception can reverse the meaning of the preceding rule. These are information dependencies, not merely formatting preferences. I would begin a chunking design by identifying which relationships must survive independent retrieval. The object returned by search should contain enough context to interpret its claims, or it should carry explicit references that let the next stage recover the missing context before answering.

This makes chunking a contract between ingestion and retrieval. Ingestion decides what counts as a retrievable unit; the answer stage often assumes each unit is meaningful on its own. If those assumptions differ, the generator may receive perfectly relevant fragments that support the wrong interpretation. I would test the boundary using documents with scope headings, footnotes, exceptions and cross-references, not only clean paragraphs of self-contained prose. A strategy that looks good on a flat text sample can fail on precisely the structured material that users consult when the details matter most.

Account for overlap before calling it free context

Consider an illustrative 10,000-token document split into windows of 500 tokens with 100-token overlap. The stride is 400. Covering the document requires 25 windows under a simple final-partial-window policy: the last starts at token 9,600 and contains 400 tokens. The stored window payload totals 12,400 tokens, an increase of 24 percent over the original. The exact overhead depends on document length and boundary handling. Overlap can preserve local continuity, but it also spends embedding work, storage and later context on repeated material. These are invented inputs for accounting, not a measured corpus.

Repeated content can distort results as well as cost. Several adjacent windows may all rank highly because they contain the same attractive sentence, crowding out independent evidence from other documents. A reranker may spend most of its budget choosing among near duplicates. I would deduplicate by source span or diversify selection before assembling the final context, while retaining enough overlap to avoid cutting necessary relationships. The right balance is not the overlap percentage with the best aesthetic appearance. It is the policy that preserves answer-bearing dependencies without consuming the retrieval budget with repeated copies of one claim.

length, width, overlap = 10_000, 500, 100
stride = width - overlap
starts = list(range(0, length, stride))
windows = [(s, min(s + width, length)) for s in starts]
assert len(windows) == 25
assert sum(end-start for start, end in windows) == 12_400

Separate the search unit from the reading unit

The best text for locating evidence need not be the best text for interpreting it. A small passage can match a narrow question precisely, while its parent section supplies definitions and qualifications. I would allow retrieval to return a passage identifier and then expand to a bounded reading context using the source structure. Expansion should follow explicit relationships such as parent heading, neighboring paragraph or referenced table, rather than blindly adding the same number of tokens on each side. The expansion budget should remain visible so one match cannot pull an entire large document into every answer.

Khattab and Zaharia’s ColBERT paper is useful here because it separates document encoding from a later interaction with query representations. It does not prescribe my proposed parent-expansion policy. The broader lesson I take is that retrieval need not collapse every design choice into one stored vector and one fixed text window. Search representations and displayed evidence can have different granularities if their identities remain connected. A system may index compact units while returning a richer, source-faithful object to the answer stage, provided it records exactly which text matched and which additional text was supplied for interpretation.

References: [1] ColBERT — Khattab and Zaharia

Preserve structure instead of flattening it away

For a technical manual, I would retain heading paths, version labels, table captions, list ancestry and stable source offsets. A chunk from a nested exception should identify the rule it modifies. A table extraction should preserve header-to-cell relationships rather than emit a sequence of unrelated values. These metadata are useful only if the retrieval and answer stages consume them. Storing a heading in a database column while omitting it from the evidence passed to the model leaves the semantic boundary broken. The representation should make required context hard to lose accidentally.

There are limits to automatic structure recovery. PDFs can place related text far apart in extraction order, and visually adjacent columns can be interleaved incorrectly. I would include ingestion quality checks and an explicit unsupported-format path rather than treating every extracted string as authoritative text. A source whose table structure cannot be recovered may need to be retrieved as a document for a different reading process. The relevant failure is not that chunking produced an unusual token count. It is that the system can no longer explain which value belonged to which label and nevertheless presents the resulting fragment as evidence.

Evaluate the evidence that survives the split

A useful chunking evaluation starts with questions and the minimal source material needed to answer them correctly. Measure whether retrieval returns that material together with necessary qualifiers, not only whether it returns a passage containing the expected keyword. Include cases where the answer spans a heading and body, a rule and exception, or two linked sections. I would annotate missing-context failures separately from missed-document failures. They require different remedies: increasing candidate count may help find another passage, but it may not repair a chunk that systematically discards the relationship needed to interpret every match.

Liu and colleagues’ Lost in the Middle paper reports that the placement of relevant information within long contexts affects performance in the evaluated models and tasks. That finding does not identify a universal optimal chunk size. It does challenge the assumption that adding more surrounding text always solves context loss. My evaluation would vary both the evidence unit and the assembly order, with fixed questions and source revisions. A larger reading context can restore a qualifier while also burying the decisive sentence among distractors. The complete retrieval-and-reading path determines whether the added material helps.

References: [2] Lost in the Middle — Liu and colleagues

Treat chunk changes as a data migration

Changing a splitter changes the indexed objects, their identifiers and often their vectors. I would version the chunking policy and rebuild references deliberately. A citation should resolve to the exact document revision and span that supported the answer, even after the latest index uses different boundaries. Otherwise a saved answer can point to a chunk identifier that now contains unrelated text. Stable document identity is helpful, but it is not enough when the meaning of a chunk number depends on whichever splitter happened to run most recently.

The strongest argument for fixed-size windows is their simplicity and predictable resource usage. That is a real advantage, especially for relatively uniform text. I would not replace them with a complicated semantic splitter merely because the latter sounds more intelligent. Start with a simple boundary rule, add source structure where it demonstrably preserves important relationships, and measure the resulting failure cases. A learned splitter can itself be nondeterministic, expensive or inconsistent across document updates. Its output still needs versioning and evidence checks; semantic in its name does not grant semantic correctness to every boundary it chooses.

Make missing context a visible retrieval outcome

When a returned passage depends on context that cannot be recovered, the system should be able to say so. I would attach a completeness status to extracted evidence: self-contained, expanded with dependencies, or missing required context. That status need not be exposed as jargon to the user. It can determine whether the answer cites a narrower claim, opens the source for further inspection or abstains. This is different from a relevance score. A passage can be highly relevant and still be incomplete in exactly the way that makes a confident answer unsafe.

The practical review question is simple: what would a careful reader misunderstand if this chunk appeared alone? Apply it to headings, tables, exceptions and version notes before optimizing token counts. Chunking cannot preserve every dependency at every scale, so the design needs both sensible units and a recovery path. I want the retrieval system to know when a fragment is a complete piece of evidence and when it is merely a pointer toward one. That distinction makes a smaller context useful without pretending that the meaning removed at its boundaries never mattered.