AI engineering · 06 of 42

Semantic Chunking

Split documents by meaning

Scroll

Split documents by meaning

Before anything can be retrieved it has to be cut into pieces. The easy way is every N tokens, which is fast and pays no attention to what it is cutting through.

Semantic chunking cuts where the meaning changes instead. Installation stays with installation, billing with billing, and a section is not sliced down the middle.

This is the least glamorous decision in a retrieval system and one of the most consequential. A fixed cut can leave one chunk holding half a billing rule and half a security warning, so a billing question retrieves a fragment of both and answers neither well. Retrieval quality is largely decided here, before anyone runs a search.

Retrieval
CUT WHERE THE MEANING CHANGES one long document Install requirements setup first run Billing plans invoices refunds Security access encryption warnings A fixed 500-token cut lands mid-sentence and mid-topic, so one chunk holds half a billing rule and half a security warning. Retrieval quality is decided here, before anyone searches: the boundaries determine what can be found later.
One long document split by topic rather than by length, so each chunk holds one idea.