Tokenization

Words become model-ready pieces

A model never sees your sentence in the form you wrote it. A tokenizer chops the text into reusable fragments and maps each one to a number, and those numbers are the only thing that travels onward.

The fragments are not words. Common words usually survive intact; rarer ones get split, so engineering might arrive as engineer plus ing. Punctuation is generally a piece of its own.

Two practical consequences. Every context limit and every line of your bill counts tokens, not words. And the ratio between the two is not constant — code, identifiers, tables and non-English text all split more aggressively than ordinary English prose, so two inputs that look the same length on screen can cost noticeably different amounts.