I animated what GPT-4o's tokenizer does to "strawberry": real token ids, a tiny BPE run, and why the space belongs to the word (6 min video) [D]

Reddit r/MachineLearning2026-11-01 18:01:28大模型,OpenAI,DeepSeek,推理思考,扩散模型,招聘HR原文 ↗

I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong.

It follows one word the whole way:

"How many r's are in strawberry?" is 8 tokens in o200k_base (the GPT-4o / GPT-5 tokenizer). " strawberry" is a single token, id 101830, so the three r's are hidden inside one id.
Why pieces of words: letters make sequences about 4x longer, whole words need a huge vocabulary and can't handle new words.
A tiny byte pair encoding run on a made-up text, merge by merge, until strawberry becomes st | raw | berry.
The space belongs to the word: "strawberry", " strawberry" and "Strawberry" are three different token sequences.
Why reasoning models count letters better: in OpenAI's 2025 paper "Why Language Models Hallucinate", DeepSeek-R1 spells DEEPSEEK out letter by letter before counting, while DeepSeek-V3 answered 2 or 3 (the answer is 1).
Bonus: the same Hindi sentence took 21 tokens with GPT-4's tokenizer and 5 with GPT-4o's.

Every split comes from running tiktoken 0.14.0. The narration is an AI voice reading my own script.