Train a real byte-level BPE tokenizer on your own text, then watch it encode and decode, the same algorithm behind GPT-style language models. Every count, merge, and token id is computed live.
BPE learns a vocabulary from data. It starts with the 256 possible bytes, then repeatedly finds the most frequent adjacent pair of tokens in the corpus and fuses it into a new token.
1 · The corpus is first pre-tokenized with the GPT-2 rule: split into word-like pieces so merges never cross word boundaries. Notice pieces keep their leading space — ␣low and low are different tokens, exactly as in real tokenizers.
2 · Each piece is turned into raw UTF-8 bytes. The starting vocabulary is every possible byte: ids 0–255. This is why byte-level BPE can encode any text, in any language, with no unknown-token fallback.
3 · Each training step counts every adjacent token pair across the corpus (weighted by how often each piece occurs), picks the most frequent pair, and merges it into one new token with the next free id (256, 257, …). Ties go to the pair seen first.
4 · The ordered list of merges is the tokenizer. Encoding later just replays these rules in learned priority order.
Unique pieces with their counts. Watch pairs fuse into single chips as merges are learned. Colored chips are learned tokens; gray chips are raw bytes.
The highlighted pair is the current winner — it merges next.
This ordered list is the entire tokenizer. Rule order = priority during encoding.
Encoding replays the learned merges by priority (rank 0 first — the exact GPT-2 loop), never by re-counting. Decoding walks the ids one by one: look up bytes, append, interpret as UTF-8. Step through both below.
Encoder — text → token ids
Each shaded group is one pre-token — merges never cross group boundaries. The highlighted pair merges next because it matches the learned rule shown below (also lit up in the vocabulary panel).
Every rule fired during this encode, in order. Notice the rule numbers only ever increase — applying a merge can never unlock an earlier rule, because earlier rules are built entirely from earlier tokens.
Decoder — token ids → text
The highlighted token is decoded next; faded tokens are already consumed.
Ids 0–255 are the raw bytes (always present). Below are the learned tokens in the order they were merged — hover any chip for its id and bytes. This is the lookup table both sides use.
Encodes the text from the encoder box with only the first k merges enabled — the same bytes get cheaper as the vocabulary grows. This is the trade-off real tokenizers tune: bigger vocabulary → fewer tokens per text.