Tokenization (BPE)

Language models never see letters or words directly. Train a byte-pair encoding tokenizer one merge at a time and watch characters fuse into subword tokens.

Beginner

1. Training corpus

51 pre-tokens, 28 distinct words, 22 distinct characters. ▁ marks a word that follows a space.

Merges0/39

2. Pair counts

Every adjacent pair of symbols, counted across the corpus (weighted by word frequency). The top pair is merged next.

Next merge #1: e + r → er (14 times)

Pair	Merged	Count
e + r	er	14
▁ + w	▁w	11
w + a	wa	10
▁ + r	▁r	9
a + l	al	9
u + n	un	9
l + k	lk	8
r + u	ru	8
▁ + t	▁t	7
n + n	nn	7
t + h	th	7
h + e	he	6

3. Vocabulary

Vocabulary size

22 chars + 0 merged

Corpus tokens

261

Chars per token

1.00

Base characters

,.abcdefghiklnorstuwy▁

Learned merges, in order

None yet. Press Step to learn the first merge.

4. Corpus after 0 merges

Each coloured box is one token; colour follows the token id.

▁the▁runner▁was▁running▁and▁the▁walker▁was▁walking.▁the▁faster▁runner▁runs▁faster▁than▁the▁slower▁walker.▁a▁newer▁runner▁is▁running▁the▁newest▁race.▁the▁slowest▁walker▁walks▁slower,▁but▁walking▁is▁still▁fun.▁runners▁and▁walkers▁were▁running▁and▁walking▁all▁day.

5. Encode new text

Uses the first 0 merges, applied in the order they were learned. Words never seen in training still split into known pieces; characters that never appeared become UNK.

▁21t17h9e6▁21f7a2s16t17e6s16t17▁21w19a2l12k11e6r15s16▁21w19e6r15e6▁21jUNKo14g8g8i10n13g8

ids: [21, 17, 9, 6, 21, 7, 2, 16, 17, 6, 16, 17, 21, 19, 2, 12, 11, 6, 15, 16, 21, 19, 6, 15, 6, 21, UNK, 14, 8, 8, 10, 13, 8]

Character-level

one token per character

BPE (0 merges)

1 unknown

Word-level

one token per word or mark

How to use this visualizer

1
Pick a preset corpus or type your own text into the training box. Training restarts whenever the corpus changes.
2
Press Step to perform one merge. The highlighted row in the pair table is the pair that gets merged.
3
Press Play to run merges automatically, or drag the Merges slider to jump to any point in training.
4
Watch the coloured corpus: neighbouring boxes fuse as their pair is merged, and the vocabulary list grows.
5
Type into the encode box to tokenize new text with the merges learned so far, and compare the token count with character-level and word-level splitting.

What is tokenization?

A neural network works on numbers, so text has to be cut into pieces called tokens, and each token is replaced by an integer id from a fixed vocabulary. The choice of pieces matters: whole words make the vocabulary enormous and fail on new words, while single characters make every sentence very long.

Byte-pair encoding (BPE) finds a middle ground. It starts from single characters and repeatedly merges the most frequent adjacent pair into a new token. Common words end up as one token, and rare words are spelled out of a few familiar subwords. GPT models, Llama and many others use a variant of this algorithm (usually on raw bytes rather than characters, so no input is ever unknown).

Granularity	Vocabulary	Sequence length	Unknown words	Trade-off
Character	Tiny (about 100)	Very long	None	Model must learn spelling from scratch
Word	Huge (100k+)	Short	Common	Every typo or new word is out of vocabulary
Subword (BPE)	Chosen (30k to 200k)	Medium	Rare	Frequent words stay whole, rare words split into pieces

The algorithm

1
Split the corpus into words (pre-tokens) and count how often each word occurs.
2
Write every word as a sequence of single characters. The set of characters is the starting vocabulary.
3
Count every adjacent pair of symbols across the corpus, weighting each pair by the frequency of its word.
4
Merge the most frequent pair everywhere it occurs and add the merged symbol to the vocabulary.
5
Repeat from step 3 until the vocabulary reaches the target size (or no pair repeats).

Choosing the next merge

(a, b)* = argmax_{(a, b)} Σ_w f(w) · count(w, a b)

f(w): how many times word w occurs in the corpus
count(w, a b): how many times symbol a is directly followed by b inside w

Vocabulary size

|V| = |base characters| + number of merges

About the ▁ marker

This visualizer prefixes words that follow a space with ▁, the SentencePiece convention. It lets the tokenizer learn that ▁run at the start of a word is different from run in the middle of "overrun". GPT-2 does the same thing with a leading space character.

Why subwords work well

Frequent words become one token

After enough merges, words like "the" or "running" are a single vocabulary entry.

Rare words still work

An unseen word such as "jogging" is spelled out of smaller known pieces instead of becoming unknown.

Merges are ordered

Encoding replays the merges in the order they were learned, so the tokenizer is fully deterministic.

Vocabulary size is a dial

More merges give shorter sequences but a larger embedding table. Real models stop at a fixed vocabulary size.

Code

Python

from collections import Counter

def train_bpe(words: Counter, num_merges: int):
    # each word starts as a tuple of characters
    vocab = {tuple(w): c for w, c in words.items()}
    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for symbols, count in vocab.items():
            for a, b in zip(symbols, symbols[1:]):
                pairs[(a, b)] += count
        if not pairs:
            break
        (a, b), freq = pairs.most_common(1)[0]
        if freq < 2:
            break
        merges.append((a, b))
        vocab = {merge(s, a, b): c for s, c in vocab.items()}
    return merges

def merge(symbols, a, b):
    out, i = [], 0
    while i < len(symbols):
        if i + 1 < len(symbols) and symbols[i] == a and symbols[i + 1] == b:
            out.append(a + b); i += 2
        else:
            out.append(symbols[i]); i += 1
    return tuple(out)

Practice problems

Problem 1: The first three merges

A corpus has the word counts hug ×10, pug ×5, pun ×12, bun ×4, hugs ×5. Every word starts as single characters. Which three merges does BPE learn first? (Count words separately, without the ▁ word-start marker the visualizer adds, so the answer differs if you paste this corpus above.)

Show solution

Pair counts: h u = 10+5 = 15, u g = 10+5+5 = 20, p u = 5+12 = 17, u n = 12+4 = 16, b u = 4, g s = 5.

Merge 1: u + g → ug (20)

Now: h ug, p ug, p u n, b u n, h ug s. Pairs: h ug = 15, p ug = 5, p u = 12, u n = 16, b u = 4, ug s = 5.

Merge 2: u + n → un (16)

Now: h ug, p ug, p un, b un, h ug s. Pairs: h ug = 15, p un = 12, p ug = 5, ug s = 5, b un = 4.

Merge 3: h + ug → hug (15)

Problem 2: Encoding unseen words

Using only the three merges from problem 1 (u+g, u+n, h+ug) and the base vocabulary {b, g, h, n, p, s, u}, how are "bug" and "thugs" tokenized?

Show solution

bug: b u g → the only applicable merge is u+g → [b, ug]. No merge joins b and ug.

bug = [b, ug], 2 tokens

thugs: t is not in the base vocabulary, so it is unknown. h u g s → u+g gives h ug s, then h+ug gives hug s.

thugs = [UNK, hug, s], 3 tokens

Byte-level BPE avoids the UNK by starting from the 256 possible byte values instead of characters.

Discussion

Questions, ideas or a different approach? Join the conversation.