Natural Language Processing

Follow text on its way through a language model: split into tokens, mapped to vectors, and turned back into words one sampled token at a time.

Key concepts

Tokenization

Text is split into subword pieces from a learned vocabulary before a model sees it

Embeddings

Each token becomes a vector, and similar meanings end up close together

Language Models

Predict a probability for every possible next token

Decoding

Temperature, top-k and top-p decide how the next token is picked from those probabilities

Visualizations

Tokenization (BPE)

Beginner

Watch byte-pair encoding merge characters into subword tokens step by step

BPESubwordsVocabularyMerges
Open

Word Embeddings

Intermediate

Explore words as vectors, cosine similarity and analogies like king − man + woman

Word2VecCosine SimilarityAnalogiesVector Space
Open

Text Generation & Sampling

Intermediate

Adjust temperature, top-k and top-p and see how they reshape next-token probabilities

TemperatureTop-kTop-pSoftmax
Open