Text Generation & Sampling

A language model only ever predicts the next token. Temperature, top-k and top-p decide how that prediction becomes a choice. Adjust them and generate text one token at a time.

Intermediate

The next-token scores here are hand-crafted and illustrative: each prompt has a small tree of a dozen or so likely continuations instead of a real model's full vocabulary. The softmax, temperature, top-k, top-p, sampling and entropy calculations are all real.

Prompt

Seed

The cat sat on the

sampled greedy0 tokens generated, sequence probability 1.00e+0

Next-token distribution

Bar length is the probability after temperature. Hatched bars are removed by top-k or top-p; the right column is the renormalised probability actually used for sampling.

Tokenlogitp after Tp(T)final

␣mat3.4

36.0%36.0%

␣floor2.6

16.2%16.2%

␣couch2.3

12.0%12.0%

␣windowsill2.1

9.8%9.8%

␣bed1.8

7.3%7.3%

␣keyboard1.5

5.4%5.4%

␣roof1.1

3.6%3.6%

␣table1.0

3.3%3.3%

␣fence0.7

2.4%2.4%

␣grass0.3

1.6%1.6%

␣laptop0.2

1.5%1.5%

␣piano-1.0

0.4%0.4%

␣moon-1.2

0.4%0.4%

␣banana-2.4

0.1%0.1%

Decoding settings

Temperature 1.00Below 1 sharpens the distribution, above 1 flattens it.Top-k offKeep only the k most likely tokens (0 = off).Top-p offKeep the smallest set of top tokens whose probabilities add up to p.

Entropy

After temperature

2.88 bits

like 7.4 equal choices

After top-k / top-p

2.88 bits

14 of 14 tokens kept

How to use this visualizer

1
Choose a prompt. "The capital of France is" is confident; "My favourite colour is" is open-ended.
2
Drag the temperature slider and watch the bars sharpen or flatten, and the orange dot move along the entropy curve.
3
Turn on top-k or top-p: hatched bars are removed and the remaining probabilities are rescaled to sum to 1.
4
Press Sample next token to draw a token using the seeded random number u, or Greedy next token to take the most likely one.
5
Keep going until the branch ends. Use Undo to step back, or change the seed and reset to get a different sample.

How text generation works

Given the text so far, a language model produces one logit (a raw score) for every token in its vocabulary. Softmax turns the logits into probabilities. A decoding strategy then picks one token, appends it to the text, and runs the model again. Everything a chatbot writes is produced this way, one token at a time.

Always taking the most likely token (greedy decoding) tends to produce safe, repetitive text. Sampling adds variety, but sampling from the full distribution occasionally picks a very unlikely token and derails the sentence. Temperature, top-k and top-p are the standard dials for balancing the two.

One token at a time

The model outputs a score for every token in its vocabulary; one is picked, appended to the text, and the process repeats.

Logits become probabilities

Softmax turns raw scores into a probability distribution that sums to 1.

Entropy measures uncertainty

A confident prediction (the capital of France) has low entropy; an open question (a favourite colour) has high entropy.

Seeds make sampling repeatable

Sampling uses random numbers. Fixing the seed reproduces the same choices, which is how this page stays deterministic.

Key formulas

Softmax with temperature

pᵢ = exp(zᵢ / T) / Σ_j exp(z_j / T)

zᵢ: logit of token i
T: temperature (T > 0); T = 1 leaves the model distribution unchanged

Top-p (nucleus) set

V_p = smallest set of most likely tokens with Σ_{i ∈ V_p} pᵢ ≥ p, then pᵢ′ = pᵢ / Σ_{j ∈ V_p} p_j

Entropy

H(p) = −Σ_i pᵢ log₂ pᵢ

H = 0: one token has probability 1
H = log₂ n: all n tokens are equally likely

Strategy	How it works	Effect
Greedy	Always take the most likely token	Deterministic; can get repetitive or bland
Temperature	Divide logits by T before softmax	T < 1 sharper, T > 1 flatter, T → 0 is greedy
Top-k	Keep only the k most likely tokens	Removes the long tail, but k is the same for confident and uncertain steps
Top-p (nucleus)	Keep the smallest set whose probability sums to p	Adapts: few tokens when confident, many when uncertain

Code

Python

import numpy as np

def sample_next(logits, temperature=1.0, top_k=0, top_p=1.0, rng=np.random.default_rng(42)):
    if temperature == 0:
        return int(np.argmax(logits))                 # greedy
    z = np.asarray(logits) / temperature
    probs = np.exp(z - z.max())
    probs /= probs.sum()

    order = np.argsort(-probs)
    keep = np.ones_like(probs, dtype=bool)
    if top_k > 0:
        keep[order[top_k:]] = False
    if top_p < 1.0:
        cum = np.cumsum(probs[order] * keep[order]) / probs[keep].sum()
        cutoff = np.searchsorted(cum, top_p) + 1      # smallest set reaching p
        keep[order[cutoff:]] = False

    probs = np.where(keep, probs, 0)
    probs /= probs.sum()
    return int(rng.choice(len(probs), p=probs))

Practice problems

Problem 1: Temperature by hand

Three tokens have logits (2, 1, 0). Compute their probabilities at T = 1 and at T = 0.5.

Show solution

T = 1: e² = 7.389, e¹ = 2.718, e⁰ = 1, sum = 11.107

p = (0.665, 0.245, 0.090)

T = 0.5: logits become (4, 2, 0). e⁴ = 54.598, e² = 7.389, e⁰ = 1, sum = 62.987

p = (0.867, 0.117, 0.016)

Halving the temperature doubled the gaps between logits, so the top token became much more likely.

Problem 2: Top-k versus top-p

Five tokens have probabilities (0.50, 0.20, 0.15, 0.10, 0.05). Which tokens survive top-p with p = 0.8, and what are their renormalised probabilities? Compare with top-k, k = 2.

Show solution

Cumulative sums: 0.50, 0.70, 0.85. The third token is the first to reach 0.8, so three tokens are kept.

Kept mass = 0.85. Renormalised: 0.50/0.85, 0.20/0.85, 0.15/0.85

top-p: (0.588, 0.235, 0.176)

Top-k with k = 2 keeps the first two: mass 0.70, so 0.50/0.70 and 0.20/0.70.

top-k: (0.714, 0.286)

Discussion

Questions, ideas or a different approach? Join the conversation.