Q-Learning

Train an agent to reach a goal by trial and error and watch its Q-table fill in

Intermediate

Q-table on the grid

Agent Start Last updateTriangle colour: Q(s, a), green high, red low. Arrow: greedy action.

Controls

Map

Algorithm

Learning rate α0.50Discount γ0.90Exploration ε0.30Decay ε by ×0.99 per episode (down to 0.01)

Speed

Episode

Step in episode

Current ε

0.300

Total steps

Last episode reward

–

Mean of last 20

–

Last temporal-difference update

Press Step or Run. Each move updates one Q-value, and the arithmetic shows up here.

Reward per episode

Steps per episode

How to use this visualizer

1
Press Run at Watch speed to follow the agent (the blue dot) one move at a time. The amber outline marks the Q-value that was just updated.
2
Each cell is split into four triangles, one per action. Colour shows Q(s, a): green for good, red for bad. The arrow is the greedy action.
3
Switch speed to Fast or Max, or press +100 episodes, to train quickly. Watch reward per episode climb and steps per episode fall.
4
Read the last temporal-difference update panel to follow the exact arithmetic behind each change.
5
Choose Cliff walk, train Q-learning for a few hundred episodes, then switch to SARSA and train again. The reward chart keeps the previous run so you can compare.
6
Turn on Edit map and tap cells to add walls, goals or pits. Editing resets the agent.

Learning by trial and error

Value iteration needs a full model of the world: every transition probability and reward. Q-learning needs none of that. The agent just acts, sees what happens, and nudges its estimate of Q(s, a), the expected return of taking action a in state s and acting well afterwards.

Each update compares the old estimate with a target built from one real step: the reward received plus the discounted value of the best action in the next state. The difference is the temporal-difference (TD) error. Repeat this enough times, while still exploring, and the Q-table converges to the optimal action values. Good values spread backwards from the goal, one episode at a time.

Q-learning is off-policy: it learns the value of acting greedily even while it behaves ε-greedily. SARSA is on-policy: it learns the value of the ε-greedy behaviour it actually follows. On the cliff walk that difference matters. Q-learning learns the shortest path right along the cliff edge and keeps falling off during exploration; SARSA accounts for its own random moves and learns a safer path one row up, earning more reward while it trains.

Learning rate α

How far each update moves Q(s, a) toward the new target. Large α learns fast but is noisy; small α is smooth but slow.

Discount γ

How much future reward counts. With γ near 1 the agent plans for the long run; with small γ it grabs nearby reward.

Exploration ε

The chance of taking a random action instead of the best known one. Decaying ε explores early and exploits later.

Episodes

One run from the start until the agent reaches a goal or pit, or hits the step limit. Learning happens across many of them.

Key formulas

Q-learning update

Q(s, a) ← Q(s, a) + α [ r + γ max_a' Q(s', a') − Q(s, a) ]

α: learning rate
γ: discount factor
r + γ max Q(s', a'): the TD target; just r when s′ is terminal

SARSA update

Q(s, a) ← Q(s, a) + α [ r + γ Q(s', a') − Q(s, a) ], where a' is the action actually chosen in s'

ε-greedy action selection

a = random action with probability ε, otherwise argmax_a Q(s, a)

Exploration is not optional

With ε = 0 the agent repeats whatever first worked and may never find a better route. Try it on the maze: set ε to 0 before training and compare with ε = 0.3.

Code

Python

import random
from collections import defaultdict

def q_learning(env, episodes=500, alpha=0.5, gamma=0.9, eps=0.3):
    Q = defaultdict(lambda: [0.0] * 4)   # Q[state][action]
    for _ in range(episodes):
        s, done = env.reset(), False
        while not done:
            if random.random() < eps:
                a = random.randrange(4)                 # explore
            else:
                a = max(range(4), key=lambda i: Q[s][i])  # exploit
            s2, r, done = env.step(a)
            target = r if done else r + gamma * max(Q[s2])
            Q[s][a] += alpha * (target - Q[s][a])       # TD update
            s = s2
        eps = max(0.01, eps * 0.99)
    return Q

Practice problems

Problem 1: One Q-learning update

Q(s, Right) = 0.50. The agent moves Right, receives r = −0.04 and lands in s′ whose Q-values are 0.80, 0.20, 0.10 and 0.30. With α = 0.1 and γ = 0.9, what is the new Q(s, Right)?

Show solution

max Q(s′, ·) = 0.80

target = −0.04 + 0.9 × 0.80 = 0.68

δ = 0.68 − 0.50 = 0.18

Q(s, Right) ← 0.50 + 0.1 × 0.18 = 0.518

Q(s, Right) = 0.518

Problem 2: The same step under SARSA

Repeat Problem 1 with SARSA. In s′ the ε-greedy policy happens to explore and picks the action whose value is 0.20. What is the new Q(s, Right), and why does this make SARSA more cautious near a cliff?

Show solution

SARSA uses Q(s′, a′) for the action actually taken: 0.20.

target = −0.04 + 0.9 × 0.20 = 0.14

δ = 0.14 − 0.50 = −0.36

Q(s, Right) ← 0.50 + 0.1 × (−0.36) = 0.464

SARSA folds the cost of its own exploratory moves into its values. Next to a cliff, those random moves sometimes fall off, so cells along the edge get low values and SARSA learns to keep its distance.

Q(s, Right) = 0.464 (Q-learning gave 0.518)

Discussion

Questions, ideas or a different approach? Join the conversation.