Multi-Layer Perceptron

A multi-layer perceptron stacks layers of neurons with non-linear activations. Configure one, train it with backpropagation right here, and watch it learn boundaries a single perceptron never could.

Intermediate

Dataset

Hidden layers: 2

Weight seed: 1

Same seed, same starting weights.

Decision boundary

Class 0 Class 1Background: predicted probability (white = 50%)

Epoch

0

Loss (BCE)

0.7087

Accuracy

54.5%

Training loss

0.710epochs

Network

Blue lines are positive weights, orange are negative; thicker means larger |w|. Neuron fill shows the activation for the input (0.5, 0.5).

inputh1 (4)h2 (4)output

How to use this visualizer

  1. 1
    Pick a dataset. Orange points are class 0, blue points are class 1.
  2. 2
    Choose 0 to 3 hidden layers and drag each slider to set its width (1 to 8 neurons). Changing the architecture re-initialises the weights from the seed.
  3. 3
    Choose the hidden activation, learning rate and batch size.
  4. 4
    Press Train to run one epoch per frame, or Step to run a single epoch. Watch the background colour (the predicted probability) reshape itself around the data.
  5. 5
    Press New seed to start from different random weights and see how the result depends on initialisation.

How an MLP learns

Each layer computes a = f(W·a_prev + b). The first layer sees the raw coordinates (x, y); every later layer sees the previous layer's outputs. The last neuron uses a sigmoid, so its output is the probability that the point is class 1. The background colour of the plot is exactly this probability evaluated on a grid.

Training minimises the binary cross-entropy loss. For every minibatch, the forward pass computes predictions, the backward pass uses the chain rule to get the gradient of the loss with respect to every weight, and stochastic gradient descent moves each weight a small step against its gradient. One epoch is one pass over all 200 points in shuffled minibatches.

Zero hidden layers

The network is logistic regression. It can only draw a straight line, so XOR and circles stay at about 50% accuracy.

One small hidden layer

A handful of tanh or ReLU units is enough for XOR and circles: each unit contributes one bend in the boundary.

Spirals need depth

Try 3 layers of 8 ReLU units. Deeper networks compose simple bends into the winding spiral boundary.

Learning rate and batch size

Too high a learning rate makes the loss jump around or blow up; batch size 1 is noisy, full batch is smooth but slow per epoch.

Key formulas

Forward pass

z(l) = W(l)a(l−1) + b(l), a(l) = f(z(l)), ŷ = σ(z(L))

W⁽ˡ⁾, b⁽ˡ⁾
weights and biases of layer l
f
hidden activation (tanh, ReLU or sigmoid)
σ
sigmoid on the output neuron

Loss and backpropagation

L = −[y·ln ŷ + (1 − y)·ln(1 − ŷ)]

δ(L) = ŷ − y, δ(l) = (W(l+1))Tδ(l+1) ⊙ f′(z(l))

∂L/∂W(l) = δ(l)(a(l−1))T, ∂L/∂b(l) = δ(l)

Minibatch SGD update

W ← W − (η / |B|) Σi∈B ∂Li/∂W

η
learning rate
B
minibatch of examples

Why initialisation matters

Weights start as small Gaussian values: He initialisation (variance 2 / fan-in) for ReLU and Glorot initialisation (variance 2 / (fan-in + fan-out)) for tanh and sigmoid. This keeps activations and gradients at a sensible scale from layer to layer. If every weight started at the same value, every neuron in a layer would receive the same gradient and they would never become different.

In code

Python
import numpy as np

def train_step(Ws, bs, X, y, lr):
    # Forward pass: tanh hidden layers, sigmoid output
    acts = [X]
    for W, b in zip(Ws[:-1], bs[:-1]):
        acts.append(np.tanh(acts[-1] @ W + b))
    p = 1 / (1 + np.exp(-(acts[-1] @ Ws[-1] + bs[-1])))

    # Backward pass: sigmoid + cross-entropy gives delta = p - y
    delta = (p - y) / len(X)
    for l in reversed(range(len(Ws))):
        gW, gb = acts[l].T @ delta, delta.sum(axis=0)
        if l > 0:
            delta = (delta @ Ws[l].T) * (1 - acts[l] ** 2)   # tanh'
        Ws[l] -= lr * gW
        bs[l] -= lr * gb

Practice problems

Problem 1: Counting parameters

How many trainable parameters (weights plus biases) does a network with 2 inputs, hidden layers of 4 and 4 neurons, and 1 output have?
Show solution

Input to hidden 1: 2 × 4 weights + 4 biases = 12

Hidden 1 to hidden 2: 4 × 4 weights + 4 biases = 20

Hidden 2 to output: 4 × 1 weights + 1 bias = 5

Total = 12 + 20 + 5 = 37 parameters

Problem 2: One backprop step by hand

A network has one tanh hidden unit and a sigmoid output: h = tanh(w₁x), ŷ = σ(w₂h), with no biases. With x = 1, w₁ = 0.5, w₂ = 1 and target y = 1, compute ∂L/∂w₂ and ∂L/∂w₁ for the cross-entropy loss.
Show solution

h = tanh(0.5) = 0.4621

ŷ = σ(1 × 0.4621) = 0.6135

Output delta: δ₂ = ŷ − y = −0.3865

∂L/∂w₂ = δ₂ × h = −0.3865 × 0.4621 = −0.1786

Hidden delta: δ₁ = w₂ × δ₂ × (1 − h²) = 1 × (−0.3865) × 0.7864 = −0.3040

∂L/∂w₁ = δ₁ × x = −0.3040

Both gradients are negative, so gradient descent increases w₁ and w₂, pushing ŷ towards 1.

Discussion

Questions, ideas or a different approach? Join the conversation.