Backpropagation Algorithm
Discover how neural networks learn by propagating errors backward and updating weights through gradient descent.
IntermediateGradient DescentWeight UpdatesError PropagationNeural Network Training
Epoch
0
Loss (MSE)
0.0000
Output
0.0000
Inputs
Training Parameters
Backpropagation Algorithm
1. Forward Pass
Calculate activations from input to output:
a = σ(Wx + b) where σ is sigmoid activation
2. Calculate Error
Compute the difference between prediction and target:
Error = Target - Output
Loss = ½(Target - Output)²
3. Backward Pass
Calculate deltas (error signals) from output to input:
δ_output = Error × σ'(output)
δ_hidden = (δ_next · W^T) × σ'(activation)
4. Update Weights
Adjust weights using gradient descent:
W_new = W_old + η × δ × a_prev
b_new = b_old + η × δ
where η is the learning rate
Key Concepts
- Sigmoid Function: σ(x) = 1/(1 + e^(-x)) - squashes values to (0,1)
- Delta (δ): Measures how much a neuron contributes to the error
- Gradient: Direction and magnitude of weight update
- Learning Rate: Controls step size during weight updates
- Weight Color: Green = positive connection, Red = negative (inhibitory)
Learning objectives
- Understand the forward and backward passes in neural networks
- Observe how errors propagate from output to input layers
- See how weights are updated using gradient descent
- Learn the role of learning rate in network training
- Visualize loss minimization over training epochs
Algorithm steps
- 1Forward pass. Compute activations from the input to the output layer. Each neuron applies weights, adds a bias, and passes the result through an activation function (sigmoid).
- 2Calculate error. Compare the network output with the target value. Compute the loss (Mean Squared Error) to measure how far off the prediction is.
- 3Backward pass. Calculate deltas (δ) for each neuron from output to input. A delta represents how much that neuron contributed to the error.
- 4Update weights. Adjust all weights and biases using gradient descent: move in the direction that reduces error, scaled by the learning rate.
Experimentation tips
Learning rate
Try high values (1.5+) to see oscillation, and low values (0.1) for slow convergence.
Watch deltas
Enable the gradient display to see error signals propagating backward.
Weight colors
Green = excitatory (positive), red = inhibitory (negative).
Training speed
Adjust the animation speed to observe individual updates or fast convergence.
Different targets
Change the target output to see the network adapt to new goals.
Real-world applications
Image recognition
Deep neural networks use backprop to learn features from millions of images for classification tasks.
Natural language processing
Language models like GPT use backpropagation to learn word patterns and generate human-like text.
Speech recognition
Voice assistants train neural networks with backprop to convert speech to text accurately.
Autonomous vehicles
Self-driving cars use neural networks trained with backprop to detect objects and make driving decisions.
Medical diagnosis
AI systems learn to detect diseases from medical images using backpropagation-trained networks.
Game AI
AlphaGo and other game-playing AIs use backprop to learn winning strategies from experience.
Key concepts explained
Activation function (sigmoid)
Squashes neuron outputs to the range (0, 1). Formula: σ(x) = 1 / (1 + e^(−x))
Delta (δ)
Error signal for each neuron. Shows how much it contributed to the total error.
Gradient
Direction and magnitude of steepest descent. Used to update weights toward lower error.
Learning rate (η)
Step size for weight updates. Too high causes oscillation, too low slows convergence.
Loss function (MSE)
Mean Squared Error: ½(target − output)². Measures prediction accuracy.
Discussion
Questions, ideas or a different approach? Join the conversation.