Image Segmentation

Label every pixel of an image. Run three classical segmentation methods for real, turn the result into semantic and instance masks, and see how deep networks like U-Net do it.

Advanced

Input image

Segmentation

Bars: pixel count per brightness (indigo = at or below t). Green: between-class variance. Dashed: Otsu's t. Solid: your t.

Threshold t118

Otsu tries every t and picks the one that makes the two groups of brightness values as separated as possible (the green curve). It only sees brightness, so the bright yellow triangle lands on the background side.

Foreground mask vs ground truth

TP: 4880
FP: 0
FN: 1102

IoU = TP / (TP + FP + FN) = 0.816

Dice = 2TP / (2TP + FP + FN) = 0.899

Semantic vs instance

Semantic segmentation gives every pixel a class. Instance segmentation also separates objects of the same class. Here the instances come from connected-component labelling of the predicted classes: touching pixels of the same class form one instance.

Ignore components smaller than (px)15

Predicted: semantic (one colour per class)

Predicted: instances (3 found)

Ground truth: semantic

Ground truth: instances (4)

Ground-truth classes: disc, rectangle, triangle. The two discs share a class (same colour on the left) but are separate instances (different colours on the right).

What segmentation is

Detection draws a box around each object. Segmentation goes further and decides, for every pixel, what it belongs to. Semantic segmentation assigns a class (disc, rectangle, background). Instance segmentation also tells the two discs apart.

The three methods in the visualizer are classical algorithms, and they really run on the pixels in your browser. They have no idea what a disc or a triangle is: they only group pixels by brightness, colour and adjacency. That is exactly why they fail in instructive ways (uneven lighting, objects as bright as the background). Deep networks learn features that cope with those problems, but nothing on this page runs a neural network: the U-Net section below is a diagram.

Thresholding

Split pixels by brightness. Otsu chooses the threshold automatically from the histogram.

Clustering

Group pixels with similar colours using k-means. Works in colour space, ignores where pixels are.

Region growing

Start at a seed pixel and absorb neighbours that look similar. Uses position and colour.

Instances

Connected-component labelling splits a class mask into separate touching blobs, one per object.

Key formulas

Otsu's method

t* = argmax_t w₀(t) · w₁(t) · (μ₀(t) − μ₁(t))²

w₀, w₁: fraction of pixels at or below t, and above t
μ₀, μ₁: mean brightness of each group

K-means (one iteration)

assign: label(p) = argmin_j ‖p − c_j‖², update: c_j = mean(S_j)

c_j: centre (mean colour) of cluster j
S_j: pixels assigned to cluster j

Mask overlap

IoU = |A ∩ B| / |A ∪ B|, Dice = 2|A ∩ B| / (|A| + |B|) = 2·IoU / (1 + IoU)

A, B: predicted and true foreground pixel sets

How to use this visualizer

1
In Threshold mode, look at the histogram: the dark objects and the light background form two humps. Otsu picks the valley automatically.
2
Switch to Errors vs truth. The yellow triangle shows up blue (missed): it is as bright as the background, so no brightness threshold can separate it.
3
Switch to K-means colours. Press Step a few times and watch the cluster centres settle. Try k = 3, 4 and 5, and tap a cluster to mark it background or foreground.
4
Switch to Region growing and tap inside an object to place a seed. Raise the tolerance until the object fills, and keep going until it leaks.
5
In the Semantic vs instance panel, compare the predicted maps with the ground truth and adjust the minimum component size to drop noise specks.
6
Try your own image with the upload button. It is processed locally; there is no ground truth for it, so IoU and Dice are hidden.

Semantic, instance and panoptic

Task	Output	Separates objects?	Typical models
Semantic segmentation	A class for every pixel	No: two people are one "person" region	U-Net, DeepLab, SegFormer
Instance segmentation	A mask per object (things only)	Yes	Mask R-CNN, YOLACT
Panoptic segmentation	Both: classes for stuff (sky, road), instances for things	Yes	Panoptic FPN, Mask2Former

Why connected components are not enough

Labelling connected blobs gives correct instances only when objects of the same class do not touch. Two overlapping people form one blob. Mask R-CNN solves this by first detecting a box per object, then predicting a separate mask inside each box.

How deep networks segment: U-Net

An encoder shrinks the image into abstract features; a decoder grows them back to full resolution; skip connections carry fine detail across.

Two 3×3 convolutions + ReLU (encoder) 2×2 max pool: halve the size, double the channels 2×2 up-convolution: double the size Skip: copy encoder features and concatenate Decoder convolutions; a final 1×1 convolution gives class scores for every pixel

The encoder is an ordinary CNN: each level applies two 3×3 convolutions, then a 2×2 max pool halves the width and height. Deep features know what is in the image but have lost where exactly.

The decoder reverses this with up-convolutions. On its own it would produce blurry boundaries, so at each level it concatenates the encoder's feature map of the same size (the skip connection). A final 1×1 convolution turns each pixel's feature vector into class scores, and the network is trained with per-pixel cross-entropy or Dice loss against labelled masks.

Code

JavaScript

// Otsu's threshold from a 256-bin histogram
function otsu(hist) {
  const total = hist.reduce((a, b) => a + b, 0);
  const sumAll = hist.reduce((s, h, v) => s + v * h, 0);
  let w0 = 0, sum0 = 0, best = 0, threshold = 0;
  for (let t = 0; t < 256; t++) {
    w0 += hist[t];
    sum0 += t * hist[t];
    const w1 = total - w0;
    if (w0 === 0 || w1 === 0) continue;
    const mu0 = sum0 / w0, mu1 = (sumAll - sum0) / w1;
    const between = (w0 / total) * (w1 / total) * (mu0 - mu1) ** 2;
    if (between > best) { best = between; threshold = t; }
  }
  return threshold;
}

// IoU and Dice of two binary masks
function scores(pred, truth) {
  let tp = 0, fp = 0, fn = 0;
  for (let i = 0; i < pred.length; i++) {
    if (pred[i] && truth[i]) tp++;
    else if (pred[i]) fp++;
    else if (truth[i]) fn++;
  }
  return { iou: tp / (tp + fp + fn), dice: (2 * tp) / (2 * tp + fp + fn) };
}

Practice problems

Problem 1: IoU and Dice

A predicted mask has 900 pixels. The true mask has 1000 pixels. They share 800 pixels. Compute the IoU and the Dice score.

Show solution

TP = 800, FP = 900 − 800 = 100, FN = 1000 − 800 = 200.

IoU = 800 / (800 + 100 + 200) = 800 / 1100 ≈ 0.727.

Dice = 2 · 800 / (900 + 1000) = 1600 / 1900 ≈ 0.842.

Check: 2 · 0.727 / (1 + 0.727) ≈ 0.842.

IoU ≈ 0.73, Dice ≈ 0.84. Dice is always at least as high as IoU.

Problem 2: Otsu on a tiny histogram

An image has 4 pixels of brightness 20 and 4 pixels of brightness 200, nothing else. Compare the between-class variance for t = 20 and t = 100. Which threshold does Otsu prefer?

Show solution

For t = 20: w₀ = 0.5, μ₀ = 20; w₁ = 0.5, μ₁ = 200. σ² = 0.5 · 0.5 · 180² = 8100.

For t = 100: the groups are exactly the same pixels, so σ² = 8100 as well.

Every t from 20 to 199 gives the same split and the same score.

It is a tie; the code above keeps the first maximum, t = 20. Any t in [20, 199] separates the image perfectly.

Discussion

Questions, ideas or a different approach? Join the conversation.