Image Segmentation

Per-pixel, not per-box

Detection drew a box around each object. Segmentation goes finer: it assigns a class to every pixel, so the output is a label map the same height and width as the input. That's what you need when the shape matters — a tumor's exact boundary, the drivable area in front of a car, the matte that separates a subject from its background.

Two flavors that solve different problems

Semantic segmentation (e.g. DeepLabV3Plus, SegFormer) — labels every pixel with a class. DeepLab leans on atrous (dilated) convolution to see multiple scales at once without losing resolution; SegFormer is the transformer-era counterpart. Reach for these when you have a fixed set of classes you care about.
Salient / boundary-aware (e.g. BASNet) — separates foreground from background cleanly, optimizing for crisp edges rather than class identity. This is the matting / cut-out use case.

All of them load through the same from_preset door, ride pretrained backbones, and run on any backend. The output convention is the thing to internalize: a class map you can argmax over the channel axis to get a single label per pixel.

Code

Load a semantic segmenter and read its output·python

import keras
import keras_cv

# Semantic segmentation: one class per pixel
segmenter = keras_cv.models.SegFormerB0.from_preset(
    "segformer_b0_ade20k"
)

logits = segmenter.predict(images)        # (B, H, W, num_classes)
masks = keras.ops.argmax(logits, axis=-1) # (B, H, W) class label per pixel

Exercise

Load SAM via from_preset. Pick a sample image. Feed it three different point prompts (image coordinates on three distinct objects) and visualize the returned masks overlaid on the image. Then give it a *box* prompt around one object and compare — note how the prompt type changes the mask. Finally, run a SegFormer semantic segmenter on the same image and contrast: SAM gives you 'this blob,' SegFormer gives you 'this is class 12.'

Hint

SAM returns multiple candidate masks per prompt with confidence scores — pick the highest, or visualize all of them to see the ambiguity it's resolving.

Per-pixel, not per-box

Two flavors that solve different problems

Code

External links

Exercise

Progress

Comments 0