Practical Vision Tasks

The four task shapes

Image classification — one label per image. Output: a vector of class logits. Loss: cross-entropy.
Object detection — multiple bounding boxes + class per box. Output: a list of (box, class, score). Models: YOLO family, DETR, Faster R-CNN.
Semantic segmentation — per-pixel class label. Output: a same-size class mask. Models: U-Net, DeepLab, SegFormer.
Instance segmentation — per-instance pixel mask. Output: per-detected-object mask. Models: Mask R-CNN, SAM, YOLO-seg.

Each task has a canonical metric: Top-K accuracy for classification, mAP for detection (mean Average Precision over IoU thresholds), mIoU for segmentation, panoptic quality for the joint problem.

Tip: Don't reinvent the wheel for these. torchvision.models.detection, ultralytics, segformer, SAM — each has a 'fit my task in 50 lines' API. Pick one based on your latency / accuracy budget and run.

The lifecycle that ships, not just trains

Vision models in production need: domain-appropriate data labelling (CVAT, Roboflow), augmentation tuned to your distribution, evaluation on per-class metrics (not just aggregate), drift monitoring once deployed. The model is a small slice of the pipeline.

The 2026 default for new vision projects

Foundation model backbone (DINOv2, SAM, CLIP) → small task-specific head → small labeled set → ship. The era of training a vision model from scratch on a million images is mostly over for application work.

Principle: Vision tasks are now mostly a labeling problem, a foundation-model-pick problem, and a metrics problem. The 'modeling' part is increasingly small.

Code

Object detection in three lines (YOLO via Ultralytics)·python

from ultralytics import YOLO

model = YOLO("yolov8n.pt")          # nano model, ~6MB
results = model("image.jpg", conf=0.25)
for r in results:
    print(r.boxes.xyxy, r.boxes.cls, r.boxes.conf)

Semantic segmentation with SAM·python

from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["vit_b"](checkpoint="sam_vit_b_01ec64.pth")
predictor = SamPredictor(sam)
predictor.set_image(image)              # numpy array
masks, scores, _ = predictor.predict(point_coords=[[300, 200]],
                                      point_labels=[1])

The four task shapes

The lifecycle that ships, not just trains

The 2026 default for new vision projects

Code

External links

Exercise

Progress

Comments 0