solvePnP: Undoing the Camera

"You know where the eyes are on a real skull. You know where they landed in the photo. solvePnP asks: what camera turns the first into the second?"

The Question solvePnP Answers

Perspective-n-Point — solvePnP — is the exact inverse of the last lesson. Forward projection took a pose and produced pixels. solvePnP takes pixels and produces the pose. Concretely: you have a set of 3-D points whose positions on a reference head you know, and you have the 2-D pixels where those same features landed in the photo. The solver finds the single rotation and translation that, run through the pinhole model, would have projected those 3-D points onto exactly those pixels. That rotation is the head's orientation — the perspective box, computed rather than eyeballed.

Correspondences Are the Fuel

The solver runs on correspondences: matched pairs of (2-D pixel, 3-D point). The 3-D half comes from the canonical head — a neutral reference face whose every landmark has a known position. The 2-D half comes from the detector (next track), which reports where each of those landmarks actually appears in this photo. They are matched by index: landmark 33 in the photo pairs with point 33 on the model. That index matching is why the detector's output and the reference model must stay topology-locked — swap or drift the indices and every correspondence lies, silently.

Minimizing Reprojection Error

There is usually no pose that hits every pixel perfectly — real faces aren't the neutral model, and detection is noisy. So solvePnP finds the pose that comes closest: it minimizes the total reprojection error, the summed pixel distance between where the 3-D points land when projected and where they were actually observed. That leftover error is not just a byproduct — it is the engine's honesty meter. A small error means the pose explains the face well; a large one means the fit is strained and should be trusted less. The same number that the solver minimizes is the number the UI later shows you as confidence.

From Rotation to the Box

The solver returns the rotation in a compact form (a rotation vector), which converts to the 3x3 rotation matrix that is the head's box orientation. Turning that matrix into human-readable yaw, pitch, and roll is its own small minefield — the naive decomposition wraps and lies on near-frontal faces — so Loomis reads the angles off the head's forward and up vectors instead. That care is the whole subject of the no-flip lesson two ahead; for now, hold the clean version: correspondences in, pose out, error as the receipt.

Code

solvePnP, correspondences to pose, with the trust meter·python

import cv2
import numpy as np

# 3D model points (canonical head) matched BY INDEX to the 2D landmarks.
object_points = CANONICAL_HEAD[LANDMARK_IDS]      # (N, 3), known on a reference head
image_points  = detected_landmarks[LANDMARK_IDS]  # (N, 2), where they landed in-photo

ok, rvec, tvec = cv2.solvePnP(
    object_points, image_points,
    camera_matrix, dist_coeffs,      # intrinsics; focal comes from the bounded sweep
    flags=cv2.SOLVEPNP_ITERATIVE,
)
R, _ = cv2.Rodrigues(rvec)           # rotation vector -> 3x3 matrix = the head's box

# The honesty meter: reproject the model and measure the leftover pixel gap.
reproj, _ = cv2.projectPoints(object_points, rvec, tvec, camera_matrix, dist_coeffs)
err = np.linalg.norm(reproj.squeeze() - image_points, axis=1).mean()
# small err -> the pose explains the face; large err -> trust this fit less.

Exercise

Imagine four known corners of a flat square card and the four pixels where they appear in a photo. Argue why, in general, that's enough to pin down one camera pose — then describe a case where it isn't unique: what if the card is photographed nearly edge-on, or the four corners are almost collinear in the image? Relate your answer to why more, well-spread landmarks make solvePnP more stable.

Hint

Each correspondence constrains the pose; enough well-spread ones over-determine it and average out noise. Degeneracies appear when the points don't span enough of the image — an edge-on card or near-collinear corners leave the depth barely constrained, so many poses reproject almost identically. A face's landmarks are stable precisely because they spread across a genuinely 3-D surface.