Reshape, View, Permute, and the Contiguous Trap

Three operations that change what data looks like

None of these operations move bytes around (unless they have to). They change the strides — the recipe PyTorch uses to walk through memory. Understanding stride is the difference between knowing tensors and using tensors.

The four operations

reshape(*shape) — most flexible. Returns a view if memory permits, otherwise copies. Always works.
view(*shape) — view-only. Errors if the tensor isn't contiguous.
transpose(d0, d1) / .T — swap two dims. Always returns a view, but the result is non-contiguous.
permute(*dims) — reorder all dims. View, non-contiguous.

The contiguous trap is the most common subtle PyTorch bug. After a transpose or permute, the tensor is laid out non-contiguously in memory. Some operations (notably .view() and many older custom CUDA kernels) require contiguous memory. The fix is either .contiguous() (which copies into a fresh contiguous block) or just using .reshape() instead of .view().

Adding and removing size-1 dims

unsqueeze(dim) adds a size-1 dim at dim. squeeze(dim=None) removes size-1 dims. Both are pure metadata changes — no copy. They're constantly needed when matching shapes for broadcasting (a row vector vs. column vector situation).

Code

Reshape vs view — when to use which·python

import torch

t = torch.arange(12)

# reshape: most flexible. Always works.
a = t.reshape(3, 4)
b = t.reshape(2, -1)   # -1 means "infer this dim" → 2 x 6
c = t.reshape(-1, 3)   # → 4 x 3

# view: identical to reshape for contiguous tensors, errors otherwise.
v = t.view(3, 4)

# Rule of thumb: use reshape unless you specifically need the
# 'fail loudly when non-contiguous' guarantee that view gives you.

Transpose, permute, and the contiguous fix·python

import torch

x = torch.randn(2, 3, 4)   # (batch=2, seq=3, features=4)

# 2D transpose shorthand
m = torch.randn(3, 4)
mt = m.T            # equivalent to m.transpose(0, 1)
print(mt.is_contiguous())  # False!

# permute reorders ALL dims
y = x.permute(0, 2, 1)     # (2, 4, 3)
print(y.is_contiguous())   # False

# y.view(-1)  # RuntimeError: view size is not compatible
y_flat = y.contiguous().view(-1)  # works
y_flat = y.reshape(-1)             # also works (reshape handles it)

# Image format conversion: NHWC → NCHW
img = torch.randn(8, 224, 224, 3)
img_pytorch = img.permute(0, 3, 1, 2).contiguous()
print(img_pytorch.shape)   # torch.Size([8, 3, 224, 224])

unsqueeze, squeeze, and broadcasting setup·python

import torch

v = torch.tensor([1, 2, 3])     # shape (3,)
v.unsqueeze(0).shape            # torch.Size([1, 3]) — row vector
v.unsqueeze(1).shape            # torch.Size([3, 1]) — column vector
v[None, :].shape                # same as unsqueeze(0)
v[:, None].shape                # same as unsqueeze(1)

# squeeze removes size-1 dims
x = torch.randn(1, 3, 1, 4)
x.squeeze().shape               # torch.Size([3, 4])
x.squeeze(0).shape              # torch.Size([3, 1, 4]) — only dim 0
x.squeeze(2).shape              # torch.Size([1, 3, 4]) — only dim 2

Reshape, View, Permute, and the Contiguous Trap

Three operations that change what data looks like

The four operations

Adding and removing size-1 dims

Code

External links

Exercise

Progress

Comments 0