Storage, Stride, and Reading Memory Maps

Tensors are layered: Storage + view metadata

A tensor is two things: a Storage (a contiguous 1-D blob of typed memory) and a view on that storage (shape, stride, offset). When you slice or transpose, the storage usually doesn't change — only the view does. That's why slicing is O(1) and why .transpose() is "free" but breaks contiguity.

What stride actually means

t.stride() returns a tuple of integers, one per dimension, telling PyTorch how many elements to step over in storage when you advance one index along that dimension. For a contiguous (3, 4) float tensor, stride is (4, 1): walking down a row jumps 4 elements; walking across a column jumps 1.

Transpose swaps the strides without touching storage: a (3, 4) tensor with stride (4, 1) becomes a (4, 3) tensor with stride (1, 4). That's why .is_contiguous() returns False after transpose — walking elements in memory order no longer matches walking the dim order.

Why this matters

You can predict whether an operation will allocate or just rewrite metadata.
You can reason about why some "trivial" transforms (e.g. NHWC → NCHW for an image batch) actually cost real time and memory.
You can debug "why does my training take 30% more memory than I expect" — usually one accidental copy.

Code

Inspect the layers under a tensor·python

import torch

t = torch.arange(12).reshape(3, 4)
print(t.shape)               # torch.Size([3, 4])
print(t.stride())            # (4, 1)  — row stride 4, col stride 1
print(t.is_contiguous())     # True
print(t.untyped_storage().size())  # 12 — single contiguous blob

# Slicing — view, no copy
row = t[1]
print(row.storage_offset())  # 4 — row 1 starts at element 4 in storage
print(row.data_ptr() == t.data_ptr() + 4 * 8)  # True (8 bytes per int64)

Transpose changes stride, not storage·python

import torch

t = torch.arange(12).reshape(3, 4)
tt = t.T
print(tt.shape)              # torch.Size([4, 3])
print(tt.stride())           # (1, 4)  — strides swapped
print(tt.is_contiguous())    # False
print(tt.data_ptr() == t.data_ptr())  # True — SAME storage

# tt.view(-1) errors. tt.reshape(-1) works (will copy under the hood).
flat = tt.contiguous().view(-1)
print(flat.data_ptr() == t.data_ptr())  # False — new allocation

Memory accounting — how much does a tensor weigh?·python

import torch

t = torch.randn(1024, 1024)            # float32 by default

elem_bytes = t.element_size()          # 4
n_elem = t.nelement()                  # 1,048,576
total_mb = elem_bytes * n_elem / 1024 / 1024
print(f"{total_mb:.1f} MB")            # 4.0 MB

# Half precision halves it
t16 = t.half()
print(f"{t16.element_size() * t16.nelement() / 1024 / 1024:.1f} MB")  # 2.0 MB

# Two views of the same storage do NOT double-count
view = t[:512, :]
print(view.untyped_storage().size())   # still 1,048,576 elements

Exercise

Build two tensors with the same values but different stride: one contiguous, one non-contiguous (e.g. via .transpose().contiguous() vs just .transpose()). Compare data_ptr() to confirm they're separate storage. Time a sum() reduction over both — the contiguous one should be measurably faster on a large enough tensor.