What Tensors Actually Are

An n-dimensional array, with two superpowers

Strip the deep-learning mystique away and a tensor is just an n-dimensional array — same idea as a NumPy ndarray. The two things that make PyTorch tensors different from NumPy arrays:

Device transparency. The same code can run on CPU, NVIDIA GPU (CUDA), or Apple Silicon GPU (MPS) by changing one argument.
Autograd. Every operation can record itself onto a graph so PyTorch can later play it backwards and compute gradients.

Conceptually, the dimensional ladder reads like this:

0D — scalar: a single number — torch.tensor(3.14)
1D — vector: a row of numbers — torch.tensor([1, 2, 3])
2D — matrix: a table — rows × cols
3D: a batch of matrices — common for sequences (batch, seq_len, features)
4D: a batch of images — (batch, channels, height, width) in PyTorch's NCHW convention

The mathematician in the room will object that "tensor" has a stricter meaning involving covariance / contravariance. In deep learning the word has been informally re-borrowed to mean "n-dim array with autograd." That's the meaning the framework, the docs, and every paper uses. Don't fight it.

One gentle warning: shape is the single most useful debugging tool you have in PyTorch. When something breaks, the first move is always print(x.shape). Internalize that habit early.

Code

Tensors at every dimensionality·python

import torch

# 0D — scalar
scalar = torch.tensor(3.14)
print(scalar.ndim, scalar.shape)  # 0  torch.Size([])

# 1D — vector
vector = torch.tensor([1.0, 2.0, 3.0])
print(vector.shape)               # torch.Size([3])

# 2D — matrix
matrix = torch.tensor([[1, 2], [3, 4], [5, 6]])
print(matrix.shape)               # torch.Size([3, 2])

# 3D — batch of sequences (batch=2, seq=3, features=4)
seq_batch = torch.randn(2, 3, 4)
print(seq_batch.shape)            # torch.Size([2, 3, 4])

# 4D — batch of images (NCHW: batch=8, channels=3, H=224, W=224)
images = torch.randn(8, 3, 224, 224)
print(images.shape)               # torch.Size([8, 3, 224, 224])

PyTorch ↔ NumPy mental model·python

import torch
import numpy as np

# A NumPy person already knows this:
arr = np.zeros((3, 4))
print(arr.shape, arr.dtype)        # (3, 4) float64

# PyTorch is the same plus device + autograd:
t = torch.zeros(3, 4)
print(t.shape, t.dtype, t.device)  # torch.Size([3, 4]) torch.float32 cpu

# Note the dtype default changed: NumPy → float64, PyTorch → float32.
# float32 is the deep-learning default because it's twice as fast on GPU
# and almost always 'good enough' precision for gradients.

Exercise

Open a Python REPL with PyTorch installed. Create one tensor at every dimensionality from 0D to 4D. For each one, print .shape, .ndim, and .numel(). Predict each value before you print it — getting your prediction wrong on .numel() for the 4D batch is the most common shape mistake people make at scale.

An n-dimensional array, with two superpowers

Code

External links

Exercise

Progress

Comments 0