Matrix Multiplication: The Ritual of Combining Spells

Why It's Not Element-Wise

Adding matrices is element-wise — easy. Multiplying matrices is not element-wise. It's a careful dance of rows and columns: each cell of the output is the dot product of one row of the left matrix and one column of the right matrix.

$(A B)_{ij} = k \sum A_{ik} B_{k j}$

The shape rule: $(m \times n) \cdot (n \times p) = (m \times p)$ . Inner dimensions must match; outer dimensions form the result. Get this rule wrong and your code crashes; get it right and you've built a neural network layer.

Composing Transformations

Matrix multiplication encodes composing transformations. Rotations and scales can be represented as matrices, and translations can too when you use homogeneous coordinates. Multiplying those matrices gives a single matrix that performs all the transformations at once. Order matters: $A B$ is generally not equal to $B A$ . Rotate-then-scale ≠ scale-then-rotate (in 3D, vividly so).

Broadcasting Sneaks Back In

NumPy/PyTorch let you "multiply" a matrix by a vector with * (element-wise) — and broadcasting will silently apply the vector across rows or columns of the matrix. This is not matrix multiplication; it's element-wise op with broadcasting. Use @ for true matrix multiplication.

The bug that bit me. Early in deep-learning work I wrote output = weights * inputs in a forward pass. The shapes happened to broadcast, the loss came out as a number, training even improved a bit. It took an hour to realize my "linear layer" was element-wise multiplication and not a real linear transformation. Use @. Every time.

Code

The two operations that look similar and aren't·python

import numpy as np

A = np.array([[1, 2],
              [3, 4]])
B = np.array([[5, 6],
              [7, 8]])

# Element-wise — wrong for linear layers
print(A * B)          # [[ 5 12] [21 32]]

# Matrix multiplication — what neural networks do
print(A @ B)          # [[19 22] [43 50]]

# Verify by hand on (0, 0): 1*5 + 2*7 = 19 ✓

Composition is multiplication, order matters·python

import numpy as np
import torch

# Composing transformations: rotate-then-scale
theta = np.pi / 4
R = torch.tensor([[np.cos(theta), -np.sin(theta)],
                  [np.sin(theta),  np.cos(theta)]], dtype=torch.float32)
S = torch.tensor([[2.0, 0.0],
                  [0.0, 0.5]])

# Composite transformation: rotate first, then scale
T = S @ R                       # apply R first, then S (read right to left)
v = torch.tensor([1.0, 0.0])
print(T @ v)                    # [√2, √2/2] — rotated 45° then scaled

Exercise

Define a 90° rotation matrix R and a nonuniform scale matrix S = diag([2, 0.5]). Apply R @ S and S @ R to the vector [1, 0]. Are the results the same? Explain why order matters.

Hint

R = [[0, -1], [1, 0]]. Read products right to left: R @ S scales first and rotates second, while S @ R rotates first and then scales along the fixed coordinate axes.

Why It's Not Element-Wise

Composing Transformations

Broadcasting Sneaks Back In

Code

External links

Exercise

Progress

Comments 2