Matrix Multiplication: The Ritual of Combining Spells

Why It's Not Element-Wise

Adding matrices is element-wise — easy. Multiplying matrices is not element-wise. It's a careful dance of rows and columns: each cell of the output is the dot product of one row of the left matrix and one column of the right matrix.

$(A B)_{ij} = k \sum A_{ik} B_{k j}$

The shape rule: $(m \times n) \cdot (n \times p) = (m \times p)$ . Inner dimensions must match; outer dimensions form the result. Get this rule wrong and your code crashes; get it right and you've built a neural network layer.

Composing Transformations

Matrix multiplication encodes composing transformations. Rotate, then scale, then translate — each is a matrix; multiplying them gives a single matrix that does all three at once. Order matters: $A B$ is generally not equal to $B A$ . Rotate-then-scale ≠ scale-then-rotate (in 3D, vividly so).

Broadcasting Sneaks Back In

NumPy/PyTorch let you "multiply" a matrix by a vector with * (element-wise) — and broadcasting will silently apply the vector across rows or columns of the matrix. This is not matrix multiplication; it's element-wise op with broadcasting. Use @ for true matrix multiplication.

The bug that bit me. Early in deep-learning work I wrote output = weights * inputs in a forward pass. The shapes happened to broadcast, the loss came out as a number, training even improved a bit. It took an hour to realize my "linear layer" was element-wise multiplication and not a real linear transformation. Use @. Every time.

Code

The two operations that look similar and aren't·python

import numpy as np

A = np.array([[1, 2],
              [3, 4]])
B = np.array([[5, 6],
              [7, 8]])

# Element-wise — wrong for linear layers
print(A * B)          # [[ 5 12] [21 32]]

# Matrix multiplication — what neural networks do
print(A @ B)          # [[19 22] [43 50]]

# Verify by hand on (0, 0): 1*5 + 2*7 = 19 ✓

Composition is multiplication, order matters·python

import torch

# Composing transformations: rotate-then-scale
theta = np.pi / 4
R = torch.tensor([[np.cos(theta), -np.sin(theta)],
                  [np.sin(theta),  np.cos(theta)]], dtype=torch.float32)
S = torch.tensor([[2.0, 0.0],
                  [0.0, 0.5]])

# Composite transformation: rotate first, then scale
T = S @ R                       # apply R first, then S (read right to left)
v = torch.tensor([1.0, 0.0])
print(T @ v)                    # [√2, √2/2] — rotated 45° then scaled

Exercise

Define a 90° rotation matrix R and a doubling scale matrix S. Compute R @ S and S @ R applied to vector [1, 0]. Are the results the same? Explain in one sentence why or why not.

Hint

R = [[0, -1], [1, 0]], S = 2*np.eye(2). Both produce the same result for these specific matrices because S is a scalar multiple of identity (it commutes with everything). Try with a non-uniform scaling like diag([2, 0.5]) and the order will matter.

Matrix Multiplication: The Ritual of Combining Spells

Why It's Not Element-Wise

Composing Transformations

Broadcasting Sneaks Back In

Code

External links

Exercise

Progress

Comments 2