Transpose: Flipping Across the Diagonal

Rows Become Columns

The transpose of a matrix $A$ , written $A^{T}$ , swaps rows and columns. The element at $(i, j)$ moves to $(j, i)$ , so a 2×3 matrix becomes 3×2. A square matrix keeps its shape but reflects across the main diagonal.

$A = (142536), A^{T} = 123456$

Why It Appears So Often

Aligning matrix-multiplication axes: a (32, 128) batch can multiply a (256, 128) weight matrix after the weight becomes (128, 256).
Defining symmetry: $A = A^{T}$ defines a symmetric matrix. Covariance and Gram matrices are common ML examples.
Backpropagation: transposed weights naturally appear when gradients pass backward through a linear layer.

In NumPy and PyTorch, a simple transpose often returns a view with changed strides rather than copying the data. That does not make every downstream use free. An operation may require contiguous memory and trigger a copy, or noncontiguous access may be slower. Inspect the memory layout when performance matters.

A transpose changes how axes are interpreted. Before using it to silence a shape error, identify which axes mean batch, input, and output.

Code

Plain transpose + symmetry check·python

import numpy as np

A = np.array([[1, 2, 3],
              [4, 5, 6]])
print(A.T)
# [[1 4]
#  [2 5]
#  [3 6]]

# Symmetric matrix — equal to its transpose
S = np.array([[1, 2, 3],
              [2, 4, 5],
              [3, 5, 6]])
print(np.allclose(S, S.T))   # True

The most common shape fix in deep learning·python

import torch

# Shape mismatch fix in one line
batch = torch.randn(32, 128)
weight = torch.randn(256, 128)

# This fails: shapes (32, 128) and (256, 128) don't align for matmul
# out = batch @ weight   # ✗

# Transpose the weight; now shapes (32, 128) @ (128, 256) work
out = batch @ weight.T
print(out.shape)             # torch.Size([32, 256])

Rows Become Columns

Why It Appears So Often

Code

Exercise

Progress

Comments 4