Matrix Multiplication and Broadcasting

Two operations every Dense layer hides under the hood

Matrix multiplication is the core operation of neural networks. A Dense layer is, mathematically, a matmul + bias add + activation. tf.matmul(a, b) requires a.shape[-1] == b.shape[-2] — the inner dimensions must contract. Leading dimensions are batch dims and must match (or broadcast).

The Python @ operator is shorthand for tf.matmul. Use whichever is more readable in context.

Broadcasting lets ops work on tensors of different shapes by implicitly expanding size-1 dimensions. The rules: right-align shapes, treat any size-1 dim as expandable, error if neither side is 1 and the sizes don't match. Adding a bias vector (n,) to a batch of activations (batch, n) works because the bias is treated as (1, n) and broadcast across the batch.

Broadcasting is silent. TF won't warn if you accidentally broadcast in an unintended way. A column vector (n, 1) when you meant a row vector (1, n) produces a different result without raising. Print the output shape after operations involving different sizes.

Code

Matrix multiplication 패턴·python

import tensorflow as tf

# Basic: (m, k) @ (k, n) → (m, n)
a = tf.constant([[1.0, 2.0], [3.0, 4.0]])     # (2, 2)
b = tf.constant([[5.0], [6.0]])                # (2, 1)
result = tf.matmul(a, b)                       # (2, 1)
result2 = a @ b                                # same — Python @ shorthand

# Batched: leading dims are batch
batch_a = tf.ones([32, 64, 128])
batch_b = tf.ones([32, 128, 10])
out = tf.matmul(batch_a, batch_b)              # (32, 64, 10)

# transpose_b avoids creating an intermediate transpose tensor
a = tf.ones([2, 3])
b = tf.ones([2, 3])
result = tf.matmul(a, b, transpose_b=True)     # (2, 3) @ (3, 2) → (2, 2)

Broadcasting 예시·python

import tensorflow as tf

x = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])  # (2, 3)

# Scalar broadcast
print(x + 10)         # [[11,12,13],[14,15,16]]

# Row vector (1, 3) broadcast to (2, 3)
row_bias = tf.constant([[10.0, 20.0, 30.0]])
print(x + row_bias)   # [[11,22,33],[14,25,36]]

# Column vector (2, 1) broadcast to (2, 3)
col_bias = tf.constant([[100.0], [200.0]])
print(x + col_bias)   # [[101,102,103],[204,205,206]]

# Higher-rank broadcasting
a = tf.ones([4, 1, 3])
b = tf.ones([1, 5, 3])
print((a + b).shape)  # (4, 5, 3)

Matrix Multiplication and Broadcasting

Two operations every Dense layer hides under the hood

Code

External links

Progress

Comments 0