Element-wise Math and Reductions

The primitives every loss function is built from

Most TF math operates element-wise — it applies a scalar function to each element independently. Add, subtract, multiply, divide, square, sqrt, exp, log, sin, sigmoid, tanh, ReLU. These are the atoms loss functions and activations are made of.

Reductions collapse one or more axes. tf.reduce_sum, tf.reduce_mean, tf.reduce_max, tf.reduce_min work globally (returning a scalar) or along a specified axis. The keepdims=True option preserves the reduced dim as size 1, which is essential when broadcasting back to the original shape.

tf.argmax and tf.argmin return the index of the max/min element — used in classification to find the predicted class from logits.

Why this matters: Mean Squared Error is tf.reduce_mean(tf.square(y_true - y_pred)). Cross-entropy is tf.math.log + tf.reduce_sum. Accuracy is tf.argmax + tf.equal + tf.reduce_mean(tf.cast(...)). Once you see these primitives, you can read any loss function.

Code

원소별 연산·python

import tensorflow as tf

x = tf.constant([[1.0, 2.0], [3.0, 4.0]])
y = tf.constant([[5.0, 6.0], [7.0, 8.0]])

x + y                # element-wise add
x * y                # element-wise multiply (NOT matmul)
tf.pow(x, 2)         # x squared
tf.sqrt(x)
tf.math.exp(x)
tf.math.log(x)
tf.math.sigmoid(x)
tf.nn.relu(x)        # max(0, x)

Reductions·python

import tensorflow as tf

x = tf.constant([[1.0, 2.0, 3.0],
                 [4.0, 5.0, 6.0]])

# Global → scalar
tf.reduce_sum(x)              # 21.0
tf.reduce_mean(x)             # 3.5

# Along axis → reduce that axis
tf.reduce_sum(x, axis=0)      # [5, 7, 9]
tf.reduce_sum(x, axis=1)      # [6, 15]

# keepdims preserves shape for broadcasting back
tf.reduce_sum(x, axis=1, keepdims=True)  # [[6], [15]] — shape (2, 1)

# argmax — used everywhere in classification
tf.argmax(x, axis=1)          # [2, 2] — index of max in each row

Element-wise Math and Reductions

The primitives every loss function is built from

Code

External links

Exercise

Progress

Comments 0