Gradient Clipping and stop_gradient

When gradients misbehave

Gradients aren't always well-behaved. Exploding gradients grow unboundedly during training (especially in RNNs and Transformers with long sequences), causing weight updates of infinite magnitude. Vanishing gradients shrink toward zero, blocking learning in early layers.

Gradient clipping is the standard remedy for explosions. Two strategies:

Clip by global norm (clipnorm=1.0): scales all gradients so their combined L2 norm ≤ max_norm. Recommended for RNNs and Transformers.
Clip by value (clipvalue=0.5): clips each gradient component independently to [-clipvalue, +clipvalue].

tf.stop_gradient blocks gradient flow through part of a computation graph. The wrapped tensor appears as a constant to the gradient computation. Common use cases: target networks in DQN reinforcement learning, momentum encoders in contrastive learning, and any setting where you compute a value but don't want it to update via backprop.

Code

Gradient clipping — 두 전략·python

import tensorflow as tf

# Strategy 1: clip by global norm — RNN/Transformer recommended
optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3, clipnorm=1.0)

# Strategy 2: clip by value
optimizer_val = tf.keras.optimizers.Adam(learning_rate=1e-3, clipvalue=0.5)

# Manual clipping in a custom loop
with tf.GradientTape() as tape:
    loss = loss_fn(model(x, training=True), y)

grads = tape.gradient(loss, model.trainable_weights)
grads, global_norm = tf.clip_by_global_norm(grads, clip_norm=1.0)
optimizer.apply_gradients(zip(grads, model.trainable_weights))

stop_gradient — DQN target pattern·python

import tensorflow as tf

# In DQN, the "target" Q-network must NOT receive gradients
# from the current loss — otherwise the target moves with the predictor
# and training diverges.

target_q = tf.stop_gradient(target_model(next_states))
loss = tf.reduce_mean(tf.square(
    current_q - (rewards + gamma * target_q)
))

# stop_gradient is also how you implement gradient blocking in
# contrastive learning, BYOL-style momentum encoders, and any
# place where you want a "snapshot" of a model's output.

Gradient Clipping and stop_gradient

When gradients misbehave

Code

Progress

Comments 0