Persistent Tapes and Higher-Order Gradients

~11 min · gradient-tape, persistent, second-derivative

Level 0Level 0

0 XP0/78 lessons0/17 achievements

0/100 XP to next level100 XP to go0% complete

When one tape isn't enough

By default, a GradientTape releases its resources after the first tape.gradient() call. Two situations require special patterns: multiple gradient computations from the same forward pass, and second-order (or higher) gradients.

Persistent tapes (tf.GradientTape(persistent=True)) allow multiple gradient() calls. The tradeoff: the tape holds references to all intermediate tensors, blocking garbage collection. Always del tape when you're done — especially in tight training loops.

Higher-order gradients use nested tapes. The outer tape records the inner tape's gradient computation, so calling outer.gradient(inner_grad, x) returns the second derivative.

For Jacobians (gradient of vector-valued functions), tape.jacobian(y, x) returns shape output_shape + input_shape.

Code

Persistent tape·python

import tensorflow as tf

x = tf.Variable(2.0)

# Default tape: one gradient() call only
with tf.GradientTape() as tape:
    y = x ** 3
dy_dx = tape.gradient(y, x)        # 3x² = 12
# tape.gradient(y, x)               # ERROR: tape released

# Persistent tape: multiple calls allowed
with tf.GradientTape(persistent=True) as tape:
    y = x ** 3

dy_dx = tape.gradient(y, x)        # 12
d2y_dx2 = tape.gradient(dy_dx, x)  # 6x = 12
del tape   # IMPORTANT for memory

Nested tape — second derivative·python

import tensorflow as tf

x = tf.Variable(2.0)

with tf.GradientTape() as t2:
    with tf.GradientTape() as t1:
        y = x * x * x       # y = x³
    dy_dx = t1.gradient(y, x)        # 3x² = 12

d2y_dx2 = t2.gradient(dy_dx, x)      # 6x = 12
print(f"f'(2) = {dy_dx.numpy()}")    # 12.0
print(f"f''(2) = {d2y_dx2.numpy()}") # 12.0

Progress

Progress is local-only — sign in to sync across devices.

← PreviousGradientTape Basics — Your First Gradient Next →Linear Regression from Scratch — The Universal Loop

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.