Checkpoints — Resume Interrupted Training

~12 min · checkpoint, training, resume

Level 0Level 0

0 XP0/78 lessons0/17 achievements

0/100 XP to next level100 XP to go0% complete

The thing that saves you from a 5-hour crash

For long-running training, you need to save snapshots so you can resume after preemption, crashes, or notebook disconnects. tf.train.Checkpoint with CheckpointManager is the right tool — unlike save_weights, it can save arbitrary Python objects including optimizer state, custom step counters, and even datasets.

CheckpointManager(max_to_keep=N) automatically deletes old checkpoints when the limit is hit. This keeps disk usage bounded during long training runs.

The pattern: build the checkpoint object, attach to a manager, restore at the start of training (no-op if no checkpoint exists), save periodically. A 6-hour training run that crashes at hour 5 should cost you 0–10 minutes of progress, not 5 hours.

Code

CheckpointManager 패턴·python

import tensorflow as tf

model     = MyModel()
optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3)
step      = tf.Variable(0)

# Track all training state
ckpt = tf.train.Checkpoint(step=step, optimizer=optimizer, model=model)

# Manager auto-deletes old checkpoints
manager = tf.train.CheckpointManager(
    ckpt, directory='./tf_ckpts', max_to_keep=3,
)

# Restore from latest if exists (resumes interrupted training)
ckpt.restore(manager.latest_checkpoint)
if manager.latest_checkpoint:
    print(f"Restored from {manager.latest_checkpoint}")
else:
    print("Starting from scratch")

for epoch in range(50):
    for batch in dataset:
        train_step(model, batch, optimizer)
        step.assign_add(1)
    if epoch % 5 == 0:
        save_path = manager.save()
        print(f"Checkpoint saved: {save_path}")

Checkpoints — Resume Interrupted Training

The thing that saves you from a 5-hour crash

Code

Progress

Comments 0