Debugging Strategies for Neural Networks

~13 min · debugging, nan, overfit, underfit

Level 0Level 0

0 XP0/78 lessons0/17 achievements

0/100 XP to next level100 XP to go0% complete

The systematic approach experienced ML engineers use

Neural networks fail in ways harder to debug than most software. Loss curves don't always reveal what went wrong, and bugs can be silent for many epochs before becoming obvious. Here's the systematic strategy.

1. Verify your data first — always. Most training failures are data bugs. Before debugging the model, check shapes, value ranges (normalized?), label distribution, and absence of NaN/inf.

2. Overfit one batch. If a model can't drive loss to ~0 on a single batch in 100 epochs, there's a fundamental bug — wrong loss, wrong output activation, label format mismatch, learning rate too small. Catching this with one batch saves hours.

3. Enable run_eagerly for full Python tracebacks. When a tracing error is opaque, compile(..., run_eagerly=True) turns the model into plain Python and gives you proper stack traces.

4. Watch for the canonical symptoms. Each common bug has a signature.

Code

데이터 sanity check + overfit one batch·python

import tensorflow as tf
import numpy as np

# Step 1: verify data
for x, y in train_dataset.take(10):
    assert not tf.reduce_any(tf.math.is_nan(x)), "NaN in inputs!"
    assert not tf.reduce_any(tf.math.is_inf(x)), "Inf in inputs!"
    print(f"x range: [{x.numpy().min():.3f}, {x.numpy().max():.3f}]")
    print(f"y unique: {np.unique(y.numpy())}")
    break

# Step 2: overfit one batch
single_x, single_y = next(iter(train_dataset))
history = model.fit(single_x, single_y, epochs=100, verbose=0)
print(f"Final loss on single batch: {history.history['loss'][-1]:.6f}")
# If not ~0, the bug is in: loss / output activation / label format / LR

Common bug signatures·python

# Symptom                       → Likely cause
# ------------------------------------------------------------
# Loss stuck at ~log(num_classes)  Init bad / LR too low — try LR x10
# Loss NaN after first batch       LR too high / no input norm /
#                                  wrong loss-activation pair
# Train acc 100%, val acc bad      Overfit — Dropout, L2, smaller model
# Train ≈ Val both bad             Underfit — bigger model / higher LR
# Output always one class          Class imbalance — class_weight or
#                                  focal loss; check final activation

# Detect NaN with a callback
class NaNDetector(tf.keras.callbacks.Callback):
    def on_batch_end(self, batch, logs=None):
        if logs and (tf.math.is_nan(logs['loss']) or
                     tf.math.is_inf(logs['loss'])):
            print(f"\n⚠️  NaN/Inf at batch {batch}!")
            self.model.stop_training = True

Exercise

Take a model that's training poorly. Run the data sanity check + overfit-one-batch. Identify which of the five canonical symptoms above matches. Apply the matching fix and verify behavior changes. Most training problems can be diagnosed in under 30 minutes with this routine.