Notebook Testing

~9 min · jupyter, notebooks, papermill

Level 0Apprentice

0 XP0/101 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Notebooks rot fast

Jupyter notebooks are everywhere in ML repos — exploratory analysis, training scripts, demo flows, tutorial pages. They also rot the moment they're committed: outputs go stale, dependencies shift, the 'just works on my laptop' state ages out.

Treat notebooks like code. Test them in CI.

Tools

papermill — execute a notebook end-to-end with parameterized inputs. Output is a new notebook with all cells run.
pytest --nbmake — runs notebooks as pytest tests.
jupyter nbconvert --execute — built-in, low-frills.
nbqa — runs ruff / black / mypy on notebooks.

What to assert

The notebook runs to completion without error.
Specific cells produce the expected output (papermill + a small test script).
Lint / format are clean (nbqa + ruff).
Outputs are stripped before commit (use nbstripout or a pre-commit hook) so diffs stay readable.

Code

Notebook test job·yaml

  notebooks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync --all-extras --dev
      - name: Lint notebooks
        run: uv run nbqa ruff notebooks/
      - name: Execute notebooks
        run: |
          for nb in notebooks/*.ipynb; do
            uv run papermill "$nb" "/tmp/$(basename $nb)" --no-progress-bar
          done

External links

Exercise

Add nbstripout as a pre-commit hook in any repo with notebooks. Add a CI job that papermill-runs each notebook end-to-end. Watch a notebook stay green across version bumps.