Multi-Version Testing

~11 min · matrix, versions, compatibility

Level 0Apprentice

0 XP0/101 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Test against the versions you support

If your README says 'works on Python 3.10–3.12', CI should prove it on every PR. Without it, the support claim is a lie waiting to be discovered.

The basic pattern: a matrix over the dimension(s) you support. Example dimensions:

Language version (Python 3.10, 3.11, 3.12, 3.13).
OS (ubuntu-latest, macos-latest, windows-latest).
Database driver (postgres 14, 15, 16).
Framework version (Django 4.2, 5.0).

Each matrix cell is its own job, runs in parallel, with its own logs. The job name in the UI is templated (e.g., test (python-3.12, ubuntu-latest)).

Cost discipline

A naive 4×3 matrix is 12 runs per PR. Cost grows fast. Two compromises:

Full matrix on main, smoke matrix on PR. PRs only test the lowest and highest version on Linux. Main runs the full matrix.
Cancel slow OSes when Linux fails. Use job-level fail-fast: true to abort the whole matrix on the first failure.

Code

Python × OS matrix with smoke vs full strategy·yaml

jobs:
  test:
    name: test (py${{ matrix.python }}, ${{ matrix.os }})
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: true
      matrix:
        python: ['3.10', '3.11', '3.12']
        os: [ubuntu-latest]
        include:
          # Add macOS only on push to main
          - python: '3.12'
            os: macos-latest
          - python: '3.12'
            os: windows-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '${{ matrix.python }}' }
      - run: pip install -e '.[dev]' && pytest -q

External links

GitHub — Using a matrix for your jobs

Exercise

Pick the language version range your project actually claims to support (check README and pyproject/package.json). Add a matrix that tests every version. If any cell fails, decide: fix the code or update the support claim.