Driver, Toolkit, NVCC — The Three Layers

Three things must agree, or nothing runs

'Install CUDA' is a misleading instruction — there isn't one thing called CUDA. There are three layers, each on a different release cadence, and they have to be compatible:

Driver — kernel-mode software that talks to the GPU silicon. Shipped by NVIDIA, updated almost weekly. The driver advertises a maximum supported CUDA runtime version.
Toolkit — userland headers, libraries (cuBLAS, cuDNN, cuFFT, etc.), and tools (NVCC, Nsight). Released every few months as 'CUDA 13.0', 'CUDA 13.2', etc.
NVCC — the compiler binary inside the toolkit. It compiles your .cu source into a fat binary containing PTX + multiple SASS variants targeted at specific compute capabilities (sm_75 Turing, sm_86 Ampere, sm_89 Ada Lovelace, sm_90 Hopper).

The compatibility rule: your driver must be ≥ the version baked into the toolkit you compiled with. A binary compiled with CUDA 13.2 toolkit will not run on a driver that only supports up to CUDA 12.4. The driver is the floor.

This sounds bureaucratic until something breaks. Then you'll be reading nvidia-smi output trying to figure out whether your driver, your toolkit, or your binary's SASS targets are the problem. The mental model is: install matching versions, and bake the right SASS into the binary.

Why does this matter for AI work? Because PyTorch, TensorFlow, JAX, and MLX all ship pre-compiled CUDA kernels for specific compute capabilities. When you upgrade your GPU from a 30-series to a 40-series, you may need a newer framework wheel that contains sm_89 SASS — otherwise everything still works through PTX JIT, just slower at startup.

Code

nvidia-smi — inspect driver + max supported CUDA version·bash

nvidia-smi
# Top-right cell shows: 'CUDA Version: 13.2'
# That is the MAX supported runtime, not the toolkit installed.
# A 13.2 driver runs binaries built with toolkit 13.2 or older.

nvidia-smi --query-gpu=driver_version,name,compute_cap --format=csv
# driver_version, name,                compute_cap
# 555.99,         NVIDIA GeForce RTX 4090, 8.9

nvcc — inspect installed toolkit·bash

nvcc --version
# nvcc: NVIDIA (R) Cuda compiler driver
# Cuda compilation tools, release 13.2, V13.2.140
# Build cuda_13.2.r13.2/compiler.34123456_0

# Compile for the architecture you actually have:
nvcc -arch=sm_89 hello.cu -o hello   # sm_89 = Ada Lovelace (RTX 4090)

# Or compile for multiple architectures (fat binary):
nvcc -gencode arch=compute_86,code=sm_86 \
     -gencode arch=compute_89,code=sm_89 \
     hello.cu -o hello

Exercise

On any machine you have access to with an NVIDIA GPU, run both nvidia-smi and nvcc --version (install nvcc if needed). Write down: (a) driver version, (b) CUDA Version field from nvidia-smi, (c) toolkit version from nvcc, (d) GPU name and compute capability. Note whether toolkit ≤ driver-CUDA-version. If you don't have an NVIDIA box, do the same on a Colab free tier (T4) or Kaggle notebook (P100/T4) — they show all three numbers in their default output.

Driver, Toolkit, NVCC — The Three Layers

Three things must agree, or nothing runs

Code

External links

Exercise

Progress

Comments 0