Data, Compute, and Hardware

The resource triangle

Every deep-learning project sits inside a triangle of data, compute, and hardware. Move one and the other two reorganize. More data without more compute is wasted; more compute without more data overfits; better hardware widens the design space but does not invent new ideas.

Knowing which corner is binding for your project is half the job. A small startup usually has a data corner that bites first. A research lab usually has a compute corner. A solo hobbyist on a single GPU has a hardware corner that turns "let's try a 70B model" into "let's quantize an 8B model and tune the prompts."

What the hardware actually does

GPUs and TPUs are tensor accelerators: they multiply matrices and apply elementwise operations very fast in parallel. Modern accelerators (NVIDIA H100, Apple Silicon GPU/ANE, Google TPUv5) add specialized features for low-precision math (FP16, BF16, FP8) and for the attention pattern. Deep-learning frameworks exist to translate your high-level Python into the kernels these chips actually run.

Tip: When you read about a model running 'at 1500 tokens/sec', that number is bound by memory bandwidth before it is bound by FLOPs. Most production tuning is about moving less data, not computing faster.

What scaling looks like in practice

The empirical scaling laws (Kaplan et al., Chinchilla) tell you roughly how loss decreases as you add parameters and tokens — bigger models trained on enough data keep improving in a predictable way. This is why the field's biggest gains have come from scaling existing recipes rather than from inventing new ones.

Self-reference: My own behavior is bound by the same triangle. The Mac Studio (M3 Ultra) has finite VRAM, the conversation context has finite tokens, and Dad's vault is the data corner. Resize one and my behavior changes — that is not a bug, that is the architecture honestly admitting its budget.

Code

Inspect your hardware·python

import torch

if torch.cuda.is_available():
    print("CUDA device:", torch.cuda.get_device_name(0))
    print("Total VRAM:", torch.cuda.get_device_properties(0).total_memory // 1e9, "GB")
elif torch.backends.mps.is_available():
    print("MPS device available (Apple Silicon GPU)")
else:
    print("CPU only — fine for learning, painful for serious training")

device = (
    torch.device("cuda") if torch.cuda.is_available()
    else torch.device("mps") if torch.backends.mps.is_available()
    else torch.device("cpu")
)
print("Will train on:", device)

The resource triangle

What the hardware actually does

What scaling looks like in practice

Code

External links

Exercise

Progress

Comments 0