Accelerate: Multi-Device, Mixed Precision, FSDP

~22 min · ops, accelerate

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

What accelerate gives you

accelerate is the abstraction below Trainer that handles: device placement, mixed precision, gradient accumulation, distributed training (DDP, FSDP, DeepSpeed), and resumability. Trainer wraps it; you can also use it directly when you have a custom training loop.

Two ways in

Through Trainer — you set TrainingArguments knobs (fsdp, deepspeed) and accelerate is plumbed for you.
Directly — Accelerator() + accelerator.prepare(model, optimizer, dataloader) + your own .backward() / .step().

FSDP vs DeepSpeed in 2026

FSDP (Fully Sharded Data Parallel, native PyTorch) is now the default for training big models — better integration, simpler config. DeepSpeed (Microsoft) is still strong for some advanced workloads but adds a runtime dependency. For most fine-tuning and pre-training at the scales we'd consider, FSDP is the right starting point.

Code

Generate an accelerate config and run·bash

# Interactive — answers go in ~/.cache/huggingface/accelerate/default_config.yaml
accelerate config

# Launch a script with the config
accelerate launch train.py

Custom training loop with accelerate·python

from accelerate import Accelerator
from torch.utils.data import DataLoader
from transformers import AutoModelForCausalLM, AutoTokenizer, get_cosine_schedule_with_warmup
import torch

acc = Accelerator(mixed_precision="bf16")  # bf16 / fp16 / no
device = acc.device

model = AutoModelForCausalLM.from_pretrained("...")
tok = AutoTokenizer.from_pretrained("...")
optim = torch.optim.AdamW(model.parameters(), lr=1e-4)

# loader = DataLoader(...)
# scheduler = get_cosine_schedule_with_warmup(optim, num_warmup_steps=100, num_training_steps=10_000)

model, optim, loader, scheduler = acc.prepare(model, optim, loader, scheduler)

for step, batch in enumerate(loader):
    out = model(**batch)
    acc.backward(out.loss)
    optim.step(); scheduler.step(); optim.zero_grad()
    if step % 50 == 0 and acc.is_main_process:
        print(step, out.loss.item())

External links

Exercise

Run accelerate config interactively. Train your IMDB Trainer setup with accelerate launch on a single GPU first. Then if you have multi-GPU access, switch to FSDP and verify the same recipe works at higher effective batch size with similar loss curve.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousONNX and MLX Conversion Next →Optimum: Hardware-Specific Optimization

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.