Reading a Model Card: The Five Questions

A model card is the standardized document a release ships alongside the weights. It tells you what the model is good for, what it isn't, and what compromises were made. Reading model cards critically is the single most useful "soft skill" in working with LLMs.

Five questions to ask of every model

What was it trained on? Source datasets, total token count, data cutoff date, deduplication and filtering strategy. Implicit biases and capabilities follow the data.
Which benchmarks were reported, and at what date? Older benchmarks (MMLU, ARC) are saturated; what's hard in 2026 is GPQA, AIME, SWE-bench, HLE. Be skeptical of benchmark numbers without context.
What's the license? Apache 2.0 / MIT (use freely), Llama-style community licenses (commercial OK with conditions), research-only licenses (no commercial use), proprietary (API only). License determines who can use it for what.
What's the context window — and how well is it actually used? "128K context" can mean anything from "trained at 128K" to "extended via YaRN with degraded recall past 32K." Always run a needle-in-a-haystack test.
Does it support the modalities and languages I need? Many models are English-first; some are explicitly multilingual; some support vision/audio. Check before you commit.

These five questions filter 90% of model-selection decisions. The remaining 10% — does it match the latent style and personality you want? — requires actually using it.

Code

Model card audit checklist·markdown

## Model card audit (fill out before adopting)

- **Architecture**: ___ params, ___ layers, ___ d_model, ___ heads, ___ KV heads
- **Tokenizer**: ___ algorithm, ___ vocab size, ___ specials
- **Context**: ___ training window / ___ extended window / ___ measured-recall depth
- **Training**: ___ tokens, ___ data sources, ___ cutoff date, ___ FLOPs, ___ time
- **Post-training**: ___ SFT recipe, ___ alignment method (RLHF/DPO/GRPO/CAI), ___ tool-use training
- **License**: ___ what can I do with it commercially?
- **Modalities**: text [Y/N] / image [Y/N] / audio [Y/N] / video [Y/N]
- **Languages**: primary ___, supported ___
- **Benchmarks**: GPQA ___ / AIME ___ / SWE-bench ___ / HLE ___
- **Known weaknesses**: ___ (read the limitations section)

Reading a Model Card: The Five Questions

Five questions to ask of every model

Code

External links

Exercise

Progress

Comments 0