Mistral Family: Efficient European Open Weights

~10 min · mistral, mixtral, moe

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Mistral AI, founded by ex-Meta and ex-Google researchers, focuses on efficient open-weight models. The lineage is short but consequential.

Model	Total params	Active params	Architecture	Context
Mistral 7B (2023)	7B	7B	Dense + sliding-window attention	32K
Mixtral 8×7B (2024)	47B	13B	MoE, top-2 of 8 experts	32K
Mixtral 8×22B (2024)	141B	39B	MoE, top-2 of 8 experts	64K
Mistral Small 3 (24B)	24B	24B	Dense	32K
Mistral Large 3 (2024)	~675B	~41B	MoE	—
Mistral Small 4 (2025)	119B	6B	MoE, 128 experts top-4	256K

Mixtral 8×22B specifics: 56 layers, d_model=6144, 48 Q heads with 8 KV heads (GQA), SwiGLU, RoPE, multilingual (English, French, Italian, German, Spanish), Apache 2.0 license. The 8×22B and Mistral Small 4 specifically demonstrated that aggressive MoE (low active / high total) could match much larger dense models.

Code

Inspect Mixtral 8×22B's MoE config·python

from transformers import AutoConfig
cfg = AutoConfig.from_pretrained("mistralai/Mixtral-8x22B-v0.1")
print("num_experts:", cfg.num_local_experts)        # 8
print("experts per token:", cfg.num_experts_per_tok) # 2
print("hidden:", cfg.hidden_size)                    # 6144
print("intermediate:", cfg.intermediate_size)        # 16384
print("num_q_heads:", cfg.num_attention_heads)       # 48
print("num_kv_heads:", cfg.num_key_value_heads)      # 8 (GQA)

External links

Exercise

Run the same prompt through Mistral 7B and Mixtral 8x7B (e.g., via together.ai or replicate). Compare quality, speed, and cost. Where does the MoE's larger total knowledge show up? Where does its 13B active param count show up in latency?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousThe Llama Lineage: Open-Weight Frontier Next →Gemini Family: Multimodal-First from Google

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.