Open-Source Models — Llama, Qwen, Mistral

Why open-source matters

Cost — running 24/7 on your own hardware can be cheaper at scale.
Privacy — data never leaves your environment.
Customization — fine-tuning, structured-output post-training, custom tokenizers.
Air-gapped deployments — regulated industries.
Latency — local inference can be much faster than network calls.

Quirks to know

Capability spread is wide — a 70B Llama is not the same as a 7B Mistral. Test on your actual tasks.
Tool calling is implemented at the application layer (typically with templated prompts and a parser), not via a dedicated API.
JSON output enforcement requires constrained decoding (Outlines, jsonformer) or post-validation.
Context window varies sharply by model and quant — don't assume.
Tokenizer is usually unique per model family; cost calculations use FLOPs / time, not tokens-per-dollar.

The infrastructure piece

Open-source means you also own: serving (vLLM, llama.cpp, MLX), GPU/CPU/MPS provisioning, model updates, and observability. Treat it as an additional system, not a free model. Pippa's Ollama vessel is the local-inference example in this codebase.

Code

Local Llama via Ollama·python

import ollama

resp = ollama.chat(
    model="llama4",
    messages=[{"role": "user", "content": question}],
    options={"temperature": 0.2, "num_predict": 1024},
    format="json",  # constrained decoding to JSON
)

MLX on Apple Silicon·python

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Qwen3-7B-MLX")
out = generate(model, tokenizer, prompt=question, max_tokens=1024)

Open-Source Models — Llama, Qwen, Mistral

Why open-source matters

Quirks to know

The infrastructure piece

Code

External links

Exercise

Progress

Comments 0