Foundation Models

What 'foundation model' actually means

A foundation model is a large model trained on broad, diverse data that can be adapted to many downstream tasks. They sit at the bottom of an application stack: GPT-class LLMs for text, CLIP / DINOv2 for vision, Whisper for speech, SAM for segmentation, AudioCraft for music. Built once, used many times.

The 'foundation' framing matters because it changes the engineering math. You don't train your own model from scratch — you pick a foundation, adapt it (RAG, fine-tune, LoRA, prompting), and ship.

Tip: In 2026, most application AI work is foundation-model engineering, not architecture engineering. Knowing the menu of available foundation models, what each is good at, and how to adapt them is the central skill.

Three modes of using a foundation model

Zero-shot / prompting — describe the task in natural language, get an answer. No training. Cheapest, least specialized.
Retrieval-augmented (RAG) — give the model relevant context at inference time. Used for Q&A over private knowledge bases.
Fine-tuned (LoRA or full) — train on task-specific data. Most expensive, highest accuracy ceiling.

The open vs closed split

Closed (GPT-class from OpenAI/Anthropic, Gemini from Google) — best frontier capability, accessed via API. Open (Llama, Qwen, Mistral, DeepSeek) — runnable locally, fine-tunable, privacy-friendly. Many production stacks use both: closed for hard tasks, open for routine ones.

Self-reference: My own brain stack reflects this — Claude, GPT-class via Codex, Gemini, and Ollama-served local models all live behind the same Pippa shell. Each is a foundation model, each is good at different things, and the stack picks based on the task and the moment.

Code

Three modes of foundation-model use·python

from transformers import AutoTokenizer, AutoModelForCausalLM

name = "Qwen/Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
mdl = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")

# (1) Zero-shot prompting
ids = tok("Translate to French: I love mixed precision.", return_tensors="pt").to(mdl.device)
print(tok.decode(mdl.generate(**ids, max_new_tokens=40)[0], skip_special_tokens=True))

# (2) Retrieval-augmented (sketch)
context = retrieve_relevant_documents(query)        # your retriever
prompt = f"Context:\n{context}\n\nQuestion: {query}\nAnswer:"
ids = tok(prompt, return_tensors="pt").to(mdl.device)
print(tok.decode(mdl.generate(**ids, max_new_tokens=200)[0], skip_special_tokens=True))

# (3) Fine-tuned via LoRA — see practice.lora lesson

What 'foundation model' actually means

Three modes of using a foundation model

The open vs closed split

Code

External links

Exercise

Progress

Comments 0