End-to-End: Hub → Train → Serve → Space

The full loop, one repo at a time

Putting the eight tracks together: a single end-to-end pipeline that exists entirely on the Hub.

Hub: pick a base model. Pin a SHA. Read the card.
Datasets: pull a curated dataset, preprocess with map, push the processed version to a private dataset repo.
Training: SFTTrainer + LoRA on the processed dataset. Push the adapter to a private model repo.
Merge + quantize: merge adapter into base, AWQ-quantize for serving, push the merged AWQ variant to a public model repo with a real model card.
Serve: TGI or vLLM pointed at the merged AWQ model.
Demo: Gradio or Docker Space wrapping the served model with a chat UI.
Iterate: collect user feedback, append to a preference dataset, run DPO, rinse repeat.

What you get from doing it this way

Reproducibility (every artifact has a SHA). Auditability (every change is a Git commit). Portability (anyone with read access can replicate the stack). Versioning (roll back at any layer with one command). Cost transparency (each step's token / GPU-hour cost shows on the Hub or your provider).

Code

End-to-end skeleton (omit lots of detail)·python

# 1. Pin and pull base
from huggingface_hub import snapshot_download
base_path = snapshot_download("meta-llama/Llama-3.2-1B-Instruct",
                               revision="<sha>", local_dir="./base")

# 2. Process and push dataset
from datasets import load_dataset
raw = load_dataset("HuggingFaceH4/no_robots", split="train")
processed = raw.map(my_preprocess, num_proc=4, remove_columns=raw.column_names)
processed.push_to_hub("yourname/no-robots-cwk", private=True)

# 3. SFT + LoRA (see training track lesson 7)
# 4. Merge + quantize
from peft import AutoPeftModelForCausalLM
m = AutoPeftModelForCausalLM.from_pretrained("./adapter")
merged = m.merge_and_unload()
merged.push_to_hub("yourname/llama-1b-cwk", private=True)

# 5. Serve via TGI / vLLM (see serving track)
# 6. Wrap in a Gradio Space (see spaces track)
# 7. Collect feedback, build preference dataset, DPO, ...

The Hub repos for the loop·yaml

# Base (third-party): meta-llama/Llama-3.2-1B-Instruct
# Dataset (yours): yourname/no-robots-cwk
# Adapter (yours, private): yourname/llama-1b-cwk-adapter
# Merged (yours, public): yourname/llama-1b-cwk
# Quantized (yours, public): yourname/llama-1b-cwk-AWQ
# Demo Space (yours): yourname/llama-1b-cwk-demo

Exercise

Plan an end-to-end pipeline for a real task you have. Sketch the six Hub repos you'd create. For each, write one sentence about: who has access, what's pinned, how it's reproduced. Run at least three steps end-to-end (e.g. dataset processing → LoRA → merged push). The remaining steps you can leave as a sketch.