C.W.K.
Stream
Lesson 01 of 05 · published

End-to-End Workflow

~28 min · workflow, end-to-end

Level 0Scout
0 XP0/48 lessons0/11 achievements
0/120 XP to next level120 XP to go0% complete

The minimum viable pipeline

An end-to-end ML project has a fixed shape: contract → data ingestion → audit → split → preprocessing pipeline → baseline → tuned model → evaluation → calibration → threshold → artifact → deployment → monitoring. Skip any step and a known failure mode appears in production.

The folder structure that scales

ml_project/
├── data/{raw,interim,processed}
├── notebooks/
├── src/
│   ├── contract.py
│   ├── features.py
│   ├── train.py
│   ├── eval.py
│   └── predict.py
├── tests/
├── artifacts/
└── README.md

Reproducibility hooks

Pin dependencies (uv, pip-tools, or poetry). Seed every random source. Log dataset hash, code commit, hyperparameters, and metrics in one experiment record. If a teammate cannot rerun your experiment in 10 minutes, the project is not reproducible.

Code

A trainable end-to-end script·python
import joblib
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from src.contract import load_contract
from src.features import build_preprocessor
from src.eval import evaluate

def main():
    contract = load_contract("contracts/churn.yaml")
    df = load_data(contract)
    X, y = df[contract.features], df[contract.target]
    pipe = Pipeline([("pre", build_preprocessor(contract)), ("clf", LogisticRegression(max_iter=1000))])
    pipe.fit(X, y)
    evaluate(pipe, X, y, contract)
    joblib.dump(pipe, contract.artifact_path)

if __name__ == "__main__":
    main()
Pin everything that influences the artifact·yaml
experiment:
  id: churn_2026_05_03_a
  data_hash: a8f1d8e9
  git_commit: 4f7c2e1
  random_seed: 7
  python: 3.12.4
  packages:
    scikit-learn: 1.5.0
    lightgbm: 4.5.0
  metrics:
    pr_auc: 0.317
    recall_at_p70: 0.62

External links

Exercise

Take your latest notebook and refactor it into a src/ module that can be invoked as python -m src.train --config contracts/your_problem.yaml. Verify a fresh kernel can produce the same artifact and metrics.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.