Data Pipelines with scikit-learn

~30 min · pipelines, sklearn, reproducibility

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Pipelines as the only path

A scikit-learn Pipeline wraps preprocessing and the estimator into a single fittable, predictable, and serializable object. The discipline is to make the pipeline the only path from raw rows to predictions. No notebook cell preprocessing, no hand-applied transformations, no "oh I forgot to scale at predict time".

Why this saves you

The pipeline serializes correctly with joblib, ships to production as one artifact, and applies identical transformations at predict time. Cross-validation runs the entire pipeline per fold, so leakage from preprocessing is prevented automatically. GridSearchCV can tune across both the preprocessing and the estimator at once.

The discipline

Treat the pipeline as the contract between training and serving. If you cannot reproduce a prediction with pipeline.predict(raw_row), the pipeline is incomplete and you have a bug waiting to ship.

Code

End-to-end pipeline with cross-validation·python

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

pre = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols),
])
pipe = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=500))])
scores = cross_val_score(pipe, X, y, cv=5, scoring="average_precision")
print("PR-AUC:", scores.mean(), "+/-", scores.std())

Save and load the pipeline as one artifact·python

import joblib

pipe.fit(X_train, y_train)
joblib.dump(pipe, "artifacts/churn_v1.joblib")

loaded = joblib.load("artifacts/churn_v1.joblib")
loaded.predict(X_new_raw_rows)  # transforms + predicts in one call

External links

Exercise

Refactor your training notebook so the only path from raw DataFrame to prediction is one fitted Pipeline saved as a joblib file. Verify by loading the pipeline in a fresh kernel and running pipe.predict(raw_row).