Data Pipelines with scikit-learn

Pipelines as the only path

A scikit-learn Pipeline wraps preprocessing and the estimator into a single fittable, predictable, and serializable object. The discipline is to make the pipeline the only path from raw rows to predictions. No notebook-cell preprocessing, no hand-applied transformations, and no separate serving implementation that can drift away.

Why this saves you

Cross-validation runs the entire pipeline per fold, so imputers, scalers, encoders, feature selectors, and the estimator are fitted on training rows only. GridSearchCV can compare preprocessing and estimator choices without opening a leakage path. The fitted pipeline then applies the same transformation at prediction time.

Make column routing explicit

Use a ColumnTransformer to declare numeric, categorical, and passthrough columns by name. Validate required columns and dtypes before transformation. Positional column selection can silently feed the wrong values after an upstream schema change.

Serialization is not immortality

joblib can store the fitted pipeline as one artifact, but pickle-based formats depend on Python classes and library versions and can execute code when loaded. Pin the environment, record the input schema and data hash, and never load an untrusted artifact.

Round-trip in a fresh process

Save the pipeline, clear the training process, load it in a new one, and score a raw row shaped exactly like a serving request. Compare predictions before and after serialization. Test missing fields, unknown categories, and reordered columns so failures are explicit rather than quietly wrong.

The discipline

Treat the pipeline as the contract between training and serving. If you cannot reproduce a prediction with pipeline.predict(raw_row) and no hidden glue, the artifact is incomplete and you have a bug waiting to ship.

Code

End-to-end pipeline with cross-validation·python

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

pre = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols),
])
pipe = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=500))])
scores = cross_val_score(pipe, X, y, cv=5, scoring="average_precision")
print("PR-AUC:", scores.mean(), "+/-", scores.std())

Save and load the pipeline as one artifact·python

import joblib

pipe.fit(X_train, y_train)
joblib.dump(pipe, "artifacts/churn_v1.joblib")

loaded = joblib.load("artifacts/churn_v1.joblib")
loaded.predict(X_new_raw_rows)  # transforms + predicts in one call

Pipelines as the only path

Why this saves you

Make column routing explicit

Serialization is not immortality

Round-trip in a fresh process

The discipline

Code

External links

Exercise

Progress

Comments 0