TFX Pipelines Overview

The full ML lifecycle, not just serving

TensorFlow Extended (TFX) is an end-to-end ML platform. While TF Serving handles serving, TFX handles the full lifecycle: data ingestion, validation, preprocessing, training, evaluation, and deployment — all in a versioned, repeatable pipeline.

A TFX pipeline is a DAG of components. Each consumes and produces versioned ML artifacts (datasets, schemas, models, statistics). ML Metadata (MLMD) tracks the lineage of every artifact — you can trace any deployed model back to the exact data it was trained on.

The components you'll see in any production TFX pipeline:

ExampleGen — ingests raw data, splits train/eval
StatisticsGen / SchemaGen / ExampleValidator — schema inference, drift detection
Transform — feature engineering with TF Transform; produces a serving-time transform
Trainer — trains model using Transform output
Evaluator — TFMA model analysis with slicing + thresholds, blocks bad deploys
Pusher — deploys validated model to TF Serving

Code

Transform — train/serve skew 방지·python

import tensorflow_transform as tft

# This function runs identically during training AND at serving time
# - eliminating train/serve skew, the most insidious production bug
def preprocessing_fn(inputs):
    age_normalized = tft.scale_to_0_1(inputs['age'])
    occupation_vocab = tft.compute_and_apply_vocabulary(inputs['occupation'])

    return {
        'age_normalized': age_normalized,
        'occupation_idx':  occupation_vocab,
        'label':           inputs['label'],
    }

Evaluator — slicing으로 회귀 방지·python

import tensorflow_model_analysis as tfma
from tfx.components import Evaluator

eval_config = tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(label_key='income')],
    slicing_specs=[
        tfma.SlicingSpec(),                     # overall metrics
        tfma.SlicingSpec(feature_keys=['age']), # slice by age group
    ],
    metrics_specs=[
        tfma.MetricsSpec(
            thresholds={
                'accuracy': tfma.MetricThreshold(
                    value_threshold=tfma.GenericValueThreshold(
                        lower_bound={'value': 0.90}),    # at least 90%
                    change_threshold=tfma.GenericChangeThreshold(
                        direction=tfma.MetricDirection.HIGHER_IS_BETTER,
                        absolute={'value': -0.01}),     # max 1% regression
                ),
            }
        ),
    ],
)

evaluator = Evaluator(
    examples=example_gen.outputs['examples'],
    model=trainer.outputs['model'],
    baseline_model=model_resolver.outputs['model'],
    eval_config=eval_config,
)

The full ML lifecycle, not just serving

Code

Progress

Comments 0