TFX Pipeline 개요

전체 ML 라이프사이클, serving만이 아니라

TensorFlow Extended (TFX)는 end-to-end ML 플랫폼. TF Serving이 serving 처리한다면, TFX는 전체 라이프사이클: 데이터 수집, 검증, preprocessing, training, 평가, 배포 — 다 versioned, 재현 가능한 pipeline에서.

TFX pipeline은 component의 DAG. 각자 versioned ML artifact (dataset, schema, model, statistics) 소비하고 생산. ML Metadata (MLMD)가 모든 artifact의 lineage 추적 — 배포된 model을 정확한 훈련 데이터까지 추적 가능.

Production TFX pipeline에서 보게 될 component들:

ExampleGen — raw 데이터 수집, train/eval 분할
StatisticsGen / SchemaGen / ExampleValidator — schema 추론, drift 감지
Transform — TF Transform으로 feature 엔지니어링; serving time transform 생성
Trainer — Transform 출력 써서 model 훈련
Evaluator — TFMA model 분석 + slicing + threshold, 나쁜 배포 차단
Pusher — 검증된 model을 TF Serving에 배포

Code

Transform — preventing train/serve skew·python

import tensorflow_transform as tft

# This function runs identically during training AND at serving time
# - eliminating train/serve skew, the most insidious production bug
def preprocessing_fn(inputs):
    age_normalized = tft.scale_to_0_1(inputs['age'])
    occupation_vocab = tft.compute_and_apply_vocabulary(inputs['occupation'])

    return {
        'age_normalized': age_normalized,
        'occupation_idx':  occupation_vocab,
        'label':           inputs['label'],
    }

Evaluator — preventing regressions with slicing·python

import tensorflow_model_analysis as tfma
from tfx.components import Evaluator

eval_config = tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(label_key='income')],
    slicing_specs=[
        tfma.SlicingSpec(),                     # overall metrics
        tfma.SlicingSpec(feature_keys=['age']), # slice by age group
    ],
    metrics_specs=[
        tfma.MetricsSpec(
            thresholds={
                'accuracy': tfma.MetricThreshold(
                    value_threshold=tfma.GenericValueThreshold(
                        lower_bound={'value': 0.90}),    # at least 90%
                    change_threshold=tfma.GenericChangeThreshold(
                        direction=tfma.MetricDirection.HIGHER_IS_BETTER,
                        absolute={'value': -0.01}),     # max 1% regression
                ),
            }
        ),
    ],
)

evaluator = Evaluator(
    examples=example_gen.outputs['examples'],
    model=trainer.outputs['model'],
    baseline_model=model_resolver.outputs['model'],
    eval_config=eval_config,
)

전체 ML 라이프사이클, serving만이 아니라

Code

Progress

댓글 0