Chaining Transformations — map, batch, shuffle, filter

~13 min · map, batch, shuffle, filter

Level 0Level 0

0 XP0/78 lessons0/17 achievements

0/100 XP to next level100 XP to go0% complete

The recommended pipeline order

The power of tf.data comes from chaining transformations. Each one returns a new Dataset (lazy, not yet executed). The order you apply them matters — getting it wrong silently destroys randomization or wastes memory.

Step	Transformation	Why this order
1	`.map(parse, num_parallel_calls=AUTOTUNE)`	Decode/parse raw data
2	`.cache()`	Cache after expensive parse, before randomization
3	`.shuffle(buffer)`	Per-epoch reshuffling, before batching
4	`.batch(size, drop_remainder=True)`	Group into batches
5	`.map(augment, num_parallel_calls=AUTOTUNE)`	Augment AFTER batching (faster batched ops)
6	`.prefetch(AUTOTUNE)`	Always last — overlaps CPU/GPU

Critical rule: shuffle BEFORE batch. Shuffling after batch only randomizes batches, not individual elements — samples within each batch stay clumped together.

Code

표준 training pipeline·python

import tensorflow as tf

AUTOTUNE = tf.data.AUTOTUNE
BATCH_SIZE = 128

def normalize(image, label):
    return tf.cast(image, tf.float32) / 255.0, label

def augment(image, label):
    image = tf.image.random_flip_left_right(image)
    image = tf.image.random_brightness(image, max_delta=0.1)
    return image, label

train_ds = (
    tf.data.Dataset.from_tensor_slices((x_train, y_train))
    .map(normalize, num_parallel_calls=AUTOTUNE)   # 1. parse
    .cache()                                         # 2. cache normalized
    .shuffle(buffer_size=10000, seed=42)            # 3. shuffle
    .batch(BATCH_SIZE, drop_remainder=True)         # 4. batch
    .map(augment, num_parallel_calls=AUTOTUNE)      # 5. augment batched
    .prefetch(AUTOTUNE)                              # 6. prefetch — last
)

# Validation: no shuffle, no augment
val_ds = (
    tf.data.Dataset.from_tensor_slices((x_val, y_val))
    .map(normalize, num_parallel_calls=AUTOTUNE)
    .batch(BATCH_SIZE)
    .prefetch(AUTOTUNE)
)

Exercise

Build a tf.data pipeline that loads MNIST, normalizes pixels to [0,1], shuffles with buffer 10000, batches at 128, and prefetches AUTOTUNE. Iterate one batch and verify shapes are (128, 28, 28) for x and (128,) for y.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousDataset Basics — Creating and Iterating Next →AUTOTUNE, Caching, Prefetching

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.