Practical Image Loading + Antipatterns

The pipeline you'll actually write at work

For most real-world image projects, you have a directory of JPEG/PNG files organized by class. Two paths: the high-level keras.utils.image_dataset_from_directory for quick experiments, or a manual pipeline with tf.io.read_file when you need control.

The four antipatterns to avoid:

Shuffling after batching — shuffles batches, not elements.
Missing prefetch — GPU stalls between batches.
Caching augmented data — same augmentation every epoch.
Hardcoded num_parallel_calls — instead of AUTOTUNE, which adapts.

When something feels slow, open TensorBoard's Input Pipeline Analyzer (Profile tab) and look for "Input Bound: 80%" — the canonical signal that data loading is the bottleneck, not model compute.

Code

Option A: image_dataset_from_directory·python

import tensorflow as tf
from tensorflow import keras

# Directory layout:
# data/train/cats/*.jpg
# data/train/dogs/*.jpg

IMG_SIZE = (224, 224)
BATCH_SIZE = 32
AUTOTUNE = tf.data.AUTOTUNE

train_ds = keras.utils.image_dataset_from_directory(
    "data/train",
    image_size=IMG_SIZE,
    batch_size=BATCH_SIZE,
    label_mode='int',           # or 'categorical' for one-hot
    shuffle=True,
    seed=42,
    validation_split=0.2,
    subset='training',
)
class_names = train_ds.class_names

normalization = keras.layers.Rescaling(1./255)

def augment_image(image, label):
    image = tf.image.random_flip_left_right(image)
    image = tf.image.random_brightness(image, 0.1)
    image = tf.image.random_contrast(image, 0.9, 1.1)
    return image, label

train_ds = (train_ds
    .map(lambda x, y: (normalization(x), y), num_parallel_calls=AUTOTUNE)
    .cache()
    .map(augment_image, num_parallel_calls=AUTOTUNE)
    .prefetch(AUTOTUNE))

Option B: 수동 pipeline (제어 더 많음)·python

import tensorflow as tf
import pathlib

data_dir = pathlib.Path("data/train")
image_paths = list(data_dir.glob("*/*.jpg"))
class_names = sorted([p.name for p in data_dir.iterdir() if p.is_dir()])
class_to_idx = {n: i for i, n in enumerate(class_names)}
labels = [class_to_idx[p.parent.name] for p in image_paths]

def load_and_preprocess(path, label):
    raw = tf.io.read_file(path)
    image = tf.io.decode_image(raw, channels=3, expand_animations=False)
    image = tf.image.resize(image, [224, 224])
    image = tf.cast(image, tf.float32) / 255.0
    return image, label

ds = (tf.data.Dataset.from_tensor_slices(
        ([str(p) for p in image_paths], labels))
    .shuffle(len(image_paths), seed=42)
    .map(load_and_preprocess, num_parallel_calls=tf.data.AUTOTUNE)
    .cache()
    .batch(32, drop_remainder=True)
    .prefetch(tf.data.AUTOTUNE))

Practical Image Loading + Antipatterns

The pipeline you'll actually write at work

Code

Progress

Comments 0