Data Pipeline Options

One fit(), many input types

One of Keras 3's quiet superpowers: model.fit() doesn't care how your data is packaged. The same training call accepts NumPy arrays, a tf.data.Dataset, a PyTorch DataLoader, a keras.utils.PyDataset, or a Pandas DataFrame — on any backend. That decoupling is what lets you keep your model code stable while you swap the data layer underneath it.

Format	Backend	Best For
NumPy arrays	All	Small datasets that fit in memory
`tf.data.Dataset`	All	Large datasets, complex pipelines
PyTorch `DataLoader`	All	PyTorch ecosystem integration
`keras.utils.PyDataset`	All	Custom data generators
Pandas DataFrame	All	Tabular data

How to choose

NumPy is the right answer when everything fits in RAM — it's the simplest path and there's no reason to overbuild. The moment your data outgrows memory, or you need streaming, on-the-fly transforms, shuffling, and prefetch, reach for a real pipeline. tf.data is the most mature and feature-complete; torch.DataLoader is the idiomatic choice once you're on the PyTorch backend (with num_workers for multi-process loading); keras.utils.PyDataset is the framework-neutral escape hatch for custom generators you control end to end.

The one gotcha that bites everyone

When you pass a batched dataset (anything iterable — tf.data, DataLoader, PyDataset), the batch_size argument to fit() is ignored. The dataset already decides its own batch size. People wire up a DataLoader(batch_size=64), then pass fit(..., batch_size=32), and get confused why batches are 64 wide. The dataset wins — set the batch size where the batching actually happens.

Code

The same fit() accepts every input format·python

# All of these work with model.fit():
model.fit(numpy_x, numpy_y)              # NumPy arrays
model.fit(tf_dataset)                     # tf.data.Dataset
model.fit(torch_dataloader)               # PyTorch DataLoader
model.fit(keras_pydataset)                # keras.utils.PyDataset

One fit(), many input types

How to choose

The one gotcha that bites everyone

Code

External links

Exercise

Progress

Comments 0