Practical: Multi-Input Image + Metadata Model

Putting the whole track to work

This capstone fuses two modalities that don't live in the same feature space: pixels and structured fields. Think of a product listing — a photo plus its price, category, and seller rating — where the right prediction needs both. You can't just stack these into one tensor; an image wants a convolutional stack, a 10-dimensional metadata vector wants Dense layers. The Functional API is what lets each modality keep its own processing path and still end up in one trainable model.

Branch, fuse, predict

The Code block is the canonical recipe, and it's every pattern from this track combined: two named inputs (lesson 03), an image branch that runs Conv → pool → GlobalAveragePooling2D down to a flat 64-dim feature vector, a metadata branch of Dense layers, then layers.concatenate to fuse and a shared head to predict. GlobalAveragePooling2D is the quiet hero — it collapses the spatial dimensions to a fixed-length vector so the two branches are concatenation-compatible regardless of input image size.

One decision dominates multi-modal design: where you fuse. This model uses late fusion — each modality is processed independently and only meets near the head. That's the right default: it's simple, each branch can be tuned in isolation, and you can swap in a pretrained image backbone without disturbing the metadata path. Early fusion (mixing the modalities before the deep layers) can be more expressive but is harder to train and to debug. Start late; move earlier only when you have evidence the modalities need to interact deeper.

Code

Multi-modal model — CNN image branch + Dense metadata branch, late-fused·python

# Image branch
image_input = keras.Input(shape=(64, 64, 3), name="image")
x = layers.Conv2D(32, 3, activation="relu")(image_input)
x = layers.MaxPooling2D(2)(x)
x = layers.Conv2D(64, 3, activation="relu")(x)
x = layers.GlobalAveragePooling2D()(x)
image_features = layers.Dense(64, activation="relu")(x)

# Metadata branch
meta_input = keras.Input(shape=(10,), name="metadata")
meta_features = layers.Dense(32, activation="relu")(meta_input)

# Merge and predict
combined = layers.concatenate([image_features, meta_features])
x = layers.Dense(64, activation="relu")(combined)
x = layers.Dropout(0.3)(x)
output = layers.Dense(5, activation="softmax")(x)

model = keras.Model(
    inputs=[image_input, meta_input],
    outputs=output,
)

Exercise

Build the image+metadata model from the Code block, then turn it into a regressor: swap the final head for layers.Dense(1) (no activation) and compile with loss='mse'. Use a small CNN (3 Conv + 1 Dense) for the image branch. Train on synthetic data and verify it trains end-to-end (loss decreases over 5 epochs).