Pooling and Recurrent Layers

Pooling: throwing away resolution on purpose

After a conv layer detects features, you usually don't need them at full spatial resolution. Pooling downsamples — shrinking the feature map while keeping the strongest signal — which cuts compute, enlarges each later neuron's receptive field, and adds a little translation tolerance.

MaxPooling2D(2) — takes the maximum value in each 2×2 window. The default choice; it keeps the strongest activation.
AveragePooling2D(2) — takes the mean instead. Smoother, used less often.
GlobalAveragePooling2D() — averages across all spatial positions, collapsing a (H, W, C) map to a single C-vector. The standard bridge between a conv backbone and a Dense head — it replaces Flatten and kills a huge pile of parameters.

Recurrent layers: state that walks the sequence

Recurrent layers (LSTM, GRU) process a sequence one timestep at a time, carrying a hidden state forward so each step's output depends on everything before it. The key knob is return_sequences: leave it False and you get a single vector summarizing the whole sequence (good for a classifier head); set it True and you get an output at every timestep — which is exactly what the next recurrent layer needs, so you must set it on every RNN except the last in a stack. Wrap a layer in Bidirectional to run it forward and backward and concatenate both readings.

Code

LSTM, GRU, and the Bidirectional wrapper·python

# LSTM: Long Short-Term Memory
layers.LSTM(64, return_sequences=True)  # Output at every timestep
layers.LSTM(64)                             # Output at last timestep only

# GRU: Gated Recurrent Unit (simpler, often comparable)
layers.GRU(64)

# Bidirectional wrapper: processes sequence forward AND backward
layers.Bidirectional(layers.LSTM(64))

Exercise

Build the same sequence classifier two ways: (a) Embedding → LSTM(64) → Dense, (b) Embedding → MultiHeadAttention → GlobalAveragePooling1D → Dense. Train both on a small text dataset (e.g. IMDB) and record wall-clock time per epoch and final validation accuracy. Note which one you'd actually ship and why.

Hint

The attention path usually trains faster per epoch on a GPU because it parallelizes across the sequence, while the LSTM must step through timesteps serially.