Essential Layers and Activation Functions

The vocabulary of every model you'll build

Keras provides over a hundred layers, but you'll use a small core constantly. Memorize these and the rest will fall into place.

Core layers: Dense (fully connected), Conv2D (image convolution), BatchNormalization, Dropout, MaxPooling2D, GlobalAveragePooling2D, Flatten, Reshape.

Activation functions: relu (default for hidden layers), gelu (transformer hidden layers), swish (EfficientNet), tanh (LSTM/GRU cells), sigmoid (binary output), softmax (multi-class output — but pair with from_logits=True in the loss for numerical stability).

The from_logits=True idiom: when using SparseCategoricalCrossentropy or BinaryCrossentropy, always set from_logits=True and skip the softmax/sigmoid in your model's output layer. The loss computes the activation internally using the numerically stable log-sum-exp formula. Adding softmax then taking log can produce NaN with extreme logits.

Code

핵심 layer 한눈에·python

from keras import layers
import tensorflow as tf

# Fully connected
dense = layers.Dense(units=128, activation='relu',
                     kernel_regularizer=tf.keras.regularizers.l2(0.01))

# Image convolution
conv = layers.Conv2D(filters=64, kernel_size=3, padding='same',
                     activation='relu')

# Regularization
bn = layers.BatchNormalization()
drop = layers.Dropout(rate=0.3)

# Pooling
max_pool = layers.MaxPooling2D(pool_size=2, strides=2)
global_avg = layers.GlobalAveragePooling2D()  # (b, h, w, c) → (b, c)

# Shape manipulation
flat = layers.Flatten()
reshape = layers.Reshape((7, 7, 32))

Activation 가이드·python

from keras import layers

# Hidden (MLP/CNN): relu, gelu (transformers), swish (EfficientNet)
relu = layers.Activation('relu')
gelu = layers.Activation('gelu')
swish = layers.Activation('swish')

# Hidden (RNN cells): tanh
# Output (binary): sigmoid — but use from_logits=True in loss
# Output (multi-class): softmax — but use from_logits=True in loss
# Output (regression): None (linear)

# LeakyReLU fixes "dead neuron" problem
leaky = layers.LeakyReLU(alpha=0.1)

Essential Layers and Activation Functions

The vocabulary of every model you'll build

Code

Progress

Comments 0