TextVectorization and Embeddings

Words → numbers, the right way

Before feeding text to a neural network, convert raw strings to dense numerical vectors. Two-step process: tokenize and encode to integers with TextVectorization, then map integers to dense vectors with an Embedding layer.

TextVectorization is a preprocessing layer that lives inside your model graph. This means tokenization is saved with the model and runs at inference without external scripts — a big advantage for production deployment.

The Embedding layer maps each integer token ID to a dense vector. These vectors are learned during training — semantically similar words end up near each other in embedding space. mask_zero=True tells downstream layers to ignore padding tokens (ID 0).

The classic property: well-trained embeddings encode semantic relationships. The famous example king - man + woman ≈ queen holds because directions in embedding space correspond to meaning. For production NLP in 2026, start with pretrained KerasHub embeddings (BERT, GPT-2, Gemma) rather than learning from scratch.

Code

TextVectorization + Embedding·python

import tensorflow as tf
from tensorflow.keras import layers

# Build vocab from training texts
vectorize = layers.TextVectorization(
    max_tokens=10000,
    output_mode='int',
    output_sequence_length=200,
    standardize='lower_and_strip_punctuation',
    split='whitespace',
)
train_texts = ["I love TensorFlow", "Deep learning is fun"]
vectorize.adapt(train_texts)
print(vectorize.get_vocabulary()[:5])    # ['', '[UNK]', 'is', ...]

# Embedding maps int IDs → dense 128-d vectors
embedding = layers.Embedding(
    input_dim=10000 + 2,    # vocab size + [PAD] + [UNK]
    output_dim=128,
    mask_zero=True,         # mask padding for variable-length sequences
)

# Full text classification pipeline
model = tf.keras.Sequential([
    vectorize,
    embedding,
    layers.Bidirectional(layers.LSTM(64, return_sequences=True)),
    layers.Bidirectional(layers.LSTM(32)),
    layers.Dense(64, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(1, activation='sigmoid'),
])

TextVectorization and Embeddings

Words → numbers, the right way

Code

Progress

Comments 0