Sequence-to-Sequence with Cross Attention

The pattern behind translation, summarization, code generation

Some tasks map one sequence to another of different length: translation (English → Korean), summarization (article → abstract), code generation (description → code). These follow the encoder-decoder pattern.

The encoder reads the entire input sequence and produces a context representation (typically a sequence of hidden states). The decoder generates output token by token, conditioned on both the encoder context (via cross-attention) and the tokens it has already generated (via self-attention).

For production 2026: don't implement seq2seq from scratch unless you're doing research. KerasHub provides T5, BART, and Whisper pretrained with a one-line load. Fine-tune them on your task; the architectural details are handled.

Code

KerasHub T5/BART/Whisper·python

import keras_hub

# T5 — general seq2seq, fine-tunable
t5 = keras_hub.models.T5CausalLM.from_preset("t5_base_multi")

# BART for summarization (pretrained on CNN/DailyMail)
bart = keras_hub.models.BartSeq2SeqLM.from_preset("bart_large_en_cnn")

input_text = "TensorFlow is an open-source machine learning framework..."
summary = bart.generate(input_text, max_length=100)
print(summary)

# Whisper — speech-to-text
whisper = keras_hub.models.WhisperAudioToText.from_preset("whisper_base_en")

Custom seq2seq 코어 — Encoder + CrossAttention·python

import tensorflow as tf
from tensorflow.keras import layers

class Encoder(tf.keras.layers.Layer):
    def __init__(self, vocab_size, d_model):
        super().__init__()
        self.embedding = layers.Embedding(vocab_size, d_model, mask_zero=True)
        self.rnn = layers.Bidirectional(
            layers.GRU(d_model // 2, return_sequences=True),
            merge_mode='concat',
        )

    def call(self, x, training=False):
        return self.rnn(self.embedding(x), training=training)


class CrossAttention(tf.keras.layers.Layer):
    def __init__(self, d_model):
        super().__init__()
        self.mha = layers.MultiHeadAttention(key_dim=d_model // 4, num_heads=4)
        self.layernorm = layers.LayerNormalization()

    def call(self, x, context, training=False):
        attn_out, _ = self.mha(
            query=x, value=context, key=context,
            return_attention_scores=True, training=training,
        )
        return self.layernorm(x + attn_out)

Sequence-to-Sequence with Cross Attention

The pattern behind translation, summarization, code generation

Code

Progress

Comments 0