Hugging Face Transformers — AutoModel and Pipelines · PyTorch Quest

One package, every Transformer architecture

The transformers library (Hugging Face) gives you a uniform interface to every major NLP, vision, audio, and multimodal Transformer. Three layers of abstraction, from highest to lowest:

Pipelines — one-line task-shaped APIs (pipeline("sentiment-analysis"), pipeline("summarization")). Best for prototyping and one-off scripts.
Auto* classes — AutoTokenizer, AutoModel, AutoModelForSequenceClassification, etc. They detect the architecture from the checkpoint name and instantiate the right class. The right level for "I want to control the model and the loop".
Specific model classes — BertModel, GPT2LMHeadModel, etc. Use only when you need architecture-specific features the Auto layer hides.

Tokenizer + Model — the standard pair

Every Transformers model comes with a matching tokenizer. The model expects token IDs (and an attention mask). The tokenizer turns text into IDs. They MUST match — load both from the same checkpoint name.

The "head" pattern

For each base model architecture there are several "head" variants for different tasks:

AutoModel — base model, returns hidden states.
AutoModelForSequenceClassification — classification head (with logits).
AutoModelForTokenClassification — per-token labels (NER).
AutoModelForCausalLM — language modeling (GPT-style).
AutoModelForSeq2SeqLM — encoder-decoder (T5-style).
AutoModelForQuestionAnswering — extractive QA (start/end positions).

Pick the one matching your task; the library hooks up the right output head automatically.

Code

Pipeline — fastest path to working code·python

# pip install transformers
from transformers import pipeline

# Sentiment classification
clf = pipeline("sentiment-analysis")
print(clf("PyTorch makes deep learning feel native."))
# [{'label': 'POSITIVE', 'score': 0.9998}]

# Other tasks — same one-line idiom
summarizer = pipeline("summarization")
qa = pipeline("question-answering")
generator = pipeline("text-generation", model="gpt2")
ner = pipeline("ner", aggregation_strategy="simple")

Auto* — the right level for fine-tuning·python

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "distilbert-base-uncased-finetuned-sst-2-english"
tok = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

# Tokenize — the inputs the model expects
inputs = tok(
    ["This tutorial is great!", "I'm bored."],
    padding=True, truncation=True, return_tensors='pt',
)
print(inputs.keys())   # dict_keys(['input_ids', 'attention_mask'])

# Forward
with torch.inference_mode():
    out = model(**inputs)
print(out.logits)
# tensor([[-1.9234, 2.0456], [1.7234, -1.8123]])

# Convert to labels
labels = ['NEGATIVE', 'POSITIVE']
preds = out.logits.argmax(-1)
for text, p in zip(["good", "bad"], preds):
    print(text, '→', labels[p])

Loading a base model + adding your own head·python

import torch
import torch.nn as nn
from transformers import AutoTokenizer, AutoModel

# Sometimes you want the encoder + a custom head (e.g. multi-task, custom loss)
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
encoder = AutoModel.from_pretrained("bert-base-uncased")

class CustomBertClassifier(nn.Module):
    def __init__(self, encoder, num_classes, dropout=0.1):
        super().__init__()
        self.encoder = encoder
        self.dropout = nn.Dropout(dropout)
        hidden = encoder.config.hidden_size
        self.classifier = nn.Linear(hidden, num_classes)

    def forward(self, input_ids, attention_mask=None):
        out = self.encoder(input_ids=input_ids, attention_mask=attention_mask)
        cls_token = out.last_hidden_state[:, 0]      # [CLS] representation
        cls_token = self.dropout(cls_token)
        return self.classifier(cls_token)

model = CustomBertClassifier(encoder, num_classes=5)

Hugging Face Transformers — AutoModel and Pipelines

One package, every Transformer architecture

Tokenizer + Model — the standard pair

The "head" pattern

Code

External links

Exercise

Progress

Comments 0