Modality Differences — Vision, Audio, Files

~14 min · providers, multimodal

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Beyond text

Modern prompts often include images, audio, or files. The mechanics differ sharply by provider.

Vision

Anthropic — image as a content block (base64 or URL); supported on most Claude models.
OpenAI — image_url content type; vision is part of the multimodal models (gpt-5.5).
Gemini — inline_data with mime_type; native multimodal across the family.

Audio

OpenAI — Whisper for transcription; Realtime API for live voice.
Gemini — native audio understanding inline.
Anthropic — text-only on most public models in 2026; pair with separate transcription.

Files

Anthropic — Files API (upload once, reference by id).
OpenAI — Files API + Assistants attachments.
Gemini — Files API for large media.

Implications for prompts

Multimodal inputs change what the prompt can refer to ("the second image," "the chart in figure 3"). Reference content explicitly. Vision tokens cost differently from text tokens; track them separately.

Code

Image content (Claude, OpenAI, Gemini)·python

# Claude
client.messages.create(model="claude-opus-4-7", messages=[
    {"role": "user", "content": [
        {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
        {"type": "text", "text": "What's in this chart?"}
    ]}
])

# OpenAI
client.chat.completions.create(model="gpt-5.5", messages=[
    {"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": data_url}},
        {"type": "text", "text": "What's in this chart?"}
    ]}
])

# Gemini
model.generate_content(["What's in this chart?", {"mime_type": "image/png", "data": img_bytes}])

External links

Exercise

Build the same vision task on two providers. Measure tokens-per-image, latency, and accuracy. Note any prompt-side differences (how you reference the image content).

Progress

Progress is local-only — sign in to sync across devices.

← PreviousSwitching Providers — What Breaks and How to Plan Next →Pricing and Rate Limits — Real Numbers

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.