Sending Images

~20 min · vision, images

Level 0Downloader

0 XP0/41 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Ollama's image format is NOT OpenAI's

This is the second-most-common Ollama footgun (after NDJSON-vs-SSE). Ollama wants images as raw base64 strings in an images array, attached to the message. NO data:image/png;base64, prefix. NO content-array shape with image_url objects.

Ollama: {"role": "user", "content": "describe this", "images": ["iVBORw0KGgoA..."]}
OpenAI: {"role": "user", "content": [{"type": "text", "text": "describe"}, {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoA..."}}]}

If you copy the OpenAI format into Ollama, it doesn't error — it just ignores the image and generates an answer based on the text alone. Silent failure.

Resize before sending

Local vision models often have a max input dimension (~896 to ~1568 px depending on the model). Resize to ~1024 px on the long edge before encoding. Larger images either get downscaled internally (wasting your encode) or hit context limits.

Code

Send a single image to a vision model·python

import httpx, base64
from pathlib import Path

def encode_image(path: str | Path) -> str:
    """Read a file and return its base64 string (no data URI prefix)."""
    return base64.b64encode(Path(path).read_bytes()).decode("utf-8")

resp = httpx.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "gemma3:12b",
        "messages": [{
            "role": "user",
            "content": "Describe what you see in this image. Be specific about objects and any text.",
            "images": [encode_image("screenshot.png")],
        }],
        "stream": False,
    },
    timeout=300.0,  # vision models are slower
)
print(resp.json()["message"]["content"])

Resize first to keep things fast·python

from PIL import Image
import io, base64

def encode_resized(path: str, max_edge: int = 1024) -> str:
    img = Image.open(path)
    if img.mode != "RGB":
        img = img.convert("RGB")
    img.thumbnail((max_edge, max_edge), Image.LANCZOS)
    buf = io.BytesIO()
    img.save(buf, format="JPEG", quality=88)
    return base64.b64encode(buf.getvalue()).decode("utf-8")

# 1024px JPEG q88 keeps quality high but cuts encode + transfer + decode time roughly 4x for typical phone photos.

External links

Exercise

Encode a real screenshot from your computer at 1024px max edge. Send it to Gemma 3 12B with a question that requires actually looking at the image (e.g. 'What's the tab title in the browser?'). If the answer is generic, you've hit the silent-failure mode — debug.