A Filename Is Not Proof

"You can tell a model 'there is a photo called lens.jpg.' That sentence is not the photo. The model can't see a filename."

The Bug That Looks Like It Works

Here's the most seductive multimodal failure. You attach an image, and somewhere in the pipeline the code sends the model a message like User attached: lens.jpg — a filename, maybe a size, maybe a caption — instead of the actual image data. The model, being a good conversationalist, responds as if it understood: "That lens looks like a 35mm prime." It looks like vision worked. It didn't. The model saw the word lens.jpg and improvised a plausible reply about a lens it never actually looked at. The bug is invisible precisely because the model is polite enough to fake competence.

What Vision Actually Requires

For a model to genuinely see an image, three things must reach its adapter, together:

The bytes. The real, decoded image data — not a path, not a URL the model can't fetch, not a description.
The media type. image/jpeg, image/png, etc., so the bytes are interpreted correctly (the next lessons dwell on this).
Correct placement. The image delivered as an image part of the request, in the shape the adapter expects — not smuggled into the text as a mention.

Anything less is not vision. A filename, a caption, an alt-text, a "[image here]" token — these are text about an image, and text about an image is exactly as useful to a vision model as a description read over the phone: better than nothing, and categorically not the same as seeing.

Supplemental vs. Substitute

The fix isn't to ban filenames and captions — they can be genuinely useful supplemental context alongside the real bytes. The rule is that they must never be a substitute for the bytes. Send the actual image data as an image, and if you also want to include the filename as a hint, fine. The failure is when the hint travels instead of the data, and the polite model papers over the gap so well that nobody notices the eyes were never open. Treat 'the model replied about the image' as zero evidence that it saw the image; the only proof is that real bytes reached the adapter.

Code

Text about an image is not the image·typescript

// WRONG: the model receives a sentence, not a picture.
const message = {
  role: "user",
  content: `Question: ${q}\nAttached: ${file.name}`, // just the filename string
};
// The model will happily 'answer' about an image it never received.

// RIGHT: the real bytes and media type travel as an image part.
const message = {
  role: "user",
  content: [
    { type: "text", text: q },
    { type: "image", mediaType: file.type, data: await file.bytes() },
  ],
};
// Now 'I see a 35mm lens' is grounded in pixels, not improvised from a filename.

Exercise

Design a test that distinguishes 'the model saw the image' from 'the model improvised from a filename.' Hint: attach an image whose filename is misleading (name it sunset.jpg but make it a photo of a keyboard) and ask what's in it. Explain what each answer proves. Then describe where in a pipeline you'd assert that real bytes — not just a name — reached the adapter.

Hint

If the model says 'a sunset', it read the filename; if it says 'a keyboard', it read the pixels. The misleading-filename trick turns the invisible bug visible. In the pipeline, assert on the presence and size of the image bytes at the adapter boundary, not on the reply.