Multimodal models

Models that accept and reason over images, audio, and sometimes video — alongside text. The architectural trick: map every modality into the same vector space the language model already understands.

What multimodal actually does

A multimodal model accepts more than text — images, audio, sometimes video — and reasons over them in the same head. You can paste a screenshot of a buggy UI and ask why it looks wrong. You can hand it a PDF page and ask what the chart shows. You can describe a problem in voice and have the answer come back the same way.

The architectural trick is that every modality gets converted into the same kind of vector that the language model already understands. Images become "image tokens." Audio becomes "audio tokens." From the model's perspective, it's all just tokens — and attention works across them indistinguishably.

That's the whole concept. Below: the encoder architecture, the practical limits (OCR, spatial reasoning, cost), and where multimodal already shines in production.

When you'd reach for it

Anywhere the input isn't naturally text:

• UI debugging and screenshot QA — paste a screenshot, get a diagnosis.
• Document understanding — PDFs with mixed text, tables, and charts.
• Accessibility — descriptions of images for screen readers.
• Image moderation and classification — replacing bespoke vision models for many tasks.
• Voice and real-time conversational agents — covered in the Voice lesson.

You wouldn't reach for multimodal when the input is already structured text — there's no benefit, and image tokens cost real money. And for high-accuracy dense-text OCR, dedicated tools (Textract, Document AI) still beat vision-LLMs.

How it's actually built

For images, the pattern (consistent across providers):

The vision encoder is typically a CLIP-style or ViT-style model trained jointly with the LLM (or projected into its space via a small adapter). The model then sees a single mixed sequence — [text] [image patches] [text] … — and attention works across all of it indistinguishably.

The same pattern (encoder per modality → shared vector space → unified attention) covers vision-in, audio-in, and increasingly real-time voice (Anthropic voice mode, OpenAI Realtime, Gemini Live). Image generation is a different architecture entirely — diffusion, covered in the Medium track lesson.

Where it bites in real life

Under the hood (optional)

A simple "describe this chart" call: read the image as base64, send it alongside a text question in a single message. ~15 lines. Skip if you don't code.

def describe_chart(image_path: str, question: str) -> str:
    with open(image_path, "rb") as f:
        b64 = base64.standard_b64encode(f.read()).decode("utf-8")
 
    resp = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {"type": "base64", "media_type": "image/png", "data": b64},
                },
                {
                    "type": "text",
                    "text": f"This is a chart from a financial report. {question}",
                },
            ],
        }],
    )
    return resp.content[0].text

Further reading

• Anthropic — Vision (using images with Claude) — official guide with pricing, formats, examples.
• OpenAI — Vision guide — same on OpenAI's side.
• Whisper paper (Radford et al. 2022) — the audio-encoder counterpart, widely used.
• OpenAI — DALL-E 3 — text-to-image example using a chat model to expand prompts.