Multi-Modal Prompting

What multi-modal prompting adds#

Everything covered so far in this track assumes a purely text-in, text-out interaction. Many modern LLMs are multi-modal: they accept images, and sometimes documents, audio, or video, alongside (or instead of) text, and can reason about their content directly in the same request. The underlying prompting principles from earlier in this track — being specific about context, task, format, and constraints — still apply in full; multi-modal prompting adds a few genuinely new considerations about what the model can and can't reliably do with non-text input, and how to combine it usefully with text instructions.

Prompting with images#

An image is typically passed alongside a text instruction describing what to do with it — extract information, describe it, answer a question about it, compare it to another image:

{
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "What's the total amount due on this invoice, and by what date?" },
        { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0KG..." } }
      ]
    }
  ]
}

A few practical realities worth knowing:

• Be as specific about the visual task as you would be about a text task. "What's in this image?" invites a generic description; "Extract the total amount due and the due date from this invoice image, and return them as JSON" gets a targeted, structured answer instead.
• Image resolution and detail matter. Small text in a photographed (rather than scanned) document, low-resolution images, or heavily compressed images reduce a model's ability to read fine detail reliably — the same way it would for a person squinting at a blurry photo.
• Models can still hallucinate about images — describing a detail that isn't actually present, or misreading text in an image — for the same underlying reason covered in this site's LLM tutorials on hallucination: the model produces a plausible-sounding answer, not a verified one, whether the source is text or an image.

Prompting with documents#

Multi-page documents (PDFs, scanned forms) are often handled by either passing them as a sequence of page images, or through a dedicated document-input capability the provider offers, which typically preserves layout information (tables, columns, headers) better than reflowing the document into plain text first would. For extraction-heavy tasks (pulling structured fields from an invoice or ID document), naming the exact expected schema — the same discipline covered in this track's advanced-prompting page for text-only structured output — matters just as much for document inputs, and for the same reason: it narrows the model's own judgment calls about what counts as a match.

Extract the following fields from this scanned invoice and return ONLY
valid JSON matching this schema:

{
  "invoice_number": string,
  "total_due": number,
  "due_date": string
}

[document attached]

Combining modalities in one prompt#

Multiple images, or images plus text context, can be combined in a single request when the task genuinely needs both — for example, comparing a product photo against a written specification, or checking whether a screenshot matches a described expected state:

{
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "Does the packaging in this photo match the approved design spec described below? List any differences.\n\nSpec: Logo top-left, red background, white text, barcode bottom-right." },
        { "type": "image_url", "image_url": { "url": "data:image/png;base64,..." } }
      ]
    }
  ]
}

Common mistakes#

• Assuming image understanding is as reliable as reading the equivalent information from clean text — small text, low resolution, unusual fonts, and cluttered layouts all measurably increase the chance of a misread, especially for fine detail like exact numbers.
• Being vague about the visual task the same way an under-specified text prompt causes vague output — "describe this image" versus a specific extraction or comparison task produces very different quality and usefulness of answer.
• Skipping the same structured-output discipline used for text tasks — a document-extraction prompt with a named JSON schema is just as important for image/document inputs as it is for pure text ones (see this track's advanced-prompting page).