Multimodal AI: Text, Images & Audio

What is multimodal AI?#

Multimodal AI can understand — and sometimes generate — more than one type of data: text, images, audio, video and documents. A multimodal model can read a screenshot of an error, look at a chart in a PDF, or listen to a lecture recording, and reason about it in text.

"Modality" just means a type of input or output. A text-only chatbot is unimodal; a model that answers questions about a photo is multimodal.

How models understand images#

A multimodal LLM usually combines three parts:

1. A vision encoder (often a Vision Transformer, ViT) splits an image into small patches and turns each patch into an embedding.
2. A projection layer maps those image embeddings into the same space the language model uses for text tokens.
3. The LLM reads image "tokens" and text tokens together and generates an answer.

To the language model, an image becomes a sequence of tokens it can attend to — just like words. That's why images also consume context and cost tokens.

Common multimodal tasks#

Understanding vs generating#

Many products mix models behind the scenes:

• Understanding images and documents → a multimodal LLM.
• Generating images → a diffusion model.
• Speech in or out → dedicated audio models.

A "multimodal app" is often several specialised models wired together, not one model doing everything.

Prompting with images: what works#

• Put the image before the question and be specific: "In the attached bank statement, list every transaction above 10,000 PKR as JSON with date, description and amount."
• Use good resolution. Tiny or blurry images lose detail; very large images cost more tokens.
• Ask for structure (JSON, tables) when you'll process the answer in code.
• Ask the model to quote what it sees before concluding — it reduces made-up readings.
• Crop to the relevant region when only part of an image matters.

Limitations to know#

• Fine details — small text, dense tables and exact counts can still be misread. Validate numbers.
• Spatial reasoning — "is A left of B?" and precise measurements are weaker than descriptions.
• Hallucinated details — a model may "see" text that isn't there, especially in low-quality images.
• Cost — every image adds many tokens; documents with many pages add up quickly.
• Privacy — images often contain faces, ID cards or personal data; handle them carefully.

A practical example: grading handwritten answers#

A teacher's assistant app could:

1. Accept a photo of a handwritten answer sheet.
2. Ask a multimodal model to transcribe the answers exactly.
3. Ask a second prompt to compare each answer with the marking scheme and suggest marks with reasons.
4. Show the transcription and suggestions to the teacher, who approves the final marks.

Splitting reading from judging makes errors easier to spot, and keeping a human in the loop keeps it fair.

Where multimodal AI is heading#

• Video understanding for lectures, sports and security footage.
• Real-time voice assistants that listen, see a camera feed and respond with speech.
• Computer-use agents that operate software through its screen like a person.

Common mistakes#

• Trusting extracted numbers blindly. Re-check totals and key fields in code.
• Sending huge images for simple questions. Resize to what the task needs.
• One prompt doing too much. Separate "read what's there" from "reason about it".
• Ignoring consent. Don't process people's photos or documents without permission.

Key takeaways#

• Multimodal AI combines text with images, audio, video and documents.
• Images become tokens through a vision encoder, so they cost context and money.
• Be specific, ask for structure, and validate details — especially numbers.
• Real products often combine several specialised models.