---
title: "Multimodal AI: definition and example – Carl-Uwe Würfel"
description: "Multimodal AI explained: how models handle text, images, audio, and video together, where they read receipts and photos, and which risks come with it."
url: https://carl-wuerfel.de/en/glossary/multimodale-ki
author: Carl-Uwe Würfel
---

1. [Home](https://carl-wuerfel.de/en)
  2. [Knowledge](https://carl-wuerfel.de/en/knowledge)
  3. [Glossary](https://carl-wuerfel.de/en/glossary)
  4. Multimodal AI

# Multimodal AI

Multimodal AI means AI models that process or generate several kinds of data, such as text, images, audio, video, and documents, and relate their content to each other.

## On this page

  1. How a model handles images, sound, and text together
  2. Native models and model chains work differently
  3. Where multimodal AI helps in operations
  4. The errors look different from text errors
  5. Photos and voices bring in GDPR and the AI Act
  6. What good implementation looks like
  7. Where the term stops working
  8. Frequently asked questions
  9. Sources

A chain of single models loses information between stages.

_Figure: A comparison based on ChatGPT’s Voice Mode as OpenAI described it in May 2024. On the left, the earlier pipeline of three models: one transcribed speech to text, GPT-3.5 or GPT-4 answered the text, and a third read the answer aloud; tone of voice, multiple speakers, and background noise never reached the language model, and responses took 2.8 or 5.4 seconds on average. On the right, GPT-4o, which processes all inputs and outputs in the same neural network and answers in 320 milliseconds on average. Below, the question for any vendor: what gets lost between the stages._

A modality is a type of data: written language, images, sound, video. A text-only language model accepts nothing but text. A multimodal model can take a photo, a question about it, and an attached PDF in one pass and answer in text; some also produce images or speech.

Google introduced Gemini in December 2023 as a model pre-trained on different modalities from the start. OpenAI announced GPT-4o in May 2024, saying it accepts any combination of text, audio, image, and video and outputs text, audio, and images. Anthropic documents image analysis for Claude.

For companies, scans, photos, and voice messages that someone used to retype become usable data. The models also misread numbers, describe things that aren't in the picture, and may process personal data with every photo. Whether a [use case](https://carl-wuerfel.de/en/glossary/use-case) pays off only shows in a test with the company's own documents.

Example

A wholesaler has drivers photograph delivery notes at drop-off, and a multimodal model reads the supplier, line items, quantities, and handwritten notes. The system matches the fields against the open purchase order and sends only notes with missing fields or mismatched quantities to a clerk. Before launch, the team built a test set of real photos, blurry ones included, and tracks errors per field.

Chapter 1 of 7

## How a model handles images, sound, and text together.

A multimodal model breaks images and sound into tokens, as a language model does with text. Anthropic spells this out for Claude: the image is split into 28 × 28 pixel blocks, each counting as one visual token. An image of 1,000 × 1,000 pixels comes to 1,296 tokens. Those tokens sit in the same context as the words of the question, and the model relates the two.

Models learn to connect images and words from pairs. A well-known building block is OpenAI's CLIP from 2021, trained on 400 million image-text pairs from the internet to predict which caption goes with which image.

In practice, resolution costs tokens and therefore money, and what a person can barely read in an image, the model won't read reliably either.

Chapter 2 of 7

## Native models and model chains work differently.

OpenAI described the difference with its own product. Before GPT-4o, ChatGPT's Voice Mode was a pipeline of three models: one transcribed speech to text, GPT-3.5 or GPT-4 answered the text, and a third read the answer aloud. Responses took 2.8 seconds on average with GPT-3.5 and 5.4 seconds with GPT-4. The language model never heard tone of voice, multiple speakers, or background noise. GPT-4o processes all inputs and outputs in the same [neural network](https://carl-wuerfel.de/en/glossary/neuronales-netz), and, per OpenAI, answers audio in 320 milliseconds on average.

Google described Gemini similarly: multimodal from the start, not stitched together from separate parts.

A chain of single models loses information between stages. In exchange, each stage can be tested and swapped on its own, and optical character recognition (OCR) is proven on clean forms. The question for any vendor is what gets lost between the stages: layout, table structure, handwriting, or tone.

Chapter 3 of 7

## Where multimodal AI helps in operations.

Invoices, receipts, and delivery notes arrive as scans, PDFs, or phone photos. The model reads the supplier, line items, and amounts and stores them as fields. With photos of damage for a complaint or insurance claim, it describes what it sees and pre-sorts the case. On forms, it also reads handwriting and stamps as long as they are legible. A recording of a meeting turns into minutes and a task list. Before recording, someone has to confirm that everyone agreed and that there is a legal basis.

In every case the model proposes, and the result is checked against existing data: the order number, the supplier master file, the sum of the line items. People only see cases with missing fields or mismatched values.

Chapter 4 of 7

## The errors look different from text errors.

A multimodal model can mix up digits or fill in a blurry amount with something plausible. Anthropic notes that Claude may hallucinate or make mistakes with low-quality, rotated, or very small images under 200 pixels, and counts objects only approximately. A 2023 study describes object [hallucination](https://carl-wuerfel.de/en/glossary/hallucination) in large vision-language models: they name objects missing from the image, especially ones that often appear alongside the objects shown.

The Open Worldwide Application Security Project (OWASP) lists multimodal [prompt injection](https://carl-wuerfel.de/en/glossary/prompt-injection) among its 2025 risks for [LLM](https://carl-wuerfel.de/en/glossary/llm) applications: an attacker hides instructions in an image and gets the model to take unauthorized actions or disclose data. Outside images are untrusted input, and a customer photo alone should never trigger a payment or a credit.

A multimodal model can’t detect forgeries either. Anthropic states that Claude cannot determine whether an image was AI-generated.

Chapter 5 of 7

## Photos and voices bring in GDPR and the AI Act.

Photos, videos, and voice recordings often contain personal data: faces, license plates, voices. The GDPR applies, usually with a data processing agreement when an outside provider runs the model. Stricter rules cover biometric data under Article 4(14) and Article 9 GDPR. Under Recital 51, though, a photo is biometric data only when specific technical means are used to uniquely identify or authenticate a person.

Since February 2, 2025, Article 5(1)(f) of the EU [AI Act](https://carl-wuerfel.de/en/ai-act), Regulation (EU) 2024/1689, has prohibited AI systems that infer the emotions of a person in the workplace or in educational institutions, except for medical or safety reasons. Analyzing employees' mood from phone calls can fall under it. Generated images, audio, and video fall under the labeling rules of Article 50, realistic depictions of real people under the [deepfake](https://carl-wuerfel.de/en/glossary/deepfake) rules.

Chapter 6 of 7

## What good implementation looks like.

It starts with a test set of the company's own documents, including crooked phone photos and handwriting. The correct value for every field is known, and accuracy is measured per field: a wrong amount is an error even if everything else is right.

In production, the system checks whether the line items add up, the order number exists, and the supplier is in the master file. Uncertain cases go to people. Images are resized to what the task needs before sending, because each image is billed in tokens by size.

The company knows which provider receives the images, in which country they are processed, and how long they are stored. When the provider changes the model, the same test set runs again.

Chapter 7 of 7

## Where the term stops working

Multimodal describes which kinds of data a model handles, not how well it handles them. In the narrow sense, a large language model (LLM) works with text only; many current foundation models are multimodal. [Generative AI](https://carl-wuerfel.de/en/glossary/generative-ki) means creating new content and overlaps with multimodal AI. Software with an upload button that runs OCR and then a text model is a chain of single models.

## Frequently asked questions

What is multimodal AI?

Multimodal AI is an AI model that processes several kinds of data, such as text, images, audio, and video, and relates them to each other. It can, for example, answer a question about a photo or summarize a voice recording.

What is the difference between unimodal and multimodal AI?

Unimodal AI handles one kind of data, such as text only, while multimodal AI handles several at once. The benefit is context: a photo shows damage that a written report only hints at.

Which models are multimodal?

Well-known examples are Google's Gemini, introduced in December 2023 as multimodal from the start, and OpenAI's GPT-4o from May 2024. Anthropic also documents image analysis for Claude. Supported inputs and outputs change with each version; the provider's documentation is the reference.

Where do companies use multimodal AI?

Companies use multimodal AI wherever information comes as a scan, photo, or recording: invoices and delivery notes, photos of damage, handwritten forms, meeting minutes. The model makes proposals that are checked against existing data.

What are the risks of multimodal AI?

Typical risks are misread numbers, invented image details, personal data in photos and recordings, and instructions hidden in images. Plausibility and spot checks catch reading errors; a rule that image content alone never triggers an action guards against hidden instructions.

How much does multimodal AI cost?

Model APIs usually bill by [token](https://carl-wuerfel.de/en/glossary/token), and images, audio, and video are converted into tokens. Larger images and longer recordings cost more. Integration, testing with real documents, and reviewing uncertain cases add to that; there is no standard price.

## Further reading

  * [**AI glossary**](https://carl-wuerfel.de/en/glossary)
  * [**AI automation with sign-off**](https://carl-wuerfel.de/en/ai-automation)
  * [**Foundation model**](https://carl-wuerfel.de/en/glossary/foundation-model)
  * [**Prompt injection**](https://carl-wuerfel.de/en/glossary/prompt-injection)
  * [**Deepfake**](https://carl-wuerfel.de/en/glossary/deepfake)
  * [ Term**LLM: Large Language Model**](https://carl-wuerfel.de/en/glossary/llm)
  * [ Term**AI labeling and disclosure**](https://carl-wuerfel.de/en/glossary/ki-kennzeichnung)
  * [ Term**Generative AI**](https://carl-wuerfel.de/en/glossary/generative-ki)

## Sources

  1. [Google: Introducing Gemini, our largest and most capable AI model, December 6, 2023](https://blog.google/technology/ai/google-gemini-ai/)
  2. [OpenAI: Hello GPT-4o, May 13, 2024](https://openai.com/index/hello-gpt-4o/)
  3. [Anthropic: Vision, Claude documentation, as of November 2026](https://platform.claude.com/docs/en/build-with-claude/vision)
  4. [Radford et al.: Learning Transferable Visual Models From Natural Language Supervision (CLIP), arXiv 2103.00020, 2021](https://arxiv.org/abs/2103.00020)
  5. [Li et al.: Evaluating Object Hallucination in Large Vision-Language Models, arXiv 2305.10355, 2023](https://arxiv.org/abs/2305.10355)
  6. [OWASP: LLM01:2025 Prompt Injection, Top 10 for LLM Applications 2025](https://genai.owasp.org/llmrisk/llm01-prompt-injection/)
  7. [European Union: Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 5(1)(f) and Article 50, 2024](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689)
  8. [European Union: Regulation (EU) 2016/679 (General Data Protection Regulation), Article 4(14), Article 9, and Recital 51, 2016](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679)
