History

Modern multimodal AI has its roots in decades of research linking multiple data sources. In the early days (1950s–1980s), AI systems were “unimodal,” handling only one type of input at a time (e.g. text or numbers). By the 1990s, separate advances in computer vision and speech recognition gave machines basic abilities to see and hear – for instance, optical character recognition (OCR) let computers read printed text, and early speech engines like Dragon NaturallySpeaking transcribed spoken words. However, these capabilities existed in silos – a program that could read couldn’t listen, and vice versa. Researchers soon began asking: why not combine these senses?

By the early 2000s, the first multimodal systems emerged, albeit in a limited form. Researchers experimented with merging video and audio streams (e.g. pairing video feeds with subtitles, or using lip movements plus sound to recognize speech in noisy settings). These early attempts were fragile and highly specialized, often custom-built for a single task like audiovisual speech recognition. Truly general multimodal AI remained elusive until the deep learning revolution of the 2010s.

In the 2010s, three breakthroughs set the stage for multimodal AI’s takeoff: big data, powerful GPUs, and new architectures. Neural networks became adept at processing images (famously recognizing cats in internet photos) and generating text. A pivotal moment came in 2015 when Google’s Show and Tell system automatically captioned images (“A group of young people playing a game of Frisbee”) – a small but critical step toward vision–language integration. Around the same time, datasets for Visual Question Answering (VQA) and image captioning grew, and researchers developed “two-stream” models with an image encoder on one side and a text encoder on the other, learning joint representations.

The Multimodal Boom (2020s): In recent years, multimodal AI has exploded. Major research labs began releasing large models that bridge modalities. OpenAI’s CLIP (2021) learned a joint image-text embedding space and could match images to natural language descriptions, greatly advancing image understanding. The generative model DALL·E (2021) went the other direction – creating novel images from text prompts – demonstrating creative vision–language synthesis. DeepMind introduced Flamingo (2022), a 80-billion-parameter visual language model that combined pre-trained vision and language transformers; Flamingo achieved state-of-the-art results across 16 vision–language benchmarks (from captioning to VQA) with few-shot learning. Around the same time, DeepMind also unveiled Gato (2022), a single Transformer-based agent that could engage in dialogue, caption images, play Atari games, and even control a robotic arm, all with one model – a milestone in generalist policy learning.

The trend continued with ever more capable systems. GPT-4 (2023) from OpenAI was introduced as a “large-scale, multimodal model” that accepts both text and image inputs. In early demos, GPT-4 showed it could describe images in detail, interpret memes, and solve visual puzzles. For example, given an image of a smartphone with a bulky VGA connector instead of a proper charger, GPT-4 could explain the joke – “the humor comes from the absurdity of plugging a large, outdated VGA connector into a small, modern smartphone port”. This was a startling leap: a chatbot understanding visual humor. By late 2023, OpenAI had also enabled GPT-4 to handle speech (with Whisper for voice input/output), and labs previewed GPT-4V (Vision) and GPT-4o (“Omni”? 2024) capable of processing text, images, and audio (and even video) in one system. Not to be outdone, Google DeepMind announced Gemini in late 2023 – a new family of multimodal models built from the ground up for integrating text, images, audio, code, and more. Google’s CEO Sundar Pichai described Gemini as a next-gen AI natively multimodal, positioned as a direct competitor to GPT-4. In short, after decades of incremental progress, the early 2020s became the inflection point when multimodal AI moved from research labs into real-world products.

Relevance

Why does multimodal AI matter, and how does it relate to human intelligence? The simple answer is: because reality is multimodal. In our daily lives, information floods in through all senses at once – sights, sounds, words, and more. Humans don’t experience the world in isolated channels; our brains constantly integrate multiple inputs to form a coherent understanding. For AI to operate in the real world and interact naturally, it must handle the same richness of input. A system that only reads text or only looks at images is bound to miss context that a person would notice.

Multimodal AI aims to replicate this multi-sensory integration. By processing different data types together, an AI can develop a more human-like understanding of situations. For example, if you’re upset, a person doesn’t just rely on your words – they also hear the strain in your voice and see the expression on your face. Similarly, an advanced AI assistant might analyze not just what a user says in text, but also their tone of voice and facial cues to gauge emotion. This leads to several benefits:

• Better decision-making: With multiple streams of information, AI can make more informed and context-aware decisions. (A self-driving car using cameras and microphones can detect an emergency vehicle by sight and sound, reacting faster and more accurately, for instance.)
• Greater empathy and alignment: Multimodal AI can detect human emotions or intentions more easily by combining cues (tone + facial expression + word choice), potentially enabling more empathetic responses. An AI therapist, for example, could notice a patient’s forced smile on video doesn’t match their anxious words.
• Safer automation: In critical applications like healthcare or transportation, relying on one modality can be dangerous. Multimodal systems (like a medical AI that reads vital signs, listens to patient symptoms, and watches for facial discomfort) provide checks and balances for safety. As one author notes, “AI with multiple senses can see what the eyes alone miss” – catching subtle indicators that a single-modality system might overlook.

In essence, multimodality brings AI a step closer to how humans perceive and learn. We evolved to use all our senses together; an AI that can do the same is not only more powerful but also more relatable. Indeed, neuroscientific research suggests that many concepts in the human brain are grounded in multiple modalities (our concept of “apple” is linked to how it looks, its crunch, its smell). Multimodal AI tries to achieve similar grounding – connecting words, images, sounds to shared meanings. This is crucial for AI that interacts with the real world. As one commentator put it, reality doesn’t come in neat boxes, and “for AI to be genuinely useful… it needs to handle this flood just like humans do.”

Current State of Multimodal AI (2024–2025)

Fast-forward to today, and we have AI models that can seemingly do it all: describe images, carry on conversations, interpret audio, and more. The capabilities of multimodal AI have advanced rapidly, with several prominent models pushing the state of the art:

• OpenAI GPT-4 and Vision Extensions: GPT-4 (released 2023) is a text-based large language model, but notably introduced a vision mode that accepts image inputs along with text. In testing, GPT-4 demonstrated impressive visual understanding – it can caption photographs, solve visual math puzzles, analyze charts, and even explain the joke in a meme. Users have shown GPT-4 images ranging from hand-drawn mockups (which it can turn into functioning website code) to complex infographics. By late 2023, OpenAI extended these capabilities with GPT-4V (vision) and voice integration in ChatGPT. This means you can now show ChatGPT a picture of, say, your broken appliance and ask for repair help, or speak a question aloud and have it respond with voice. Early adopters have found GPT-4’s image analysis surprisingly strong, though not infallible. For example, in medical imaging, GPT-4 Vision can recognize what modality an image is (X-ray, CT, etc.) with near-perfect accuracy and identify anatomical regions ~87% of the time, but it correctly diagnosed pathologies in only ~35% of cases in one radiology study. This underscores that while current models handle general vision-language tasks well, they may still struggle with expert domains or detailed reasoning about images. Nevertheless, GPT-4’s multimodal capabilities are already being applied in real products – from helping blind users “see” (via image-to-text descriptions) to debugging UI designs from screenshots.
• Google DeepMind Gemini: Announced at the end of 2023, Gemini represents the cutting edge of multimodal AI from Google. It’s a suite of models (Ultra, Pro, etc.) designed with multimodality in mind. Early reports show Gemini is extremely powerful: Google claims it’s the biggest leap in generative AI since the introduction of GPT-4. In evaluations, Gemini Ultra achieved state-of-the-art results on 30 out of 32 academic benchmarks used in LLM research – a sweeping improvement across language, reasoning, and multimodal tasks. Notably, Gemini Ultra became the first model to outperform human experts on the tough MMLU exam (a collection of questions from 57 subjects ranging from math to medicine), scoring 90%. On multimodal benchmarks, Gemini also shines: it scored ~59.4% on a new multimodal reasoning test (MMMU) and surpassed previous best models on image-based tasks. For example, on visual question answering and image comprehension tasks, it outperformed OpenAI’s vision model (GPT-4V) without needing any external OCR tool for reading text in images – indicating Gemini has native vision–language integration robust enough to read text from images by itself. Under the hood, Google attributes Gemini’s performance to training it “from the start” on diverse modalities and fine-tuning for multimodal reasoning. In practice, this means Gemini can fluidly combine different inputs – e.g. analyzing an image, some accompanying text, and even audio – within one unified AI. (Google has started deploying Gemini in its products, and it’s powering the latest Bard and Vertex AI services with features like image understanding and tool use.)
• Meta’s Multimodal Efforts: Meta (Facebook) has also been actively researching multimodal AI, albeit with a more open-source approach. In 2023, Meta’s AI lab introduced ImageBind, a unique model that learns a joint embedding for six different modalities – images, text, audio, and even sensor data like depth (3D), thermal (infrared), and IMU (motion) readings. ImageBind is only a research prototype (no direct consumer app yet), but it shows how future AI might bind together visual, auditory, and physical sensory information. Meta open-sourced this model, signaling how future generative AI could create “multisensory” experiences – imagine an AI that could see a video, hear the sound, and even anticipate the physical feel of the scene. On the language model front, Meta’s LLaMA (which is text-only) is evolving into multimodal versions as well. In late 2024 Meta released Llama 3.2, including two multimodal model variants (at 11B and 90B parameters) that can accept images alongside text. These models are available with open weights for researchers, marking an important step in democratizing multimodal AI. While details are technical, one approach used is connecting image tokens into the LLM’s input stream via a cross-modal attention mechanism – essentially giving the language model “eyes.” Early community projects like LLaVA and MiniGPT-4 similarly bolted image encoders onto LLaMA to create open-source vision-chatbots, but Meta’s own multimodal LLaMA suggests official support for images and perhaps audio in future flagship models.
• Others (Audio and Video): Multimodal AI isn’t just about static images and text. There’s active progress in combining text with audio (speech or music) and video (temporal sequences of images). For example, OpenAI’s Whisper and other speech models can transcribe audio to text; when integrated with an LLM, they enable voice conversations with AI. OpenAI’s new GPT-4o (nicknamed Omni) reportedly goes further, handling real-time audio and video inputs in addition to text. Meanwhile, companies are tackling video understanding by extending vision-language models to handle sequences of frames (e.g. asking an AI to watch a video clip and answer questions about it). NVIDIA’s recent research on cross-modal temporal understanding is an example, where an AI answers questions about a video by combining visual and audio context. These are still emerging capabilities – current models may summarize short videos or identify actions (“The person in this video is juggling while a dog barks in the background”), but true deep understanding of long videos is a frontier. Nonetheless, the trend is clear: AI models are steadily incorporating more data types (images, sound, video, even touch sensors) under one roof.

In summary, the state of the art in 2024–2025 is that multimodal AI has arrived in both research and deployed systems. Pioneering models like GPT-4 and Gemini showcase high-level vision-language reasoning, and a host of others (from Meta’s open models to DeepMind’s Flamingo and Gato) demonstrate that integrating modalities often leads to stronger performance. Benchmarks for multimodal tasks – from classic ones like VQAv2 (visual Q&A) to new comprehensive tests – are being dominated by these large models. That said, current models are not without limitations: they can still exhibit errors or biases inherited from training data (see Future Directions), and specialized tasks may require fine-tuning or human expert oversight. But compared to even five years ago, today’s multimodal AI can see, hear, and read in a way that approaches a basic human-like contextual understanding, which is a dramatic transformation in capability.