Cognitive

Part 4 · Generative and Multimodal Models

Multimodal Models

Multimodal AI connects text, images, audio, and video in one shared understanding.

3 min readBuilds on Convolutional Networks and Vision, The Transformer

Contrastive training pulls matching images and captions together.

Key concepts

  • Encoders convert each type of input into embeddings.
  • CLIP-style training pulls matching images and captions together and pushes mismatches apart.
  • Images are often split into patches, and audio is often turned into a spectrogram.
  • Unified models can reason across inputs, such as a chart and a question.
  • Vision-language models can still invent details that are not in an image.

Contrastive training places matching images and text close together in a shared space.

Figure 21.1. Contrastive training places matching images and text close together in a shared space.
Audio is often converted to a spectrogram before encoding.

Formal view and evidence

  • Established. Contrastive objectives such as InfoNCE maximize agreement for positive pairs relative to negatives. CLIP learns a shared image-text embedding from natural-language supervision.
  • Debated. Whether multimodal training meaningfully addresses the symbol grounding problem, or whether physical interaction is required.
  • Open problem. Long-video understanding, accurate spatial reasoning, and reducing object hallucination.

Common misconception

Common belief

"If AI describes a picture, it sees it as we do."

In practice

It processes learned representations and can confidently describe things that are not there.

Key takeaways

  • Encoders map different kinds of input into comparable representations.
  • Contrastive learning aligns matching pairs of data.
  • Multimodal models can still describe things that are not present.