Back to glossary

Vision Transformer

An architecture that applies the Transformer model, originally designed for NLP, to image recognition tasks by treating image patches as tokens.

AbbreviationViT
Categoryai

Overview

Vision Transformers (ViTs) represent a shift away from Convolutional Neural Networks (CNNs) for computer vision. They demonstrate that the same self-attention mechanisms that revolutionized language processing can also be highly effective for images.

How it Works

  1. Patch Partitioning: An image is split into fixed-size patches (e.g., 16x16 pixels).
  2. Linear Projection: Each patch is flattened and mapped to an embedding vector.
  3. Positional Encoding: Information about the patch's location in the image is added.
  4. Transformer Encoder: The sequence of patch embeddings is processed by standard Transformer layers.

Advantages

  • Scalability: ViTs often perform better than CNNs when trained on extremely large datasets.
  • Global Context: Self-attention allows the model to relate any two parts of an image regardless of their distance.

Limitations

ViTs typically require more data to reach the same performance as CNNs on smaller datasets because they lack the 'inductive biases' (like translation invariance) inherent in convolutions.