{
  "schemaVersion": 1,
  "id": "atlas-lesson-image-text-contrastive-learning",
  "title": "Pair images and captions in a shared representation",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Compute a paired image/text similarity loss and test a shortcut with held-out compositions.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/image-text-contrastive-learning/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-multimodal",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/image-text-contrastive-learning-40b0c75d5fba4ae0.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#image-text-contrastive-learning",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "6d0147d470e5b2945591e22129d461360ef261d9710d928144abbc2fb8220eda",
  "contentLength": 6704,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-image-text-contrastive-learning.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-image-text-contrastive-learning.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Pair images and captions in a shared representation",
      "start": 43,
      "end": 474
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 474,
      "end": 1547
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1547,
      "end": 1575
    },
    {
      "id": "section-4",
      "title": "Pair images and captions in a shared representation · worked relation",
      "start": 1575,
      "end": 2790
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2790,
      "end": 3232
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3232,
      "end": 3639
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3639,
      "end": 4083
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4083,
      "end": 4325
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4325,
      "end": 4354
    },
    {
      "id": "section-10",
      "title": "What does a low contrastive loss directly train?",
      "start": 4354,
      "end": 4935
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4935,
      "end": 5562
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5562,
      "end": 5898
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5898,
      "end": 5921
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5921,
      "end": 6187
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6187,
      "end": 6704
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#image-text-contrastive-learning"
    },
    {
      "title": "Learning Transferable Visual Models From Natural Language Supervision",
      "url": "https://arxiv.org/html/2103.00020v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/image-text-contrastive-learning-40b0c75d5fba4ae0.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Embedding table",
      "url": "https://semiaifoundry.com/learn/atlas/embedding/"
    },
    {
      "title": "Loss, gradients & optimizer",
      "url": "https://semiaifoundry.com/learn/atlas/loss-optimizer/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Patches, projection and visual instruction tuning",
      "url": "https://semiaifoundry.com/learn/atlas/visual-language-bridge/"
    },
    {
      "title": "Audio representation, alignment and transcript loss",
      "url": "https://semiaifoundry.com/learn/atlas/speech-audio-sequence-objectives/"
    },
    {
      "title": "Train a denoiser and sample iteratively",
      "url": "https://semiaifoundry.com/learn/atlas/denoising-generative-objectives/"
    },
    {
      "title": "Predict temporal representations without generating pixels",
      "url": "https://semiaifoundry.com/learn/atlas/video-latent-prediction/"
    }
  ],
  "recordHash": "52f104c3d30eb936bf1982cd8102a2c61807144d6d6167486ae445ea7aee110e",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-multimodal · Atlas v1.3.0\n\n# Pair images and captions in a shared representation\n\nCompute a paired image/text similarity loss and test a shortcut with held-out compositions.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#image-text-contrastive-learning)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveCompute a paired image/text similarity loss and test a shortcut with held-out compositions.\n\n## The mechanism\n\nContrastive image/text learning places both modalities into a comparable representation space and trains matched pairs to score above alternatives in a batch. It supplies a retrieval-like training signal rather than an instruction to generate a caption token by token. Normalization and temperature determine the similarity scale. Negative pairs are constructed from the batch and may include legitimate alternative captions, so label assumptions matter. A representation that recognizes red and square separately still needs evaluation on the previously unseen red-square combination. The lifecycle decoder does not train image encoders; this is a worked multimodal extension.\n\n1.  Encode each image and caption into vectors and state the normalization convention.\n2.  Form the full pairwise similarity matrix and identify the true paired diagonal.\n3.  Compute image-to-text and text-to-image cross-entropies separately, then their stated mean.\n4.  Compare correct retrieval on held-out compositions and controls that remove color or shape information.\n\n## Science and assumptions\n\n### Pair images and captions in a shared representation · worked relation\n\n$$\ns_{ij}=\\frac{u_i^\\mathsf T v_j}{\\tau},\\quad L=-\\frac1{2N}\\sum_i\\left[\\log\\frac{e^{s_{ii}}}{\\sum_j e^{s_{ij}}}+\\log\\frac{e^{s_{ii}}}{\\sum_j e^{s_{ji}}}\\right]\n$$\n\nCompute a paired image/text similarity loss and test a shortcut with held-out compositions.\n\nSymbols and units\n\nu\\_i,v\\_j\n\nUnit-normalized image and text feature vectors\n\ndimensionless\n\ntau\n\nPositive contrastive temperature\n\ndimensionless\n\ns\\_ij\n\nPair similarity logit\n\ndimensionless\n\nN\n\nPaired examples in the batch\n\npairs\n\nL\n\nMean symmetric directional cross-entropy\n\nnats\n\nFor two authored paired examples, let the normalized similarity logits be \\[\\[1,0\\],\\[0,1\\]\\] at temperature 1. In either row or column, the matching probability is e/(e+1)=0.731059. Symmetric mean contrastive loss is −ln0.731059=0.313262 nats per directional example. If both images are encoded identically, no deterministic dot-product comparison can separate their captions without another distinguishing feature.\n\nAssumptions Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic.\n\n## Engineering example\n\nFor two authored paired examples, let the normalized similarity logits be \\[\\[1,0\\],\\[0,1\\]\\] at temperature 1. In either row or column, the matching probability is e/(e+1)=0.731059. Symmetric mean contrastive loss is −ln0.731059=0.313262 nats per directional example. If both images are encoded identically, no deterministic dot-product comparison can separate their captions without another distinguishing feature.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: A complete autoregressive caption decoder.\n\nPaired supervision can use rich natural-language labels while inheriting ambiguity, source bias and shortcuts in the image/text pairs.\n\nTeach modality-specific representation, objective and evaluation choices. Compute a paired image/text similarity loss and test a shortcut with held-out compositions.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nCompute a paired image/text similarity loss and test a shortcut with held-out compositions.\n\nTwo uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic.\n\n### Interfaces\n\nInputs and state Input\n\nEncode each image and caption into vectors and state the normalization convention.\n\nResult and check Output\n\nCompute a paired image/text similarity loss and test a shortcut with held-out compositions.\n\n## Check your understanding\n\n### What does a low contrastive loss directly train?\n\n1.  Guaranteed understanding of every unseen composition.\n2.  The matched pair to receive high probability among the batch alternatives.\n3.  A complete autoregressive caption decoder.\n\nAnswer and reasoning\n\nThe matched pair to receive high probability among the batch alternatives.\n\nThe matched pair to receive high probability among the batch alternatives. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic.\n\n## Evidence and scope\n\nMode: worked example. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic.\n\n## Sources\n\n-   [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/html/2103.00020v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/image-text-contrastive-learning-40b0c75d5fba4ae0.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Embedding table](https://semiaifoundry.com/learn/atlas/embedding/)\n-   [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n\n### Related lessons\n\n-   [Patches, projection and visual instruction tuning](https://semiaifoundry.com/learn/atlas/visual-language-bridge/)\n-   [Audio representation, alignment and transcript loss](https://semiaifoundry.com/learn/atlas/speech-audio-sequence-objectives/)\n-   [Train a denoiser and sample iteratively](https://semiaifoundry.com/learn/atlas/denoising-generative-objectives/)\n-   [Predict temporal representations without generating pixels](https://semiaifoundry.com/learn/atlas/video-latent-prediction/)"
}