# Pair images and captions in a shared representation Canonical: https://semiaifoundry.com/learn/atlas/image-text-contrastive-learning/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-multimodal · Atlas v1.3.0 # Pair images and captions in a shared representation Compute a paired image/text similarity loss and test a shortcut with held-out compositions. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#image-text-contrastive-learning)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveCompute a paired image/text similarity loss and test a shortcut with held-out compositions. ## The mechanism Contrastive image/text learning places both modalities into a comparable representation space and trains matched pairs to score above alternatives in a batch. It supplies a retrieval-like training signal rather than an instruction to generate a caption token by token. Normalization and temperature determine the similarity scale. Negative pairs are constructed from the batch and may include legitimate alternative captions, so label assumptions matter. A representation that recognizes red and square separately still needs evaluation on the previously unseen red-square combination. The lifecycle decoder does not train image encoders; this is a worked multimodal extension. 1. Encode each image and caption into vectors and state the normalization convention. 2. Form the full pairwise similarity matrix and identify the true paired diagonal. 3. Compute image-to-text and text-to-image cross-entropies separately, then their stated mean. 4. Compare correct retrieval on held-out compositions and controls that remove color or shape information. ## Science and assumptions ### Pair images and captions in a shared representation · worked relation $$ s_{ij}=\frac{u_i^\mathsf T v_j}{\tau},\quad L=-\frac1{2N}\sum_i\left[\log\frac{e^{s_{ii}}}{\sum_j e^{s_{ij}}}+\log\frac{e^{s_{ii}}}{\sum_j e^{s_{ji}}}\right] $$ Compute a paired image/text similarity loss and test a shortcut with held-out compositions. Symbols and units u\_i,v\_j Unit-normalized image and text feature vectors dimensionless tau Positive contrastive temperature dimensionless s\_ij Pair similarity logit dimensionless N Paired examples in the batch pairs L Mean symmetric directional cross-entropy nats For two authored paired examples, let the normalized similarity logits be \[\[1,0\],\[0,1\]\] at temperature 1. In either row or column, the matching probability is e/(e+1)=0.731059. Symmetric mean contrastive loss is −ln0.731059=0.313262 nats per directional example. If both images are encoded identically, no deterministic dot-product comparison can separate their captions without another distinguishing feature. Assumptions Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic. ## Engineering example For two authored paired examples, let the normalized similarity logits be \[\[1,0\],\[0,1\]\] at temperature 1. In either row or column, the matching probability is e/(e+1)=0.731059. Symmetric mean contrastive loss is −ln0.731059=0.313262 nats per directional example. If both images are encoded identically, no deterministic dot-product comparison can separate their captions without another distinguishing feature. ## Distinctions and tradeoffs It is a mistake to conclude: A complete autoregressive caption decoder. Paired supervision can use rich natural-language labels while inheriting ambiguity, source bias and shortcuts in the image/text pairs. Teach modality-specific representation, objective and evaluation choices. Compute a paired image/text similarity loss and test a shortcut with held-out compositions. ## Specifications and interfaces Learning mode worked example The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Compute a paired image/text similarity loss and test a shortcut with held-out compositions. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic. ### Interfaces Inputs and state Input Encode each image and caption into vectors and state the normalization convention. Result and check Output Compute a paired image/text similarity loss and test a shortcut with held-out compositions. ## Check your understanding ### What does a low contrastive loss directly train? 1. Guaranteed understanding of every unseen composition. 2. The matched pair to receive high probability among the batch alternatives. 3. A complete autoregressive caption decoder. Answer and reasoning The matched pair to receive high probability among the batch alternatives. The matched pair to receive high probability among the batch alternatives. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic. ## Evidence and scope Mode: worked example. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: worked example. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic. ## Sources - [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/html/2103.00020v1) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/image-text-contrastive-learning-40b0c75d5fba4ae0.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Embedding table](https://semiaifoundry.com/learn/atlas/embedding/) - [Loss, gradients & optimizer](https://semiaifoundry.com/learn/atlas/loss-optimizer/) - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/) ### Related lessons - [Patches, projection and visual instruction tuning](https://semiaifoundry.com/learn/atlas/visual-language-bridge/) - [Audio representation, alignment and transcript loss](https://semiaifoundry.com/learn/atlas/speech-audio-sequence-objectives/) - [Train a denoiser and sample iteratively](https://semiaifoundry.com/learn/atlas/denoising-generative-objectives/) - [Predict temporal representations without generating pixels](https://semiaifoundry.com/learn/atlas/video-latent-prediction/)