{"id":"image-text-contrastive-learning","name":"Pair images and captions in a shared representation","shortName":"Pair images and captions in a shared representation","role":"Compute a paired image/text similarity loss and test a shortcut with held-out compositions.","description":"Contrastive image/text learning places both modalities into a comparable representation space and trains matched pairs to score above alternatives in a batch. It supplies a retrieval-like training signal rather than an instruction to generate a caption token by token. Normalization and temperature determine the similarity scale. Negative pairs are constructed from the batch and may include legitimate alternative captions, so label assumptions matter. A representation that recognizes red and square separately still needs evaluation on the previously unseen red-square combination. The lifecycle decoder does not train image encoders; this is a worked multimodal extension.","mechanism":["Encode each image and caption into vectors and state the normalization convention.","Form the full pairwise similarity matrix and identify the true paired diagonal.","Compute image-to-text and text-to-image cross-entropies separately, then their stated mean.","Compare correct retrieval on held-out compositions and controls that remove color or shape information."],"learningObjective":"Compute a paired image/text similarity loss and test a shortcut with held-out compositions.","engineeringExample":"For two authored paired examples, let the normalized similarity logits be [[1,0],[0,1]] at temperature 1. In either row or column, the matching probability is e/(e+1)=0.731059. Symmetric mean contrastive loss is −ln0.731059=0.313262 nats per directional example. If both images are encoded identically, no deterministic dot-product comparison can separate their captions without another distinguishing feature.","evidenceNotes":"Mode: worked example. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Compute a paired image/text similarity loss and test a shortcut with held-out compositions.","context":"Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Encode each image and caption into vectors and state the normalization convention."},{"name":"Result and check","direction":"out","signal":"Compute a paired image/text similarity loss and test a shortcut with held-out compositions."}],"tradeoff":"Paired supervision can use rich natural-language labels while inheriting ambiguity, source bias and shortcuts in the image/text pairs.","whyAI":"Teach modality-specific representation, objective and evaluation choices. Compute a paired image/text similarity loss and test a shortcut with held-out compositions.","misconception":"It is a mistake to conclude: A complete autoregressive caption decoder.","check":{"question":"What does a low contrastive loss directly train?","options":["Guaranteed understanding of every unseen composition.","The matched pair to receive high probability among the batch alternatives.","A complete autoregressive caption decoder."],"answer":1,"explanation":"The matched pair to receive high probability among the batch alternatives. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic."},"prerequisites":["embedding","loss-optimizer","evaluation"],"related":["embedding","loss-optimizer","evaluation","visual-language-bridge","speech-audio-sequence-objectives","denoising-generative-objectives","video-latent-prediction"],"sources":[{"title":"Learning Transferable Visual Models From Natural Language Supervision","url":"https://arxiv.org/html/2103.00020v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-multimodal","subbranch":"Pair images and captions in a shared representation","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic.","science":[{"title":"Pair images and captions in a shared representation · worked relation","equation":"s_{ij}=\\frac{u_i^\\mathsf T v_j}{\\tau},\\quad L=-\\frac1{2N}\\sum_i\\left[\\log\\frac{e^{s_{ii}}}{\\sum_j e^{s_{ij}}}+\\log\\frac{e^{s_{ii}}}{\\sum_j e^{s_{ji}}}\\right]","explanation":"For two authored paired examples, let the normalized similarity logits be [[1,0],[0,1]] at temperature 1. In either row or column, the matching probability is e/(e+1)=0.731059. Symmetric mean contrastive loss is −ln0.731059=0.313262 nats per directional example. If both images are encoded identically, no deterministic dot-product comparison can separate their captions without another distinguishing feature.","assumptions":"Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic."}],"history":[]}