Pair images and captions in a shared representation

The mechanism

Contrastive image/text learning places both modalities into a comparable representation space and trains matched pairs to score above alternatives in a batch. It supplies a retrieval-like training signal rather than an instruction to generate a caption token by token. Normalization and temperature determine the similarity scale. Negative pairs are constructed from the batch and may include legitimate alternative captions, so label assumptions matter. A representation that recognizes red and square separately still needs evaluation on the previously unseen red-square combination. The lifecycle decoder does not train image encoders; this is a worked multimodal extension.

Encode each image and caption into vectors and state the normalization convention.
Form the full pairwise similarity matrix and identify the true paired diagonal.
Compute image-to-text and text-to-image cross-entropies separately, then their stated mean.
Compare correct retrieval on held-out compositions and controls that remove color or shape information.

Science and assumptions

Pair images and captions in a shared representation · worked relation

s_{ij}=\frac{u_i^\mathsf T v_j}{\tau},\quad L=-\frac1{2N}\sum_i\left[\log\frac{e^{s_{ii}}}{\sum_j e^{s_{ij}}}+\log\frac{e^{s_{ii}}}{\sum_j e^{s_{ji}}}\right]

Compute a paired image/text similarity loss and test a shortcut with held-out compositions.

Symbols and units

u_i,v_j: Unit-normalized image and text feature vectors
dimensionless
tau: Positive contrastive temperature
dimensionless
s_ij: Pair similarity logit
dimensionless
N: Paired examples in the batch
pairs
L: Mean symmetric directional cross-entropy
nats

For two authored paired examples, let the normalized similarity logits be [[1,0],[0,1]] at temperature 1. In either row or column, the matching probability is e/(e+1)=0.731059. Symmetric mean contrastive loss is −ln0.731059=0.313262 nats per directional example. If both images are encoded identically, no deterministic dot-product comparison can separate their captions without another distinguishing feature.

Assumptions Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic.

Engineering example

Distinctions and tradeoffs

It is a mistake to conclude: A complete autoregressive caption decoder.

Paired supervision can use rich natural-language labels while inheriting ambiguity, source bias and shortcuts in the image/text pairs.

Teach modality-specific representation, objective and evaluation choices. Compute a paired image/text similarity loss and test a shortcut with held-out compositions.

Specifications and interfaces

Learning mode: worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
Worked contract: Compute a paired image/text similarity loss and test a shortcut with held-out compositions.
Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic.

Interfaces

Inputs and state Input: Encode each image and caption into vectors and state the normalization convention.
Result and check Output: Compute a paired image/text similarity loss and test a shortcut with held-out compositions.

Check your understanding

What does a low contrastive loss directly train?

Guaranteed understanding of every unseen composition.
The matched pair to receive high probability among the batch alternatives.
A complete autoregressive caption decoder.

Answer and reasoning

The matched pair to receive high probability among the batch alternatives.

The matched pair to receive high probability among the batch alternatives. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic.

Evidence and scope

Mode: worked example. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Two uniquely matched pairs, unit vectors and temperature one in the example. False-negative captions and general image understanding are outside this exact arithmetic.