Patches, projection and visual instruction tuning

The mechanism

A vision-language bridge converts image features into inputs with the language model’s required width. Shape compatibility allows the computation to run, but does not prove that the generated answer depends on the image. The trainable state can change across stages: one recipe may initially freeze the vision encoder and language model while fitting a projector, then update additional components for instruction following. The original LLaVA paper documents such a staged choice; other architectures differ. The worked matrix here exposes the interface without executing an image model.

1. Declare patch count, visual-feature width and language input width.
2. Multiply by a declared projection and preserve patch ordering.
3. Identify frozen and trainable components separately for each training stage.
4. Compare image swaps, blank-image inputs and text-only baselines on questions whose answers truly depend on the image.

Science and assumptions

Engineering example

Two patch rows Z=[[1,2],[3,4]] have visual width 2. With projector W=[[1,0,1],[0,1,−1]] of shape 2×3, visual tokens are ZW=[[1,2,−1],[3,4,−1]], now width 3. The projector has 6 trainable entries if it has no bias. Replacing both patches with zeros makes projected tokens zero, but a language prior could still produce a plausible answer; that would not establish image grounding.

Distinctions and tradeoffs

It is a mistake to conclude: The projected tensor has the expected width.

A lightweight projector limits trainable state but may constrain cross-modal alignment. Larger or jointly trained interfaces require new data and evaluation evidence.

Teach modality-specific representation, objective and evaluation choices. Trace visual feature dimensions into a language model and distinguish alignment from visual use.

Specifications and interfaces

Learning mode

worked example

The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.

Worked contract

Trace visual feature dimensions into a language model and distinguish alignment from visual use.

Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based.

Interfaces

Inputs and state Input

Declare patch count, visual-feature width and language input width.

Result and check Output

Trace visual feature dimensions into a language model and distinguish alignment from visual use.

Check your understanding

Evidence and scope

Mode: worked example. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based.

Sources

• Visual Instruction Tuning
• OpenVLA: An Open-Source Vision-Language-Action Model

Published lesson data · Atlas releases and licensing

Continue exploring

Foundations

• Pair images and captions in a shared representation
• Self-attention
• Instruction tuning and assistant-only loss

Related lessons

• Audio representation, alignment and transcript loss
• Train a denoiser and sample iteratively
• Predict temporal representations without generating pixels