# Patches, projection and visual instruction tuning

Canonical: https://semiaifoundry.com/learn/atlas/visual-language-bridge/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

model-lifecycle-multimodal · Atlas v1.3.0

# Patches, projection and visual instruction tuning

Trace visual feature dimensions into a language model and distinguish alignment from visual use.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#visual-language-bridge)Mathematical, data and behavior space; logical dimensions are not physical distance

Learning objectiveTrace visual feature dimensions into a language model and distinguish alignment from visual use.

## The mechanism

A vision-language bridge converts image features into inputs with the language model’s required width. Shape compatibility allows the computation to run, but does not prove that the generated answer depends on the image. The trainable state can change across stages: one recipe may initially freeze the vision encoder and language model while fitting a projector, then update additional components for instruction following. The original LLaVA paper documents such a staged choice; other architectures differ. The worked matrix here exposes the interface without executing an image model.

1.  Declare patch count, visual-feature width and language input width.
2.  Multiply by a declared projection and preserve patch ordering.
3.  Identify frozen and trainable components separately for each training stage.
4.  Compare image swaps, blank-image inputs and text-only baselines on questions whose answers truly depend on the image.

## Science and assumptions

### Patches, projection and visual instruction tuning · worked relation

$$
H_v=Z_vW,\qquad Z_v\in\mathbb R^{N\times d_v},\quad W\in\mathbb R^{d_v\times d_l}
$$

Trace visual feature dimensions into a language model and distinguish alignment from visual use.

Symbols and units

H\_v

Projected visual tokens in the declared row-vector convention

dimensionless

Z\_v

Patch feature matrix

dimensionless

W

Projection weights with no bias in this example

dimensionless

N

Patch count

patches

d\_v,d\_l

Visual and language widths

features

Two patch rows Z=\[\[1,2\],\[3,4\]\] have visual width 2. With projector W=\[\[1,0,1\],\[0,1,−1\]\] of shape 2×3, visual tokens are ZW=\[\[1,2,−1\],\[3,4,−1\]\], now width 3. The projector has 6 trainable entries if it has no bias. Replacing both patches with zeros makes projected tokens zero, but a language prior could still produce a plausible answer; that would not establish image grounding.

Assumptions Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based.

## Engineering example

Two patch rows Z=\[\[1,2\],\[3,4\]\] have visual width 2. With projector W=\[\[1,0,1\],\[0,1,−1\]\] of shape 2×3, visual tokens are ZW=\[\[1,2,−1\],\[3,4,−1\]\], now width 3. The projector has 6 trainable entries if it has no bias. Replacing both patches with zeros makes projected tokens zero, but a language prior could still produce a plausible answer; that would not establish image grounding.

## Distinctions and tradeoffs

It is a mistake to conclude: The projected tensor has the expected width.

A lightweight projector limits trainable state but may constrain cross-modal alignment. Larger or jointly trained interfaces require new data and evaluation evidence.

Teach modality-specific representation, objective and evaluation choices. Trace visual feature dimensions into a language model and distinguish alignment from visual use.

## Specifications and interfaces

Learning mode

worked example

The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.

Worked contract

Trace visual feature dimensions into a language model and distinguish alignment from visual use.

Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based.

### Interfaces

Inputs and state Input

Declare patch count, visual-feature width and language input width.

Result and check Output

Trace visual feature dimensions into a language model and distinguish alignment from visual use.

## Check your understanding

### Which observation tests visual use beyond mere shape compatibility?

1.  The projected tensor has the expected width.
2.  The answer sounds plausible with a blank image.
3.  Changing the image changes the answer correctly on an image-dependent question.

Answer and reasoning

Changing the image changes the answer correctly on an image-dependent question.

Changing the image changes the answer correctly on an image-dependent question. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based.

## Evidence and scope

Mode: worked example. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based.

## Sources

-   [Visual Instruction Tuning](https://arxiv.org/html/2304.08485v2)
-   [OpenVLA: An Open-Source Vision-Language-Action Model](https://arxiv.org/html/2406.09246v1)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/visual-language-bridge-ceae6b82d67e72bf.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Pair images and captions in a shared representation](https://semiaifoundry.com/learn/atlas/image-text-contrastive-learning/)
-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)
-   [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/)

### Related lessons

-   [Audio representation, alignment and transcript loss](https://semiaifoundry.com/learn/atlas/speech-audio-sequence-objectives/)
-   [Train a denoiser and sample iteratively](https://semiaifoundry.com/learn/atlas/denoising-generative-objectives/)
-   [Predict temporal representations without generating pixels](https://semiaifoundry.com/learn/atlas/video-latent-prediction/)