{"id":"visual-language-bridge","name":"Patches, projection and visual instruction tuning","shortName":"Patches, projection and visual instruction tuning","role":"Trace visual feature dimensions into a language model and distinguish alignment from visual use.","description":"A vision-language bridge converts image features into inputs with the language model’s required width. Shape compatibility allows the computation to run, but does not prove that the generated answer depends on the image. The trainable state can change across stages: one recipe may initially freeze the vision encoder and language model while fitting a projector, then update additional components for instruction following. The original LLaVA paper documents such a staged choice; other architectures differ. The worked matrix here exposes the interface without executing an image model.","mechanism":["Declare patch count, visual-feature width and language input width.","Multiply by a declared projection and preserve patch ordering.","Identify frozen and trainable components separately for each training stage.","Compare image swaps, blank-image inputs and text-only baselines on questions whose answers truly depend on the image."],"learningObjective":"Trace visual feature dimensions into a language model and distinguish alignment from visual use.","engineeringExample":"Two patch rows Z=[[1,2],[3,4]] have visual width 2. With projector W=[[1,0,1],[0,1,−1]] of shape 2×3, visual tokens are ZW=[[1,2,−1],[3,4,−1]], now width 3. The projector has 6 trainable entries if it has no bias. Replacing both patches with zeros makes projected tokens zero, but a language prior could still produce a plausible answer; that would not establish image grounding.","evidenceNotes":"Mode: worked example. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Trace visual feature dimensions into a language model and distinguish alignment from visual use.","context":"Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Declare patch count, visual-feature width and language input width."},{"name":"Result and check","direction":"out","signal":"Trace visual feature dimensions into a language model and distinguish alignment from visual use."}],"tradeoff":"A lightweight projector limits trainable state but may constrain cross-modal alignment. Larger or jointly trained interfaces require new data and evaluation evidence.","whyAI":"Teach modality-specific representation, objective and evaluation choices. Trace visual feature dimensions into a language model and distinguish alignment from visual use.","misconception":"It is a mistake to conclude: The projected tensor has the expected width.","check":{"question":"Which observation tests visual use beyond mere shape compatibility?","options":["The projected tensor has the expected width.","The answer sounds plausible with a blank image.","Changing the image changes the answer correctly on an image-dependent question."],"answer":2,"explanation":"Changing the image changes the answer correctly on an image-dependent question. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based."},"prerequisites":["image-text-contrastive-learning","attention","sft-chat-loss-masks"],"related":["image-text-contrastive-learning","attention","sft-chat-loss-masks","speech-audio-sequence-objectives","denoising-generative-objectives","video-latent-prediction"],"sources":[{"title":"Visual Instruction Tuning","url":"https://arxiv.org/html/2304.08485v2"},{"title":"OpenVLA: An Open-Source Vision-Language-Action Model","url":"https://arxiv.org/html/2406.09246v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-multimodal","subbranch":"Patches, projection and visual instruction tuning","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based.","science":[{"title":"Patches, projection and visual instruction tuning · worked relation","equation":"H_v=Z_vW,\\qquad Z_v\\in\\mathbb R^{N\\times d_v},\\quad W\\in\\mathbb R^{d_v\\times d_l}","explanation":"Two patch rows Z=[[1,2],[3,4]] have visual width 2. With projector W=[[1,0,1],[0,1,−1]] of shape 2×3, visual tokens are ZW=[[1,2,−1],[3,4,−1]], now width 3. The projector has 6 trainable entries if it has no bias. Replacing both patches with zeros makes projected tokens zero, but a language prior could still produce a plausible answer; that would not establish image grounding.","assumptions":"Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based."}],"history":[]}