{
  "schemaVersion": 1,
  "id": "atlas-lesson-visual-language-bridge",
  "title": "Patches, projection and visual instruction tuning",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Trace visual feature dimensions into a language model and distinguish alignment from visual use.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/visual-language-bridge/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-multimodal",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/visual-language-bridge-ceae6b82d67e72bf.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#visual-language-bridge",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "91cc7e6ff8ed38dac54cb125c451db4385d522992fd059d3d76f772198586972",
  "contentLength": 6499,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-visual-language-bridge.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-visual-language-bridge.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Patches, projection and visual instruction tuning",
      "start": 43,
      "end": 473
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 473,
      "end": 1424
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1424,
      "end": 1452
    },
    {
      "id": "section-4",
      "title": "Patches, projection and visual instruction tuning · worked relation",
      "start": 1452,
      "end": 2574
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2574,
      "end": 2997
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2997,
      "end": 3443
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3443,
      "end": 3887
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3887,
      "end": 4119
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4119,
      "end": 4148
    },
    {
      "id": "section-10",
      "title": "Which observation tests visual use beyond mere shape compatibility?",
      "start": 4148,
      "end": 4754
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4754,
      "end": 5371
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5371,
      "end": 5750
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5750,
      "end": 5773
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5773,
      "end": 6101
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6101,
      "end": 6499
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#visual-language-bridge"
    },
    {
      "title": "Visual Instruction Tuning",
      "url": "https://arxiv.org/html/2304.08485v2"
    },
    {
      "title": "OpenVLA: An Open-Source Vision-Language-Action Model",
      "url": "https://arxiv.org/html/2406.09246v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/visual-language-bridge-ceae6b82d67e72bf.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Pair images and captions in a shared representation",
      "url": "https://semiaifoundry.com/learn/atlas/image-text-contrastive-learning/"
    },
    {
      "title": "Self-attention",
      "url": "https://semiaifoundry.com/learn/atlas/attention/"
    },
    {
      "title": "Instruction tuning and assistant-only loss",
      "url": "https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/"
    },
    {
      "title": "Audio representation, alignment and transcript loss",
      "url": "https://semiaifoundry.com/learn/atlas/speech-audio-sequence-objectives/"
    },
    {
      "title": "Train a denoiser and sample iteratively",
      "url": "https://semiaifoundry.com/learn/atlas/denoising-generative-objectives/"
    },
    {
      "title": "Predict temporal representations without generating pixels",
      "url": "https://semiaifoundry.com/learn/atlas/video-latent-prediction/"
    }
  ],
  "recordHash": "2c4b29b8e6565b11983cfc3f3722ce7247419737a02c09bca36bc3f3a3e8e225",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-multimodal · Atlas v1.3.0\n\n# Patches, projection and visual instruction tuning\n\nTrace visual feature dimensions into a language model and distinguish alignment from visual use.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#visual-language-bridge)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveTrace visual feature dimensions into a language model and distinguish alignment from visual use.\n\n## The mechanism\n\nA vision-language bridge converts image features into inputs with the language model’s required width. Shape compatibility allows the computation to run, but does not prove that the generated answer depends on the image. The trainable state can change across stages: one recipe may initially freeze the vision encoder and language model while fitting a projector, then update additional components for instruction following. The original LLaVA paper documents such a staged choice; other architectures differ. The worked matrix here exposes the interface without executing an image model.\n\n1.  Declare patch count, visual-feature width and language input width.\n2.  Multiply by a declared projection and preserve patch ordering.\n3.  Identify frozen and trainable components separately for each training stage.\n4.  Compare image swaps, blank-image inputs and text-only baselines on questions whose answers truly depend on the image.\n\n## Science and assumptions\n\n### Patches, projection and visual instruction tuning · worked relation\n\n$$\nH_v=Z_vW,\\qquad Z_v\\in\\mathbb R^{N\\times d_v},\\quad W\\in\\mathbb R^{d_v\\times d_l}\n$$\n\nTrace visual feature dimensions into a language model and distinguish alignment from visual use.\n\nSymbols and units\n\nH\\_v\n\nProjected visual tokens in the declared row-vector convention\n\ndimensionless\n\nZ\\_v\n\nPatch feature matrix\n\ndimensionless\n\nW\n\nProjection weights with no bias in this example\n\ndimensionless\n\nN\n\nPatch count\n\npatches\n\nd\\_v,d\\_l\n\nVisual and language widths\n\nfeatures\n\nTwo patch rows Z=\\[\\[1,2\\],\\[3,4\\]\\] have visual width 2. With projector W=\\[\\[1,0,1\\],\\[0,1,−1\\]\\] of shape 2×3, visual tokens are ZW=\\[\\[1,2,−1\\],\\[3,4,−1\\]\\], now width 3. The projector has 6 trainable entries if it has no bias. Replacing both patches with zeros makes projected tokens zero, but a language prior could still produce a plausible answer; that would not establish image grounding.\n\nAssumptions Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based.\n\n## Engineering example\n\nTwo patch rows Z=\\[\\[1,2\\],\\[3,4\\]\\] have visual width 2. With projector W=\\[\\[1,0,1\\],\\[0,1,−1\\]\\] of shape 2×3, visual tokens are ZW=\\[\\[1,2,−1\\],\\[3,4,−1\\]\\], now width 3. The projector has 6 trainable entries if it has no bias. Replacing both patches with zeros makes projected tokens zero, but a language prior could still produce a plausible answer; that would not establish image grounding.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: The projected tensor has the expected width.\n\nA lightweight projector limits trainable state but may constrain cross-modal alignment. Larger or jointly trained interfaces require new data and evaluation evidence.\n\nTeach modality-specific representation, objective and evaluation choices. Trace visual feature dimensions into a language model and distinguish alignment from visual use.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nTrace visual feature dimensions into a language model and distinguish alignment from visual use.\n\nAuthored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based.\n\n### Interfaces\n\nInputs and state Input\n\nDeclare patch count, visual-feature width and language input width.\n\nResult and check Output\n\nTrace visual feature dimensions into a language model and distinguish alignment from visual use.\n\n## Check your understanding\n\n### Which observation tests visual use beyond mere shape compatibility?\n\n1.  The projected tensor has the expected width.\n2.  The answer sounds plausible with a blank image.\n3.  Changing the image changes the answer correctly on an image-dependent question.\n\nAnswer and reasoning\n\nChanging the image changes the answer correctly on an image-dependent question.\n\nChanging the image changes the answer correctly on an image-dependent question. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based.\n\n## Evidence and scope\n\nMode: worked example. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Authored fixed features and a linear projector, not an executed vision encoder or replication of an entire VLM. Matrix orientation is explicitly row-vector based.\n\n## Sources\n\n-   [Visual Instruction Tuning](https://arxiv.org/html/2304.08485v2)\n-   [OpenVLA: An Open-Source Vision-Language-Action Model](https://arxiv.org/html/2406.09246v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/visual-language-bridge-ceae6b82d67e72bf.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Pair images and captions in a shared representation](https://semiaifoundry.com/learn/atlas/image-text-contrastive-learning/)\n-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)\n-   [Instruction tuning and assistant-only loss](https://semiaifoundry.com/learn/atlas/sft-chat-loss-masks/)\n\n### Related lessons\n\n-   [Audio representation, alignment and transcript loss](https://semiaifoundry.com/learn/atlas/speech-audio-sequence-objectives/)\n-   [Train a denoiser and sample iteratively](https://semiaifoundry.com/learn/atlas/denoising-generative-objectives/)\n-   [Predict temporal representations without generating pixels](https://semiaifoundry.com/learn/atlas/video-latent-prediction/)"
}