{
  "schemaVersion": 1,
  "id": "atlas-lesson-video-latent-prediction",
  "title": "Predict temporal representations without generating pixels",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Compare masked feature prediction with pixel generation and test temporal information use.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/video-latent-prediction/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-multimodal",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Scalar authored feature targets and a declared L1 coordinate mean. Worked EMA arithmetic does not replicate an encoder, dataset or the full V-JEPA 2 training recipe. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/video-latent-prediction-427031a2feb6a9dd.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#video-latent-prediction",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "6ff8f756a3f38df95a66cad1c18a03c99e0647c2eba6e49e479b71a5a174cb04",
  "contentLength": 6464,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-video-latent-prediction.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-video-latent-prediction.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Predict temporal representations without generating pixels",
      "start": 43,
      "end": 471
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 471,
      "end": 1534
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1534,
      "end": 1562
    },
    {
      "id": "section-4",
      "title": "Predict temporal representations without generating pixels · worked relation",
      "start": 1562,
      "end": 2785
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2785,
      "end": 3206
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3206,
      "end": 3635
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3635,
      "end": 4076
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4076,
      "end": 4341
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4341,
      "end": 4370
    },
    {
      "id": "section-10",
      "title": "What does zero error on the two representation targets establish?",
      "start": 4370,
      "end": 4944
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4944,
      "end": 5567
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5567,
      "end": 5834
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5834,
      "end": 5857
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5857,
      "end": 6076
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6076,
      "end": 6464
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#video-latent-prediction"
    },
    {
      "title": "V-JEPA 2",
      "url": "https://arxiv.org/html/2506.09985v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/video-latent-prediction-427031a2feb6a9dd.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Pair images and captions in a shared representation",
      "url": "https://semiaifoundry.com/learn/atlas/image-text-contrastive-learning/"
    },
    {
      "title": "Self-attention",
      "url": "https://semiaifoundry.com/learn/atlas/attention/"
    },
    {
      "title": "Patches, projection and visual instruction tuning",
      "url": "https://semiaifoundry.com/learn/atlas/visual-language-bridge/"
    },
    {
      "title": "Audio representation, alignment and transcript loss",
      "url": "https://semiaifoundry.com/learn/atlas/speech-audio-sequence-objectives/"
    },
    {
      "title": "Train a denoiser and sample iteratively",
      "url": "https://semiaifoundry.com/learn/atlas/denoising-generative-objectives/"
    }
  ],
  "recordHash": "c43ed8adcfc97a28f01e20d601de120fa68a04b846ff52bec7a10207e8eda2b7",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-multimodal · Atlas v1.3.0\n\n# Predict temporal representations without generating pixels\n\nCompare masked feature prediction with pixel generation and test temporal information use.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#video-latent-prediction)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveCompare masked feature prediction with pixel generation and test temporal information use.\n\n## The mechanism\n\nVideo prediction can target learned representations of hidden regions rather than reconstruct every pixel. A target encoder and a predictor have different roles; a stop-gradient or moving-average target prevents treating the target as an unconstrained jointly changing answer. V-JEPA 2 describes masked video representation prediction before adding action-conditioned training. That observational stage does not by itself specify what a robot action will cause. Shuffling time or removing motion cues provides a useful control for whether temporal information matters. The scalar example below illustrates target accounting rather than executing a learned video encoder.\n\n1.  Mark visible context and hidden target positions without leaking target features into the predictor input.\n2.  Record how the target encoder is updated and where gradients stop.\n3.  Compare predicted and target features under the declared norm and denominator.\n4.  Test temporal order and held-out motion, then evaluate a downstream task separately from feature loss.\n\n## Science and assumptions\n\n### Predict temporal representations without generating pixels · worked relation\n\n$$\nL=\\frac1M\\sum_{j\\in mask}|\\hat z_j-\\operatorname{sg}(z_j)|,\\qquad \\bar\\theta^+=\\rho\\bar\\theta+(1-\\rho)\\theta\n$$\n\nCompare masked feature prediction with pixel generation and test temporal information use.\n\nSymbols and units\n\nL\n\nMean absolute masked-feature error\n\nfeature units\n\nM\n\nScored scalar feature coordinates\n\ncoordinates\n\nz\\_j,hat z\\_j\n\nTarget and predicted representation coordinates\n\nfeature units\n\nsg\n\nStop-gradient operation\n\noperator\n\ntheta,bar theta\n\nOnline and target-encoder parameters\n\nparameters\n\nrho\n\nMoving-average coefficient\n\ndimensionless\n\nAn authored target representation for two masked frames is \\[2,4\\], while the predictor emits \\[1.5,3\\]. Mean absolute feature error is (0.5+1)/2=0.75. Predicting \\[2,4\\] gives zero for these two scalar targets, yet says nothing about unseen texture or whether a control action is correct. A moving-average target coefficient 0.9 combines old parameter 2 and current online parameter 3 into 2.1.\n\nAssumptions Scalar authored feature targets and a declared L1 coordinate mean. Worked EMA arithmetic does not replicate an encoder, dataset or the full V-JEPA 2 training recipe.\n\n## Engineering example\n\nAn authored target representation for two masked frames is \\[2,4\\], while the predictor emits \\[1.5,3\\]. Mean absolute feature error is (0.5+1)/2=0.75. Predicting \\[2,4\\] gives zero for these two scalar targets, yet says nothing about unseen texture or whether a control action is correct. A moving-average target coefficient 0.9 combines old parameter 2 and current online parameter 3 into 2.1.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Every future pixel is realistic.\n\nLatent prediction can focus learning on retained features while discarding details. Features useful for one downstream task may omit another task’s needed information.\n\nTeach modality-specific representation, objective and evaluation choices. Compare masked feature prediction with pixel generation and test temporal information use.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nCompare masked feature prediction with pixel generation and test temporal information use.\n\nScalar authored feature targets and a declared L1 coordinate mean. Worked EMA arithmetic does not replicate an encoder, dataset or the full V-JEPA 2 training recipe.\n\n### Interfaces\n\nInputs and state Input\n\nMark visible context and hidden target positions without leaking target features into the predictor input.\n\nResult and check Output\n\nCompare masked feature prediction with pixel generation and test temporal information use.\n\n## Check your understanding\n\n### What does zero error on the two representation targets establish?\n\n1.  Every future pixel is realistic.\n2.  The model can control any robot.\n3.  Those two target coordinates were reproduced under the stated representation.\n\nAnswer and reasoning\n\nThose two target coordinates were reproduced under the stated representation.\n\nThose two target coordinates were reproduced under the stated representation. Scalar authored feature targets and a declared L1 coordinate mean. Worked EMA arithmetic does not replicate an encoder, dataset or the full V-JEPA 2 training recipe.\n\n## Evidence and scope\n\nMode: worked example. Scalar authored feature targets and a declared L1 coordinate mean. Worked EMA arithmetic does not replicate an encoder, dataset or the full V-JEPA 2 training recipe. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Scalar authored feature targets and a declared L1 coordinate mean. Worked EMA arithmetic does not replicate an encoder, dataset or the full V-JEPA 2 training recipe.\n\n## Sources\n\n-   [V-JEPA 2](https://arxiv.org/html/2506.09985v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/video-latent-prediction-427031a2feb6a9dd.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Pair images and captions in a shared representation](https://semiaifoundry.com/learn/atlas/image-text-contrastive-learning/)\n-   [Self-attention](https://semiaifoundry.com/learn/atlas/attention/)\n\n### Related lessons\n\n-   [Patches, projection and visual instruction tuning](https://semiaifoundry.com/learn/atlas/visual-language-bridge/)\n-   [Audio representation, alignment and transcript loss](https://semiaifoundry.com/learn/atlas/speech-audio-sequence-objectives/)\n-   [Train a denoiser and sample iteratively](https://semiaifoundry.com/learn/atlas/denoising-generative-objectives/)"
}