{"id":"video-latent-prediction","name":"Predict temporal representations without generating pixels","shortName":"Predict temporal representations without generating pixels","role":"Compare masked feature prediction with pixel generation and test temporal information use.","description":"Video prediction can target learned representations of hidden regions rather than reconstruct every pixel. A target encoder and a predictor have different roles; a stop-gradient or moving-average target prevents treating the target as an unconstrained jointly changing answer. V-JEPA 2 describes masked video representation prediction before adding action-conditioned training. That observational stage does not by itself specify what a robot action will cause. Shuffling time or removing motion cues provides a useful control for whether temporal information matters. The scalar example below illustrates target accounting rather than executing a learned video encoder.","mechanism":["Mark visible context and hidden target positions without leaking target features into the predictor input.","Record how the target encoder is updated and where gradients stop.","Compare predicted and target features under the declared norm and denominator.","Test temporal order and held-out motion, then evaluate a downstream task separately from feature loss."],"learningObjective":"Compare masked feature prediction with pixel generation and test temporal information use.","engineeringExample":"An authored target representation for two masked frames is [2,4], while the predictor emits [1.5,3]. Mean absolute feature error is (0.5+1)/2=0.75. Predicting [2,4] gives zero for these two scalar targets, yet says nothing about unseen texture or whether a control action is correct. A moving-average target coefficient 0.9 combines old parameter 2 and current online parameter 3 into 2.1.","evidenceNotes":"Mode: worked example. Scalar authored feature targets and a declared L1 coordinate mean. Worked EMA arithmetic does not replicate an encoder, dataset or the full V-JEPA 2 training recipe. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Compare masked feature prediction with pixel generation and test temporal information use.","context":"Scalar authored feature targets and a declared L1 coordinate mean. Worked EMA arithmetic does not replicate an encoder, dataset or the full V-JEPA 2 training recipe."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Mark visible context and hidden target positions without leaking target features into the predictor input."},{"name":"Result and check","direction":"out","signal":"Compare masked feature prediction with pixel generation and test temporal information use."}],"tradeoff":"Latent prediction can focus learning on retained features while discarding details. Features useful for one downstream task may omit another task’s needed information.","whyAI":"Teach modality-specific representation, objective and evaluation choices. Compare masked feature prediction with pixel generation and test temporal information use.","misconception":"It is a mistake to conclude: Every future pixel is realistic.","check":{"question":"What does zero error on the two representation targets establish?","options":["Every future pixel is realistic.","The model can control any robot.","Those two target coordinates were reproduced under the stated representation."],"answer":2,"explanation":"Those two target coordinates were reproduced under the stated representation. Scalar authored feature targets and a declared L1 coordinate mean. Worked EMA arithmetic does not replicate an encoder, dataset or the full V-JEPA 2 training recipe."},"prerequisites":["image-text-contrastive-learning","attention"],"related":["image-text-contrastive-learning","attention","visual-language-bridge","speech-audio-sequence-objectives","denoising-generative-objectives"],"sources":[{"title":"V-JEPA 2","url":"https://arxiv.org/html/2506.09985v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-multimodal","subbranch":"Predict temporal representations without generating pixels","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Scalar authored feature targets and a declared L1 coordinate mean. Worked EMA arithmetic does not replicate an encoder, dataset or the full V-JEPA 2 training recipe.","science":[{"title":"Predict temporal representations without generating pixels · worked relation","equation":"L=\\frac1M\\sum_{j\\in mask}|\\hat z_j-\\operatorname{sg}(z_j)|,\\qquad \\bar\\theta^+=\\rho\\bar\\theta+(1-\\rho)\\theta","explanation":"An authored target representation for two masked frames is [2,4], while the predictor emits [1.5,3]. Mean absolute feature error is (0.5+1)/2=0.75. Predicting [2,4] gives zero for these two scalar targets, yet says nothing about unseen texture or whether a control action is correct. A moving-average target coefficient 0.9 combines old parameter 2 and current online parameter 3 into 2.1.","assumptions":"Scalar authored feature targets and a declared L1 coordinate mean. Worked EMA arithmetic does not replicate an encoder, dataset or the full V-JEPA 2 training recipe."}],"history":[]}