{"id":"training-tier-placement","level":"software","name":"Training state across HBM, host memory and storage","shortName":"Training state across HBM, host memory and storage","role":"Distinguish mutable training state, recomputation, offload and durable checkpoints.","description":"Training retains values for the forward pass, computes gradients and updates parameters and optimizer state. The fastest tier must therefore accommodate more than inference weights. Activation checkpointing stores selected intermediates and recomputes others; persistent checkpoints instead save restartable state outside the live execution path. State sharding changes ownership across devices, while CPU/NVMe offload moves both capacity and work across interfaces. A read-oriented flash tier cannot simply inherit the role of frequently updated optimizer or gradient storage: its write service, software support and movement schedule must be demonstrated.","mechanism":["Separate parameters, gradients, master weights, optimizer moments, live activations and temporary buffers.","Set precision and sharding rules independently for each class of state.","Choose where optimizer computation occurs, keeping its mutable state and transfer requirements consistent.","Model recomputation and offload as extra work or movement rather than free capacity.","Save version-consistent durable checkpoints and measure recovery alongside steady step time."],"science":[{"title":"State inventory before sharding","equation":"Mstate = P (bw + bg + bm + bo) + A + O","explanation":"Per-parameter byte terms represent the explicitly chosen training representation. bo includes all optimizer-state values per parameter.","assumptions":"Dense unsharded inventory; omit absent state such as a separate master copy, then apply actual partition/replication factors by tensor class."}],"specs":[{"label":"Two checkpoint meanings","value":"Activation recomputation versus durable restart","context":"Saving an activation is not saving a recoverable training run."},{"label":"Offload boundary","value":"Data and computation placement","context":"A CPU optimizer also consumes CPU compute and memory bandwidth."}],"interfaces":[{"name":"Forward/backward tensors","direction":"in","signal":"Activations, gradients and parameter version"},{"name":"Update and durable state","direction":"out","signal":"New parameters, optimizer state and recoverable checkpoint"}],"tradeoff":"Offload and recomputation enable larger models at the cost of movement or extra arithmetic; the best choice depends on overlap and recovery requirements.","whyAI":"Useful training progress depends on completed and recoverable updates, not the largest parameter count that can be allocated.","prerequisites":["training-memory-state","memory-working-set-placement"],"related":["checkpoint-retained-work","parallelism-communication-budget","hbf-read-tier-contract"],"misconception":"Training repeatedly updates gradient/optimizer/parameter state, so read bandwidth alone cannot characterize an appropriate tier.","check":{"question":"Which state is inherently write-active during ordinary optimizer updates?","options":["Gradients and optimizer state","Only the immutable tokenizer vocabulary","Only a finalized archival checkpoint"],"answer":0,"explanation":"Training repeatedly updates gradient/optimizer/parameter state, so read bandwidth alone cannot characterize an appropriate tier."},"engineeringExample":"Synthetic mixed-precision Adam inventory: 1 billion parameters × (2 weight + 2 gradient + 4 master + 8 moment bytes) = 16 GB before activations, workspace, sharding or replication. It is one representation, not a rule for every optimizer.","evidenceNotes":"DeepSpeed documents optimizer offload to the host CPU; ZeRO-Infinity studies GPU/CPU/NVMe state management. Neither source establishes an HBF training implementation or its write endurance.","sources":[{"title":"DeepSpeed — ZeRO-Offload tutorial","url":"https://www.deepspeed.ai/tutorials/zero-offload/"},{"title":"ZeRO-Infinity: Breaking the GPU Memory Wall","url":"https://arxiv.org/abs/2104.07857"}],"learningObjective":"Distinguish mutable training state, recomputation, offload and durable checkpoints.","workstream":"W11","visualFamily":"model-workload","scale":"Tensor state over a training step and across memory tiers; abstraction","kind":"system","labId":"engineering","branch":"Software","subbranch":"Working-set placement across memory tiers","scope":"Enable the optimizer-state option to budget weights and master/moment state; gradients, activations, workspaces and actual tier transfers are omitted.","deepLabId":"footprint","investigationId":"execution","labScope":"Enable the optimizer-state option to budget weights and master/moment state; gradients, activations, workspaces and actual tier transfers are omitted."}