The mechanism
The bounded lifecycle model is a complete causal decoder rather than a collection of supplied-gradient formulas. A nine-symbol lookup feeds one width-six attention head and a width-twelve nonlinear block. Residual paths carry activations around each sublayer; normalization controls their local scale. A separate output head predicts nine next-token probabilities. Every learned coefficient participates in the same reverse-mode graph. Fixed positions, masks and tokenizer are part of the computation but are not trainable parameters. The exact small architecture is a teaching choice, so its behavior must be measured on its own documented symbol task.
- Initialize all learned weights from the saved seed; keep fixed positions and normalization convention separate.
- Trace embeddings through causal Q/K/V mixing, residuals, tanh features and output logits.
- Form target losses, backpropagate through all eleven tensor groups, and apply the selected optimizer.
- Compare reserved copy and swap cases with a train-fitted baseline and preserve the entire experiment state.
Science and assumptions
Assemble an initialized causal language model · worked relation
Join tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.
Symbols and units
- P
- Learned scalar coefficients in this declared decoder
parameters
- V
- Vocabulary size, 9
tokens
- d
- Hidden and single-head width, 6
features
- h
- Feed-forward width, 12
features
The exact parameter ledger is 9×6 embedding + 4×6×6 attention projections + (6×12+12+12×6+6) feed-forward + (6×9+9) output =423. Fixed positions and gain-free RMS normalization add no parameters. At binary64 payload size the weights occupy 423×8=3384 bytes, excluding runtime objects and activations. In the default tested run, 320 actual updates produce 8/8 copy development answers; this finite toy result is not language competence.
Assumptions One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic.
Engineering example
The exact parameter ledger is 9×6 embedding + 4×6×6 attention projections + (6×12+12+12×6+6) feed-forward + (6×9+9) output =423. Fixed positions and gain-free RMS normalization add no parameters. At binary64 payload size the weights occupy 423×8=3384 bytes, excluding runtime objects and activations. In the default tested run, 320 actual updates produce 8/8 copy development answers; this finite toy result is not language competence.
Distinctions and tradeoffs
It is a mistake to conclude: They are ignored by causal attention.
A small model permits complete inspection and exact tests. Its narrow vocabulary and task leave most real language phenomena outside the evidence.
Join existing forward mechanisms into an initialized model with real gradients and measurable predictions. Join tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.
Specifications and interfaces
- Learning mode
- executable core
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
- Worked contract
- Join tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.
One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic.
Interfaces
- Inputs and state Input
- Initialize all learned weights from the saved seed; keep fixed positions and normalization convention separate.
- Result and check Output
- Join tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.
Check your understanding
Why are the fixed positional vectors absent from the 423 count?
- They affect the forward computation but are not updated as learned parameters.
- They are ignored by causal attention.
- All vectors in a model are necessarily trainable.
Answer and reasoning
They affect the forward computation but are not updated as learned parameters.
They affect the forward computation but are not updated as learned parameters. One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic.
Evidence and scope
Mode: executable core. One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.
Model lifecycle studio: executable core. One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic.