{"id":"tiny-causal-decoder","name":"Assemble an initialized causal language model","shortName":"Assemble an initialized causal language model","role":"Join tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.","description":"The bounded lifecycle model is a complete causal decoder rather than a collection of supplied-gradient formulas. A nine-symbol lookup feeds one width-six attention head and a width-twelve nonlinear block. Residual paths carry activations around each sublayer; normalization controls their local scale. A separate output head predicts nine next-token probabilities. Every learned coefficient participates in the same reverse-mode graph. Fixed positions, masks and tokenizer are part of the computation but are not trainable parameters. The exact small architecture is a teaching choice, so its behavior must be measured on its own documented symbol task.","mechanism":["Initialize all learned weights from the saved seed; keep fixed positions and normalization convention separate.","Trace embeddings through causal Q/K/V mixing, residuals, tanh features and output logits.","Form target losses, backpropagate through all eleven tensor groups, and apply the selected optimizer.","Compare reserved copy and swap cases with a train-fitted baseline and preserve the entire experiment state."],"learningObjective":"Join tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.","engineeringExample":"The exact parameter ledger is 9×6 embedding + 4×6×6 attention projections + (6×12+12+12×6+6) feed-forward + (6×9+9) output =423. Fixed positions and gain-free RMS normalization add no parameters. At binary64 payload size the weights occupy 423×8=3384 bytes, excluding runtime objects and activations. In the default tested run, 320 actual updates produce 8/8 copy development answers; this finite toy result is not language competence.","evidenceNotes":"Mode: executable core. One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"executable core","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Join tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.","context":"One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Initialize all learned weights from the saved seed; keep fixed positions and normalization convention separate."},{"name":"Result and check","direction":"out","signal":"Join tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model."}],"tradeoff":"A small model permits complete inspection and exact tests. Its narrow vocabulary and task leave most real language phenomena outside the evidence.","whyAI":"Join existing forward mechanisms into an initialized model with real gradients and measurable predictions. Join tensor shapes, causal probabilities, gradients and optimizer state into one reproducible trained model.","misconception":"It is a mistake to conclude: They are ignored by causal attention.","check":{"question":"Why are the fixed positional vectors absent from the 423 count?","options":["They affect the forward computation but are not updated as learned parameters.","They are ignored by causal attention.","All vectors in a model are necessarily trainable."],"answer":0,"explanation":"They affect the forward computation but are not updated as learned parameters. One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic."},"prerequisites":["embedding","attention","feed-forward","residual-normalization","output-head","reverse-mode-autodiff"],"related":["embedding","attention","feed-forward","residual-normalization","output-head","reverse-mode-autodiff"],"sources":[{"title":"Attention Is All You Need","url":"https://arxiv.org/html/1706.03762v7"},{"title":"2 OLMo 2 Furious","url":"https://arxiv.org/html/2501.00656v1"},{"title":"Root Mean Square Layer Normalization","url":"https://arxiv.org/html/1910.07467v1"},{"title":"PyTorch Adam algorithm","url":"https://docs.pytorch.org/docs/2.14/generated/torch.optim.Adam.html"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-base-model","subbranch":"Assemble an initialized causal language model","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: executable core. One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic.","science":[{"title":"Assemble an initialized causal language model · worked relation","equation":"P=Vd+4d^2+(dh+h+hd+d)+(dV+V)=423","explanation":"The exact parameter ledger is 9×6 embedding + 4×6×6 attention projections + (6×12+12+12×6+6) feed-forward + (6×9+9) output =423. Fixed positions and gain-free RMS normalization add no parameters. At binary64 payload size the weights occupy 423×8=3384 bytes, excluding runtime objects and activations. In the default tested run, 320 actual updates produce 8/8 copy development answers; this finite toy result is not language competence.","assumptions":"One head, separate input/output weights, two feed-forward biases, output bias, fixed sinusoidal positions, gain-free RMS normalization, tanh activation and binary64 arithmetic."}],"history":[]}