{"id":"model-system-regression-contract","name":"Version the entire evaluated system","shortName":"Version the entire evaluated system","role":"Attach an evaluation result to the complete system configuration and apply a stated rollback rule.","description":"A useful output depends on more than model weights. Tokenizer, prompt template, retrieved sources, tool schemas, verifier, decoding settings and environment state can change behavior. A result without these identities cannot distinguish model improvement from a changed test harness. Regression testing keeps a stable case set and scoring contract while changing a declared component. Development feedback is used for iteration; a final set exposed during repeated choices becomes part of that feedback process. The lifecycle checkpoint identifies its own narrow model/data/config state, while a deployed system needs a wider record.","mechanism":["Freeze the evaluation cases, score rules and required system identities.","Change one declared component and retain the old configuration for comparison.","Score acquisition and protected failure slices at both identities.","Apply predeclared rollback thresholds and record the actual component change responsible for the comparison."],"learningObjective":"Attach an evaluation result to the complete system configuration and apply a stated rollback rule.","engineeringExample":"An authored baseline passes 18/20 task cases and all 4 source-support cases. A new retriever passes 19/20 task cases but only 3/4 source-support cases. Under the predeclared rule “task accuracy at least 90% and all four source checks pass,” the candidate fails despite higher aggregate task accuracy. The source-support count is a separate gate, not a fifth task silently mixed into a new denominator.","evidenceNotes":"Mode: worked example. Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Attach an evaluation result to the complete system configuration and apply a stated rollback rule.","context":"Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Freeze the evaluation cases, score rules and required system identities."},{"name":"Result and check","direction":"out","signal":"Attach an evaluation result to the complete system configuration and apply a stated rollback rule."}],"tradeoff":"A strict protected slice can block useful aggregate improvements, but silently changing its threshold after seeing results defeats the intended evidence contract.","whyAI":"Measure quality, uncertainty and failure under a stated system/task contract. Attach an evaluation result to the complete system configuration and apply a stated rollback rule.","misconception":"It is a mistake to conclude: Every improvement in accuracy requires automatic acceptance.","check":{"question":"Why does the candidate fail the stated gate?","options":["One protected source-support case regressed despite higher task accuracy.","Every improvement in accuracy requires automatic acceptance.","Nineteen successes is lower than eighteen."],"answer":0,"explanation":"One protected source-support case regressed despite higher task accuracy. Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available."},"prerequisites":["evaluation","checkpoint","tool-loop"],"related":["evaluation","checkpoint","tool-loop","probability-calibration-abstention","judge-rubric-agreement","retrieval-supported-generation","test-time-search-verification"],"sources":[{"title":"Holistic Evaluation of Language Models","url":"https://arxiv.org/html/2211.09110v1"},{"title":"Tülu 3: Pushing Frontiers in Open Language Model Post-Training","url":"https://arxiv.org/html/2411.15124v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-evaluation","subbranch":"Version the entire evaluated system","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available.","science":[{"title":"Version the entire evaluated system · worked relation","equation":"\\operatorname{accept}=\\mathbf1[S_{task}\\ge\\tau]\\mathbf1[F_{protected}=0]","explanation":"An authored baseline passes 18/20 task cases and all 4 source-support cases. A new retriever passes 19/20 task cases but only 3/4 source-support cases. Under the predeclared rule “task accuracy at least 90% and all four source checks pass,” the candidate fails despite higher aggregate task accuracy. The source-support count is a separate gate, not a fifth task silently mixed into a new denominator.","assumptions":"Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available."}],"history":[]}