Version the entire evaluated system

The mechanism

A useful output depends on more than model weights. Tokenizer, prompt template, retrieved sources, tool schemas, verifier, decoding settings and environment state can change behavior. A result without these identities cannot distinguish model improvement from a changed test harness. Regression testing keeps a stable case set and scoring contract while changing a declared component. Development feedback is used for iteration; a final set exposed during repeated choices becomes part of that feedback process. The lifecycle checkpoint identifies its own narrow model/data/config state, while a deployed system needs a wider record.

1. Freeze the evaluation cases, score rules and required system identities.
2. Change one declared component and retain the old configuration for comparison.
3. Score acquisition and protected failure slices at both identities.
4. Apply predeclared rollback thresholds and record the actual component change responsible for the comparison.

Science and assumptions

Engineering example

An authored baseline passes 18/20 task cases and all 4 source-support cases. A new retriever passes 19/20 task cases but only 3/4 source-support cases. Under the predeclared rule “task accuracy at least 90% and all four source checks pass,” the candidate fails despite higher aggregate task accuracy. The source-support count is a separate gate, not a fifth task silently mixed into a new denominator.

Distinctions and tradeoffs

It is a mistake to conclude: Every improvement in accuracy requires automatic acceptance.

A strict protected slice can block useful aggregate improvements, but silently changing its threshold after seeing results defeats the intended evidence contract.

Measure quality, uncertainty and failure under a stated system/task contract. Attach an evaluation result to the complete system configuration and apply a stated rollback rule.

Specifications and interfaces

Learning mode

worked example

The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.

Worked contract

Attach an evaluation result to the complete system configuration and apply a stated rollback rule.

Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available.

Interfaces

Inputs and state Input

Freeze the evaluation cases, score rules and required system identities.

Result and check Output

Attach an evaluation result to the complete system configuration and apply a stated rollback rule.

Check your understanding

Evidence and scope

Mode: worked example. Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available.

Sources

• Holistic Evaluation of Language Models
• Tülu 3: Pushing Frontiers in Open Language Model Post-Training

Published lesson data · Atlas releases and licensing

Continue exploring

Foundations

• Evaluation & the AGI frontier
• Distributed checkpoint and recovery state
• Tools, memory & agent loop

Related lessons

• Confidence, errors and refusal to guess
• Human and model judgments under a rubric
• From lexical retrieval to model answers with evidence
• Sampling more candidates versus selecting correct ones