{
  "schemaVersion": 1,
  "id": "atlas-lesson-model-system-regression-contract",
  "title": "Version the entire evaluated system",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Attach an evaluation result to the complete system configuration and apply a stated rollback rule.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/model-system-regression-contract/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-evaluation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/model-system-regression-contract-a5ad234ff0723e81.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#model-system-regression-contract",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "ceb77b59857c4157287fcfe7a0f9eda679a758e4dd5f7ee29de48f7d35d059df",
  "contentLength": 6643,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-model-system-regression-contract.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-model-system-regression-contract.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Version the entire evaluated system",
      "start": 43,
      "end": 473
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 473,
      "end": 1471
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1471,
      "end": 1499
    },
    {
      "id": "section-4",
      "title": "Version the entire evaluated system · worked relation",
      "start": 1499,
      "end": 2574
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2574,
      "end": 3001
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3001,
      "end": 3465
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3465,
      "end": 3920
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3920,
      "end": 4159
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4159,
      "end": 4188
    },
    {
      "id": "section-10",
      "title": "Why does the candidate fail the stated gate?",
      "start": 4188,
      "end": 4773
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4773,
      "end": 5408
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5408,
      "end": 5820
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5820,
      "end": 5843
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5843,
      "end": 6130
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6130,
      "end": 6643
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#model-system-regression-contract"
    },
    {
      "title": "Holistic Evaluation of Language Models",
      "url": "https://arxiv.org/html/2211.09110v1"
    },
    {
      "title": "Tülu 3: Pushing Frontiers in Open Language Model Post-Training",
      "url": "https://arxiv.org/html/2411.15124v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/model-system-regression-contract-a5ad234ff0723e81.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Distributed checkpoint and recovery state",
      "url": "https://semiaifoundry.com/learn/atlas/checkpoint/"
    },
    {
      "title": "Tools, memory & agent loop",
      "url": "https://semiaifoundry.com/learn/atlas/tool-loop/"
    },
    {
      "title": "Confidence, errors and refusal to guess",
      "url": "https://semiaifoundry.com/learn/atlas/probability-calibration-abstention/"
    },
    {
      "title": "Human and model judgments under a rubric",
      "url": "https://semiaifoundry.com/learn/atlas/judge-rubric-agreement/"
    },
    {
      "title": "From lexical retrieval to model answers with evidence",
      "url": "https://semiaifoundry.com/learn/atlas/retrieval-supported-generation/"
    },
    {
      "title": "Sampling more candidates versus selecting correct ones",
      "url": "https://semiaifoundry.com/learn/atlas/test-time-search-verification/"
    }
  ],
  "recordHash": "7c1e6d798c08c9c9b668c34b4ffac9e1063c5873415673274837b65a967a0071",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-evaluation · Atlas v1.3.0\n\n# Version the entire evaluated system\n\nAttach an evaluation result to the complete system configuration and apply a stated rollback rule.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#model-system-regression-contract)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveAttach an evaluation result to the complete system configuration and apply a stated rollback rule.\n\n## The mechanism\n\nA useful output depends on more than model weights. Tokenizer, prompt template, retrieved sources, tool schemas, verifier, decoding settings and environment state can change behavior. A result without these identities cannot distinguish model improvement from a changed test harness. Regression testing keeps a stable case set and scoring contract while changing a declared component. Development feedback is used for iteration; a final set exposed during repeated choices becomes part of that feedback process. The lifecycle checkpoint identifies its own narrow model/data/config state, while a deployed system needs a wider record.\n\n1.  Freeze the evaluation cases, score rules and required system identities.\n2.  Change one declared component and retain the old configuration for comparison.\n3.  Score acquisition and protected failure slices at both identities.\n4.  Apply predeclared rollback thresholds and record the actual component change responsible for the comparison.\n\n## Science and assumptions\n\n### Version the entire evaluated system · worked relation\n\n$$\n\\operatorname{accept}=\\mathbf1[S_{task}\\ge\\tau]\\mathbf1[F_{protected}=0]\n$$\n\nAttach an evaluation result to the complete system configuration and apply a stated rollback rule.\n\nSymbols and units\n\naccept\n\nBoolean acceptance under the declared policy\n\ndimensionless\n\nS\\_task\n\nTask success fraction\n\ndimensionless\n\ntau\n\nMinimum task success threshold\n\ndimensionless\n\nF\\_protected\n\nFailures on explicitly protected cases\n\ncases\n\nAn authored baseline passes 18/20 task cases and all 4 source-support cases. A new retriever passes 19/20 task cases but only 3/4 source-support cases. Under the predeclared rule “task accuracy at least 90% and all four source checks pass,” the candidate fails despite higher aggregate task accuracy. The source-support count is a separate gate, not a fifth task silently mixed into a new denominator.\n\nAssumptions Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available.\n\n## Engineering example\n\nAn authored baseline passes 18/20 task cases and all 4 source-support cases. A new retriever passes 19/20 task cases but only 3/4 source-support cases. Under the predeclared rule “task accuracy at least 90% and all four source checks pass,” the candidate fails despite higher aggregate task accuracy. The source-support count is a separate gate, not a fifth task silently mixed into a new denominator.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Every improvement in accuracy requires automatic acceptance.\n\nA strict protected slice can block useful aggregate improvements, but silently changing its threshold after seeing results defeats the intended evidence contract.\n\nMeasure quality, uncertainty and failure under a stated system/task contract. Attach an evaluation result to the complete system configuration and apply a stated rollback rule.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nAttach an evaluation result to the complete system configuration and apply a stated rollback rule.\n\nAuthored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available.\n\n### Interfaces\n\nInputs and state Input\n\nFreeze the evaluation cases, score rules and required system identities.\n\nResult and check Output\n\nAttach an evaluation result to the complete system configuration and apply a stated rollback rule.\n\n## Check your understanding\n\n### Why does the candidate fail the stated gate?\n\n1.  One protected source-support case regressed despite higher task accuracy.\n2.  Every improvement in accuracy requires automatic acceptance.\n3.  Nineteen successes is lower than eighteen.\n\nAnswer and reasoning\n\nOne protected source-support case regressed despite higher task accuracy.\n\nOne protected source-support case regressed despite higher task accuracy. Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available.\n\n## Evidence and scope\n\nMode: worked example. Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Authored finite regression contract; thresholds are chosen before evaluation and do not certify open-ended model safety. System component identities must remain available.\n\n## Sources\n\n-   [Holistic Evaluation of Language Models](https://arxiv.org/html/2211.09110v1)\n-   [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/model-system-regression-contract-a5ad234ff0723e81.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n-   [Distributed checkpoint and recovery state](https://semiaifoundry.com/learn/atlas/checkpoint/)\n-   [Tools, memory & agent loop](https://semiaifoundry.com/learn/atlas/tool-loop/)\n\n### Related lessons\n\n-   [Confidence, errors and refusal to guess](https://semiaifoundry.com/learn/atlas/probability-calibration-abstention/)\n-   [Human and model judgments under a rubric](https://semiaifoundry.com/learn/atlas/judge-rubric-agreement/)\n-   [From lexical retrieval to model answers with evidence](https://semiaifoundry.com/learn/atlas/retrieval-supported-generation/)\n-   [Sampling more candidates versus selecting correct ones](https://semiaifoundry.com/learn/atlas/test-time-search-verification/)"
}