{
  "schemaVersion": 1,
  "id": "atlas-lesson-probability-calibration-abstention",
  "title": "Confidence, errors and refusal to guess",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Evaluate probability quality and selective error using distinct calibration and final-test data.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/probability-calibration-abstention/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-evaluation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Binary Brier convention shown, not sum over both class coordinates. Authored predictions; no calibration fitting feature is implemented in the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/probability-calibration-abstention-ffb9054290f936c8.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#probability-calibration-abstention",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "1e95b7b22a61bf242816eb2289bba2ca24dd373d748c2e54d4e28f00b7a628b2",
  "contentLength": 7470,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-probability-calibration-abstention.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-probability-calibration-abstention.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Confidence, errors and refusal to guess",
      "start": 43,
      "end": 475
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 475,
      "end": 1894
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1894,
      "end": 1922
    },
    {
      "id": "section-4",
      "title": "Confidence, errors and refusal to guess · worked relation",
      "start": 1922,
      "end": 3060
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 3060,
      "end": 3826
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3826,
      "end": 4268
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 4268,
      "end": 4708
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4708,
      "end": 4928
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4928,
      "end": 4957
    },
    {
      "id": "section-10",
      "title": "How should a threshold answering zero cases be reported?",
      "start": 4957,
      "end": 5407
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 5407,
      "end": 6016
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 6016,
      "end": 6408
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 6408,
      "end": 6431
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 6431,
      "end": 6741
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6741,
      "end": 7470
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#probability-calibration-abstention"
    },
    {
      "title": "On Calibration of Modern Neural Networks",
      "url": "https://arxiv.org/html/1706.04599v2"
    },
    {
      "title": "Holistic Evaluation of Language Models",
      "url": "https://arxiv.org/html/2211.09110v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/probability-calibration-abstention-ffb9054290f936c8.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Output head & decoding",
      "url": "https://semiaifoundry.com/learn/atlas/output-head/"
    },
    {
      "title": "Sampling variability and independent evidence",
      "url": "https://semiaifoundry.com/learn/atlas/sampling-estimation-uncertainty/"
    },
    {
      "title": "Human and model judgments under a rubric",
      "url": "https://semiaifoundry.com/learn/atlas/judge-rubric-agreement/"
    },
    {
      "title": "From lexical retrieval to model answers with evidence",
      "url": "https://semiaifoundry.com/learn/atlas/retrieval-supported-generation/"
    },
    {
      "title": "Sampling more candidates versus selecting correct ones",
      "url": "https://semiaifoundry.com/learn/atlas/test-time-search-verification/"
    },
    {
      "title": "Version the entire evaluated system",
      "url": "https://semiaifoundry.com/learn/atlas/model-system-regression-contract/"
    },
    {
      "title": "Model artifacts, rollout and drift",
      "url": "https://semiaifoundry.com/learn/atlas/model-artifact-rollout-drift/"
    },
    {
      "title": "Filtering, duplication and contamination",
      "url": "https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/"
    }
  ],
  "recordHash": "93ae814baa9a9230ec22c83b1dda3973f73d2b6972e3b1185b3eec67f0906295",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-evaluation · Atlas v1.3.0\n\n# Confidence, errors and refusal to guess\n\nEvaluate probability quality and selective error using distinct calibration and final-test data.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#probability-calibration-abstention)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveEvaluate probability quality and selective error using distinct calibration and final-test data.\n\n## The mechanism\n\nConfidence is a numerical prediction about outcomes, not a guarantee attached to a fluent answer. Calibration asks whether cases assigned similar probabilities succeed at corresponding frequencies under a stated task and distribution. Temperature scaling fits a scalar on development logits; it usually preserves the class ordering while changing probabilities. Abstention adds a decision policy: only answer when confidence passes a threshold. Lower error among answered cases can be purchased by answering fewer cases, so coverage belongs beside selective error. If no cases are answered, selective error is undefined rather than zero.\n\nThe identified-evaluation experience fits the sensor model on eight training rows, selects a scalar temperature using four separate calibration rows, and audits six diagnostic rows. Threshold and acceptance controls do not refit the weights. Empty probability bins and an empty accepted set retain unavailable metrics rather than becoming perfect scores. The small audit is exploratory, and repeated recipe selection against it is development feedback.\n\n1.  Fit any calibration parameter on development data only.\n2.  Score probability quality with an explicit Brier or log-loss convention.\n3.  Choose an abstention threshold before opening final cases.\n4.  Report answered count, error among answered cases and failures by slice, including empty denominators.\n\n## Science and assumptions\n\n### Confidence, errors and refusal to guess · worked relation\n\n$$\nB=\\frac1N\\sum_i(p_i-y_i)^2,\\quad C=\\frac{|A|}{N},\\quad R=\\frac{\\sum_{i\\in A}\\mathbf1[\\hat y_i\\ne y_i]}{|A|}\\ (|A|>0)\n$$\n\nEvaluate probability quality and selective error using distinct calibration and final-test data.\n\nSymbols and units\n\nB\n\nBinary one-probability Brier score\n\ndimensionless\n\np\\_i\n\nProbability of class one\n\ndimensionless\n\ny\\_i\n\nBinary class-one label\n\ndimensionless\n\nA\n\nCases passing the confidence threshold\n\nset\n\nN\n\nAll evaluated cases\n\ncases\n\nC\n\nAnswer coverage\n\ndimensionless\n\nR\n\nError conditional on answering\n\ndimensionless\n\nBinary probabilities \\[0.8,0.6,0.4,0.2\\] for class one with targets \\[1,0,1,0\\] have Brier score (0.04+0.36+0.36+0.04)/4=0.2. Predicted-class confidence is \\[0.8,0.6,0.6,0.8\\]. Threshold 0.75 answers two cases, both correct: coverage 1/2 and selective error 0/2. Threshold 0.9 answers none, so coverage is zero and selective error has no denominator.\n\nAssumptions Binary Brier convention shown, not sum over both class coordinates. Authored predictions; no calibration fitting feature is implemented in the lifecycle core.\n\n## Engineering example\n\nBinary probabilities \\[0.8,0.6,0.4,0.2\\] for class one with targets \\[1,0,1,0\\] have Brier score (0.04+0.36+0.36+0.04)/4=0.2. Predicted-class confidence is \\[0.8,0.6,0.6,0.8\\]. Threshold 0.75 answers two cases, both correct: coverage 1/2 and selective error 0/2. Threshold 0.9 answers none, so coverage is zero and selective error has no denominator.\n\nThe default executable audit uses temperature 0.5 selected on calibration rows. Its six diagnostic decisions are correct, mean binary log loss is approximately 0.13357 and Brier score is 0.02864. Requiring chosen-class probability exactly 1 accepts zero cases, so conditional error is unavailable. These are computed synthetic fixtures, not calibration evidence for a deployed classifier.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: Zero coverage and proven zero risk.\n\nAbstention can reduce errors among answered cases while excluding difficult or underrepresented slices. Calibration learned on one distribution can fail after a shift.\n\nMeasure quality, uncertainty and failure under a stated system/task contract. Evaluate probability quality and selective error using distinct calibration and final-test data.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nEvaluate probability quality and selective error using distinct calibration and final-test data.\n\nBinary Brier convention shown, not sum over both class coordinates. Authored predictions; no calibration fitting feature is implemented in the lifecycle core.\n\n### Interfaces\n\nInputs and state Input\n\nFit any calibration parameter on development data only.\n\nResult and check Output\n\nEvaluate probability quality and selective error using distinct calibration and final-test data.\n\n## Check your understanding\n\n### How should a threshold answering zero cases be reported?\n\n1.  Zero coverage and proven zero risk.\n2.  Perfect calibration.\n3.  Zero coverage and undefined selective error.\n\nAnswer and reasoning\n\nZero coverage and undefined selective error.\n\nZero coverage and undefined selective error. Binary Brier convention shown, not sum over both class coordinates. Authored predictions; no calibration fitting feature is implemented in the lifecycle core.\n\n## Evidence and scope\n\nMode: worked example. Binary Brier convention shown, not sum over both class coordinates. Authored predictions; no calibration fitting feature is implemented in the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Binary Brier convention shown, not sum over both class coordinates. Authored predictions; no calibration fitting feature is implemented in the lifecycle core.\n\n## Sources\n\n-   [On Calibration of Modern Neural Networks](https://arxiv.org/html/1706.04599v2)\n-   [Holistic Evaluation of Language Models](https://arxiv.org/html/2211.09110v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/probability-calibration-abstention-ffb9054290f936c8.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n-   [Output head & decoding](https://semiaifoundry.com/learn/atlas/output-head/)\n-   [Sampling variability and independent evidence](https://semiaifoundry.com/learn/atlas/sampling-estimation-uncertainty/)\n\n### Related lessons\n\n-   [Human and model judgments under a rubric](https://semiaifoundry.com/learn/atlas/judge-rubric-agreement/)\n-   [From lexical retrieval to model answers with evidence](https://semiaifoundry.com/learn/atlas/retrieval-supported-generation/)\n-   [Sampling more candidates versus selecting correct ones](https://semiaifoundry.com/learn/atlas/test-time-search-verification/)\n-   [Version the entire evaluated system](https://semiaifoundry.com/learn/atlas/model-system-regression-contract/)\n-   [Model artifacts, rollout and drift](https://semiaifoundry.com/learn/atlas/model-artifact-rollout-drift/)\n-   [Filtering, duplication and contamination](https://semiaifoundry.com/learn/atlas/data-quality-dedup-audit/)"
}