{"id":"probability-calibration-abstention","name":"Confidence, errors and refusal to guess","shortName":"Confidence, errors and refusal to guess","role":"Evaluate probability quality and selective error using distinct calibration and final-test data.","description":"Confidence is a numerical prediction about outcomes, not a guarantee attached to a fluent answer. Calibration asks whether cases assigned similar probabilities succeed at corresponding frequencies under a stated task and distribution. Temperature scaling fits a scalar on development logits; it usually preserves the class ordering while changing probabilities. Abstention adds a decision policy: only answer when confidence passes a threshold. Lower error among answered cases can be purchased by answering fewer cases, so coverage belongs beside selective error. If no cases are answered, selective error is undefined rather than zero.\n\nThe identified-evaluation experience fits the sensor model on eight training rows, selects a scalar temperature using four separate calibration rows, and audits six diagnostic rows. Threshold and acceptance controls do not refit the weights. Empty probability bins and an empty accepted set retain unavailable metrics rather than becoming perfect scores. The small audit is exploratory, and repeated recipe selection against it is development feedback.","mechanism":["Fit any calibration parameter on development data only.","Score probability quality with an explicit Brier or log-loss convention.","Choose an abstention threshold before opening final cases.","Report answered count, error among answered cases and failures by slice, including empty denominators."],"learningObjective":"Evaluate probability quality and selective error using distinct calibration and final-test data.","engineeringExample":"Binary probabilities [0.8,0.6,0.4,0.2] for class one with targets [1,0,1,0] have Brier score (0.04+0.36+0.36+0.04)/4=0.2. Predicted-class confidence is [0.8,0.6,0.6,0.8]. Threshold 0.75 answers two cases, both correct: coverage 1/2 and selective error 0/2. Threshold 0.9 answers none, so coverage is zero and selective error has no denominator.\n\nThe default executable audit uses temperature 0.5 selected on calibration rows. Its six diagnostic decisions are correct, mean binary log loss is approximately 0.13357 and Brier score is 0.02864. Requiring chosen-class probability exactly 1 accepts zero cases, so conditional error is unavailable. These are computed synthetic fixtures, not calibration evidence for a deployed classifier.","evidenceNotes":"Mode: worked example. Binary Brier convention shown, not sum over both class coordinates. Authored predictions; no calibration fitting feature is implemented in the lifecycle core. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Evaluate probability quality and selective error using distinct calibration and final-test data.","context":"Binary Brier convention shown, not sum over both class coordinates. Authored predictions; no calibration fitting feature is implemented in the lifecycle core."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Fit any calibration parameter on development data only."},{"name":"Result and check","direction":"out","signal":"Evaluate probability quality and selective error using distinct calibration and final-test data."}],"tradeoff":"Abstention can reduce errors among answered cases while excluding difficult or underrepresented slices. Calibration learned on one distribution can fail after a shift.","whyAI":"Measure quality, uncertainty and failure under a stated system/task contract. Evaluate probability quality and selective error using distinct calibration and final-test data.","misconception":"It is a mistake to conclude: Zero coverage and proven zero risk.","check":{"question":"How should a threshold answering zero cases be reported?","options":["Zero coverage and proven zero risk.","Perfect calibration.","Zero coverage and undefined selective error."],"answer":2,"explanation":"Zero coverage and undefined selective error. Binary Brier convention shown, not sum over both class coordinates. Authored predictions; no calibration fitting feature is implemented in the lifecycle core."},"prerequisites":["evaluation","output-head","sampling-estimation-uncertainty"],"related":["evaluation","output-head","sampling-estimation-uncertainty","judge-rubric-agreement","retrieval-supported-generation","test-time-search-verification","model-system-regression-contract","model-artifact-rollout-drift","data-quality-dedup-audit"],"sources":[{"title":"On Calibration of Modern Neural Networks","url":"https://arxiv.org/html/1706.04599v2"},{"title":"Holistic Evaluation of Language Models","url":"https://arxiv.org/html/2211.09110v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-evaluation","subbranch":"Confidence, errors and refusal to guess","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Binary Brier convention shown, not sum over both class coordinates. Authored predictions; no calibration fitting feature is implemented in the lifecycle core.","science":[{"title":"Confidence, errors and refusal to guess · worked relation","equation":"B=\\frac1N\\sum_i(p_i-y_i)^2,\\quad C=\\frac{|A|}{N},\\quad R=\\frac{\\sum_{i\\in A}\\mathbf1[\\hat y_i\\ne y_i]}{|A|}\\ (|A|>0)","explanation":"Binary probabilities [0.8,0.6,0.4,0.2] for class one with targets [1,0,1,0] have Brier score (0.04+0.36+0.36+0.04)/4=0.2. Predicted-class confidence is [0.8,0.6,0.6,0.8]. Threshold 0.75 answers two cases, both correct: coverage 1/2 and selective error 0/2. Threshold 0.9 answers none, so coverage is zero and selective error has no denominator.","assumptions":"Binary Brier convention shown, not sum over both class coordinates. Authored predictions; no calibration fitting feature is implemented in the lifecycle core."}],"history":[],"checks":[]}