# Human and model judgments under a rubric Canonical: https://semiaifoundry.com/learn/atlas/judge-rubric-agreement/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html model-lifecycle-evaluation · Atlas v1.3.0 # Human and model judgments under a rubric Separate agreement, task correctness and preference under a reproducible judging rubric. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#judge-rubric-agreement)Mathematical, data and behavior space; logical dimensions are not physical distance Learning objectiveSeparate agreement, task correctness and preference under a reproducible judging rubric. ## The mechanism A judge turns an output and rubric into a score. A model judge, a human judge and an exact checker can disagree because they observe different evidence or optimize different criteria. Agreement measures consistency; it does not prove that the shared answer is correct. Pair order and output length can change judgments without changing factual content. A useful evaluation preserves the rubric, raw responses, order assignment and disagreements rather than collapsing them into one unexplained score. The examples below are authored labels, not records of recruited human participants. 1. Define correctness, source support and style as separate rubric dimensions. 2. Counterbalance pair order and retain each judgment with its evaluator identity. 3. Compare both agreement and correctness against the appropriate reference. 4. Review disagreements and shared errors rather than interpreting consensus as ground truth. ## Science and assumptions ### Human and model judgments under a rubric · worked relation $$ G=\frac1N\sum_i\mathbf1[j_i^{A}=j_i^{B}],\qquad S_A=\frac1N\sum_i\mathbf1[j_i^{A}=y_i] $$ Separate agreement, task correctness and preference under a reproducible judging rubric. Symbols and units G Observed pairwise judge agreement dimensionless S\_A Judge A correctness against the declared reference dimensionless j\_i^A,j\_i^B Judge labels category y\_i Task reference label category N Paired cases cases For four binary cases the reference is \[1,1,0,0\]. Judge A emits \[1,1,1,0\] and B emits \[1,0,1,0\]. They agree on cases 1,3,4: agreement is 3/4. A is correct on 3/4, B on 2/4. Their agreement on case 3 is a shared error. If the first response is preferred 7/10 before swapping order but only 3/10 afterward, content and presentation order need separate investigation. Assumptions Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference. ## Engineering example For four binary cases the reference is \[1,1,0,0\]. Judge A emits \[1,1,1,0\] and B emits \[1,0,1,0\]. They agree on cases 1,3,4: agreement is 3/4. A is correct on 3/4, B on 2/4. Their agreement on case 3 is a shared error. If the first response is preferred 7/10 before swapping order but only 3/10 afterward, content and presentation order need separate investigation. ## Distinctions and tradeoffs It is a mistake to conclude: The reference must be wrong because two judges agree. Automated judges can scale scoring but can also share systematic biases. Human judgments require a clear rubric and do not become specialist review merely by being labeled human. Measure quality, uncertainty and failure under a stated system/task contract. Separate agreement, task correctness and preference under a reproducible judging rubric. ## Specifications and interfaces Learning mode worked example The executable decoder has 423 parameters; worked extensions do not imply an additional simulator. Worked contract Separate agreement, task correctness and preference under a reproducible judging rubric. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference. ### Interfaces Inputs and state Input Define correctness, source support and style as separate rubric dimensions. Result and check Output Separate agreement, task correctness and preference under a reproducible judging rubric. ## Check your understanding ### What does agreement on case 3 establish? 1. Both judges make the same error relative to this reference. 2. The reference must be wrong because two judges agree. 3. Agreement equals factual correctness by definition. Answer and reasoning Both judges make the same error relative to this reference. Both judges make the same error relative to this reference. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference. ## Evidence and scope Mode: worked example. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied. Model lifecycle studio: worked example. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference. ## Sources - [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1) - [Holistic Evaluation of Language Models](https://arxiv.org/html/2211.09110v1) - [Training language models to follow instructions with human feedback](https://arxiv.org/html/2203.02155v1) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/judge-rubric-agreement-ab6bcee550c9490b.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Foundations - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/) - [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/) ### Related lessons - [Confidence, errors and refusal to guess](https://semiaifoundry.com/learn/atlas/probability-calibration-abstention/) - [From lexical retrieval to model answers with evidence](https://semiaifoundry.com/learn/atlas/retrieval-supported-generation/) - [Sampling more candidates versus selecting correct ones](https://semiaifoundry.com/learn/atlas/test-time-search-verification/) - [Version the entire evaluated system](https://semiaifoundry.com/learn/atlas/model-system-regression-contract/)