{
  "schemaVersion": 1,
  "id": "atlas-lesson-judge-rubric-agreement",
  "title": "Human and model judgments under a rubric",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Separate agreement, task correctness and preference under a reproducible judging rubric.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/judge-rubric-agreement/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "model-lifecycle-evaluation",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "Mode: worked example. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/judge-rubric-agreement-ab6bcee550c9490b.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#judge-rubric-agreement",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "ccce96986180ac4684f85623b423d129464b91b48042c22fad68e84418f76616",
  "contentLength": 6462,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-judge-rubric-agreement.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-judge-rubric-agreement.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Human and model judgments under a rubric",
      "start": 43,
      "end": 448
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 448,
      "end": 1391
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1391,
      "end": 1419
    },
    {
      "id": "section-4",
      "title": "Human and model judgments under a rubric · worked relation",
      "start": 1419,
      "end": 2467
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2467,
      "end": 2863
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2863,
      "end": 3326
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3326,
      "end": 3768
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3768,
      "end": 4000
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4000,
      "end": 4029
    },
    {
      "id": "section-10",
      "title": "What does agreement on case 3 establish?",
      "start": 4029,
      "end": 4567
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4567,
      "end": 5196
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5196,
      "end": 5709
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5709,
      "end": 5732
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5732,
      "end": 5944
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 5944,
      "end": 6462
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#judge-rubric-agreement"
    },
    {
      "title": "Tülu 3: Pushing Frontiers in Open Language Model Post-Training",
      "url": "https://arxiv.org/html/2411.15124v1"
    },
    {
      "title": "Holistic Evaluation of Language Models",
      "url": "https://arxiv.org/html/2211.09110v1"
    },
    {
      "title": "Training language models to follow instructions with human feedback",
      "url": "https://arxiv.org/html/2203.02155v1"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/judge-rubric-agreement-ab6bcee550c9490b.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Learn a reward from pairwise choices",
      "url": "https://semiaifoundry.com/learn/atlas/preference-reward-model/"
    },
    {
      "title": "Confidence, errors and refusal to guess",
      "url": "https://semiaifoundry.com/learn/atlas/probability-calibration-abstention/"
    },
    {
      "title": "From lexical retrieval to model answers with evidence",
      "url": "https://semiaifoundry.com/learn/atlas/retrieval-supported-generation/"
    },
    {
      "title": "Sampling more candidates versus selecting correct ones",
      "url": "https://semiaifoundry.com/learn/atlas/test-time-search-verification/"
    },
    {
      "title": "Version the entire evaluated system",
      "url": "https://semiaifoundry.com/learn/atlas/model-system-regression-contract/"
    }
  ],
  "recordHash": "fca6201957a91423b7b46346bec6a144ec803596f9f222c24b8f9f45599af301",
  "contentFormat": "text/markdown",
  "content": "model-lifecycle-evaluation · Atlas v1.3.0\n\n# Human and model judgments under a rubric\n\nSeparate agreement, task correctness and preference under a reproducible judging rubric.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#judge-rubric-agreement)Mathematical, data and behavior space; logical dimensions are not physical distance\n\nLearning objectiveSeparate agreement, task correctness and preference under a reproducible judging rubric.\n\n## The mechanism\n\nA judge turns an output and rubric into a score. A model judge, a human judge and an exact checker can disagree because they observe different evidence or optimize different criteria. Agreement measures consistency; it does not prove that the shared answer is correct. Pair order and output length can change judgments without changing factual content. A useful evaluation preserves the rubric, raw responses, order assignment and disagreements rather than collapsing them into one unexplained score. The examples below are authored labels, not records of recruited human participants.\n\n1.  Define correctness, source support and style as separate rubric dimensions.\n2.  Counterbalance pair order and retain each judgment with its evaluator identity.\n3.  Compare both agreement and correctness against the appropriate reference.\n4.  Review disagreements and shared errors rather than interpreting consensus as ground truth.\n\n## Science and assumptions\n\n### Human and model judgments under a rubric · worked relation\n\n$$\nG=\\frac1N\\sum_i\\mathbf1[j_i^{A}=j_i^{B}],\\qquad S_A=\\frac1N\\sum_i\\mathbf1[j_i^{A}=y_i]\n$$\n\nSeparate agreement, task correctness and preference under a reproducible judging rubric.\n\nSymbols and units\n\nG\n\nObserved pairwise judge agreement\n\ndimensionless\n\nS\\_A\n\nJudge A correctness against the declared reference\n\ndimensionless\n\nj\\_i^A,j\\_i^B\n\nJudge labels\n\ncategory\n\ny\\_i\n\nTask reference label\n\ncategory\n\nN\n\nPaired cases\n\ncases\n\nFor four binary cases the reference is \\[1,1,0,0\\]. Judge A emits \\[1,1,1,0\\] and B emits \\[1,0,1,0\\]. They agree on cases 1,3,4: agreement is 3/4. A is correct on 3/4, B on 2/4. Their agreement on case 3 is a shared error. If the first response is preferred 7/10 before swapping order but only 3/10 afterward, content and presentation order need separate investigation.\n\nAssumptions Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference.\n\n## Engineering example\n\nFor four binary cases the reference is \\[1,1,0,0\\]. Judge A emits \\[1,1,1,0\\] and B emits \\[1,0,1,0\\]. They agree on cases 1,3,4: agreement is 3/4. A is correct on 3/4, B on 2/4. Their agreement on case 3 is a shared error. If the first response is preferred 7/10 before swapping order but only 3/10 afterward, content and presentation order need separate investigation.\n\n## Distinctions and tradeoffs\n\nIt is a mistake to conclude: The reference must be wrong because two judges agree.\n\nAutomated judges can scale scoring but can also share systematic biases. Human judgments require a clear rubric and do not become specialist review merely by being labeled human.\n\nMeasure quality, uncertainty and failure under a stated system/task contract. Separate agreement, task correctness and preference under a reproducible judging rubric.\n\n## Specifications and interfaces\n\nLearning mode\n\nworked example\n\nThe executable decoder has 423 parameters; worked extensions do not imply an additional simulator.\n\nWorked contract\n\nSeparate agreement, task correctness and preference under a reproducible judging rubric.\n\nAuthored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference.\n\n### Interfaces\n\nInputs and state Input\n\nDefine correctness, source support and style as separate rubric dimensions.\n\nResult and check Output\n\nSeparate agreement, task correctness and preference under a reproducible judging rubric.\n\n## Check your understanding\n\n### What does agreement on case 3 establish?\n\n1.  Both judges make the same error relative to this reference.\n2.  The reference must be wrong because two judges agree.\n3.  Agreement equals factual correctness by definition.\n\nAnswer and reasoning\n\nBoth judges make the same error relative to this reference.\n\nBoth judges make the same error relative to this reference. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference.\n\n## Evidence and scope\n\nMode: worked example. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.\n\nModel lifecycle studio: worked example. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference.\n\n## Sources\n\n-   [Tülu 3: Pushing Frontiers in Open Language Model Post-Training](https://arxiv.org/html/2411.15124v1)\n-   [Holistic Evaluation of Language Models](https://arxiv.org/html/2211.09110v1)\n-   [Training language models to follow instructions with human feedback](https://arxiv.org/html/2203.02155v1)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/judge-rubric-agreement-ab6bcee550c9490b.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n-   [Learn a reward from pairwise choices](https://semiaifoundry.com/learn/atlas/preference-reward-model/)\n\n### Related lessons\n\n-   [Confidence, errors and refusal to guess](https://semiaifoundry.com/learn/atlas/probability-calibration-abstention/)\n-   [From lexical retrieval to model answers with evidence](https://semiaifoundry.com/learn/atlas/retrieval-supported-generation/)\n-   [Sampling more candidates versus selecting correct ones](https://semiaifoundry.com/learn/atlas/test-time-search-verification/)\n-   [Version the entire evaluated system](https://semiaifoundry.com/learn/atlas/model-system-regression-contract/)"
}