{
  "schemaVersion": 1,
  "id": "atlas-lesson-dataset-human-judgment-lineage",
  "title": "Human judgments, disagreement and dataset lineage",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Makes the people and decisions behind training labels visible in the resulting dataset.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/dataset-human-judgment-lineage/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "data-evidence-foundations",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/dataset-human-judgment-lineage-0d84a36e42386aa5.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#dataset-human-judgment-lineage",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "5f8fd7836d362400ca18e17f774bc34d9547f0250a1d556e93e0aa9feba5feb0",
  "contentLength": 5757,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-dataset-human-judgment-lineage.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-dataset-human-judgment-lineage.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Human judgments, disagreement and dataset lineage",
      "start": 42,
      "end": 460
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 460,
      "end": 1519
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1519,
      "end": 1547
    },
    {
      "id": "section-4",
      "title": "Human judgments, disagreement and dataset lineage",
      "start": 1547,
      "end": 2387
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2387,
      "end": 2748
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2748,
      "end": 3297
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3297,
      "end": 3882
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3882,
      "end": 4169
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4169,
      "end": 4198
    },
    {
      "id": "section-10",
      "title": "What happens to original agreement after the review-category rule is applied?",
      "start": 4198,
      "end": 4710
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 4710,
      "end": 5158
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5158,
      "end": 5459
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5459,
      "end": 5482
    },
    {
      "id": "section-14",
      "title": "Related lessons",
      "start": 5482,
      "end": 5757
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#dataset-human-judgment-lineage"
    },
    {
      "title": "Gebru et al. · Datasheets for Datasets",
      "url": "https://arxiv.org/abs/1803.09010"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/dataset-human-judgment-lineage-0d84a36e42386aa5.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Training data pipeline",
      "url": "https://semiaifoundry.com/learn/atlas/training-data/"
    },
    {
      "title": "Post-training & feedback",
      "url": "https://semiaifoundry.com/learn/atlas/post-training/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    }
  ],
  "recordHash": "9133583b22bbe538b63e59deb31fd2e7dd89a615dddc31c49a66c4c840807d23",
  "contentFormat": "text/markdown",
  "content": "data-evidence-foundations · Atlas v1.3.0\n\n# Human judgments, disagreement and dataset lineage\n\nMakes the people and decisions behind training labels visible in the resulting dataset.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#dataset-human-judgment-lineage)Mathematical and data space; dimensions are not physical distance\n\nLearning objectivePreserve original judgments through an adjudication rule and distinguish label availability from truth.\n\n## The mechanism\n\nA label is produced by a process: an example is selected, a task is explained, a person or system makes a judgment, and a policy decides what becomes training material. Disagreement can reveal ambiguity, differing interpretations or poor instructions. Erasing the earlier judgments after adjudication makes those causes harder to study. The specimen retains two fictional annotations for each example and applies a declared rule to unresolved cases. The result is a lineage table containing both source judgments and the selected training label. The rule is inspectable and changeable; it is not presented as a way to infer truth automatically.\n\n1.  Retain the example identifier, each original annotation and the relevant instruction/version context.\n2.  Measure direct agreement without changing the labels to make the agreement look higher.\n3.  Apply a declared review-category policy to disagreements or retain them as unresolved.\n4.  Count which examples now have usable labels while preserving the disagreement and selection history.\n\n## Science and assumptions\n\n### Human judgments, disagreement and dataset lineage\n\n$$\na=\\frac{1}{n}\\sum_{i=1}^{n}\\mathbf1[y_i^{(1)}=y_i^{(2)}]\n$$\n\nDirect agreement is the fraction of examples on which the two original labels match.\n\nSymbols and units\n\na\n\nDirect observed agreement\n\nfraction\n\nn\n\nExamples with both original judgments\n\ncount\n\nyᵢ⁽¹⁾,yᵢ⁽²⁾\n\nOriginal categorical labels\n\nlabel\n\nSix fictional examples contain three agreements and three disagreements, so direct agreement is 3/6. With unresolved cases excluded, three training labels remain. Sending disagreements to the review category retains six labels, but original agreement remains 3/6. The extra labels reflect a policy decision, not newly discovered truth.\n\nAssumptions Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric.\n\n## Engineering example\n\nSix fictional examples contain three agreements and three disagreements, so direct agreement is 3/6. With unresolved cases excluded, three training labels remain. Sending disagreements to the review category retains six labels, but original agreement remains 3/6. The extra labels reflect a policy decision, not newly discovered truth.\n\n## Distinctions and tradeoffs\n\nAgreement is not proof of truth, and an adjudicated label does not by itself establish representative coverage or permission to use the data.\n\nAdjudication can improve consistency or usability, but it can also suppress meaningful disagreement or change which populations are retained. Documenting the rule makes the effect inspectable.\n\nSupervised tuning, preference data and model evaluation all incorporate judgments. Their provenance and selection can affect what behavior the system learns or appears to satisfy.\n\n## Specifications and interfaces\n\nTeaching specimen\n\nAnnotation lineage\n\nSix fictional examples contain three agreements and three disagreements, so direct agreement is 3/6. With unresolved cases excluded, three training labels remain. Sending disagreements to the review category retains six labels, but original agreement remains 3/6. The extra labels reflect a policy decision, not newly discovered truth.\n\nEvidence boundary\n\nExplicit computed example\n\nTwo annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric.\n\n### Interfaces\n\nInputs and convention Input\n\nRetain the example identifier, each original annotation and the relevant instruction/version context.\n\nComputed result and scope Output\n\nPreserve original judgments through an adjudication rule and distinguish label availability from truth.\n\n## Check your understanding\n\n### What happens to original agreement after the review-category rule is applied?\n\n1.  It remains 3/6 because the original judgments are preserved\n2.  It becomes 6/6 by definition\n3.  It proves the selected labels are objectively correct\n\nAnswer and reasoning\n\nIt remains 3/6 because the original judgments are preserved\n\nIt remains 3/6 because the original judgments are preserved. Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric.\n\n## Evidence and scope\n\nThe workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric.\n\nOpen Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library.\n\n## Sources\n\n-   [Gebru et al. · Datasheets for Datasets](https://arxiv.org/abs/1803.09010)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/dataset-human-judgment-lineage-0d84a36e42386aa5.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Related lessons\n\n-   [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/)\n-   [Post-training & feedback](https://semiaifoundry.com/learn/atlas/post-training/)\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)"
}