# Human judgments, disagreement and dataset lineage Canonical: https://semiaifoundry.com/learn/atlas/dataset-human-judgment-lineage/ Revision: 2026-10-07 Rights: https://semiaifoundry.com/ai-atlas/licenses.html data-evidence-foundations · Atlas v1.3.0 # Human judgments, disagreement and dataset lineage Makes the people and decisions behind training labels visible in the resulting dataset. [Explore in Atlas](https://semiaifoundry.com/ai-atlas/#dataset-human-judgment-lineage)Mathematical and data space; dimensions are not physical distance Learning objectivePreserve original judgments through an adjudication rule and distinguish label availability from truth. ## The mechanism A label is produced by a process: an example is selected, a task is explained, a person or system makes a judgment, and a policy decides what becomes training material. Disagreement can reveal ambiguity, differing interpretations or poor instructions. Erasing the earlier judgments after adjudication makes those causes harder to study. The specimen retains two fictional annotations for each example and applies a declared rule to unresolved cases. The result is a lineage table containing both source judgments and the selected training label. The rule is inspectable and changeable; it is not presented as a way to infer truth automatically. 1. Retain the example identifier, each original annotation and the relevant instruction/version context. 2. Measure direct agreement without changing the labels to make the agreement look higher. 3. Apply a declared review-category policy to disagreements or retain them as unresolved. 4. Count which examples now have usable labels while preserving the disagreement and selection history. ## Science and assumptions ### Human judgments, disagreement and dataset lineage $$ a=\frac{1}{n}\sum_{i=1}^{n}\mathbf1[y_i^{(1)}=y_i^{(2)}] $$ Direct agreement is the fraction of examples on which the two original labels match. Symbols and units a Direct observed agreement fraction n Examples with both original judgments count yᵢ⁽¹⁾,yᵢ⁽²⁾ Original categorical labels label Six fictional examples contain three agreements and three disagreements, so direct agreement is 3/6. With unresolved cases excluded, three training labels remain. Sending disagreements to the review category retains six labels, but original agreement remains 3/6. The extra labels reflect a policy decision, not newly discovered truth. Assumptions Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric. ## Engineering example Six fictional examples contain three agreements and three disagreements, so direct agreement is 3/6. With unresolved cases excluded, three training labels remain. Sending disagreements to the review category retains six labels, but original agreement remains 3/6. The extra labels reflect a policy decision, not newly discovered truth. ## Distinctions and tradeoffs Agreement is not proof of truth, and an adjudicated label does not by itself establish representative coverage or permission to use the data. Adjudication can improve consistency or usability, but it can also suppress meaningful disagreement or change which populations are retained. Documenting the rule makes the effect inspectable. Supervised tuning, preference data and model evaluation all incorporate judgments. Their provenance and selection can affect what behavior the system learns or appears to satisfy. ## Specifications and interfaces Teaching specimen Annotation lineage Six fictional examples contain three agreements and three disagreements, so direct agreement is 3/6. With unresolved cases excluded, three training labels remain. Sending disagreements to the review category retains six labels, but original agreement remains 3/6. The extra labels reflect a policy decision, not newly discovered truth. Evidence boundary Explicit computed example Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric. ### Interfaces Inputs and convention Input Retain the example identifier, each original annotation and the relevant instruction/version context. Computed result and scope Output Preserve original judgments through an adjudication rule and distinguish label availability from truth. ## Check your understanding ### What happens to original agreement after the review-category rule is applied? 1. It remains 3/6 because the original judgments are preserved 2. It becomes 6/6 by definition 3. It proves the selected labels are objectively correct Answer and reasoning It remains 3/6 because the original judgments are preserved It remains 3/6 because the original judgments are preserved. Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric. ## Evidence and scope The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric. Open Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library. ## Sources - [Gebru et al. · Datasheets for Datasets](https://arxiv.org/abs/1803.09010) [Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/dataset-human-judgment-lineage-0d84a36e42386aa5.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html) ## Continue exploring ### Related lessons - [Training data pipeline](https://semiaifoundry.com/learn/atlas/training-data/) - [Post-training & feedback](https://semiaifoundry.com/learn/atlas/post-training/) - [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)