{"id":"dataset-human-judgment-lineage","name":"Human judgments, disagreement and dataset lineage","shortName":"Annotation lineage","role":"Makes the people and decisions behind training labels visible in the resulting dataset.","description":"A label is produced by a process: an example is selected, a task is explained, a person or system makes a judgment, and a policy decides what becomes training material. Disagreement can reveal ambiguity, differing interpretations or poor instructions. Erasing the earlier judgments after adjudication makes those causes harder to study. The specimen retains two fictional annotations for each example and applies a declared rule to unresolved cases. The result is a lineage table containing both source judgments and the selected training label. The rule is inspectable and changeable; it is not presented as a way to infer truth automatically.","mechanism":["Retain the example identifier, each original annotation and the relevant instruction/version context.","Measure direct agreement without changing the labels to make the agreement look higher.","Apply a declared review-category policy to disagreements or retain them as unresolved.","Count which examples now have usable labels while preserving the disagreement and selection history."],"learningObjective":"Preserve original judgments through an adjudication rule and distinguish label availability from truth.","engineeringExample":"Six fictional examples contain three agreements and three disagreements, so direct agreement is 3/6. With unresolved cases excluded, three training labels remain. Sending disagreements to the review category retains six labels, but original agreement remains 3/6. The extra labels reflect a policy decision, not newly discovered truth.","evidenceNotes":"The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric.","specs":[{"label":"Teaching specimen","value":"Annotation lineage","context":"Six fictional examples contain three agreements and three disagreements, so direct agreement is 3/6. With unresolved cases excluded, three training labels remain. Sending disagreements to the review category retains six labels, but original agreement remains 3/6. The extra labels reflect a policy decision, not newly discovered truth."},{"label":"Evidence boundary","value":"Explicit computed example","context":"Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric."}],"interfaces":[{"name":"Inputs and convention","direction":"in","signal":"Retain the example identifier, each original annotation and the relevant instruction/version context."},{"name":"Computed result and scope","direction":"out","signal":"Preserve original judgments through an adjudication rule and distinguish label availability from truth."}],"tradeoff":"Adjudication can improve consistency or usability, but it can also suppress meaningful disagreement or change which populations are retained. Documenting the rule makes the effect inspectable.","whyAI":"Supervised tuning, preference data and model evaluation all incorporate judgments. Their provenance and selection can affect what behavior the system learns or appears to satisfy.","misconception":"Agreement is not proof of truth, and an adjudicated label does not by itself establish representative coverage or permission to use the data.","check":{"question":"What happens to original agreement after the review-category rule is applied?","options":["It remains 3/6 because the original judgments are preserved","It becomes 6/6 by definition","It proves the selected labels are objectively correct"],"answer":0,"explanation":"It remains 3/6 because the original judgments are preserved. Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric."},"prerequisites":[],"related":["training-data","post-training","evaluation"],"sources":[{"title":"Gebru et al. · Datasheets for Datasets","url":"https://arxiv.org/abs/1803.09010"}],"level":"model","scale":"Mathematical and data space; dimensions are not physical distance","kind":"model","branch":"data-evidence-foundations","subbranch":"Annotation lineage","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Open Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library.","science":[{"title":"Human judgments, disagreement and dataset lineage","equation":"Direct agreement is the fraction of examples on which the two original labels match.","explanation":"Six fictional examples contain three agreements and three disagreements, so direct agreement is 3/6. With unresolved cases excluded, three training labels remain. Sending disagreements to the review category retains six labels, but original agreement remains 3/6. The extra labels reflect a policy decision, not newly discovered truth.","assumptions":"Two annotations per fictional example; direct agreement is not chance-adjusted and is not a truth or representativeness metric."}]}