{"id":"judge-rubric-agreement","name":"Human and model judgments under a rubric","shortName":"Human and model judgments under a rubric","role":"Separate agreement, task correctness and preference under a reproducible judging rubric.","description":"A judge turns an output and rubric into a score. A model judge, a human judge and an exact checker can disagree because they observe different evidence or optimize different criteria. Agreement measures consistency; it does not prove that the shared answer is correct. Pair order and output length can change judgments without changing factual content. A useful evaluation preserves the rubric, raw responses, order assignment and disagreements rather than collapsing them into one unexplained score. The examples below are authored labels, not records of recruited human participants.","mechanism":["Define correctness, source support and style as separate rubric dimensions.","Counterbalance pair order and retain each judgment with its evaluator identity.","Compare both agreement and correctness against the appropriate reference.","Review disagreements and shared errors rather than interpreting consensus as ground truth."],"learningObjective":"Separate agreement, task correctness and preference under a reproducible judging rubric.","engineeringExample":"For four binary cases the reference is [1,1,0,0]. Judge A emits [1,1,1,0] and B emits [1,0,1,0]. They agree on cases 1,3,4: agreement is 3/4. A is correct on 3/4, B on 2/4. Their agreement on case 3 is a shared error. If the first response is preferred 7/10 before swapping order but only 3/10 afterward, content and presentation order need separate investigation.","evidenceNotes":"Mode: worked example. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.","specs":[{"label":"Learning mode","value":"worked example","context":"The executable decoder has 423 parameters; worked extensions do not imply an additional simulator."},{"label":"Worked contract","value":"Separate agreement, task correctness and preference under a reproducible judging rubric.","context":"Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference."}],"interfaces":[{"name":"Inputs and state","direction":"in","signal":"Define correctness, source support and style as separate rubric dimensions."},{"name":"Result and check","direction":"out","signal":"Separate agreement, task correctness and preference under a reproducible judging rubric."}],"tradeoff":"Automated judges can scale scoring but can also share systematic biases. Human judgments require a clear rubric and do not become specialist review merely by being labeled human.","whyAI":"Measure quality, uncertainty and failure under a stated system/task contract. Separate agreement, task correctness and preference under a reproducible judging rubric.","misconception":"It is a mistake to conclude: The reference must be wrong because two judges agree.","check":{"question":"What does agreement on case 3 establish?","options":["Both judges make the same error relative to this reference.","The reference must be wrong because two judges agree.","Agreement equals factual correctness by definition."],"answer":0,"explanation":"Both judges make the same error relative to this reference. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference."},"prerequisites":["evaluation","preference-reward-model"],"related":["evaluation","preference-reward-model","probability-calibration-abstention","retrieval-supported-generation","test-time-search-verification","model-system-regression-contract"],"sources":[{"title":"Tülu 3: Pushing Frontiers in Open Language Model Post-Training","url":"https://arxiv.org/html/2411.15124v1"},{"title":"Holistic Evaluation of Language Models","url":"https://arxiv.org/html/2211.09110v1"},{"title":"Training language models to follow instructions with human feedback","url":"https://arxiv.org/html/2203.02155v1"}],"level":"model","scale":"Mathematical, data and behavior space; logical dimensions are not physical distance","kind":"model","branch":"model-lifecycle-evaluation","subbranch":"Human and model judgments under a rubric","workstream":"W11","visualFamily":"mathematical","labId":"library","labScope":"Model lifecycle studio: worked example. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference.","science":[{"title":"Human and model judgments under a rubric · worked relation","equation":"G=\\frac1N\\sum_i\\mathbf1[j_i^{A}=j_i^{B}],\\qquad S_A=\\frac1N\\sum_i\\mathbf1[j_i^{A}=y_i]","explanation":"For four binary cases the reference is [1,1,0,0]. Judge A emits [1,1,1,0] and B emits [1,0,1,0]. They agree on cases 1,3,4: agreement is 3/4. A is correct on 3/4, B on 2/4. Their agreement on case 3 is a shared error. If the first response is preferred 7/10 before swapping order but only 3/10 afterward, content and presentation order need separate investigation.","assumptions":"Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference."}],"history":[]}