Human and model judgments under a rubric

The mechanism

A judge turns an output and rubric into a score. A model judge, a human judge and an exact checker can disagree because they observe different evidence or optimize different criteria. Agreement measures consistency; it does not prove that the shared answer is correct. Pair order and output length can change judgments without changing factual content. A useful evaluation preserves the rubric, raw responses, order assignment and disagreements rather than collapsing them into one unexplained score. The examples below are authored labels, not records of recruited human participants.

Define correctness, source support and style as separate rubric dimensions.
Counterbalance pair order and retain each judgment with its evaluator identity.
Compare both agreement and correctness against the appropriate reference.
Review disagreements and shared errors rather than interpreting consensus as ground truth.

Science and assumptions

Human and model judgments under a rubric · worked relation

G=\frac1N\sum_i\mathbf1[j_i^{A}=j_i^{B}],\qquad S_A=\frac1N\sum_i\mathbf1[j_i^{A}=y_i]

Separate agreement, task correctness and preference under a reproducible judging rubric.

Symbols and units

G: Observed pairwise judge agreement
dimensionless
S_A: Judge A correctness against the declared reference
dimensionless
j_i^A,j_i^B: Judge labels
category
y_i: Task reference label
category
N: Paired cases
cases

For four binary cases the reference is [1,1,0,0]. Judge A emits [1,1,1,0] and B emits [1,0,1,0]. They agree on cases 1,3,4: agreement is 3/4. A is correct on 3/4, B on 2/4. Their agreement on case 3 is a shared error. If the first response is preferred 7/10 before swapping order but only 3/10 afterward, content and presentation order need separate investigation.

Assumptions Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference.

Engineering example

Distinctions and tradeoffs

It is a mistake to conclude: The reference must be wrong because two judges agree.

Automated judges can scale scoring but can also share systematic biases. Human judgments require a clear rubric and do not become specialist review merely by being labeled human.

Measure quality, uncertainty and failure under a stated system/task contract. Separate agreement, task correctness and preference under a reproducible judging rubric.

Specifications and interfaces

Learning mode: worked example
The executable decoder has 423 parameters; worked extensions do not imply an additional simulator.
Worked contract: Separate agreement, task correctness and preference under a reproducible judging rubric.
Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference.

Interfaces

Inputs and state Input: Define correctness, source support and style as separate rubric dimensions.
Result and check Output: Separate agreement, task correctness and preference under a reproducible judging rubric.

Check your understanding

What does agreement on case 3 establish?

Both judges make the same error relative to this reference.
The reference must be wrong because two judges agree.
Agreement equals factual correctness by definition.

Answer and reasoning

Both judges make the same error relative to this reference.

Both judges make the same error relative to this reference. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference.

Evidence and scope

Mode: worked example. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference. Numerical examples are authored calculations, except explicitly identified automated core fixtures. Primary sources support the scoped mechanism; no human specialist review or learner outcome is implied.

Model lifecycle studio: worked example. Authored binary labels and exact case matching; agreement here is raw agreement, not chance-corrected kappa. Human preference tasks may lack a single factual reference.