The mechanism
A recommender selects candidates and ranks them for a user or context. Sparse categorical features often become embedding lookups, while dense features and interactions contribute to a score. DLRM provides a primary example of this combination. Evaluation must also account for what the system chose to expose: a user cannot click an item that was never shown, and position can affect clicks independently of relevance. Treating every unclicked or unshown item as an equally observed negative can therefore distort learning. The worked ranking metrics below use authored labels and exposure counts; they do not train a recommender or implement a counterfactual estimator. Position-biased feedback research motivates why observational click totals alone are insufficient evidence of quality.
- Candidate generation limits the items available to the ranker for the current user or context.
- The scoring model combines declared features or learned embeddings to order those candidates.
- The exposure log records which items were actually shown and where, separately from clicks.
- Evaluation uses independent relevance/outcome evidence and states how missing exposure and feedback bias are handled.
Science and assumptions
Ranking changes what is observed · explicit relationship
Precision at k is the fraction of the first k ranked items labeled relevant under the declared reference.
Symbols and units
- k
- Positive evaluated list length
items
- yᵢ
- Independent binary relevance label at rank i
0 or 1
- 1[·]
- Indicator equal to one when its condition holds
dimensionless
For a three-item displayed list with independent relevance labels [1,0,1], precision at three is 2/3. Now suppose item A was shown 100 times and clicked 20 times, while B was shown 10 times and clicked four times. Raw clicks favor A, but their observed click-through ratios are 0.20 and 0.40 under different exposures. Neither comparison by itself removes user or position confounding. An item never shown has no observed opportunity to be clicked; calling it a confirmed negative invents evidence.
Assumptions The relevance labels here are authored and independent of clicks. This metric does not correct exposure bias, missing labels or changing user utility.
Engineering example
For a three-item displayed list with independent relevance labels [1,0,1], precision at three is 2/3. Now suppose item A was shown 100 times and clicked 20 times, while B was shown 10 times and clicked four times. Raw clicks favor A, but their observed click-through ratios are 0.20 and 0.40 under different exposures. Neither comparison by itself removes user or position confounding. An item never shown has no observed opportunity to be clicked; calling it a confirmed negative invents evidence.
Distinctions and tradeoffs
More clicks do not automatically mean greater relevance under unequal or biased exposure.
Exploration can improve evidence about less-exposed items while changing short-term user experience and the distribution used for training.
Recommendation workloads connect embedding-heavy computation to feedback generated by their own decisions.
Specifications and interfaces
- Mechanism boundary
- What does a click label leave unobserved?
The relevance labels here are authored and independent of clicks. This metric does not correct exposure bias, missing labels or changing user utility.
- Evidence type
- Authored worked example
Worked explanation with a bounded numeric or state example. This lesson does not claim an executing implementation of an external model or platform.
Interfaces
- Declared input and state Input
- Candidate generation limits the items available to the ranker for the current user or context.
- Checked outcome Output
- Compute a ranking metric and explain why exposure and position prevent raw clicks from being neutral labels.
Check your understanding
An item was never displayed. What does its zero-click count establish?
- That every user dislikes it
- That no click opportunity was observed in this log
- That its relevance score must be zero
Answer and reasoning
That no click opportunity was observed in this log
Without exposure there is no observed opportunity for a click; relevance remains unestablished by that log.
Evidence and scope
This is a worked explanation, not an executing external platform, model or robot. The quantities and fixtures are authored. Source scope is recorded in the automation learning review manifest; no source benchmark or private system behavior is claimed. The relevance labels here are authored and independent of clicks. This metric does not correct exposure bias, missing labels or changing user utility.
The worked example explains this mechanism. General numerical labs are a reference; they do not execute this lesson’s full mechanism.