{"id":"recommendation-ranking-feedback","name":"Ranking changes what is observed","shortName":"Ranking changes what is observed","role":"Candidate retrieval, sparse embeddings, ranking, exposure and feedback bias.","description":"A recommender selects candidates and ranks them for a user or context. Sparse categorical features often become embedding lookups, while dense features and interactions contribute to a score. DLRM provides a primary example of this combination. Evaluation must also account for what the system chose to expose: a user cannot click an item that was never shown, and position can affect clicks independently of relevance. Treating every unclicked or unshown item as an equally observed negative can therefore distort learning. The worked ranking metrics below use authored labels and exposure counts; they do not train a recommender or implement a counterfactual estimator. Position-biased feedback research motivates why observational click totals alone are insufficient evidence of quality.","mechanism":["Candidate generation limits the items available to the ranker for the current user or context.","The scoring model combines declared features or learned embeddings to order those candidates.","The exposure log records which items were actually shown and where, separately from clicks.","Evaluation uses independent relevance/outcome evidence and states how missing exposure and feedback bias are handled."],"learningObjective":"Compute a ranking metric and explain why exposure and position prevent raw clicks from being neutral labels.","engineeringExample":"For a three-item displayed list with independent relevance labels [1,0,1], precision at three is 2/3. Now suppose item A was shown 100 times and clicked 20 times, while B was shown 10 times and clicked four times. Raw clicks favor A, but their observed click-through ratios are 0.20 and 0.40 under different exposures. Neither comparison by itself removes user or position confounding. An item never shown has no observed opportunity to be clicked; calling it a confirmed negative invents evidence.","misconception":"More clicks do not automatically mean greater relevance under unequal or biased exposure.","check":{"question":"An item was never displayed. What does its zero-click count establish?","options":["That every user dislikes it","That no click opportunity was observed in this log","That its relevance score must be zero"],"answer":1,"explanation":"Without exposure there is no observed opportunity for a click; relevance remains unestablished by that log."},"checks":[],"prerequisites":["embedding","training-data","evaluation"],"related":["tool-loop","evaluation"],"specs":[{"label":"Mechanism boundary","value":"What does a click label leave unobserved?","context":"The relevance labels here are authored and independent of clicks. This metric does not correct exposure bias, missing labels or changing user utility."},{"label":"Evidence type","value":"Authored worked example","context":"Worked explanation with a bounded numeric or state example. This lesson does not claim an executing implementation of an external model or platform."}],"interfaces":[{"name":"Declared input and state","direction":"in","signal":"Candidate generation limits the items available to the ranker for the current user or context."},{"name":"Checked outcome","direction":"out","signal":"Compute a ranking metric and explain why exposure and position prevent raw clicks from being neutral labels."}],"tradeoff":"Exploration can improve evidence about less-exposed items while changing short-term user experience and the distribution used for training.","whyAI":"Recommendation workloads connect embedding-heavy computation to feedback generated by their own decisions.","sourceIds":["automation-dlrm","automation-ranking-bias"],"sources":[{"title":"Deep Learning Recommendation Model for Personalization and Recommendation Systems","url":"https://arxiv.org/abs/1906.00091"},{"title":"Joachims, Swaminathan and Schnabel · Unbiased Learning-to-Rank with Biased Feedback","url":"https://arxiv.org/abs/1608.04468"}],"science":[{"title":"Ranking changes what is observed · explicit relationship","equation":"Precision at k is the fraction of the first k ranked items labeled relevant under the declared reference.","explanation":"For a three-item displayed list with independent relevance labels [1,0,1], precision at three is 2/3. Now suppose item A was shown 100 times and clicked 20 times, while B was shown 10 times and clicked four times. Raw clicks favor A, but their observed click-through ratios are 0.20 and 0.40 under different exposures. Neither comparison by itself removes user or position confounding. An item never shown has no observed opportunity to be clicked; calling it a confirmed negative invents evidence.","assumptions":"The relevance labels here are authored and independent of clicks. This metric does not correct exposure bias, missing labels or changing user utility."}],"evidenceNotes":"This is a worked explanation, not an executing external platform, model or robot. The quantities and fixtures are authored. Source scope is recorded in the automation learning review manifest; no source benchmark or private system behavior is claimed. The relevance labels here are authored and independent of clicks. This metric does not correct exposure bias, missing labels or changing user utility.","level":"model","branch":"Models become evaluated applications","subbranch":"Ranking changes what is observed","scale":"Logical state, task evidence and declared units; no physical assembly scale","kind":"system","workstream":"W11","visualFamily":"software-flow","labId":"library","labScope":"The worked example explains this mechanism. General numerical labs are a reference; they do not execute this lesson’s full mechanism.","history":[],"workedTrace":{"labels":[1,0,1],"k":3,"precision":0.6666666666666666,"shown":[100,10],"clicks":[20,4],"observedCtr":[0.2,0.4]}}