{"id":"retrieval-indexing-evaluation","name":"Retrieval from source document to answer","shortName":"Retrieval from source document to answer","role":"Chunking, lexical versus dense retrieval, provenance, reranking and answer validation are distinct stages.","description":"Retrieval converts a source collection into an index and selects material for a particular query. Chunk boundaries, identifiers and metadata determine which evidence can be returned. A lexical index matches terms; a dense index uses learned representations. Their scores are different objects, and neither is automatically a calibrated probability that an answer is correct. Reranking, evidence selection and answer generation or extraction are further stages. The current lexical application exposes TF–IDF and a deterministic selector, including a case where the correct passage is retrieved but the wrong entity’s fact is chosen. A neural retrieval-and-generation pipeline requires actual learned artifacts and recorded outputs; the original RAG paper supplies method context, not evidence that the current lexical example implements that model.","mechanism":["Documents and chunks retain stable source identities, versions and task-relevant metadata.","An index maps each query to a ranked candidate set under a declared lexical or learned scoring method.","Relevance labels evaluate retrieval separately from the answer selector or generator.","Task correctness, source support and justified abstention are scored with their own denominators."],"learningObjective":"Locate indexing, ranking and answer-selection errors and calculate each evaluation denominator correctly.","engineeringExample":"Suppose two chunks are relevant and the top three results contain one of them. Recall at three is 1/2, regardless of the third result’s high similarity score. On a separate five-question answer set, four questions receive answers and all four answers are supported by their cited passages, but only three answer the requested entity correctly. Source support among answers is 4/4 while task accuracy over all questions is 3/5. Better retrieval recall may help, yet it cannot by itself repair a selector that uses the wrong entity.","misconception":"A relevant source or a high similarity score does not establish a correct answer.","check":{"question":"Four supported answers contain three task-correct answers among five questions. What is task accuracy?","options":["100%","75%","60%"],"answer":2,"explanation":"Task accuracy uses all five questions, including unanswered or incorrect cases; it is 3/5."},"checks":[],"prerequisites":["embedding","evaluation","tool-loop"],"related":["tool-loop","evaluation"],"specs":[{"label":"Mechanism boundary","value":"Which error came from indexing, ranking or answer selection?","context":"Reference relevance and task labels are fixed independently of the method. Empty relevance sets require a separate convention rather than division by zero."},{"label":"Evidence type","value":"Authored worked example","context":"Worked explanation with a bounded numeric or state example. This lesson does not claim an executing implementation of an external model or platform."}],"interfaces":[{"name":"Declared input and state","direction":"in","signal":"Documents and chunks retain stable source identities, versions and task-relevant metadata."},{"name":"Checked outcome","direction":"out","signal":"Locate indexing, ranking and answer-selection errors and calculate each evaluation denominator correctly."}],"tradeoff":"Larger contexts may improve evidence recall while adding irrelevant material and increasing downstream interpretation cost.","whyAI":"Grounding is a pipeline of inspectable decisions, not simply attaching a citation to generated text.","sourceIds":["automation-rag","automation-tau-bench"],"sources":[{"title":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks","url":"https://arxiv.org/abs/2005.11401"},{"title":"τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains","url":"https://arxiv.org/abs/2406.12045"}],"science":[{"title":"Retrieval from source document to answer · explicit relationship","equation":"Retrieval recall counts retrieved relevant items over all relevant items; task accuracy counts correct outcomes over all evaluated questions.","explanation":"Suppose two chunks are relevant and the top three results contain one of them. Recall at three is 1/2, regardless of the third result’s high similarity score. On a separate five-question answer set, four questions receive answers and all four answers are supported by their cited passages, but only three answer the requested entity correctly. Source support among answers is 4/4 while task accuracy over all questions is 3/5. Better retrieval recall may help, yet it cannot by itself repair a selector that uses the wrong entity.","assumptions":"Reference relevance and task labels are fixed independently of the method. Empty relevance sets require a separate convention rather than division by zero."}],"evidenceNotes":"This is a worked explanation, not an executing external platform, model or robot. The quantities and fixtures are authored. Source scope is recorded in the automation learning review manifest; no source benchmark or private system behavior is claimed. Reference relevance and task labels are fixed independently of the method. Empty relevance sets require a separate convention rather than division by zero.","level":"model","branch":"Models become evaluated applications","subbranch":"Retrieval from source document to answer","scale":"Logical state, task evidence and declared units; no physical assembly scale","kind":"system","workstream":"W11","visualFamily":"software-flow","labId":"library","labScope":"The worked example explains this mechanism. General numerical labs are a reference; they do not execute this lesson’s full mechanism.","history":[],"workedTrace":{"relevant":2,"retrievedRelevant":1,"recall":0.5,"questions":5,"answers":4,"supported":4,"correct":3,"accuracy":0.6}}