{"id":"coding-agent-patch-evidence","name":"A code patch needs execution evidence","shortName":"A code patch needs execution evidence","role":"Repository context, patch application, sandboxed commands, hidden tests and result artifacts.","description":"A coding assistant produces a proposed change in a particular repository state. The patch is not the completed task: it must apply to the intended tree, run in an appropriate environment and satisfy a behavior contract without breaking other required behavior. Visible tests are useful evidence but can be overfit, especially if the same system writes both implementation and checks around one example. Independent regression cases and the exact source identity make the claim assessable. SWE-bench motivates repository issue-to-patch evaluation rather than isolated code completion. The worked example here illustrates the logic of a testable patch; it does not run a coding agent or reproduce a benchmark score. A sandbox controls process effects, while tests evaluate function behavior, and those are different boundaries.\n\nThe application-contract experience executes four authored clamp patches under an explicit finite ordered-bound task. An identity function and an upper-only clamp both pass two interior examples. Boundary cases and an independently branched oracle expose the omitted behavior. The interface executes whitelisted functions corresponding to the displayed source; it does not run arbitrary imported code or claim a code-generating model.","mechanism":["The task identifies the repository revision and intended behavior before the patch is applied.","The proposed diff changes code in that concrete tree and is inspected as an artifact.","Tests run on the resulting tree with identified inputs, including cases independent of the proposal.","The final report distinguishes observed passing checks, untested scope and any execution failure."],"learningObjective":"Evaluate a code change using independent behavioral evidence and an identified source tree.","engineeringExample":"Suppose a mean function incorrectly divides every sum by two. It returns 6 for [2,4,6] instead of 4. A patch that always returns 4 passes that one visible test but fails [1,9], whose mean is 5. A genuine nonempty-list implementation divides the sum by the list length and returns both expected results. The empty-list behavior is a separate contract rather than an accidental division by zero. A test log should identify which patched tree was executed, so a passing result cannot be attributed to a later untested change.\n\nThe independent finite oracle checks 7 choices of lower bound, all upper bounds at least as large through 3, and 11 integer inputs from −5 to 5: 28 × 11 = 308 triples. The ordered clamp passes every triple and all ten visible boundary examples. This remains a finite test domain, not a proof over every floating-point input.","misconception":"A patch that passes its own demonstration is not automatically a correct repair.","check":{"question":"A patch hard-codes 4 and passes mean([2,4,6]). Which test exposes that overfit?","options":["Repeat the same input","Check mean([1,9]) = 5","Rename the test"],"answer":1,"explanation":"An independent input requires behavior beyond the hard-coded visible answer."},"checks":[],"prerequisites":["sandbox-egress-secrets","evaluation"],"related":["tool-loop","evaluation","temporal-forecast-validation","telemetry-traces-evaluation"],"specs":[{"label":"Mechanism boundary","value":"What demonstrates that a generated change works?","context":"Exact small-number worked behavior; finite-number rounding and empty-input policy require separate tests. No coding model or repository execution is claimed for this example."},{"label":"Evidence type","value":"Authored worked example","context":"Worked explanation with a bounded numeric or state example. This lesson does not claim an executing implementation of an external model or platform."}],"interfaces":[{"name":"Declared input and state","direction":"in","signal":"The task identifies the repository revision and intended behavior before the patch is applied."},{"name":"Checked outcome","direction":"out","signal":"Evaluate a code change using independent behavioral evidence and an identified source tree."}],"tradeoff":"Broader tests increase evidence but cannot prove arbitrary program correctness; the task and tested boundary remain explicit.","whyAI":"Coding agents connect context, tool execution and artifacts, so source identity and independent checks are essential.","sourceIds":["automation-swebench","automation-linux-namespaces"],"sources":[{"title":"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?","url":"https://arxiv.org/abs/2310.06770v3"},{"title":"Linux man-pages · namespaces(7)","url":"https://man7.org/linux/man-pages/man7/namespaces.7.html"}],"science":[{"title":"A code patch needs execution evidence · explicit relationship","equation":"For a nonempty input list, the mean is its sum divided by its actual length.","explanation":"Suppose a mean function incorrectly divides every sum by two. It returns 6 for [2,4,6] instead of 4. A patch that always returns 4 passes that one visible test but fails [1,9], whose mean is 5. A genuine nonempty-list implementation divides the sum by the list length and returns both expected results. The empty-list behavior is a separate contract rather than an accidental division by zero. A test log should identify which patched tree was executed, so a passing result cannot be attributed to a later untested change.","assumptions":"Exact small-number worked behavior; finite-number rounding and empty-input policy require separate tests. No coding model or repository execution is claimed for this example."}],"evidenceNotes":"This is a worked explanation, not an executing external platform, model or robot. The quantities and fixtures are authored. Source scope is recorded in the automation learning review manifest; no source benchmark or private system behavior is claimed. Exact small-number worked behavior; finite-number rounding and empty-input policy require separate tests. No coding model or repository execution is claimed for this example.","level":"model","branch":"Models become evaluated applications","subbranch":"A code patch needs execution evidence","scale":"Logical state, task evidence and declared units; no physical assembly scale","kind":"system","workstream":"W11","visualFamily":"software-flow","labId":"library","labScope":"The worked example explains this mechanism. General numerical labs are a reference; they do not execute this lesson’s full mechanism.","history":[],"workedTrace":{"first":[2,4,6],"firstMean":4,"second":[1,9],"secondMean":5,"hardcodedPatch":4}}