{
  "schemaVersion": 1,
  "id": "atlas-lesson-sampling-estimation-uncertainty",
  "title": "Sampling variability and independent evidence",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Shows why a measured success rate varies even when the underlying teaching probability is held fixed.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/sampling-estimation-uncertainty/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "mathematical-foundations",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "The workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Independent identically distributed Bernoulli trials and known p. No interval estimation, multiple-comparison correction or guarantee of real evaluation representativeness is implied.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/sampling-estimation-uncertainty-5273ca7b00504d7c.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#sampling-estimation-uncertainty",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "ea7d1250c039384f0102642a246b2596397df9f1d1c2c908c9a8f115da4bb6af",
  "contentLength": 6314,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-sampling-estimation-uncertainty.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-sampling-estimation-uncertainty.md",
  "sections": [
    {
      "id": "section-1",
      "title": "Sampling variability and independent evidence",
      "start": 41,
      "end": 496
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 496,
      "end": 1692
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1692,
      "end": 1720
    },
    {
      "id": "section-4",
      "title": "Sampling variability and independent evidence",
      "start": 1720,
      "end": 2672
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2672,
      "end": 3024
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 3024,
      "end": 3544
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 3544,
      "end": 4178
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 4178,
      "end": 4505
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 4505,
      "end": 4534
    },
    {
      "id": "section-10",
      "title": "What does the plotted distribution assume?",
      "start": 4534,
      "end": 5086
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 5086,
      "end": 5590
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 5590,
      "end": 5942
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 5942,
      "end": 5965
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 5965,
      "end": 6113
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 6113,
      "end": 6314
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#sampling-estimation-uncertainty"
    },
    {
      "title": "Goodfellow, Bengio and Courville · Probability and information theory",
      "url": "https://www.deeplearningbook.org/contents/prob.html"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/sampling-estimation-uncertainty-5273ca7b00504d7c.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Probability, likelihood and stable information",
      "url": "https://semiaifoundry.com/learn/atlas/probability-conditioning-information/"
    },
    {
      "title": "Evaluation & the AGI frontier",
      "url": "https://semiaifoundry.com/learn/atlas/evaluation/"
    },
    {
      "title": "Scaling laws & resource allocation",
      "url": "https://semiaifoundry.com/learn/atlas/scaling-laws/"
    }
  ],
  "recordHash": "04966d9201d50c958fe67b1d497e245df8aeffd2f92798fe5004df52d894e2d4",
  "contentFormat": "text/markdown",
  "content": "mathematical-foundations · Atlas v1.3.0\n\n# Sampling variability and independent evidence\n\nShows why a measured success rate varies even when the underlying teaching probability is held fixed.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#sampling-estimation-uncertainty)Mathematical and data space; dimensions are not physical distance\n\nLearning objectiveReconcile an exact sampling distribution with its mean and variance, and identify when a claimed sample count is not independent.\n\n## The mechanism\n\nA finite evaluation contains a particular sample of outcomes. To reason about variability, this specimen begins with a known synthetic success probability and independent Bernoulli trials. Enumerating all possible success counts produces the exact binomial distribution of observed rates. Increasing the number of independent trials concentrates that distribution. The qualifier independent matters: duplicated observations, repeated attempts on the same task or shared hidden causes can violate this model. This is a forward sampling calculation conditional on a known probability. It does not automatically supply a confidence interval for an unknown production system or justify a chosen classroom sample size.\n\n1.  Declare the event counted as success, the number of independent trials and the fixed probability used by the model.\n2.  For every possible count k, combine the number of outcome sequences with their success/failure probabilities.\n3.  Divide counts by n and calculate the probability-weighted mean and variance of the rate.\n4.  Compare different n values while keeping p fixed; explain why correlated duplicates cannot be treated as fresh independent trials.\n\n## Science and assumptions\n\n### Sampling variability and independent evidence\n\n$$\n\\Pr(K=k)={n\\choose k}p^k(1-p)^{n-k},\\quad \\mathbb E[K/n]=p,\\quad \\mathrm{Var}(K/n)=\\frac{p(1-p)}n\n$$\n\nThe binomial count distribution has rate mean p and variance p times one minus p divided by the independent sample count.\n\nSymbols and units\n\nn\n\nIndependent trial count\n\ncount\n\nK,k\n\nSuccess count and a possible count\n\ncount\n\np\n\nDeclared success probability\n\nprobability\n\nAt p=0.5 and n=4, success counts 0 through 4 have probabilities \\[1,4,6,4,1\\]/16. The expected rate is 0.5 and its standard deviation is 0.25. With n=16 independent trials the standard deviation becomes 0.125. Copying four original outcomes four times does not create the same distribution as sixteen independent observations.\n\nAssumptions Independent identically distributed Bernoulli trials and known p. No interval estimation, multiple-comparison correction or guarantee of real evaluation representativeness is implied.\n\n## Engineering example\n\nAt p=0.5 and n=4, success counts 0 through 4 have probabilities \\[1,4,6,4,1\\]/16. The expected rate is 0.5 and its standard deviation is 0.25. With n=16 independent trials the standard deviation becomes 0.125. Copying four original outcomes four times does not create the same distribution as sixteen independent observations.\n\n## Distinctions and tradeoffs\n\nA displayed rate is not an exact population property, and repeated correlated trials do not necessarily reduce uncertainty like independent samples.\n\nMore evaluations consume time and resources. Their informational value depends on task diversity, sampling and dependence, not solely the count.\n\nModel comparisons, reliability estimates and learning studies need transparent denominators and sampling assumptions. A small observed improvement may be consistent with ordinary variability.\n\n## Specifications and interfaces\n\nTeaching specimen\n\nSampling uncertainty\n\nAt p=0.5 and n=4, success counts 0 through 4 have probabilities \\[1,4,6,4,1\\]/16. The expected rate is 0.5 and its standard deviation is 0.25. With n=16 independent trials the standard deviation becomes 0.125. Copying four original outcomes four times does not create the same distribution as sixteen independent observations.\n\nEvidence boundary\n\nExplicit computed example\n\nIndependent identically distributed Bernoulli trials and known p. No interval estimation, multiple-comparison correction or guarantee of real evaluation representativeness is implied.\n\n### Interfaces\n\nInputs and convention Input\n\nDeclare the event counted as success, the number of independent trials and the fixed probability used by the model.\n\nComputed result and scope Output\n\nReconcile an exact sampling distribution with its mean and variance, and identify when a claimed sample count is not independent.\n\n## Check your understanding\n\n### What does the plotted distribution assume?\n\n1.  Every evaluation task has identical difficulty in the real world\n2.  A fixed known probability and independent Bernoulli trials\n3.  The probability was proved by the atlas\n\nAnswer and reasoning\n\nA fixed known probability and independent Bernoulli trials\n\nA fixed known probability and independent Bernoulli trials. Independent identically distributed Bernoulli trials and known p. No interval estimation, multiple-comparison correction or guarantee of real evaluation representativeness is implied.\n\n## Evidence and scope\n\nThe workbench executes the declared small mechanism. Numeric examples are authored and independently checked, not measurements from the cited research. Independent identically distributed Bernoulli trials and known p. No interval estimation, multiple-comparison correction or guarantee of real evaluation representativeness is implied.\n\nOpen Mathematical foundations for the bounded executable mechanism and its record. General numerical labs are an additional reference library.\n\n## Sources\n\n-   [Goodfellow, Bengio and Courville · Probability and information theory](https://www.deeplearningbook.org/contents/prob.html)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/sampling-estimation-uncertainty-5273ca7b00504d7c.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Probability, likelihood and stable information](https://semiaifoundry.com/learn/atlas/probability-conditioning-information/)\n\n### Related lessons\n\n-   [Evaluation & the AGI frontier](https://semiaifoundry.com/learn/atlas/evaluation/)\n-   [Scaling laws & resource allocation](https://semiaifoundry.com/learn/atlas/scaling-laws/)"
}