{
  "schemaVersion": 1,
  "id": "atlas-lesson-cuda",
  "title": "CUDA programming and runtime layer",
  "category": "learning",
  "format": "Atlas lesson",
  "description": "Provides a software model and interfaces for executing parallel work on NVIDIA GPUs.",
  "canonicalUrl": "https://semiaifoundry.com/learn/atlas/cuda/",
  "publishedDate": null,
  "revisionDate": "2026-10-07",
  "author": null,
  "topic": "Execution software",
  "series": null,
  "license": null,
  "rightsUrl": "https://semiaifoundry.com/ai-atlas/licenses.html",
  "version": "1.3.0",
  "status": "Published educational lesson",
  "evidenceScope": "This applies the lesson’s fixed-workload model, not a guaranteed CUDA speedup. Transfers, launches, changed algorithms, problem size and CPU contention can change both the parallel fraction and overhead.",
  "contentCoverage": "published-lesson",
  "artifactUrl": "https://semiaifoundry.com/ai-atlas/lesson-data/cuda-38a2940bde1cde9f.json",
  "interactiveUrl": "https://semiaifoundry.com/ai-atlas/#cuda",
  "related": [],
  "supersedes": [],
  "supersededBy": [],
  "contentHash": "62e4fd337d6b095fd3a9f4a3afea346e4c4f84700647f96eef809d91cc701832",
  "contentLength": 5019,
  "jsonUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-cuda.json",
  "markdownUrl": "https://semiaifoundry.com/agents/publications/atlas-lesson-cuda.md",
  "sections": [
    {
      "id": "section-1",
      "title": "CUDA programming and runtime layer",
      "start": 35,
      "end": 370
    },
    {
      "id": "section-2",
      "title": "The mechanism",
      "start": 370,
      "end": 1373
    },
    {
      "id": "section-3",
      "title": "Science and assumptions",
      "start": 1373,
      "end": 1401
    },
    {
      "id": "section-4",
      "title": "Amdahl speedup bound",
      "start": 1401,
      "end": 2196
    },
    {
      "id": "section-5",
      "title": "Engineering example",
      "start": 2196,
      "end": 2454
    },
    {
      "id": "section-6",
      "title": "Distinctions and tradeoffs",
      "start": 2454,
      "end": 2841
    },
    {
      "id": "section-7",
      "title": "Specifications and interfaces",
      "start": 2841,
      "end": 3120
    },
    {
      "id": "section-8",
      "title": "Interfaces",
      "start": 3120,
      "end": 3262
    },
    {
      "id": "section-9",
      "title": "Check your understanding",
      "start": 3262,
      "end": 3291
    },
    {
      "id": "section-10",
      "title": "Where can an AI framework use CUDA?",
      "start": 3291,
      "end": 3682
    },
    {
      "id": "section-11",
      "title": "Evidence and scope",
      "start": 3682,
      "end": 4063
    },
    {
      "id": "section-12",
      "title": "Sources",
      "start": 4063,
      "end": 4428
    },
    {
      "id": "section-13",
      "title": "Continue exploring",
      "start": 4428,
      "end": 4451
    },
    {
      "id": "section-14",
      "title": "Foundations",
      "start": 4451,
      "end": 4629
    },
    {
      "id": "section-15",
      "title": "Related lessons",
      "start": 4629,
      "end": 5019
    }
  ],
  "citations": [
    {
      "title": "Explore in Atlas",
      "url": "https://semiaifoundry.com/ai-atlas/#cuda"
    },
    {
      "title": "NVIDIA — CUDA FAQ",
      "url": "https://developer.nvidia.com/cuda/faq"
    },
    {
      "title": "Hill and Marty — Amdahl’s Law in the Multicore Era",
      "url": "https://research.cs.wisc.edu/multifacet/amdahl/"
    },
    {
      "title": "Published lesson data",
      "url": "https://semiaifoundry.com/ai-atlas/lesson-data/cuda-38a2940bde1cde9f.json"
    },
    {
      "title": "Atlas releases and licensing",
      "url": "https://semiaifoundry.com/ai-atlas/licenses.html"
    },
    {
      "title": "Host CPU",
      "url": "https://semiaifoundry.com/learn/atlas/host-cpu/"
    },
    {
      "title": "Streaming multiprocessor",
      "url": "https://semiaifoundry.com/learn/atlas/streaming-multiprocessor/"
    },
    {
      "title": "GPU kernel and thread hierarchy",
      "url": "https://semiaifoundry.com/learn/atlas/kernel/"
    },
    {
      "title": "Compiler, intermediate code, and device executable",
      "url": "https://semiaifoundry.com/learn/atlas/compiler/"
    },
    {
      "title": "Collective communication library",
      "url": "https://semiaifoundry.com/learn/atlas/nccl/"
    },
    {
      "title": "Inference serving and request scheduling",
      "url": "https://semiaifoundry.com/learn/atlas/serving/"
    }
  ],
  "recordHash": "aa0ae2b28d979b0fa3f570ef5676ff1c39d5caa7b1c274f5bf382bc55ded3fe3",
  "contentFormat": "text/markdown",
  "content": "Execution software · Atlas v1.3.0\n\n# CUDA programming and runtime layer\n\nProvides a software model and interfaces for executing parallel work on NVIDIA GPUs.\n\n[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#cuda)Software abstraction; no physical size\n\nLearning objectiveApply Amdahl’s bound to distinguish acceleration of device work from whole-program speedup.\n\n## The mechanism\n\nCUDA connects host-side programs to GPU execution. A program prepares data, invokes device work, and coordinates memory and synchronization through runtime or driver interfaces and libraries. The programming model gives developers a way to describe many related operations without manually addressing every physical arithmetic lane. Frameworks commonly use CUDA indirectly through optimized libraries or generated kernels. The atlas represents this layer as a translation boundary between tensor operations and device launches, not as a physical chip component. CUDA alone is not an AI model, a complete training framework, or a promise of optimal performance. Data movement, launch structure, numerical formats, and kernel behavior still determine how well the application uses the hardware.\n\n1.  Prepare operations and data from the host-side application.\n2.  Invoke GPU kernels or accelerated library routines.\n3.  Coordinate completion and use the produced results in later work.\n\n## Science and assumptions\n\n### Amdahl speedup bound\n\n$$\nS(N)=\\frac{1}{(1-p)+p/N}\n$$\n\nOverall speedup is the reciprocal of unaccelerated time fraction plus the accelerated fraction divided by its N-fold speedup.\n\nSymbols and units\n\nS(N)\n\nOverall speedup with N-fold acceleration of the selected fraction\n\ndimensionless\n\np\n\nAccelerable fraction of baseline execution time\n\nfraction\n\nN\n\nAcceleration factor for that fraction\n\npositive dimensionless ratio\n\nIf fraction p of baseline time can be accelerated N-fold, the remaining fraction still limits total speedup. Host work and transfers can therefore dominate after a GPU kernel becomes very fast.\n\nAssumptions Fixed workload, ideal acceleration of the selected fraction, and no added overhead; real GPU applications may have additional communication and scaling effects.\n\n## Engineering example\n\nSynthetic program spends 90% of its original time in work accelerated by 10×, with the other 10% unchanged. Ideal speedup is 1/(0.1 + 0.9/10) ≈ 5.26×. Even unlimited acceleration of that 90% would cap the same fixed workload at 10×.\n\n## Distinctions and tradeoffs\n\nCUDA is a programming platform, not a neural network and not a transistor-level execution diagram.\n\nHigh-level interfaces improve programmability. Efficient execution still requires suitable algorithms and memory behavior.\n\nCUDA is one major bridge from AI frameworks to accelerators. It helps software turn tensor operations into executable device work.\n\n## Specifications and interfaces\n\nExecution roles\n\nHost + device\n\nCooperating CPU-side control and GPU-side work.\n\nWork expression\n\nKernels and libraries\n\nFrameworks may call these indirectly.\n\nScope\n\nNVIDIA GPU platform\n\nOther accelerator ecosystems have different interfaces.\n\n### Interfaces\n\nApplication operations Input\n\nCode, tensors, and launch requests\n\nGPU execution Output\n\nDevice work and synchronized results\n\n## Check your understanding\n\n### Where can an AI framework use CUDA?\n\n1.  Only during wafer fabrication\n2.  Through runtime interfaces, libraries, or generated GPU kernels\n3.  Only after a model reaches AGI\n\nAnswer and reasoning\n\nThrough runtime interfaces, libraries, or generated GPU kernels\n\nFramework operations can dispatch work through several CUDA-layer mechanisms, often without users writing kernels directly.\n\n## Evidence and scope\n\nThis applies the lesson’s fixed-workload model, not a guaranteed CUDA speedup. Transfers, launches, changed algorithms, problem size and CPU contention can change both the parallel fraction and overhead.\n\nUse the execution investigation for the shared tensor-to-memory-to-result thread; no current standalone calculator models this lesson’s full mechanism.\n\n## Sources\n\n-   [NVIDIA — CUDA FAQ](https://developer.nvidia.com/cuda/faq)\n-   [Hill and Marty — Amdahl’s Law in the Multicore Era](https://research.cs.wisc.edu/multifacet/amdahl/)\n\n[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/cuda-38a2940bde1cde9f.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)\n\n## Continue exploring\n\n### Foundations\n\n-   [Host CPU](https://semiaifoundry.com/learn/atlas/host-cpu/)\n-   [Streaming multiprocessor](https://semiaifoundry.com/learn/atlas/streaming-multiprocessor/)\n\n### Related lessons\n\n-   [GPU kernel and thread hierarchy](https://semiaifoundry.com/learn/atlas/kernel/)\n-   [Compiler, intermediate code, and device executable](https://semiaifoundry.com/learn/atlas/compiler/)\n-   [Collective communication library](https://semiaifoundry.com/learn/atlas/nccl/)\n-   [Inference serving and request scheduling](https://semiaifoundry.com/learn/atlas/serving/)"
}