{"id":"runtime-device-dispatch","level":"software","name":"Runtime, driver and device dispatch","shortName":"Runtime, driver and device dispatch","role":"Turns compiled work into ordered execution with real buffer lifetimes and completion events.","description":"Compilation produces an executable plan, but a runtime must allocate or bind buffers, place work on devices and submit commands through drivers. Device queues execute kernels and transfers according to dependencies. Asynchronous submission lets the CPU continue before the GPU finishes, so a returned launch call is not evidence that a tensor result is ready. Events and synchronization establish when consumers can safely read or recycle storage. The driver also manages device access and address mappings. This completes the graph-to-hardware chain: a logical operation becomes instructions operating on physical operands under an enforced execution order.\n\nBuffer ownership and completion events connect an asynchronous command queue to correct values. Host code returning from enqueue does not establish that a device has finished reading or writing a buffer. A dependent consumer must wait for the required event, while unrelated work may overlap if the runtime and hardware permit it. The resulting timeline is a causal dependency model; a drawing of software layers alone cannot demonstrate the hazards.","specs":[{"label":"Submission versus completion","value":"Two distinct timestamps","context":"CPU enqueue time is not device execution time."},{"label":"Dependency","value":"Event or ordering constraint","context":"Correctness requires transfers and producers to finish before consumers read."}],"interfaces":[{"name":"Compiled executable and inputs","direction":"in","signal":"Device code, buffer bindings and scheduling dependencies"},{"name":"Device completion","direction":"out","signal":"Results, events, timing and error status"}],"tradeoff":"Asynchronous queues allow overlap but complicate ownership and measurement; excessive global synchronization simplifies reasoning while creating idle gaps.","whyAI":"Short decode kernels can expose launch and synchronization overhead, while large training kernels benefit from overlapping input transfer and communication.","related":["cuda","kernel","compiler","graph-tensor-ir","compiler-layout-lowering","telemetry-traces-evaluation"],"sources":[{"title":"CUDA C++ Programming Guide","url":"https://docs.nvidia.com/cuda/cuda-c-programming-guide/"},{"title":"XLA:GPU Architecture Overview","url":"https://openxla.org/xla/gpu_architecture"}],"scale":"Software abstraction layers mapped to physical memory and compute","kind":"workload","branch":"Model–machine co-design","subbranch":"Graph to executable operation","mechanism":["Bind actual input buffers and verify the executable matches the selected device and shape assumptions.","Submit transfers and kernels to execution queues with explicit or guaranteed dependencies.","Let the device fetch instructions and stage operands through its memory hierarchy into execution resources.","Record completion events and defer dependent reads or buffer reuse until those events establish safety.","Collect synchronized timing and error information so performance and correctness reflect finished device work.","A buffer remains live until all required consumers complete, according to the actual stream/event dependency contract.","Timing reports distinguish CPU submission work, queued delay, device service and synchronization rather than treating them as one interchangeable duration."],"misconception":"A fast host launch call does not mean the kernel completed quickly; asynchronous APIs separate enqueueing from execution.","check":{"question":"When is an asynchronously filled buffer safe for a dependent consumer?","options":["After the required completion dependency is satisfied","As soon as the launch function returns","Only after the computer restarts"],"answer":0,"explanation":"The dependency must establish that the producer and any transfer have finished before the consumer accesses the buffer."},"science":[{"title":"Launch-bound useful time","equation":"Ttotal ≥ Nlaunch·Llaunch + Tserial","explanation":"Nlaunch is launch count, Llaunch seconds per non-overlapped launch, and Tserial remaining unavoidable serial work in seconds.","assumptions":"A simplified bound; graphs, batching and overlapping submission can reduce exposed launch overhead."}],"prerequisites":["kernel","compiler-layout-lowering"],"learningObjective":"Trace submission, device execution and completion without confusing their timing.","workstream":"W11","visualFamily":"software-flow","engineeringExample":"Synthetic sequence: a CPU enqueues a 5 ms device operation in 20 μs. Timing only the call reports submission overhead; an event after completion is needed to measure the operation’s elapsed device work.\n\nSuppose a CPU submits a kernel in 0.02 ms and the kernel needs 5 ms of device service. Reusing its input at 0.02 ms is unsafe under a contract requiring the buffer through completion. With immediate service and no other overhead, the earliest safe reuse is 5 ms after submission began. Waiting 5 ms by an assumed timer is still weaker evidence than the actual completion event when queueing can vary.","evidenceNotes":"Queue, graph and event semantics are stack-specific. The displayed path is a representative CUDA/OpenXLA flow, not a claim that every runtime uses identical objects.\n\nThis is a worked dependency and lifetime example. It does not execute a CUDA stream, driver or real device event and does not change the original bounded calculator scope.","checks":[{"question":"What permits safe reuse of a buffer consumed asynchronously?","options":["The CPU enqueue call returned","The required device consumer completion is established","The buffer name has not changed"],"answer":1,"explanation":"Submission and completion are different events; the consumer lifetime determines reuse."}],"labId":"token-system-path","scope":"Follow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms.","investigationId":"execution","labScope":"Follow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms."}