Runtime, driver and device dispatch

The mechanism

Compilation produces an executable plan, but a runtime must allocate or bind buffers, place work on devices and submit commands through drivers. Device queues execute kernels and transfers according to dependencies. Asynchronous submission lets the CPU continue before the GPU finishes, so a returned launch call is not evidence that a tensor result is ready. Events and synchronization establish when consumers can safely read or recycle storage. The driver also manages device access and address mappings. This completes the graph-to-hardware chain: a logical operation becomes instructions operating on physical operands under an enforced execution order.

Buffer ownership and completion events connect an asynchronous command queue to correct values. Host code returning from enqueue does not establish that a device has finished reading or writing a buffer. A dependent consumer must wait for the required event, while unrelated work may overlap if the runtime and hardware permit it. The resulting timeline is a causal dependency model; a drawing of software layers alone cannot demonstrate the hazards.

Bind actual input buffers and verify the executable matches the selected device and shape assumptions.
Submit transfers and kernels to execution queues with explicit or guaranteed dependencies.
Let the device fetch instructions and stage operands through its memory hierarchy into execution resources.
Record completion events and defer dependent reads or buffer reuse until those events establish safety.
Collect synchronized timing and error information so performance and correctness reflect finished device work.
A buffer remains live until all required consumers complete, according to the actual stream/event dependency contract.
Timing reports distinguish CPU submission work, queued delay, device service and synchronization rather than treating them as one interchangeable duration.

Science and assumptions

Launch-bound useful time

T_{\mathrm{total}}\ge N_{\mathrm{launch}}L_{\mathrm{launch}}+T_{\mathrm{serial}}

Total time is at least nonoverlapped launch count times launch overhead plus other unavoidable serial work.

Symbols and units

T_total: Total execution-time lower bound
s
N_launch: Nonoverlapped launch count
launches
L_launch: Exposed overhead per such launch
s/launch
T_serial: Other unavoidable serial work, excluding those launch costs
s

Nlaunch is launch count, Llaunch seconds per non-overlapped launch, and Tserial remaining unavoidable serial work in seconds.

Assumptions A simplified bound; graphs, batching and overlapping submission can reduce exposed launch overhead.

Engineering example

Synthetic sequence: a CPU enqueues a 5 ms device operation in 20 μs. Timing only the call reports submission overhead; an event after completion is needed to measure the operation’s elapsed device work.

Suppose a CPU submits a kernel in 0.02 ms and the kernel needs 5 ms of device service. Reusing its input at 0.02 ms is unsafe under a contract requiring the buffer through completion. With immediate service and no other overhead, the earliest safe reuse is 5 ms after submission began. Waiting 5 ms by an assumed timer is still weaker evidence than the actual completion event when queueing can vary.

Distinctions and tradeoffs

A fast host launch call does not mean the kernel completed quickly; asynchronous APIs separate enqueueing from execution.

Asynchronous queues allow overlap but complicate ownership and measurement; excessive global synchronization simplifies reasoning while creating idle gaps.

Short decode kernels can expose launch and synchronization overhead, while large training kernels benefit from overlapping input transfer and communication.

Specifications and interfaces

Submission versus completion: Two distinct timestamps
CPU enqueue time is not device execution time.
Dependency: Event or ordering constraint
Correctness requires transfers and producers to finish before consumers read.

Interfaces

Compiled executable and inputs Input: Device code, buffer bindings and scheduling dependencies
Device completion Output: Results, events, timing and error status

Check your understanding

When is an asynchronously filled buffer safe for a dependent consumer?

After the required completion dependency is satisfied
As soon as the launch function returns
Only after the computer restarts

Answer and reasoning

After the required completion dependency is satisfied

The dependency must establish that the producer and any transfer have finished before the consumer accesses the buffer.

What permits safe reuse of a buffer consumed asynchronously?

The CPU enqueue call returned
The required device consumer completion is established
The buffer name has not changed

Answer and reasoning

The required device consumer completion is established

Submission and completion are different events; the consumer lifetime determines reuse.

Evidence and scope

Queue, graph and event semantics are stack-specific. The displayed path is a representative CUDA/OpenXLA flow, not a claim that every runtime uses identical objects.

This is a worked dependency and lifetime example. It does not execute a CUDA stream, driver or real device event and does not change the original bounded calculator scope.

Follow a modeled batched decode critical path and energy budget; prefill, expert routing and scheduling differences are comparison context rather than separately simulated algorithms.