{"id":"aws-neuron-dataflow","level":"compute","name":"Trainium and Inferentia execution paths","shortName":"Trainium and Inferentia execution paths","role":"Combines specialized engines, local buffers and a compiler-managed accelerator runtime.","description":"An AWS Neuron program targets a machine with multiple kinds of execution and data movement resources. Matrix-heavy work, elementwise work and transfers need compatible schedules and buffer layouts. Trainium2 and Inferentia2 are named chip generations, while Trn2 and Inf2 describe instance families that assemble chips with host and interconnect resources. AWS also distinguishes physical and logical NeuronCore configurations, making a core count meaningful only with its definition. The compiler and runtime are part of the deployment path: supported operators, executable shapes and collective placement determine whether an application can use the disclosed hardware effectively.","specs":[{"label":"Specimen","value":"Trainium2 and Inferentia2","context":"Historical disclosed generations, not interchangeable chip specifications."},{"label":"Memory units","value":"GiB capacity; TB/s link rate","context":"Preserve the source’s binary capacity and decimal bandwidth labels."}],"interfaces":[{"name":"Neuron executable","direction":"in","signal":"Operations, shapes, buffer assignments and placements"},{"name":"Specialized engines","direction":"out","signal":"Tensor results and collective data transfers"}],"tradeoff":"Explicit scheduling can improve reuse and overlap, but unusual shapes or unsupported operators can require recompilation, custom kernels or fallback execution.","whyAI":"Training and serving need a complete supported model path; a high matrix peak does not establish end-to-end operator coverage or latency.","related":["systolic-array-dataflow","compiler-layout-lowering","runtime-device-dispatch"],"sources":[{"title":"Trainium2 Architecture","url":"https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium2.html"},{"title":"Amazon EC2 Inf2 Architecture","url":"https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/inf2-arch.html"}],"scale":"Processing element to accelerator package; array tiles exaggerated","kind":"architecture","branch":"Processing architectures","subbranch":"Explicit dataflow and custom accelerators","mechanism":["Choose a chip and instance boundary before counting cores, memory or interconnect resources.","Compile the model into operations supported by the selected Neuron software and hardware generation.","Place live tiles in on-chip buffers and arrange transfers from HBM to feed the relevant compute engines.","Schedule collective transfers and independent arithmetic where dependencies permit useful overlap.","Profile the complete application and record software versions, logical-core mode and shape configuration with any result."],"misconception":"Trainium and Inferentia names do not mean training and inference are physically exclusive operations; support and performance depend on the generation and software path.","check":{"question":"Why must an Inf2 instance memory total be distinguished from an Inferentia2 chip value?","options":["The instance can contain several accelerator chips","Instances contain no host resources","Memory capacity is always bandwidth"],"answer":0,"explanation":"An instance aggregates a defined configuration of chips and hosts; comparing its total against a single chip changes the denominator."},"science":[{"title":"Engine schedule bound","equation":"Tstep ≥ max(Ttensor, Tvector, Ttransfer)","explanation":"Each term is the time required on one resource class in seconds. Parallel engines only overlap work allowed by dependencies.","assumptions":"A lower bound for sufficiently buffered execution; engine contention and serial edges add time."}],"prerequisites":["systolic-array-dataflow","compiler"],"learningObjective":"Identify chip, instance and logical-core boundaries in an accelerator disclosure.","workstream":"W07","visualFamily":"architecture-array","engineeringExample":"Synthetic graph: 5 ms of matrix work, 2 ms of vector work and 4 ms of transfers could approach a 5 ms bound if fully overlapped. A serial dependency ordering can instead require 11 ms.","evidenceNotes":"Published memory values are retained in evidence claims. The example is synthetic and does not imply equal features or performance across Trainium2 and Inferentia2.","labId":"library","scope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.","investigationId":"execution","labScope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions."}