# Trainium and Inferentia execution paths

Canonical: https://semiaifoundry.com/learn/atlas/aws-neuron-dataflow/
Revision: 2026-10-07
Rights: https://semiaifoundry.com/ai-atlas/licenses.html

Processing architectures · Atlas v1.3.0

# Trainium and Inferentia execution paths

Combines specialized engines, local buffers and a compiler-managed accelerator runtime.

[Explore in Atlas](https://semiaifoundry.com/ai-atlas/#aws-neuron-dataflow)Processing element to accelerator package; array tiles exaggerated

Learning objectiveIdentify chip, instance and logical-core boundaries in an accelerator disclosure.

## The mechanism

An AWS Neuron program targets a machine with multiple kinds of execution and data movement resources. Matrix-heavy work, elementwise work and transfers need compatible schedules and buffer layouts. Trainium2 and Inferentia2 are named chip generations, while Trn2 and Inf2 describe instance families that assemble chips with host and interconnect resources. AWS also distinguishes physical and logical NeuronCore configurations, making a core count meaningful only with its definition. The compiler and runtime are part of the deployment path: supported operators, executable shapes and collective placement determine whether an application can use the disclosed hardware effectively.

1.  Choose a chip and instance boundary before counting cores, memory or interconnect resources.
2.  Compile the model into operations supported by the selected Neuron software and hardware generation.
3.  Place live tiles in on-chip buffers and arrange transfers from HBM to feed the relevant compute engines.
4.  Schedule collective transfers and independent arithmetic where dependencies permit useful overlap.
5.  Profile the complete application and record software versions, logical-core mode and shape configuration with any result.

## Science and assumptions

### Engine schedule bound

$$
T_{\mathrm{step}}\ge\max(T_{\mathrm{tensor}},T_{\mathrm{vector}},T_{\mathrm{transfer}})
$$

Step time is at least the largest resource-time requirement among tensor, vector and transfer engines.

Symbols and units

T\_step

Step duration lower bound

s

T\_tensor

Time required on tensor engines

s

T\_vector

Time required on vector engines

s

T\_transfer

Time required on transfer resources

s

Each term is the time required on one resource class in seconds. Parallel engines only overlap work allowed by dependencies.

Assumptions A lower bound for sufficiently buffered execution; engine contention and serial edges add time.

## Engineering example

Synthetic graph: 5 ms of matrix work, 2 ms of vector work and 4 ms of transfers could approach a 5 ms bound if fully overlapped. A serial dependency ordering can instead require 11 ms.

## Distinctions and tradeoffs

Trainium and Inferentia names do not mean training and inference are physically exclusive operations; support and performance depend on the generation and software path.

Explicit scheduling can improve reuse and overlap, but unusual shapes or unsupported operators can require recompilation, custom kernels or fallback execution.

Training and serving need a complete supported model path; a high matrix peak does not establish end-to-end operator coverage or latency.

## Specifications and interfaces

Specimen

Trainium2 and Inferentia2

Historical disclosed generations, not interchangeable chip specifications.

Memory units

GiB capacity; TB/s link rate

Preserve the source’s binary capacity and decimal bandwidth labels.

### Interfaces

Neuron executable Input

Operations, shapes, buffer assignments and placements

Specialized engines Output

Tensor results and collective data transfers

## Check your understanding

### Why must an Inf2 instance memory total be distinguished from an Inferentia2 chip value?

1.  The instance can contain several accelerator chips
2.  Instances contain no host resources
3.  Memory capacity is always bandwidth

Answer and reasoning

The instance can contain several accelerator chips

An instance aggregates a defined configuration of chips and hosts; comparing its total against a single chip changes the denominator.

## Evidence and scope

Published memory values are retained in evidence claims. The example is synthetic and does not imply equal features or performance across Trainium2 and Inferentia2.

No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.

## Sources

-   [Trainium2 Architecture](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trainium2.html)
-   [Amazon EC2 Inf2 Architecture](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/inf2-arch.html)

[Published lesson data](https://semiaifoundry.com/ai-atlas/lesson-data/aws-neuron-dataflow-a76c9fd8110a4cf1.json) · [Atlas releases and licensing](https://semiaifoundry.com/ai-atlas/licenses.html)

## Continue exploring

### Foundations

-   [Systolic arrays and tensor dataflow](https://semiaifoundry.com/learn/atlas/systolic-array-dataflow/)
-   [Compiler, intermediate code, and device executable](https://semiaifoundry.com/learn/atlas/compiler/)

### Related lessons

-   [Compiler layout, lowering and buffer planning](https://semiaifoundry.com/learn/atlas/compiler-layout-lowering/)
-   [Runtime, driver and device dispatch](https://semiaifoundry.com/learn/atlas/runtime-device-dispatch/)