The mechanism
Custom inference chips express a workload choice, not a single architecture category. Microsoft’s Maia 200 announcement describes an inference-focused accelerator with a memory hierarchy and system connections designed together. Meta’s historical MTIA v1 disclosure centers ranking and recommendation inference and uses substantial SRAM with LPDDR5; its second-generation co-design discussion is a separate specimen. Those examples show why every custom accelerator need not reproduce a GPU’s balance of matrix arithmetic, cache and HBM. A recommendation lookup and a long-context language-model decode have different working sets and reuse, so comparing them solely by nominal operations per second hides the design rationale.
- Measure the actual operator mix and service objectives instead of assuming every AI workload resembles dense training.
- Estimate which working sets can remain in SRAM and which must cross a DRAM interface.
- Choose arithmetic, memory and networking resources that jointly address the measured bottleneck.
- Co-develop the compiler and kernels so the intended data placement is achievable by production models.
- Separate announcement, disclosed architecture and measured deployment evidence when judging the resulting platform.
Science and assumptions
Weighted service cost
Cost per accepted result is the charged cost rate divided by the running accepted-result rate times operational availability.
a explicitly names the source availability term. Cost and result rates must use the same accounting interval and service target.
Symbols and units
- C_result
- Time-based service cost per accepted result
currency/result
- C_time
- Charged service cost rate over the full accounting interval
currency/s
- R_useful
- Accepted-result rate while running
results/s
- a
- Operational availability fraction over that interval
fraction
Ctime is currency/s, Ruseful is accepted results/s while running, and availability is a dimensionless operational fraction.
Assumptions Consistent service quality and accounting boundary; model mix and fixed development costs need separate treatment.
Engineering example
Synthetic comparison: a lookup set with high reuse that fits local SRAM can avoid repeated external reads; doubling arithmetic alone leaves that traffic unchanged. A different model whose weights exceed SRAM requires a new memory budget.
Distinctions and tradeoffs
Custom means designed for a chosen workload and organization; it does not automatically mean faster, cheaper or appropriate for every model.
A provider can optimize frequent operations aggressively, but must fund the compiler, kernels and adaptation needed when models change.
Large service volumes can justify specialization only when the resulting platform delivers acceptable quality, reliability and useful throughput.
Specifications and interfaces
- Maia specimen
- Maia 200, January 2026 announcement
Provider-reported inference design and deployment statements.
- MTIA specimen
- MTIA v1 and separate 2024 co-design discussion
Historical ranking/recommendation context, not a claim about all future MTIA generations.
Interfaces
- Production workload traces Input
- Operator shapes, memory reuse and latency targets
- Specialized platform Output
- Supported inference graph under a defined service target
Check your understanding
What evidence best motivates more on-chip SRAM for an inference design?
- A useful working set is repeatedly accessed and can fit there
- A competitor advertises more peak operations
- The chip name contains AI
Answer and reasoning
A useful working set is repeatedly accessed and can fit there
Measured reuse and a fitting working set provide a causal link between SRAM allocation and reduced external traffic.
Evidence and scope
Maia specifications are provider-reported. The inaccessible next-generation Meta blog was replaced by accessible first-generation and engineering sources; no unsupported current MTIA peak is claimed.
No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.