{"id":"inference-custom-silicon","level":"compute","name":"Custom inference silicon: Maia and MTIA","shortName":"Custom inference silicon","role":"Allocates compute and memory around a provider’s measured inference needs.","description":"Custom inference chips express a workload choice, not a single architecture category. Microsoft’s Maia 200 announcement describes an inference-focused accelerator with a memory hierarchy and system connections designed together. Meta’s historical MTIA v1 disclosure centers ranking and recommendation inference and uses substantial SRAM with LPDDR5; its second-generation co-design discussion is a separate specimen. Those examples show why every custom accelerator need not reproduce a GPU’s balance of matrix arithmetic, cache and HBM. A recommendation lookup and a long-context language-model decode have different working sets and reuse, so comparing them solely by nominal operations per second hides the design rationale.","specs":[{"label":"Maia specimen","value":"Maia 200, January 2026 announcement","context":"Provider-reported inference design and deployment statements."},{"label":"MTIA specimen","value":"MTIA v1 and separate 2024 co-design discussion","context":"Historical ranking/recommendation context, not a claim about all future MTIA generations."}],"interfaces":[{"name":"Production workload traces","direction":"in","signal":"Operator shapes, memory reuse and latency targets"},{"name":"Specialized platform","direction":"out","signal":"Supported inference graph under a defined service target"}],"tradeoff":"A provider can optimize frequent operations aggressively, but must fund the compiler, kernels and adaptation needed when models change.","whyAI":"Large service volumes can justify specialization only when the resulting platform delivers acceptable quality, reliability and useful throughput.","related":["chip-workload-contract","memory-interface-choice","chip-volume-economics"],"sources":[{"title":"Maia 200: The AI accelerator built for inference","url":"https://blogs.microsoft.com/blog/2026/01/26/maia-200-the-ai-accelerator-built-for-inference/"},{"title":"MTIA v1: Meta’s first-generation AI inference accelerator","url":"https://ai.meta.com/blog/meta-training-inference-accelerator-AI-MTIA/"},{"title":"Inside the hardware and co-design of MTIA","url":"https://engineering.fb.com/2024/08/22/ml-applications/meta-mtia-hardware-co-design/"}],"scale":"Processing element to accelerator package; array tiles exaggerated","kind":"architecture","branch":"Processing architectures","subbranch":"Explicit dataflow and custom accelerators","mechanism":["Measure the actual operator mix and service objectives instead of assuming every AI workload resembles dense training.","Estimate which working sets can remain in SRAM and which must cross a DRAM interface.","Choose arithmetic, memory and networking resources that jointly address the measured bottleneck.","Co-develop the compiler and kernels so the intended data placement is achievable by production models.","Separate announcement, disclosed architecture and measured deployment evidence when judging the resulting platform."],"misconception":"Custom means designed for a chosen workload and organization; it does not automatically mean faster, cheaper or appropriate for every model.","check":{"question":"What evidence best motivates more on-chip SRAM for an inference design?","options":["A useful working set is repeatedly accessed and can fit there","A competitor advertises more peak operations","The chip name contains AI"],"answer":0,"explanation":"Measured reuse and a fitting working set provide a causal link between SRAM allocation and reduced external traffic."},"science":[{"title":"Weighted service cost","equation":"Cresult = Ctime / (Ruseful × availability)","explanation":"Ctime is currency/s, Ruseful is accepted results/s while running, and availability is a dimensionless operational fraction.","assumptions":"Consistent service quality and accounting boundary; model mix and fixed development costs need separate treatment."}],"prerequisites":["memory-hierarchy-route","host-cpu"],"learningObjective":"Explain custom inference design through workload and software requirements rather than brand categories.","workstream":"W07","visualFamily":"architecture-array","engineeringExample":"Synthetic comparison: a lookup set with high reuse that fits local SRAM can avoid repeated external reads; doubling arithmetic alone leaves that traffic unchanged. A different model whose weights exceed SRAM requires a new memory budget.","evidenceNotes":"Maia specifications are provider-reported. The inaccessible next-generation Meta blog was replaced by accessible first-generation and engineering sources; no unsupported current MTIA peak is claimed.","labId":"library","scope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions.","investigationId":"execution","labScope":"No current calculator directly models this lesson’s main mechanism. Use the library for related bounds while retaining their stated omissions."}