Three years after pausing work on reverse-engineering Apple’s Neural Engine (ANE), a renewed investigation has mapped the full internal architecture of the ANE in the M1 chip. The ANE, first introduced in the A11 Bionic in 2017, was designed primarily for convolutional neural networks (CNNs) with predictable data reuse patterns. However, the rise of transformer models, which rely on less predictable memory access patterns, challenged these assumptions.
The M1 ANE features 16 compute cores, each containing 128 parallel multiply-accumulate (MAC) units capable of processing FP16 or INT8 data. These cores perform dot products fundamental to neural network operations. However, the unique strength of the ANE lies not in the MAC units themselves but in the specialized dataflow architecture that manages how data moves through the cores, optimized for dense image processing workloads.
Unlike general-purpose GPUs, the ANE operates as a fixed-function dataflow engine without a traditional instruction set architecture. Instead, it executes tasks described by precompiled command streams that configure hardware registers directly. This design enables deterministic, low-latency execution but limits flexibility.
Memory architecture plays a critical role in ANE performance. The chip includes a 2 MiB shared L2 SRAM and per-core 64 KiB kernel memory (KMem) and L1 staging areas. Kernels (weights) are loaded into private per-core KMem, while input and output tiles reside in shared L2 memory. This separation reflects an operand asymmetry assumption: kernels are reused extensively, while inputs and outputs stream through.
Bandwidth constraints significantly impact the ANE’s throughput. Although the ANE achieves 11 TOP/s at 68 GB/s system DRAM bandwidth, the arithmetic intensity required to saturate compute units demands 162 operations per byte of DRAM traffic, indicating that local memory reuse is essential. The ANE cannot stream all operands directly from DRAM without severe bottlenecks.
Comparisons with the GPU on the same system reveal that the ANE’s DMA engines for kernel and tile data achieve lower bandwidth individually (38 GB/s and 60 GB/s respectively) than the GPU’s 78 GB/s. Moreover, these DMA engines operate serially rather than in parallel, further limiting effective bandwidth.
This analysis helps explain Apple’s architectural shift in the upcoming M5 chip, where ANE cores are integrated within GPU cores, signaling a move away from standalone NPUs. The ANE’s fixed-function design and memory limitations make it less suited for modern transformer workloads, which benefit from the flexibility and higher bandwidth of GPUs.
Understanding the ANE’s architecture and constraints provides insight into the evolving landscape of AI hardware, highlighting the trade-offs between specialized accelerators and general-purpose GPUs in supporting diverse machine learning models.