This article details the first comparative benchmark of commercially available μNPU platforms, alongside fine-grained, independent results for several previously unevaluated platforms.
Our analysis uncovers expected trends and surprising gaps between hardware specifications and measured performance. Firstly, peak _giga operations per second_ (GOPS), an accelerator's compute throughput, is a weak predictor of real-world performance [3]. Secondly, memory movement can dominate end-to-end latency, unlike in larger-scale hardware architectures.
The µNPU design accelerates tensor operations using systolic arrays of processing elements (PEs). Each PE contains local multiply-accumulate units and often local weight memory, reducing contention and improving parallelism. The PE array connects to larger buffers and on-chip memory, which may be accelerator SRAM, tightly coupled memory, shared MCU SRAM, or external memory.
The benchmark covers a diverse set of com-mercially available platforms, alongside MCUs without dedicated neural hardware to quantify the benefit of acceleration. Our selection spans 4.8 to 512 GOPS, 128 KB to 2 MB SRAM, and arithmetic support from 1-bit quantized to floating-point execution.
{
"x_axis": {
"label": "Peak Power",
"unit": "W"
},
"y_axis": {
"label": "Peak Performance",
"unit": "GOPs/sec"
},
"series": [
{
"name": "MCU-scale NPUs",
"points": [
{
"x": 0.05,
"y": 5,
"label": "MAX78k"
},
{
"x": 0.1,
"y": 10,
"label": "GAP8"
},
{
"x": 0.12,
"y": 15,
"label": "GAP9"
},
{
"x": 0.15,
"y": 60,
"label": "HX6538 WE2"
}
]
},
{
"name": "SoC NPUs/TPUs",
"points": [
{
"x": 2,
"y": 3000,
"label": "Coral TPU"
},
{
"x": 10,
"y": 20000,
"label": "Xavier NX"
},
{
"x": 30,
"y": 30000,
"label": "Xavier AGX"
}
]
},
{
"name": "High-end GPU cluster",
"points": [
{
"x": 300,
"y": 100000,
"label": "V100"
},
{
"x": 400,
"y": 300000,
"label": "A100"
},
{
"x": 700,
"y": 1000000,
"label": "H100"
}
]
}
],
"precision": "estimated"
}TABLE 1. The platforms used in our benchmark, and their specifications [10-13].
| MCU | CPU(s) | NPU | Flash | RAM | GOPs (max) | Bit Cap. | |---|---|---|---|---|---|---| | NPUs | | | | | | | | MAX78000 | Cortex-M4; RISC-V | MAXIM-own | 512 KB | 512 KB NPU; 128 KB CPU | 30 | 1, 2, 4, 8 | | HX-WE2 (Corstone-300) | Cortex-M55 | Ethos-U55 | 16 MB | 2 MB SRAM; 512 KB TCM | 512 | 8, 16, 32 | | NXP-MCXN947 | Cortex-M33 (2) | eIQ Neutron | 2 MB | 512 KB | 4.8 | 8 | | GAP8 | RISC-V | GAP-own | 20 MB L3 | 512 KB L2; 8 MB L3 | 22.65 | 8, 16 | | Baseline MCUs | | | | | | | | STM32H74A3ZI | Cortex-M7 | - | 2 MB | 1.4 MB | - | 8, 16, 32 | | ESP32 | Tensilica LX6 | - | 4 MB | 520 KB | - | 8, 16, 32 |
8- or 16-bit acceleration. With 512 KB L2 RAM, up to 8 MB L3 SRAM, and 20 MB flash, it can store larger models or model ensembles.
Our benchmark also includes the STM32H7A3ZI and ESP32-S3 MCUs without neural hardware (although the latter has an extended instruction set for vector operations).
µNPUs are primarily optimized for convolutional models, with our benchmark suite covering a range of such models, varying convolutional depth, width, activation memory, parameter count, and post-processing requirements.
Ensuring uniformity is non-trivial: platforms expose different operator sets,
TABLE 2. The various models used in our benchmark. | Model | Input | #Params | #Layers | Conv Depth | Max Conv Ch. | Peak Act. Ram (MB, B=1)1 | Lifetime Pressure (MB·steps)2 | MMACs | MFLOPs | |---|---|---|---|---|---|---|---|---|---| | CIFAR10-NAS | 3×32×32 | 298,762 | 11 | 10 | 128 | 0.07 | 18.39 | 74.25 | 36.38 | | ResidualNet | 3×32×32 | 383,012 | 14 | 14 | 512 | 0.14 | 2.98 | 37.78 | 18.46 | | SimpleNet | 3×32×32 | 383,012 | 14 | 14 | 512 | 0.14 | 2.45 | 38.00 | 18.46 | | AI8XAutoEnc | 3×256 | 136,800 | 7 | 2 | 128 | 0.07 | 2.89 | 0.55 | 0.20 | | YoloV1 | 3×96×96 | 40,700 | 20 | 20 | 32 | 0.02 | 0.80 | 43.83 | 21.22 | _Note: MACs/FLOPs are forward-only. Peak activation RAM and Lifetime Pressure are measured with batch=1 and INT8 activations._
{
"x_axis": {
"label": null,
"unit": null
},
"y_axis": {
"label": "Power",
"unit": "mW"
},
"series": [
{
"name": "Initialization",
"points": [
{
"x": "CIFAR10-NAS / MAX78K (C)",
"y": 0
},
{
"x": "CIFAR10-NAS / MAX78K (R)",
"y": 0
},
{
"x": "CIFAR10-NAS / GAP8",
"y": 100
},
{
"x": "CIFAR10-NAS / HX-WFE2 (S)",
"y": 95
},
{
"x": "CIFAR10-NAS / HX-WFE2 (P)",
"y": 95
},
{
"x": "CIFAR10-NAS / NVP-MCXN847",
"y": 25
},
{
"x": "ResidualNet / MAX78K (C)",
"y": 0
},
{
"x": "ResidualNet / MAX78K (R)",
"y": 0
},
{
"x": "ResidualNet / GAP8",
"y": 145
},
{
"x": "ResidualNet / HX-WFE2 (S)",
"y": 95
},
{
"x": "ResidualNet / HX-WFE2 (P)",
"y": 95
},
{
"x": "ResidualNet / NVP-MCXN847",
"y": 25
},
{
"x": "SimpleNet / MAX78K (C)",
"y": 10
},
{
"x": "SimpleNet / MAX78K (R)",
"y": 10
},
{
"x": "SimpleNet / GAP8",
"y": 125
},
{
"x": "SimpleNet / HX-WFE2 (S)",
"y": 95
},
{
"x": "SimpleNet / HX-WFE2 (P)",
"y": 95
},
{
"x": "SimpleNet / NVP-MCXN847",
"y": 25
},
{
"x": "AI8XAutoEnc / MAX78K (C)",
"y": 0
},
{
"x": "AI8XAutoEnc / MAX78K (R)",
"y": 0
},
{
"x": "AI8XAutoEnc / GAP8",
"y": 90
},
{
"x": "AI8XAutoEnc / HX-WFE2 (S)",
"y": 95
},
{
"x": "AI8XAutoEnc / HX-WFE2 (P)",
"y": 95
},
{
"x": "AI8XAutoEnc / NVP-MCXN847",
"y": 25
},
{
"x": "YoloV1 / MAX78K (C)",
"y": 10
},
{
"x": "YoloV1 / MAX78K (R)",
"y": 10
},
{
"x": "YoloV1 / GAP8",
"y": 95
},
{
"x": "YoloV1 / HX-WFE2 (S)",
"y": 95
},
{
"x": "YoloV1 / HX-WFE2 (P)",
"y": 95
},
{
"x": "YoloV1 / NVP-MCXN847",
"y": 25
}
]
},
{
"name": "Memory I/O",
"points": [
{
"x": "CIFAR10-NAS / MAX78K (C)",
"y": 25
},
{
"x": "CIFAR10-NAS / MAX78K (R)",
"y": 25
},
{
"x": "CIFAR10-NAS / GAP8",
"y": 30
},
{
"x": "CIFAR10-NAS / HX-WFE2 (S)",
"y": 115
},
{
"x": "CIFAR10-NAS / HX-WFE2 (P)",
"y": 100
},
{
"x": "CIFAR10-NAS / NVP-MCXN847",
"y": 30
},
{
"x": "ResidualNet / MAX78K (C)",
"y": 25
},
{
"x": "ResidualNet / MAX78K (R)",
"y": 25
},
{
"x": "ResidualNet / GAP8",
"y": 35
},
{
"x": "ResidualNet / HX-WFE2 (S)",
"y": 100
},
{
"x": "ResidualNet / HX-WFE2 (P)",
"y": 100
},
{
"x": "ResidualNet / NVP-MCXN847",
"y": 30
},
{
"x": "SimpleNet / MAX78K (C)",
"y": 25
},
{
"x": "SimpleNet / MAX78K (R)",
"y": 25
},
{
"x": "SimpleNet / GAP8",
"y": 35
},
{
"x": "SimpleNet / HX-WFE2 (S)",
"y": 100
},
{
"x": "SimpleNet / HX-WFE2 (P)",
"y": 100
},
{
"x": "SimpleNet / NVP-MCXN847",
"y": 30
},
{
"x": "AI8XAutoEnc / MAX78K (C)",
"y": 25
},
{
"x": "AI8XAutoEnc / MAX78K (R)",
"y": 25
},
{
"x": "AI8XAutoEnc / GAP8",
"y": 35
},
{
"x": "AI8XAutoEnc / HX-WFE2 (S)",
"y": 95
},
{
"x": "AI8XAutoEnc / HX-WFE2 (P)",
"y": 95
},
{
"x": "AI8XAutoEnc / NVP-MCXN847",
"y": 30
},
{
"x": "YoloV1 / MAX78K (C)",
"y": 40
},
{
"x": "YoloV1 / MAX78K (R)",
"y": 25
},
{
"x": "YoloV1 / GAP8",
"y": 40
},
{
"x": "YoloV1 / HX-WFE2 (S)",
"y": 100
},
{
"x": "YoloV1 / HX-WFE2 (P)",
"y": 105
},
{
"x": "YoloV1 / NVP-MCXN847",
"y": 30
}
]
},
{
"name": "Inference",
"points": [
{
"x": "CIFAR10-NAS / MAX78K (C)",
"y": 80
},
{
"x": "CIFAR10-NAS / MAX78K (R)",
"y": 80
},
{
"x": "CIFAR10-NAS / GAP8",
"y": 125
},
{
"x": "CIFAR10-NAS / HX-WFE2 (S)",
"y": 115
},
{
"x": "CIFAR10-NAS / HX-WFE2 (P)",
"y": 115
},
{
"x": "CIFAR10-NAS / NVP-MCXN847",
"y": 35
},
{
"x": "ResidualNet / MAX78K (C)",
"y": 50
},
{
"x": "ResidualNet / MAX78K (R)",
"y": 50
},
{
"x": "ResidualNet / GAP8",
"y": 115
},
{
"x": "ResidualNet / HX-WFE2 (S)",
"y": 115
},
{
"x": "ResidualNet / HX-WFE2 (P)",
"y": 115
},
{
"x": "ResidualNet / NVP-MCXN847",
"y": 35
},
{
"x": "SimpleNet / MAX78K (C)",
"y": 50
},
{
"x": "SimpleNet / MAX78K (R)",
"y": 50
},
{
"x": "SimpleNet / GAP8",
"y": 130
},
{
"x": "SimpleNet / HX-WFE2 (S)",
"y": 115
},
{
"x": "SimpleNet / HX-WFE2 (P)",
"y": 115
},
{
"x": "SimpleNet / NVP-MCXN847",
"y": 35
},
{
"x": "AI8XAutoEnc / MAX78K (C)",
"y": 80
},
{
"x": "AI8XAutoEnc / MAX78K (R)",
"y": 80
},
{
"x": "AI8XAutoEnc / GAP8",
"y": 85
},
{
"x": "AI8XAutoEnc / HX-WFE2 (S)",
"y": 95
},
{
"x": "AI8XAutoEnc / HX-WFE2 (P)",
"y": 100
},
{
"x": "AI8XAutoEnc / NVP-MCXN847",
"y": 35
},
{
"x": "YoloV1 / MAX78K (C)",
"y": 25
},
{
"x": "YoloV1 / MAX78K (R)",
"y": 25
},
{
"x": "YoloV1 / GAP8",
"y": 120
},
{
"x": "YoloV1 / HX-WFE2 (S)",
"y": 115
},
{
"x": "YoloV1 / HX-WFE2 (P)",
"y": 115
},
{
"x": "YoloV1 / NVP-MCXN847",
"y": 35
}
]
},
{
"name": "Post-Processing",
"points": [
{
"x": "CIFAR10-NAS / MAX78K (C)",
"y": 25
},
{
"x": "CIFAR10-NAS / MAX78K (R)",
"y": 25
},
{
"x": "CIFAR10-NAS / GAP8",
"y": 30
},
{
"x": "CIFAR10-NAS / HX-WFE2 (S)",
"y": 90
},
{
"x": "CIFAR10-NAS / HX-WFE2 (P)",
"y": 90
},
{
"x": "CIFAR10-NAS / NVP-MCXN847",
"y": 25
},
{
"x": "ResidualNet / MAX78K (C)",
"y": 25
},
{
"x": "ResidualNet / MAX78K (R)",
"y": 25
},
{
"x": "ResidualNet / GAP8",
"y": 35
},
{
"x": "ResidualNet / HX-WFE2 (S)",
"y": 90
},
{
"x": "ResidualNet / HX-WFE2 (P)",
"y": 90
},
{
"x": "ResidualNet / NVP-MCXN847",
"y": 25
},
{
"x": "SimpleNet / MAX78K (C)",
"y": 25
},
{
"x": "SimpleNet / MAX78K (R)",
"y": 25
},
{
"x": "SimpleNet / GAP8",
"y": 35
},
{
"x": "SimpleNet / HX-WFE2 (S)",
"y": 90
},
{
"x": "SimpleNet / HX-WFE2 (P)",
"y": 90
},
{
"x": "SimpleNet / NVP-MCXN847",
"y": 25
},
{
"x": "AI8XAutoEnc / MAX78K (C)",
"y": 25
},
{
"x": "AI8XAutoEnc / MAX78K (R)",
"y": 25
},
{
"x": "AI8XAutoEnc / GAP8",
"y": 35
},
{
"x": "AI8XAutoEnc / HX-WFE2 (S)",
"y": 90
},
{
"x": "AI8XAutoEnc / HX-WFE2 (P)",
"y": 90
},
{
"x": "AI8XAutoEnc / NVP-MCXN847",
"y": 25
},
{
"x": "YoloV1 / MAX78K (C)",
"y": 25
},
{
"x": "YoloV1 / MAX78K (R)",
"y": 25
},
{
"x": "YoloV1 / GAP8",
"y": 40
},
{
"x": "YoloV1 / HX-WFE2 (S)",
"y": 90
},
{
"x": "YoloV1 / HX-WFE2 (P)",
"y": 90
},
{
"x": "YoloV1 / NVP-MCXN847",
"y": 25
}
]
},
{
"name": "Idle",
"points": [
{
"x": "CIFAR10-NAS / MAX78K (C)",
"y": 25
},
{
"x": "CIFAR10-NAS / MAX78K (R)",
"y": 25
},
{
"x": "CIFAR10-NAS / GAP8",
"y": 30
},
{
"x": "CIFAR10-NAS / HX-WFE2 (S)",
"y": 90
},
{
"x": "CIFAR10-NAS / HX-WFE2 (P)",
"y": 90
},
{
"x": "CIFAR10-NAS / NVP-MCXN847",
"y": 25
},
{
"x": "ResidualNet / MAX78K (C)",
"y": 25
},
{
"x": "ResidualNet / MAX78K (R)",
"y": 25
},
{
"x": "ResidualNet / GAP8",
"y": 35
},
{
"x": "ResidualNet / HX-WFE2 (S)",
"y": 90
},
{
"x": "ResidualNet / HX-WFE2 (P)",
"y": 90
},
{
"x": "ResidualNet / NVP-MCXN847",
"y": 25
},
{
"x": "SimpleNet / MAX78K (C)",
"y": 25
},
{
"x": "SimpleNet / MAX78K (R)",
"y": 25
},
{
"x": "SimpleNet / GAP8",
"y": 35
},
{
"x": "SimpleNet / HX-WFE2 (S)",
"y": 90
},
{
"x": "SimpleNet / HX-WFE2 (P)",
"y": 90
},
{
"x": "SimpleNet / NVP-MCXN847",
"y": 25
},
{
"x": "AI8XAutoEnc / MAX78K (C)",
"y": 25
},
{
"x": "AI8XAutoEnc / MAX78K (R)",
"y": 25
},
{
"x": "AI8XAutoEnc / GAP8",
"y": 35
},
{
"x": "AI8XAutoEnc / HX-WFE2 (S)",
"y": 90
},
{
"x": "AI8XAutoEnc / HX-WFE2 (P)",
"y": 90
},
{
"x": "AI8XAutoEnc / NVP-MCXN847",
"y": 25
},
{
"x": "YoloV1 / MAX78K (C)",
"y": 25
},
{
"x": "YoloV1 / MAX78K (R)",
"y": 25
},
{
"x": "YoloV1 / GAP8",
"y": 40
},
{
"x": "YoloV1 / HX-WFE2 (S)",
"y": 90
},
{
"x": "YoloV1 / HX-WFE2 (P)",
"y": 90
},
{
"x": "YoloV1 / NVP-MCXN847",
"y": 25
}
]
}
],
"precision": "estimated"
}memory layouts, and fallback behaviour. Certain operations are not universally supported, necessitating their implementation as CPU-side post-processing steps for relevant models. Each platform also has its own set of supported operator layouts; the MAX78000, for example, only supports 1D convolution with kernel sizes 1 to 9 and 2D with sizes of 1 by 1 or 3 by 3. Unsupported operations will fail to compile or be mapped to CPU execution and incur latency penalties.
We redesign our benchmark models around a shared, mutually compatible operator subset, and aim to avoid platform-specific structural rewrites during compilation. We use INT8 post-training quantization (PTQ), the only universally supported scheme across our platforms. The resulting models are, consequently, not necessarily optimal for each platform. Quantization-aware training and mixed-precision search are outside our scope.
The various benchmark platforms also support a range of model formats, from vendor-optimized variants of general model formats, e.g., LiteRT, to platform-specific formats. Our compiler workflow ingests Torch / LiteRT models along with various compiler flags (i.e., representative calibration data for PTQ, model input dimensions, platform-specific compiler options) and emits the binary / compiled model required by each platform.
Ethos-U55 models on the HX-WE2 are compiled using Vela [9]. We evaluate both its _Size_ mode, minimizing SRAM usage, and _Performance_ mode, minimizing latency using available arena cache3.
We measure latency, power and energyefficiency, in terms of number of inference operations per mJ. The impacts of various platform-specific performance optimizations are out of scope. Latency can be considered proportional to throughput, since batching and other amortizations are not practical on the given hardware.
Latency: Latency is measured using each platform's internal timer. We fix CPU/ NPU frequency at 100 MHz. We run 10 consecutive inferences and report mean and standard deviation to account for run-to- run variability4.
Power and energy-efficiency: We measure power using a high voltage power monitor at 50 kHz and 3.3 V, also averaging readings over 10 inferences. We report energy-efficiency as the number of inferences per mJ consumed.
We break execution into stages and measure per-stage overheads:
1 Peak activation RAM: maximum live activation footprint during the forward pass, with batch=1 and INT8 activations, excluding parameter storage.
2 Lifetime Pressure: sum over all tracked activations of size(t) in MB × lifetime_steps(t), where lifetime_steps(t) counts the number of subsequent leaf-module steps over which activation t must remain live until its last use. Larger values indicate reduced opportunities for buffer reuse.
3 HX-WE2 (S) denotes model compilation with the Size optimization, and (P) with Performance.
4Runs show limited variance under bare-metal execution.
EXPERIMENTAL METHODS
Our benchmarks show that μNPU performance is governed by the complete execution path, and regularly dominated by initialiization or data movement. GOPS alone does not accurately predict deployed performance.
Dedicated neural hardware provides large latency gains over MCU-only execution on convolution-heavy workloads. On CIFAR10-NAS, HX-WE2(P) runs in roughly 11.1 ms, compared with 550 ms on the STM32H7A3ZI and 555 ms on the ESP32, approximately a 50× reduction. On YoloV1, HX-WE2(P) runs in 11.4 ms, compared with 340 ms and 1166 ms on the STM32H7A3ZI and ESP32, approximately 30× and 102× reductions. These gains are not universal: for lightweight models relying more on fully connected layers than convolutions, the STM32H7A3ZI remains competitive.
The MAX78000, with its Cortex-M4 active5, provides the best overall efficiency when NPU initialization is included, with sub-30 ms end-to-end latency across our benchmarks. HX-WE2(P) is nevertheless faster end-to-end: 1.93× faster than the MAX78000 Cortex-M4 configuration and 3.07× faster than its RISC-V configuration, but at 3.13× and 3.33× greater average power draw.
This is counterintuitive because the MAX78000 can achieve lower accelerator-only latency. At 100 MHz, the MAX78000 Cortex-M4 configuration gives an average 2.48× inference-latency improvement over HX-WE2 (P). HX-WE2 wins end-to-end because of lower I/O overheads: on the MAX78000, I/O takes 6.10× longer than inference with the Cortex-M4 active and 9.80× longer with the RISC-V. In the worst case, ResidualNet on the RISC-V spends 44.89 ms on I/O but only 2.96 ms on inference, meaning over 90% of end-to-end time is I/O, largely from moving weights into accelerator memory [7, 8].
Vela's _Performance_ optimizations mostly reduce HX-WE2 latency compared with Size, but this is non-uniform across models, and energy-efficiency can degrade as model complexity grows.
The NXP-MCXN947 is the clearest example of GOPS being a weak proxy for deployed performance. Although its eIQ Neutron accelerator is rated at only 4.8 GOPS, it achieves consistent sub-30 ms end-to-end latency, comparable to the MAX78000, due to low initialization and memory-transfer overheads.
Hardware-aware model design. Neural architecture search should target platform-specifics constraints, alongside activation lifetime, buffer reuse, operator support, and data movement.
Improved neural operator support. Most platforms support limited operators, optimized for linear and convolutional layers plus pooling, element-wise activation, and addition, but excludes operators central to other model families. Unsupported layers can, in principle, be offloaded to the CPU, but this requires a tightly integrated CPU/ NPU software stack that minimizes data transfer and synchronization overheads. Fine-grained CPU fallback is still limited on most evaluated platforms, and post-inference reads of intermediate activations can be latency-expensive.
Dynamic execution. Static deployment restricts early-exit networks [14], conditional computation, runtime model selection, and other input-adaptive methods for reducing latency and energy.
Standardized model formats and profiling. Unified deployment flows, reliable latency/ energy profiling, and memory-aware profilers would reduce cross-platform engineering overhead and expose bottlenecks earlier [15].
The MAX78000 achieves the strongest GOPS/mW efficiency and lowest idle power. Its main limitation lies with I/O: data movement can vastly exceed NPU execution time. Use it when models can remain resident and power-gating can be exploited.
Despite only supporting 4.8 GOPS, the MCXN947 remains competitive because of low initialization and I/O overheads.
While it does not achieve competitive latency, its 8 MB L3 SRAM and 20 MB flash are valuable for hosting larger models or model ensembles.
{
"x_axis": {
"label": "End-to-End Latency",
"unit": "ms"
},
"y_axis": {
"label": "Average Power",
"unit": "mW"
},
"series": [
{
"name": "MAX78k (C)",
"points": [
{
"x": 0,
"y": 10
},
{
"x": 10,
"y": 12
},
{
"x": 13,
"y": 9
},
{
"x": 18,
"y": 7
},
{
"x": 24,
"y": 4
},
{
"x": 26,
"y": 6
},
{
"x": 27,
"y": 9
},
{
"x": 30,
"y": 4
},
{
"x": 40,
"y": 3
},
{
"x": 50,
"y": 2
}
]
},
{
"name": "MAX78k (R)",
"points": [
{
"x": 0,
"y": 5
},
{
"x": 13,
"y": 8
},
{
"x": 18,
"y": 7
},
{
"x": 24,
"y": 4
},
{
"x": 26,
"y": 6
},
{
"x": 27,
"y": 9
},
{
"x": 30,
"y": 4
},
{
"x": 40,
"y": 3
},
{
"x": 50,
"y": 2
}
]
},
{
"name": "GAP8",
"points": [
{
"x": 100,
"y": 3
},
{
"x": 12,
"y": 18
},
{
"x": 48,
"y": 6
},
{
"x": 50,
"y": 7
}
]
},
{
"name": "HX-WE2 (S)",
"points": [
{
"x": 0,
"y": 85
},
{
"x": 0,
"y": 35
},
{
"x": 10,
"y": 36
},
{
"x": 11,
"y": 35
},
{
"x": 12,
"y": 33
}
]
},
{
"name": "HX-WE2 (P)",
"points": [
{
"x": 0,
"y": 40
},
{
"x": 0,
"y": 38
},
{
"x": 10,
"y": 39
},
{
"x": 11,
"y": 36
}
]
},
{
"name": "NXP-MCXN947",
"points": [
{
"x": 0,
"y": 83
},
{
"x": 12,
"y": 35
},
{
"x": 13,
"y": 8
},
{
"x": 18,
"y": 7
},
{
"x": 20,
"y": 6
},
{
"x": 26,
"y": 4
}
]
},
{
"name": "STM32",
"points": [
{
"x": 0,
"y": 20
},
{
"x": 270,
"y": 1
},
{
"x": 320,
"y": 1
},
{
"x": 380,
"y": 1
}
]
},
{
"name": "ESP32",
"points": [
{
"x": 380,
"y": 1
},
{
"x": 550,
"y": 1
},
{
"x": 1150,
"y": 1
}
]
}
],
"precision": "estimated"
}5 MAX78k (C) denotes use of its Cortex-M CPU, and (R) its RISC-V CPU.
Baseline MCUs: viable for convolution-light models, particularly those dominated by fully connected layers or simple 1D operations.
We normalize CPU and NPU operating frequencies, improving comparability but not capturing each platform's peak performance. INT8 quantization, again, improves comparability but hides platform-specific advantages such as MAX78000 sub-byte quantization or HX-WE2 floating-point acceleration. Our shared NPU-compatible operator set also excludes platform-specific redesigns, QAT, operator fusion, and broader layer support.
Our evaluation reveals both expected trends and surprising disparities. Dedicated accelerators can deliver up to two orders of magnitude lower latency than MCUs on convolution-heavy models, yet GOPS alone does not explain real-world performance. As embedded deployments become more diverse and memory-intensive, the next gains will come not only from higher arithmetic throughput, but also from reducing data movement, broadening operator support, and improving deployment-level profiling. ■
Glossary
GOPS - Giga Operations Per Second. Peak arithmetic throughput of an accelerator.
Systolic array. Grid of processing elements, each with a multiply-accumulate unit and local memory, that passes data between neighbouring elements to parallelize tensor operations.
Activation lifetime. How long an inter-mediate tensor remains in memory before its final use, used to estimate memory pressure.
Quantization-aware training (QAT). Training a model while simulating quantization effects, often improving accuracy.
Operator fusion. Combining multiple model operations into one optimized execution step to reduce memory movement and overhead.
Josh Millar is a PhD candidate in the Department of Computing at Imperial College London, U.K. His research focuses on machine learning for embedded systems, with an emphasis on microcontroller-scale accelerators, low-power inference, and energy-aware system design for resource-constrained sensing platforms.
Yushan Huang is a senior researcher at 2012 Labs, Huawei, Beijing, China. He received a PhD from Imperial College London, U.K. His research focuses on low-energy on-device machine learning and the resource characterization of accelerator-equipped microcontrollers.
Sarab Sethi is an Assistant Professor in Ecosystem Sensing at Imperial College London, U.K., where he leads the Ecosystem Sensing research group. His research focuses on developing scalable, real-time sensing systems and machine-learning approaches for understanding natural environments, with a particular focus on acoustic sensing for autonomous biodiversity monitoring. His work bridges embedded systems, ecological data analysis, and practical conservation decision-making.
Hamed Haddadi is the Professor of Human-Centred Systems in the Department of Computing at Imperial College London, U.K. In his industrial role, he is the Chief Scientist at Brave Software, where he works on developing privacy-preserving analytics protocols. He is also the Co-Founder and CSO of Mulini SRL, focusing on securing consumer and industrial cyber-physical systems. His research spans IoT, applied machine learning, security, and privacy-preserving systems, including the design of AIoT platforms that can process data efficiently on-device while respecting users' privacy.
Anil Madhavapeddy is the Professor of Planetary Computing at the University of Cambridge Computer Laboratory, U.K. He co-directs the Centre for Earth Observation and the Cambridge Centre for Carbon Credits, with the aim of accelerating natural climate solutions that contribute towards ending deforestation and improving biodiversity globally. He has decades of experience constructing large-scale systems and has contributed to many open-source projects such as OCaml, Docker, Xen, and OpenBSD.
REFERENCES