Benchmarking Ultra-Low-Power μNPUs

Benchmarking Ultra-Low-Power μNPUs

Josh Millar1, Yushan Huang2, Sarab Sethi1, Hamed Haddadi1, Anil Madhavapeddy3
1Imperial College London, London; 2Imperial College London / Huawei; 3University of Cambridge, Cambridge
GetMobile 30 pp. 11-12
Abstract. Microcontrollers (MCUs) are widely used in resource-constrained environments due to their form factor and low cost and power consumption. Performing model inference on MCU-scale hardware improves privacy by running locally, lowers operating costs for model vendors, and eliminates dependence on network connectivity [1, 2]. However, deployments remain constrained by limited memory, throughput, and compute. The growing computational demands of modern models have catalyzed specialized accelerators across the computing spectrum, from data centers to embedded systems. At this resource-constrained end, MCU-scale neural processing units, or µNPUs, have emerged to provide real-time or near-real-time inference within milliwatt-scale power budgets.

This article details the first comparative benchmark of commercially available μNPU platforms, alongside fine-grained, independent results for several previously unevaluated platforms.

Our analysis uncovers expected trends and surprising gaps between hardware specifications and measured performance. Firstly, peak _giga operations per second_ (GOPS), an accelerator's compute throughput, is a weak predictor of real-world performance [3]. Secondly, memory movement can dominate end-to-end latency, unlike in larger-scale hardware architectures.

INSIDE A μNPU

The µNPU design accelerates tensor operations using systolic arrays of processing elements (PEs). Each PE contains local multiply-accumulate units and often local weight memory, reducing contention and improving parallelism. The PE array connects to larger buffers and on-chip memory, which may be accelerator SRAM, tightly coupled memory, shared MCU SRAM, or external memory.

METHODOLOGY Platforms

The benchmark covers a diverse set of com-mercially available platforms, alongside MCUs without dedicated neural hardware to quantify the benefit of acceleration. Our selection spans 4.8 to 512 GOPS, 128 KB to 2 MB SRAM, and arithmetic support from 1-bit quantized to floating-point execution.

figure
Figure description (VLM): A log-log scatter plot comparing AI accelerators, with Peak Power (W) on the x-axis (10^-1 to 10^3) and Peak Performance (GOPs/sec) on the y-axis (10^1 to 10^6). Individual devices (e.g., MAX78k, GAP8, GAP9, HX6538 WE2, Coral TPU, Xavier NX, Xavier AGX, V100, A100, H100) are shown as colored dots grouped by dashed ellipses into MCU-scale NPUs, SoC NPUs/TPUs, and a high-end GPU cluster.
Figure data (VLM re-extraction)
{
  "x_axis": {
    "label": "Peak Power",
    "unit": "W"
  },
  "y_axis": {
    "label": "Peak Performance",
    "unit": "GOPs/sec"
  },
  "series": [
    {
      "name": "MCU-scale NPUs",
      "points": [
        {
          "x": 0.05,
          "y": 5,
          "label": "MAX78k"
        },
        {
          "x": 0.1,
          "y": 10,
          "label": "GAP8"
        },
        {
          "x": 0.12,
          "y": 15,
          "label": "GAP9"
        },
        {
          "x": 0.15,
          "y": 60,
          "label": "HX6538 WE2"
        }
      ]
    },
    {
      "name": "SoC NPUs/TPUs",
      "points": [
        {
          "x": 2,
          "y": 3000,
          "label": "Coral TPU"
        },
        {
          "x": 10,
          "y": 20000,
          "label": "Xavier NX"
        },
        {
          "x": 30,
          "y": 30000,
          "label": "Xavier AGX"
        }
      ]
    },
    {
      "name": "High-end GPU cluster",
      "points": [
        {
          "x": 300,
          "y": 100000,
          "label": "V100"
        },
        {
          "x": 400,
          "y": 300000,
          "label": "A100"
        },
        {
          "x": 700,
          "y": 1000000,
          "label": "H100"
        }
      ]
    }
  ],
  "precision": "estimated"
}
FIGURE 1. Performance of representative hardware accelerators [4]. µNPUs are uniquely suited for low-power, wearable systems.

TABLE 1. The platforms used in our benchmark, and their specifications [10-13].

| MCU | CPU(s) | NPU | Flash | RAM | GOPs (max) | Bit Cap. | |---|---|---|---|---|---|---| | NPUs | | | | | | | | MAX78000 | Cortex-M4; RISC-V | MAXIM-own | 512 KB | 512 KB NPU; 128 KB CPU | 30 | 1, 2, 4, 8 | | HX-WE2 (Corstone-300) | Cortex-M55 | Ethos-U55 | 16 MB | 2 MB SRAM; 512 KB TCM | 512 | 8, 16, 32 | | NXP-MCXN947 | Cortex-M33 (2) | eIQ Neutron | 2 MB | 512 KB | 4.8 | 8 | | GAP8 | RISC-V | GAP-own | 20 MB L3 | 512 KB L2; 8 MB L3 | 22.65 | 8, 16 | | Baseline MCUs | | | | | | | | STM32H74A3ZI | Cortex-M7 | - | 2 MB | 1.4 MB | - | 8, 16, 32 | | ESP32 | Tensilica LX6 | - | 4 MB | 520 KB | - | 8, 16, 32 |

8- or 16-bit acceleration. With 512 KB L2 RAM, up to 8 MB L3 SRAM, and 20 MB flash, it can store larger models or model ensembles.

Our benchmark also includes the STM32H7A3ZI and ESP32-S3 MCUs without neural hardware (although the latter has an extended instruction set for vector operations).

Models and Cross-Platform Uniformity

µNPUs are primarily optimized for convolutional models, with our benchmark suite covering a range of such models, varying convolutional depth, width, activation memory, parameter count, and post-processing requirements.

Ensuring uniformity is non-trivial: platforms expose different operator sets,

TABLE 2. The various models used in our benchmark. | Model | Input | #Params | #Layers | Conv Depth | Max Conv Ch. | Peak Act. Ram (MB, B=1)1 | Lifetime Pressure (MB·steps)2 | MMACs | MFLOPs | |---|---|---|---|---|---|---|---|---|---| | CIFAR10-NAS | 3×32×32 | 298,762 | 11 | 10 | 128 | 0.07 | 18.39 | 74.25 | 36.38 | | ResidualNet | 3×32×32 | 383,012 | 14 | 14 | 512 | 0.14 | 2.98 | 37.78 | 18.46 | | SimpleNet | 3×32×32 | 383,012 | 14 | 14 | 512 | 0.14 | 2.45 | 38.00 | 18.46 | | AI8XAutoEnc | 3×256 | 136,800 | 7 | 2 | 128 | 0.07 | 2.89 | 0.55 | 0.20 | | YoloV1 | 3×96×96 | 40,700 | 20 | 20 | 32 | 0.02 | 0.80 | 43.83 | 21.22 | _Note: MACs/FLOPs are forward-only. Peak activation RAM and Lifetime Pressure are measured with batch=1 and INT8 activations._

figure
Figure description (VLM): A set of five grouped bar charts (CIFAR10-NAS, ResidualNet, SimpleNet, AI8XAutoEnc, YoloV1) showing power consumption in mW on the y-axis across hardware platforms (MAX78K (C), MAX78K (R), GAP8, HX-WFE2 (S), HX-WFE2 (P), NVP-MCXN847) on the x-axis. Each group of bars is color-coded by processing stage: Initialization, Memory I/O, Inference, Post-Processing, and Idle, as defined in the shared legend at the bottom.
Figure data (VLM re-extraction)
{
  "x_axis": {
    "label": null,
    "unit": null
  },
  "y_axis": {
    "label": "Power",
    "unit": "mW"
  },
  "series": [
    {
      "name": "Initialization",
      "points": [
        {
          "x": "CIFAR10-NAS / MAX78K (C)",
          "y": 0
        },
        {
          "x": "CIFAR10-NAS / MAX78K (R)",
          "y": 0
        },
        {
          "x": "CIFAR10-NAS / GAP8",
          "y": 100
        },
        {
          "x": "CIFAR10-NAS / HX-WFE2 (S)",
          "y": 95
        },
        {
          "x": "CIFAR10-NAS / HX-WFE2 (P)",
          "y": 95
        },
        {
          "x": "CIFAR10-NAS / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "ResidualNet / MAX78K (C)",
          "y": 0
        },
        {
          "x": "ResidualNet / MAX78K (R)",
          "y": 0
        },
        {
          "x": "ResidualNet / GAP8",
          "y": 145
        },
        {
          "x": "ResidualNet / HX-WFE2 (S)",
          "y": 95
        },
        {
          "x": "ResidualNet / HX-WFE2 (P)",
          "y": 95
        },
        {
          "x": "ResidualNet / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "SimpleNet / MAX78K (C)",
          "y": 10
        },
        {
          "x": "SimpleNet / MAX78K (R)",
          "y": 10
        },
        {
          "x": "SimpleNet / GAP8",
          "y": 125
        },
        {
          "x": "SimpleNet / HX-WFE2 (S)",
          "y": 95
        },
        {
          "x": "SimpleNet / HX-WFE2 (P)",
          "y": 95
        },
        {
          "x": "SimpleNet / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "AI8XAutoEnc / MAX78K (C)",
          "y": 0
        },
        {
          "x": "AI8XAutoEnc / MAX78K (R)",
          "y": 0
        },
        {
          "x": "AI8XAutoEnc / GAP8",
          "y": 90
        },
        {
          "x": "AI8XAutoEnc / HX-WFE2 (S)",
          "y": 95
        },
        {
          "x": "AI8XAutoEnc / HX-WFE2 (P)",
          "y": 95
        },
        {
          "x": "AI8XAutoEnc / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "YoloV1 / MAX78K (C)",
          "y": 10
        },
        {
          "x": "YoloV1 / MAX78K (R)",
          "y": 10
        },
        {
          "x": "YoloV1 / GAP8",
          "y": 95
        },
        {
          "x": "YoloV1 / HX-WFE2 (S)",
          "y": 95
        },
        {
          "x": "YoloV1 / HX-WFE2 (P)",
          "y": 95
        },
        {
          "x": "YoloV1 / NVP-MCXN847",
          "y": 25
        }
      ]
    },
    {
      "name": "Memory I/O",
      "points": [
        {
          "x": "CIFAR10-NAS / MAX78K (C)",
          "y": 25
        },
        {
          "x": "CIFAR10-NAS / MAX78K (R)",
          "y": 25
        },
        {
          "x": "CIFAR10-NAS / GAP8",
          "y": 30
        },
        {
          "x": "CIFAR10-NAS / HX-WFE2 (S)",
          "y": 115
        },
        {
          "x": "CIFAR10-NAS / HX-WFE2 (P)",
          "y": 100
        },
        {
          "x": "CIFAR10-NAS / NVP-MCXN847",
          "y": 30
        },
        {
          "x": "ResidualNet / MAX78K (C)",
          "y": 25
        },
        {
          "x": "ResidualNet / MAX78K (R)",
          "y": 25
        },
        {
          "x": "ResidualNet / GAP8",
          "y": 35
        },
        {
          "x": "ResidualNet / HX-WFE2 (S)",
          "y": 100
        },
        {
          "x": "ResidualNet / HX-WFE2 (P)",
          "y": 100
        },
        {
          "x": "ResidualNet / NVP-MCXN847",
          "y": 30
        },
        {
          "x": "SimpleNet / MAX78K (C)",
          "y": 25
        },
        {
          "x": "SimpleNet / MAX78K (R)",
          "y": 25
        },
        {
          "x": "SimpleNet / GAP8",
          "y": 35
        },
        {
          "x": "SimpleNet / HX-WFE2 (S)",
          "y": 100
        },
        {
          "x": "SimpleNet / HX-WFE2 (P)",
          "y": 100
        },
        {
          "x": "SimpleNet / NVP-MCXN847",
          "y": 30
        },
        {
          "x": "AI8XAutoEnc / MAX78K (C)",
          "y": 25
        },
        {
          "x": "AI8XAutoEnc / MAX78K (R)",
          "y": 25
        },
        {
          "x": "AI8XAutoEnc / GAP8",
          "y": 35
        },
        {
          "x": "AI8XAutoEnc / HX-WFE2 (S)",
          "y": 95
        },
        {
          "x": "AI8XAutoEnc / HX-WFE2 (P)",
          "y": 95
        },
        {
          "x": "AI8XAutoEnc / NVP-MCXN847",
          "y": 30
        },
        {
          "x": "YoloV1 / MAX78K (C)",
          "y": 40
        },
        {
          "x": "YoloV1 / MAX78K (R)",
          "y": 25
        },
        {
          "x": "YoloV1 / GAP8",
          "y": 40
        },
        {
          "x": "YoloV1 / HX-WFE2 (S)",
          "y": 100
        },
        {
          "x": "YoloV1 / HX-WFE2 (P)",
          "y": 105
        },
        {
          "x": "YoloV1 / NVP-MCXN847",
          "y": 30
        }
      ]
    },
    {
      "name": "Inference",
      "points": [
        {
          "x": "CIFAR10-NAS / MAX78K (C)",
          "y": 80
        },
        {
          "x": "CIFAR10-NAS / MAX78K (R)",
          "y": 80
        },
        {
          "x": "CIFAR10-NAS / GAP8",
          "y": 125
        },
        {
          "x": "CIFAR10-NAS / HX-WFE2 (S)",
          "y": 115
        },
        {
          "x": "CIFAR10-NAS / HX-WFE2 (P)",
          "y": 115
        },
        {
          "x": "CIFAR10-NAS / NVP-MCXN847",
          "y": 35
        },
        {
          "x": "ResidualNet / MAX78K (C)",
          "y": 50
        },
        {
          "x": "ResidualNet / MAX78K (R)",
          "y": 50
        },
        {
          "x": "ResidualNet / GAP8",
          "y": 115
        },
        {
          "x": "ResidualNet / HX-WFE2 (S)",
          "y": 115
        },
        {
          "x": "ResidualNet / HX-WFE2 (P)",
          "y": 115
        },
        {
          "x": "ResidualNet / NVP-MCXN847",
          "y": 35
        },
        {
          "x": "SimpleNet / MAX78K (C)",
          "y": 50
        },
        {
          "x": "SimpleNet / MAX78K (R)",
          "y": 50
        },
        {
          "x": "SimpleNet / GAP8",
          "y": 130
        },
        {
          "x": "SimpleNet / HX-WFE2 (S)",
          "y": 115
        },
        {
          "x": "SimpleNet / HX-WFE2 (P)",
          "y": 115
        },
        {
          "x": "SimpleNet / NVP-MCXN847",
          "y": 35
        },
        {
          "x": "AI8XAutoEnc / MAX78K (C)",
          "y": 80
        },
        {
          "x": "AI8XAutoEnc / MAX78K (R)",
          "y": 80
        },
        {
          "x": "AI8XAutoEnc / GAP8",
          "y": 85
        },
        {
          "x": "AI8XAutoEnc / HX-WFE2 (S)",
          "y": 95
        },
        {
          "x": "AI8XAutoEnc / HX-WFE2 (P)",
          "y": 100
        },
        {
          "x": "AI8XAutoEnc / NVP-MCXN847",
          "y": 35
        },
        {
          "x": "YoloV1 / MAX78K (C)",
          "y": 25
        },
        {
          "x": "YoloV1 / MAX78K (R)",
          "y": 25
        },
        {
          "x": "YoloV1 / GAP8",
          "y": 120
        },
        {
          "x": "YoloV1 / HX-WFE2 (S)",
          "y": 115
        },
        {
          "x": "YoloV1 / HX-WFE2 (P)",
          "y": 115
        },
        {
          "x": "YoloV1 / NVP-MCXN847",
          "y": 35
        }
      ]
    },
    {
      "name": "Post-Processing",
      "points": [
        {
          "x": "CIFAR10-NAS / MAX78K (C)",
          "y": 25
        },
        {
          "x": "CIFAR10-NAS / MAX78K (R)",
          "y": 25
        },
        {
          "x": "CIFAR10-NAS / GAP8",
          "y": 30
        },
        {
          "x": "CIFAR10-NAS / HX-WFE2 (S)",
          "y": 90
        },
        {
          "x": "CIFAR10-NAS / HX-WFE2 (P)",
          "y": 90
        },
        {
          "x": "CIFAR10-NAS / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "ResidualNet / MAX78K (C)",
          "y": 25
        },
        {
          "x": "ResidualNet / MAX78K (R)",
          "y": 25
        },
        {
          "x": "ResidualNet / GAP8",
          "y": 35
        },
        {
          "x": "ResidualNet / HX-WFE2 (S)",
          "y": 90
        },
        {
          "x": "ResidualNet / HX-WFE2 (P)",
          "y": 90
        },
        {
          "x": "ResidualNet / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "SimpleNet / MAX78K (C)",
          "y": 25
        },
        {
          "x": "SimpleNet / MAX78K (R)",
          "y": 25
        },
        {
          "x": "SimpleNet / GAP8",
          "y": 35
        },
        {
          "x": "SimpleNet / HX-WFE2 (S)",
          "y": 90
        },
        {
          "x": "SimpleNet / HX-WFE2 (P)",
          "y": 90
        },
        {
          "x": "SimpleNet / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "AI8XAutoEnc / MAX78K (C)",
          "y": 25
        },
        {
          "x": "AI8XAutoEnc / MAX78K (R)",
          "y": 25
        },
        {
          "x": "AI8XAutoEnc / GAP8",
          "y": 35
        },
        {
          "x": "AI8XAutoEnc / HX-WFE2 (S)",
          "y": 90
        },
        {
          "x": "AI8XAutoEnc / HX-WFE2 (P)",
          "y": 90
        },
        {
          "x": "AI8XAutoEnc / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "YoloV1 / MAX78K (C)",
          "y": 25
        },
        {
          "x": "YoloV1 / MAX78K (R)",
          "y": 25
        },
        {
          "x": "YoloV1 / GAP8",
          "y": 40
        },
        {
          "x": "YoloV1 / HX-WFE2 (S)",
          "y": 90
        },
        {
          "x": "YoloV1 / HX-WFE2 (P)",
          "y": 90
        },
        {
          "x": "YoloV1 / NVP-MCXN847",
          "y": 25
        }
      ]
    },
    {
      "name": "Idle",
      "points": [
        {
          "x": "CIFAR10-NAS / MAX78K (C)",
          "y": 25
        },
        {
          "x": "CIFAR10-NAS / MAX78K (R)",
          "y": 25
        },
        {
          "x": "CIFAR10-NAS / GAP8",
          "y": 30
        },
        {
          "x": "CIFAR10-NAS / HX-WFE2 (S)",
          "y": 90
        },
        {
          "x": "CIFAR10-NAS / HX-WFE2 (P)",
          "y": 90
        },
        {
          "x": "CIFAR10-NAS / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "ResidualNet / MAX78K (C)",
          "y": 25
        },
        {
          "x": "ResidualNet / MAX78K (R)",
          "y": 25
        },
        {
          "x": "ResidualNet / GAP8",
          "y": 35
        },
        {
          "x": "ResidualNet / HX-WFE2 (S)",
          "y": 90
        },
        {
          "x": "ResidualNet / HX-WFE2 (P)",
          "y": 90
        },
        {
          "x": "ResidualNet / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "SimpleNet / MAX78K (C)",
          "y": 25
        },
        {
          "x": "SimpleNet / MAX78K (R)",
          "y": 25
        },
        {
          "x": "SimpleNet / GAP8",
          "y": 35
        },
        {
          "x": "SimpleNet / HX-WFE2 (S)",
          "y": 90
        },
        {
          "x": "SimpleNet / HX-WFE2 (P)",
          "y": 90
        },
        {
          "x": "SimpleNet / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "AI8XAutoEnc / MAX78K (C)",
          "y": 25
        },
        {
          "x": "AI8XAutoEnc / MAX78K (R)",
          "y": 25
        },
        {
          "x": "AI8XAutoEnc / GAP8",
          "y": 35
        },
        {
          "x": "AI8XAutoEnc / HX-WFE2 (S)",
          "y": 90
        },
        {
          "x": "AI8XAutoEnc / HX-WFE2 (P)",
          "y": 90
        },
        {
          "x": "AI8XAutoEnc / NVP-MCXN847",
          "y": 25
        },
        {
          "x": "YoloV1 / MAX78K (C)",
          "y": 25
        },
        {
          "x": "YoloV1 / MAX78K (R)",
          "y": 25
        },
        {
          "x": "YoloV1 / GAP8",
          "y": 40
        },
        {
          "x": "YoloV1 / HX-WFE2 (S)",
          "y": 90
        },
        {
          "x": "YoloV1 / HX-WFE2 (P)",
          "y": 90
        },
        {
          "x": "YoloV1 / NVP-MCXN847",
          "y": 25
        }
      ]
    }
  ],
  "precision": "estimated"
}
FIGURE 2. Power consumption breakdown for each stage, model, and µNPU.

memory layouts, and fallback behaviour. Certain operations are not universally supported, necessitating their implementation as CPU-side post-processing steps for relevant models. Each platform also has its own set of supported operator layouts; the MAX78000, for example, only supports 1D convolution with kernel sizes 1 to 9 and 2D with sizes of 1 by 1 or 3 by 3. Unsupported operations will fail to compile or be mapped to CPU execution and incur latency penalties.

We redesign our benchmark models around a shared, mutually compatible operator subset, and aim to avoid platform-specific structural rewrites during compilation. We use INT8 post-training quantization (PTQ), the only universally supported scheme across our platforms. The resulting models are, consequently, not necessarily optimal for each platform. Quantization-aware training and mixed-precision search are outside our scope.

Compilation and Deployment

The various benchmark platforms also support a range of model formats, from vendor-optimized variants of general model formats, e.g., LiteRT, to platform-specific formats. Our compiler workflow ingests Torch / LiteRT models along with various compiler flags (i.e., representative calibration data for PTQ, model input dimensions, platform-specific compiler options) and emits the binary / compiled model required by each platform.

Ethos-U55 models on the HX-WE2 are compiled using Vela [9]. We evaluate both its _Size_ mode, minimizing SRAM usage, and _Performance_ mode, minimizing latency using available arena cache3.

Measurements and Metrics

We measure latency, power and energyefficiency, in terms of number of inference operations per mJ. The impacts of various platform-specific performance optimizations are out of scope. Latency can be considered proportional to throughput, since batching and other amortizations are not practical on the given hardware.

Latency: Latency is measured using each platform's internal timer. We fix CPU/ NPU frequency at 100 MHz. We run 10 consecutive inferences and report mean and standard deviation to account for run-to- run variability4.

Power and energy-efficiency: We measure power using a high voltage power monitor at 50 kHz and 3.3 V, also averaging readings over 10 inferences. We report energy-efficiency as the number of inferences per mJ consumed.

Breaking Down Inference

We break execution into stages and measure per-stage overheads:

1 Peak activation RAM: maximum live activation footprint during the forward pass, with batch=1 and INT8 activations, excluding parameter storage.

2 Lifetime Pressure: sum over all tracked activations of size(t) in MB × lifetime_steps(t), where lifetime_steps(t) counts the number of subsequent leaf-module steps over which activation t must remain live until its last use. Larger values indicate reduced opportunities for buffer reuse.

3 HX-WE2 (S) denotes model compilation with the Size optimization, and (P) with Performance.

4Runs show limited variance under bare-metal execution.

EXPERIMENTAL METHODS

RESULTS AND DISCUSSION

Our benchmarks show that μNPU performance is governed by the complete execution path, and regularly dominated by initialiization or data movement. GOPS alone does not accurately predict deployed performance.

Dedicated neural hardware provides large latency gains over MCU-only execution on convolution-heavy workloads. On CIFAR10-NAS, HX-WE2(P) runs in roughly 11.1 ms, compared with 550 ms on the STM32H7A3ZI and 555 ms on the ESP32, approximately a 50× reduction. On YoloV1, HX-WE2(P) runs in 11.4 ms, compared with 340 ms and 1166 ms on the STM32H7A3ZI and ESP32, approximately 30× and 102× reductions. These gains are not universal: for lightweight models relying more on fully connected layers than convolutions, the STM32H7A3ZI remains competitive.

The MAX78000, with its Cortex-M4 active5, provides the best overall efficiency when NPU initialization is included, with sub-30 ms end-to-end latency across our benchmarks. HX-WE2(P) is nevertheless faster end-to-end: 1.93× faster than the MAX78000 Cortex-M4 configuration and 3.07× faster than its RISC-V configuration, but at 3.13× and 3.33× greater average power draw.

This is counterintuitive because the MAX78000 can achieve lower accelerator-only latency. At 100 MHz, the MAX78000 Cortex-M4 configuration gives an average 2.48× inference-latency improvement over HX-WE2 (P). HX-WE2 wins end-to-end because of lower I/O overheads: on the MAX78000, I/O takes 6.10× longer than inference with the Cortex-M4 active and 9.80× longer with the RISC-V. In the worst case, ResidualNet on the RISC-V spends 44.89 ms on I/O but only 2.96 ms on inference, meaning over 90% of end-to-end time is I/O, largely from moving weights into accelerator memory [7, 8].

Vela's _Performance_ optimizations mostly reduce HX-WE2 latency compared with Size, but this is non-uniform across models, and energy-efficiency can degrade as model complexity grows.

The NXP-MCXN947 is the clearest example of GOPS being a weak proxy for deployed performance. Although its eIQ Neutron accelerator is rated at only 4.8 GOPS, it achieves consistent sub-30 ms end-to-end latency, comparable to the MAX78000, due to low initialization and memory-transfer overheads.

WHAT NEXT FOR μNPUs?

Hardware-aware model design. Neural architecture search should target platform-specifics constraints, alongside activation lifetime, buffer reuse, operator support, and data movement.

Improved neural operator support. Most platforms support limited operators, optimized for linear and convolutional layers plus pooling, element-wise activation, and addition, but excludes operators central to other model families. Unsupported layers can, in principle, be offloaded to the CPU, but this requires a tightly integrated CPU/ NPU software stack that minimizes data transfer and synchronization overheads. Fine-grained CPU fallback is still limited on most evaluated platforms, and post-inference reads of intermediate activations can be latency-expensive.

Dynamic execution. Static deployment restricts early-exit networks [14], conditional computation, runtime model selection, and other input-adaptive methods for reducing latency and energy.

Standardized model formats and profiling. Unified deployment flows, reliable latency/ energy profiling, and memory-aware profilers would reduce cross-platform engineering overhead and expose bottlenecks earlier [15].

PRACTICAL GUIDANCE FOR DEVELOPERS

The MAX78000 achieves the strongest GOPS/mW efficiency and lowest idle power. Its main limitation lies with I/O: data movement can vastly exceed NPU execution time. Use it when models can remain resident and power-gating can be exploited.

Despite only supporting 4.8 GOPS, the MCXN947 remains competitive because of low initialization and I/O overheads.

While it does not achieve competitive latency, its 8 MB L3 SRAM and 20 MB flash are valuable for hosting larger models or model ensembles.

figure
Figure description (VLM): A scatter plot of Average Power (mW) versus End-to-End Latency (ms), with points color-coded by hardware platform (e.g., MAX78k, GAP8, HX-WE2, NXP-MCXN947, STM32, ESP32) and shape-coded by dataset (CIFAR10-NAS, ResidualNet, SimpleNet, AI8XAutoEnc, YoloV1). An inset scatter plot in the upper-middle zooms into the low-latency (10-50 ms) region. No exact numeric values are printed on the points.
Figure data (VLM re-extraction)
{
  "x_axis": {
    "label": "End-to-End Latency",
    "unit": "ms"
  },
  "y_axis": {
    "label": "Average Power",
    "unit": "mW"
  },
  "series": [
    {
      "name": "MAX78k (C)",
      "points": [
        {
          "x": 0,
          "y": 10
        },
        {
          "x": 10,
          "y": 12
        },
        {
          "x": 13,
          "y": 9
        },
        {
          "x": 18,
          "y": 7
        },
        {
          "x": 24,
          "y": 4
        },
        {
          "x": 26,
          "y": 6
        },
        {
          "x": 27,
          "y": 9
        },
        {
          "x": 30,
          "y": 4
        },
        {
          "x": 40,
          "y": 3
        },
        {
          "x": 50,
          "y": 2
        }
      ]
    },
    {
      "name": "MAX78k (R)",
      "points": [
        {
          "x": 0,
          "y": 5
        },
        {
          "x": 13,
          "y": 8
        },
        {
          "x": 18,
          "y": 7
        },
        {
          "x": 24,
          "y": 4
        },
        {
          "x": 26,
          "y": 6
        },
        {
          "x": 27,
          "y": 9
        },
        {
          "x": 30,
          "y": 4
        },
        {
          "x": 40,
          "y": 3
        },
        {
          "x": 50,
          "y": 2
        }
      ]
    },
    {
      "name": "GAP8",
      "points": [
        {
          "x": 100,
          "y": 3
        },
        {
          "x": 12,
          "y": 18
        },
        {
          "x": 48,
          "y": 6
        },
        {
          "x": 50,
          "y": 7
        }
      ]
    },
    {
      "name": "HX-WE2 (S)",
      "points": [
        {
          "x": 0,
          "y": 85
        },
        {
          "x": 0,
          "y": 35
        },
        {
          "x": 10,
          "y": 36
        },
        {
          "x": 11,
          "y": 35
        },
        {
          "x": 12,
          "y": 33
        }
      ]
    },
    {
      "name": "HX-WE2 (P)",
      "points": [
        {
          "x": 0,
          "y": 40
        },
        {
          "x": 0,
          "y": 38
        },
        {
          "x": 10,
          "y": 39
        },
        {
          "x": 11,
          "y": 36
        }
      ]
    },
    {
      "name": "NXP-MCXN947",
      "points": [
        {
          "x": 0,
          "y": 83
        },
        {
          "x": 12,
          "y": 35
        },
        {
          "x": 13,
          "y": 8
        },
        {
          "x": 18,
          "y": 7
        },
        {
          "x": 20,
          "y": 6
        },
        {
          "x": 26,
          "y": 4
        }
      ]
    },
    {
      "name": "STM32",
      "points": [
        {
          "x": 0,
          "y": 20
        },
        {
          "x": 270,
          "y": 1
        },
        {
          "x": 320,
          "y": 1
        },
        {
          "x": 380,
          "y": 1
        }
      ]
    },
    {
      "name": "ESP32",
      "points": [
        {
          "x": 380,
          "y": 1
        },
        {
          "x": 550,
          "y": 1
        },
        {
          "x": 1150,
          "y": 1
        }
      ]
    }
  ],
  "precision": "estimated"
}
FIGURE 3. A visualization of average end-to-end latency vs. power draw for evaluated models and platforms. The inset graph provides a magnified view of platforms with lower end-to-end latency.

5 MAX78k (C) denotes use of its Cortex-M CPU, and (R) its RISC-V CPU.

Baseline MCUs: viable for convolution-light models, particularly those dominated by fully connected layers or simple 1D operations.

Limitations

We normalize CPU and NPU operating frequencies, improving comparability but not capturing each platform's peak performance. INT8 quantization, again, improves comparability but hides platform-specific advantages such as MAX78000 sub-byte quantization or HX-WE2 floating-point acceleration. Our shared NPU-compatible operator set also excludes platform-specific redesigns, QAT, operator fusion, and broader layer support.

CONCLUSIONS

Our evaluation reveals both expected trends and surprising disparities. Dedicated accelerators can deliver up to two orders of magnitude lower latency than MCUs on convolution-heavy models, yet GOPS alone does not explain real-world performance. As embedded deployments become more diverse and memory-intensive, the next gains will come not only from higher arithmetic throughput, but also from reducing data movement, broadening operator support, and improving deployment-level profiling. ■

Glossary

GOPS - Giga Operations Per Second. Peak arithmetic throughput of an accelerator.

Systolic array. Grid of processing elements, each with a multiply-accumulate unit and local memory, that passes data between neighbouring elements to parallelize tensor operations.

Activation lifetime. How long an inter-mediate tensor remains in memory before its final use, used to estimate memory pressure.

Quantization-aware training (QAT). Training a model while simulating quantization effects, often improving accuracy.

Operator fusion. Combining multiple model operations into one optimized execution step to reduce memory movement and overhead.

Josh Millar is a PhD candidate in the Department of Computing at Imperial College London, U.K. His research focuses on machine learning for embedded systems, with an emphasis on microcontroller-scale accelerators, low-power inference, and energy-aware system design for resource-constrained sensing platforms.

Yushan Huang is a senior researcher at 2012 Labs, Huawei, Beijing, China. He received a PhD from Imperial College London, U.K. His research focuses on low-energy on-device machine learning and the resource characterization of accelerator-equipped microcontrollers.

Sarab Sethi is an Assistant Professor in Ecosystem Sensing at Imperial College London, U.K., where he leads the Ecosystem Sensing research group. His research focuses on developing scalable, real-time sensing systems and machine-learning approaches for understanding natural environments, with a particular focus on acoustic sensing for autonomous biodiversity monitoring. His work bridges embedded systems, ecological data analysis, and practical conservation decision-making.

Hamed Haddadi is the Professor of Human-Centred Systems in the Department of Computing at Imperial College London, U.K. In his industrial role, he is the Chief Scientist at Brave Software, where he works on developing privacy-preserving analytics protocols. He is also the Co-Founder and CSO of Mulini SRL, focusing on securing consumer and industrial cyber-physical systems. His research spans IoT, applied machine learning, security, and privacy-preserving systems, including the design of AIoT platforms that can process data efficiently on-device while respecting users' privacy.

Anil Madhavapeddy is the Professor of Planetary Computing at the University of Cambridge Computer Laboratory, U.K. He co-directs the Centre for Earth Observation and the Cambridge Centre for Carbon Credits, with the aim of accelerating natural climate solutions that contribute towards ending deforestation and improving biodiversity globally. He has decades of experience constructing large-scale systems and has contributed to many open-source projects such as OCaml, Docker, Xen, and OpenBSD.

REFERENCES

References

  1. Saha S.S.; Sandha S.S.; Srivastava M. (2022) Machine learning for microcontrollerclass hardware: A review. IEEE Sensors Journal 22, 21362-21390
    raw S.S. Saha, S.S. Sandha, and M. Srivastava. 2022. Machine learning for microcontroller-class hardware: A review. _IEEE Sensors Journal_, 22(22):21362–21390.
  2. Lin J.; Chen W.-M.; Lin Y.; Gan C.; Han S. (2020) MCUNet: Tiny deep learning on IoT devices. NeurIPS 33, 11711-11722
    raw J. Lin, W.-M. Chen, Y. Lin, C. Gan, S. Han, et al. 2020. MCUNet: Tiny deep learning on IoT devices. _NeurIPS_, 33:11711–11722.
  3. Sze V.; Chen Y.-H.; Yang T.-J.; Emer J. S. (2020) How to evaluate deep neural network processors: TOPS/W (Alone) considered harmful. IEEE Solid-State Circuits Magazine 12, 28-41
    raw V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer. 2020. How to evaluate deep neural network processors: TOPS/W (Alone) considered harmful. _IEEE Solid-State Circuits Magazine_, 12(3):28–41.
  4. Moss A.; Lee H.; Xun L.; Min C.; Kawsar F.; Montanari A. (2022) Ultra-low-power DNN accelerators for IoT: Resource characterization of the MAX78000. SenSys 934-940
    raw A. Moss, H. Lee, L. Xun, C. Min, F. Kawsar, and A. Montanari. 2022. Ultra-low-power DNN accelerators for IoT: Resource characterization of the MAX78000. 2022. _SenSys_, 934–940.
  5. Clay M.; Grecos C.; Shirvaikar M.; Richey B. (2022) Benchmarking the MAX78000 artificial intelligence microcontroller for deep learning applications. Real-Time Image Processing and Deep Learning 2022, SPIE
    raw M. Clay, C. Grecos, M. Shirvaikar, and B. Richey. 2022. Benchmarking the MAX78000 artificial intelligence microcontroller for deep learning applications. _Real-Time Image Processing and Deep Learning_ 2022, SPIE.
  6. Huang Y.; Gong T.; Jang S.; Kawsar F.; Min C. (2024) Energy characterization of tiny AI accelerator-equipped microcontrollers. HumanSys 1-6
    raw Y. Huang, T. Gong, S. Jang, F. Kawsar, and C. Min. 2024. Energy characterization of tiny AI accelerator-equipped microcontrollers. 2024. _HumanSys_, 1–6.
  7. Jeon C.; Gong T.; Yi J.; Kawar F.; Min C. (2025) TinyMem: Boosting multi-DNN inference on tiny AI accelerators with weight memory virtualization. HotMobile 1-6
    raw C. Jeon, T. Gong, J. Yi, F. Kawar, and C. Min. 2025. TinyMem: Boosting multi-DNN inference on tiny AI accelerators with weight memory virtualization. 2025. _HotMobile_, 1-6.
  8. Gong T.; Kawsar F.; Min C. (2025) DEX: Data channel extension for efficient CNN inference on tiny AI accelerators. NeurIPS 37, 43925-43951
    raw T. Gong, F. Kawsar, and C. Min. 2025. DEX: Data channel extension for efficient CNN inference on tiny AI accelerators. 2025. _NeurIPS_, 37:43925–43951.
  9. Arm Ltd. (2024) Ethos-U Vela Compiler. Arm Developer Documentation
    raw Arm Ltd. “Ethos-U Vela Compiler.” Arm Developer Documentation, Version 1.3, 2024. https://developer.arm.com/documentation/109267/0103/Tool-support-for-the-Arm-Ethos-U-NPU/Ethos-U-Vela-compiler
  10. Analog Devices (2024) MAX78000 Artificial Intelligence Microcontroller with Ultra-Low-Power Convolutional Neural Network Accelerator. Datasheet 5
    raw Analog Devices. “MAX78000 Artificial Intelligence Microcontroller with Ultra-Low-Power Convolutional Neural Network Accelerator.” Datasheet, Rev. 5, 2024. https://www.analog.com/ media/en/technical-documentation/data-sheets/ max78000.pdf
  11. GreenWaves Technologies (2020) GAP8 IoT Application Processor. Product Brief 1
    raw GreenWaves Technologies. “GAP8 IoT Applica- tion Processor.” Product Brief, Version 1.9, 2020.
  12. Himax Technologies (2023) HX6538-A WE2 AI Processor. Preliminary Datasheet
    raw Himax Technologies. “HX6538-A WE2 AI Processor.” Preliminary Datasheet, Version 01, Jul. 2023. https://files.seedstudio.com/wiki/grove-vision-ai-v2/HX6538_datasheet.pdf
  13. NXP Semiconductors (2024) FRDM-MCXN947 Board User Manual. User Manual UM12018
    raw NXP Semiconductors. “FRDM-MCXN947 Board User Manual.” User Manual UM12018, Rev. 2.0, Aug. 2024. https://docs.nxp.com/bundle/UM12018/page/topics/Evk_overview.html
  14. Laskaridis S.; Kouris A.; Lane N.D. (2021) Adaptive inference through early-exit networks: Design, challenges and directions. Éditions Maison des Langues: EMDL 1-6
    raw S. Laskaridis, A. Kouris, and N.D. Lane. 2021. Adaptive inference through early-exit networks: Design, challenges and directions." _Éditions Maison des Langues: EMDL_, 1-6.
  15. Reddi V.J.; Kanter D.; Mattson P.; Duke J.; Nguyen T.; Chukka R.; Shiring K.; Tan K.-S.; Charlebois M.; Chou W. (2022) MLPerf mobile inference benchmark: An industrystandard open-source machine learning benchmark for on-device AI. Proceedings of Machine Learning and Systems 4, 352-369
    raw V.J. Reddi, D. Kanter, P. Mattson, J. Duke, T. Nguyen, R. Chukka, K. Shiring, K.-S. Tan, M. Charlebois, W. Chou, et al. 2022. MLPerf mobile inference benchmark: An industry-standard open-source machine learning benchmark for on-device AI. _Proceedings of Machine Learning and Systems_, 4:352–369.