EThos-U55 / U65: Arm ML Processor for Cortex-M Systems

Running a MobileNet-v2 or DS-CNN keyword-spotting model purely on a Cortex-M CPU works, but it burns cycles and power that a battery-powered product often can't spare — CMSIS-NN on an M55 still spends the bulk of its time on MAC-heavy conv layers. Arm's answer is the Ethos-U microNPU family: a small, in-order, fixed-function accelerator that sits beside a Cortex-M (or Cortex-M/Ethos-U combined as an "ML island") and offloads exactly the operators that dominate CNN inference — convolution, depthwise convolution, pooling, and fully-connected layers — while the CPU handles control flow and unsupported ops.

Where It Fits in the System

Ethos-U55 and U65 are NPUs, not standalone processors. They have no instruction fetch of their own in the traditional sense — a Cortex-M (typically M33, M55, or M85) programs them via a command stream generated by the Vela compiler, and the NPU executes that stream autonomously, raising an interrupt on completion. The CPU stays free to run RTOS tasks, sensor drivers, or other DSP work while inference executes in the background.

 Sensor → DMA → SRAM buffer
                   │
   Cortex-M33/M55  │   Ethos-U55/U65 (NPU)
   ┌─────────────┐ │   ┌───────────────────┐
   │ RTOS, glue  │◄┼──►│ MAC array          │
   │ code, unsup-│ │   │ weight decoder     │
   │ ported ops  │ │   │ DMA + SRAM buffers │
   └─────────────┘ │   └───────────────────┘
          ▲         AXI/AHB bus, shared SRAM/Flash
          └── interrupt on command-stream completion

The key architectural distinction between the two variants:

Both are licensed as part of Arm's Corstone reference subsystems (Corstone-300, -310, -1000) alongside the CPU, making them common in silicon like Alif Semiconductor's Ensemble series or NXP's newer i.MX RT parts.

The Compute Model: What Actually Gets Offloaded

The U55/U65 accelerates a fixed set of quantized INT8/INT16 operators defined by the TOSA-like operator set Arm publishes for Ethos-U:

• Conv2D, depthwise Conv2D, transpose Conv2D
• Fully connected (matmul)
• Average/max pooling
• Elementwise add/mul, LUT-based activations (ReLU, tanh, sigmoid approximated via lookup)
• Resize (nearest/bilinear) for some networks

Anything outside that set — custom ops, unsupported activation functions, or FP32 layers — falls back to the Cortex-M CPU running CMSIS-NN or a reference kernel. This is why operator coverage, not raw MAC count, is usually the first-order driver of end-to-end latency: a network with even one CPU-fallback layer in the middle of the graph forces a stall while the CPU catches up, plus a memory round-trip.

MAC array and quantization

The MAC array operates exclusively on quantized integer tensors — INT8 weights/activations as the baseline, with INT16 activation support on U55/U65 for higher-precision layers. There is no native FP32 or FP16 path; a trained FP32 model must go through post-training quantization or quantization-aware training before Vela can compile it. The per-layer scale/zero-point (affine quantization) parameters are baked into the command stream, so requantization between layers happens in hardware without CPU intervention.

Vela: Compiling a Model for Ethos-U

The toolchain flow is:

TFLite Micro (.tflite, INT8) → Vela compiler → optimized .tflite + command stream
                                              → deployed alongside TFLM runtime

Vela performs:

1. Operator partitioning — splits the graph into NPU-supported subgraphs and CPU-fallback subgraphs.
2. Tensor scheduling — decides how activations are tiled and staged through the small on-chip SRAM buffer (typically tens of KB), since intermediate feature maps rarely fit entirely on-chip.
3. Weight compression — Ethos-U weights are stored in a compressed format read directly by the weight decoder, reducing Flash footprint and read bandwidth.
4. Command stream generation — the sequence of NPU register/DMA operations the CPU hands off at inference time.

The compiler also reports, per layer, whether it mapped to the NPU or fell back to CPU — this report is the primary tool for finding accidental CPU stalls in a graph (e.g., an unsupported PAD mode or a resize op not in the supported list).

Worked Example: Sizing a Person-Detection Pipeline

Take a MobileNet-v2 (0.35×, 96×96 input) person-detection model with roughly 8 MMAC (million multiply-accumulates) per inference frame, fully mapped onto an Ethos-U55 configured for 128 MACs/cycle at 200 MHz.

Step 1 — theoretical peak cycles:

MACs per frame     = 8 × 10⁶
MACs/cycle (NPU)   = 128
Ideal cycles       = 8×10⁶ / 128 = 62,500 cycles

Step 2 — convert to time at 200 MHz:

t_ideal = 62,500 / 200×10⁶ Hz = 312.5 µs

Step 3 — apply realistic utilization. Depthwise-heavy networks like MobileNet rarely hit 100% MAC utilization because depthwise convolution has low arithmetic intensity relative to weight/activation movement; 40–60% utilization is typical for Ethos-U on MobileNet-class graphs. Assume 50%:

t_actual ≈ t_ideal / 0.5 = 625 µs

Step 4 — check against a frame budget. At a target of 10 fps (100 ms/frame budget), 625 µs leaves >99% of the frame time free for the CPU to do sensor read, preprocessing, and post-processing — confirming the NPU is not the bottleneck here and the CPU-side image resize/crop (see Image Preprocessing on MCU) is likely the larger cost.

Verification: 8 MMAC at 128 MACs/cycle and 50% utilization should also equal roughly 2× the ideal-cycle time; 312.5 µs × 2 = 625 µs — consistent. Power-wise, at ~1 mW/MHz-ish scaling typical for these small NPUs (implementation-dependent), 625 µs of active NPU time per 100 ms frame corresponds to a duty cycle low enough that average power is dominated by the CPU's preprocessing path and any always-on sensor front end, not the NPU itself — the usual outcome for U55 designs and the reason it targets battery/coin-cell products.

Practical Design Implications

• SRAM sizing dominates system cost. Because the NPU stages activations through on-chip SRAM, undersized SRAM forces more DMA traffic to/from Flash or external memory, eroding the utilization assumed above — always check Vela's memory report, not just the operator-mapping report.
• Check the fallback report before trusting a datasheet MAC number. A network with 95% NPU coverage on paper can still be CPU-bound if the remaining 5% sits on the critical path with no pipelining opportunity.
• U65's AXI/DRAM path changes the tradeoff. For larger models (person + object detection combined, or higher-resolution input), U65's ability to address external DRAM avoids the SRAM-fitting problem but reintroduces DRAM latency/power as a new constraint — a subsystem-level (Cortex-A-class) decision, not a pure MCU one.
• Quantization strategy is not optional. Since the MAC array is integer-only, model accuracy after INT8 quantization (post-training or QAT) must be validated before committing to hardware sizing — a poorly quantized model may need per-channel scaling or QAT to recover accuracy, which changes the compiled command stream size.
• It composes with CMSIS-NN, not replaces it. Cortex-M fallback layers still run through CMSIS-NN kernels, so overall latency is the sum of NPU time and CPU/CMSIS-NN time for the unsupported subgraphs — profile both.

Key Takeaways

• Ethos-U55/U65 are command-driven microNPUs paired with a Cortex-M (U55, AHB, MCU SRAM/Flash) or Cortex-M/A subsystem (U65, AXI, external DRAM), not standalone processors.
• They accelerate a fixed INT8/INT16 operator set (conv, depthwise conv, pooling, FC, elementwise, LUT activations); unsupported ops fall back to CPU/CMSIS-NN, which is usually the dominant source of unexpected latency.
• The Vela compiler partitions the graph, schedules on-chip SRAM tiling, compresses weights, and emits the command stream — its per-layer mapping and memory reports are the primary tuning tools.
• Real-world MAC utilization for depthwise-heavy CNNs like MobileNet is typically 40–60% of peak, not 100% — size performance and power budgets from measured/estimated utilization, not the peak MACs/cycle spec.
• On-chip SRAM capacity, not MAC count alone, is often the practical limiter for U55-class designs; U65's DRAM path trades that constraint for external-memory latency and power.