MCU Inference Benchmarks: MLPerf Tiny Results

Choosing an MCU for an embedded vision or audio model is guesswork without a common yardstick — a vendor's "AI-ready" claim means nothing until you know what workload, what quantization, and what latency it was measured under. MLPerf Tiny is the closest thing the industry has to a standardized dyno test for microcontroller inference, and reading its results correctly (not just the leaderboard number) is a skill every engineer picking silicon for a TinyML product needs.

What MLPerf Tiny Actually Measures

MLPerf Tiny is a benchmark suite from MLCommons (the same body behind datacenter MLPerf) targeting systems in the sub-1W, memory-constrained class — Cortex-M, RISC-V MCUs, and small NPUs. It defines four fixed reference workloads so that every submitter runs the same model architecture and dataset, not a vendor-optimized substitute:

Two metrics are reported per benchmark, and both matter independently:

• Latency — median inference time over repeated runs, in milliseconds, measured through the EEMBC EnergyRunner harness on real hardware (not simulation).
• Energy per inference — microjoules, measured with an external power monitor (e.g., an EEMBC-approved energy shield), optional in the closed division but where most differentiation actually happens.

Accuracy is fixed at (or above) a reference threshold — submitters can quantize and optimize, but the model must clear a minimum top-1/AUC bar. This decouples "fast but wrong" from legitimate results: a 0.3 ms KWS inference that fails accuracy doesn't get a leaderboard entry.

Closed vs. Open Division — Why It Changes What the Numbers Mean

This is the single most misread part of MLPerf Tiny, and it's why two submissions with the same benchmark name can differ 10× in latency for reasons that have nothing to do with silicon speed.

• Closed division: the reference model architecture is fixed. Submitters may quantize (typically INT8), retrain, and apply hardware-specific kernel scheduling, but they cannot change the network topology. This isolates hardware + software stack (compiler, kernel library, accelerator) as the variable — the only fair cross-chip comparison.
• Open division: the model itself can be replaced (different architecture, pruned, distilled, different input resolution) as long as the accuracy target is still met. This showcases what a chip can do with the "wrong" workload for it swapped out for something friendlier — useful for algorithm research, but not comparable across submitters.

Practical rule: when comparing two MCUs from a leaderboard, only compare closed-division numbers to each other, and only within the same benchmark. Mixing a closed VWW number against an open VWW number (or comparing VWW latency to KWS latency) produces a meaningless ratio.

Reading a Result Correctly: A Worked Comparison

Suppose a leaderboard shows two closed-division VWW submissions:

Device A (Cortex-M4F @ 80 MHz, CMSIS-NN):     latency = 38.2 ms,  energy = 950 µJ
Device B (Cortex-M55 + Ethos-U55 @ 200 MHz):  latency = 2.1 ms,   energy = 180 µJ

Two numbers to derive beyond the raw table:

1. Throughput ratio:

Speedup = 38.2 ms / 2.1 ms ≈ 18.2×

2. Energy-delay product (EDP) — the metric that actually matters for a battery-powered always-on sensor, since it penalizes designs that are fast only by burning more power:

EDP_A = 950 µJ × 38.2 ms ≈ 36.3 µJ·ms
EDP_B = 180 µJ × 2.1 ms  ≈ 0.378 µJ·ms
EDP ratio ≈ 96×

Device B isn't just 18× faster — its energy-delay product is ~96× better, meaning the NPU-assisted path wins on both axes simultaneously (an accelerator amortizes fixed overhead and cuts active cycles, so both latency and energy drop together, not one at the expense of the other). This is the check to run before trusting a single "X× faster" marketing headline: compute EDP, not just latency ratio, because a chip can win latency while losing badly on energy (e.g., a high-clock core with no low-power idle path).

Verification: 950 µJ over 38.2 ms implies an average power draw of 950 µJ / 38.2 ms ≈ 24.9 mW for device A; 180 µJ / 2.1 ms ≈ 85.7 mW for device B. Device B draws over 3× the power while running, but finishes so much faster that total energy per inference is still 5.3× lower (950/180 ≈ 5.3×) — consistent with the EDP calculation. The numbers close.

Why Results Vary So Much Between Submissions

• Compiler/kernel library: the same Cortex-M4 running hand-tuned CMSIS-NN INT8 kernels vs. a generic C reference implementation can differ 5–10× in latency for identical silicon.
• Clock and memory configuration: submissions often run at a specific voltage/frequency operating point tuned for the energy metric, not the chip's maximum rated clock — comparing against a datasheet's "max MHz" figure will not reproduce the leaderboard number.
• Quantization scheme: INT8 post-training quantization is near-universal in closed submissions, but the calibration dataset and per-channel vs. per-tensor scaling affect both accuracy margin and achievable latency.
• Memory placement: whether weights/activations run from on-chip SRAM vs. external flash/PSRAM with wait states materially changes latency; the harness reports what was actually measured, but two "same MCU" boards with different memory strapping are not really the same test.

Practical Implications for Design Decisions

• Use closed-division numbers for silicon selection, and always pull both latency and energy — a chip that tops the latency table but omits the energy submission may simply not have run it, not necessarily be efficient.
• Match the benchmark to your workload class, not just the chip family: a strong KWS score says little about VWW (CNN-on-image) performance, since kernel mixes (depthwise conv vs. dense FC) stress different parts of a kernel library and memory subsystem.
• Treat the leaderboard as a stack benchmark, not a silicon benchmark — CMSIS-NN version, compiler version, and quantization toolchain are part of every result; the same die under an immature software stack will underperform its ceiling.
• Recompute EDP yourself when comparing candidates for a battery-life-constrained product; MLPerf Tiny's own summary tables emphasize latency, but energy is usually the binding constraint in the field.
• Verify accuracy, not just speed, when a result looks unusually fast — under the closed division it must clear the reference accuracy floor, but open-division "fast" results may have traded accuracy for latency in ways your application can't tolerate.

Key Takeaways

• MLPerf Tiny standardizes four workloads (VWW, IC, KWS, AD) with a fixed reference model per benchmark, measuring latency (ms) and energy (µJ) on real hardware via the EEMBC EnergyRunner harness.
• Closed division fixes the model architecture (only quantization/optimization allowed) and is the only division valid for cross-chip comparison; open division allows model substitution and is not comparable across submitters.
• Always compare like-for-like: same benchmark, same division, and derive energy-delay product (EDP = energy × latency) rather than trusting a single latency ratio, since it exposes designs that are fast only by drawing more power.
• Results depend heavily on compiler/kernel library maturity, quantization scheme, clock/voltage operating point, and memory placement — the leaderboard measures a hardware+software stack, not silicon in isolation.
• For product decisions, prioritize the benchmark closest to your actual workload and recompute EDP against your own duty cycle, since MLPerf Tiny's headline metric (latency) may not be the constraint that limits your battery life.