TinyML Benchmarking: eAI, TinyMLPerf, MLCommons

Selecting an MCU or NPU for an edge-AI product used to mean trusting vendor slide decks quoting "X TOPS" or "Y ms inference" numbers measured on cherry-picked models with unstated quantization, memory placement, and clocking assumptions. TinyML benchmarking standardizes this so that a number from one vendor is comparable to a number from another. Getting this wrong means selecting silicon based on a synthetic peak-throughput figure that has nothing to do with the actual model, memory footprint, and power budget of the product you're shipping.

Why Vendor Numbers Aren't Comparable by Default

A raw "TOPS" figure describes peak MAC throughput of the datapath under ideal conditions — 100% utilization, no memory stalls, best-case data type. Real inference on real models rarely reaches that ceiling because:

• Memory bandwidth often bottlenecks before compute does, especially for depthwise-separable convolutions with low arithmetic intensity.
• Quantization scheme (INT8 symmetric vs. asymmetric, per-tensor vs. per-channel) changes both accuracy and cycle count, but isn't always disclosed.
• Model chosen for the demo may not resemble your workload — a vendor benchmarking MobileNet-v1 on image classification tells you little about a DS-CNN keyword-spotting workload's latency.
• Measurement boundary varies: some numbers include only the NPU compute kernel, others include DMA setup, preprocessing, and postprocessing overhead — a difference that can be 2–5× in reported latency.

TinyML benchmarking suites exist to close these gaps by fixing the model, dataset, quantization rules, and measurement methodology, so the only variable left is the hardware/software stack under test.

MLPerf Tiny: The Reference Suite

MLPerf Tiny (developed under MLCommons, the same organization behind datacenter MLPerf) is the closest thing the industry has to a standardized TinyML benchmark. It targets microcontroller-class and low-power edge devices — sub-1 mW to a few hundred mW — and defines four reference tasks:

Two submission tracks matter:

• Closed division: model architecture and quantization are fixed exactly as specified. This isolates hardware/runtime efficiency as the only variable — the fair apples-to-apples comparison.
• Open division: submitters can retrain, prune, or re-architect the model, as long as it performs the same task. This shows what's achievable with hardware-aware optimization, but is not directly comparable across submitters.

Each submission reports three numbers together, not in isolation:

Accuracy (%)         — task quality after quantization
Latency (ms)          — single-inference wall-clock time
Energy (µJ/inference) — energy consumed per inference

Reporting only one of these is close to meaningless: a submitter can trivially win on latency by dropping accuracy, or win on energy by underclocking at the cost of latency. MLPerf Tiny requires all three so trade-offs are visible.

Energy measurement methodology is arguably the suite's most valuable contribution: it specifies how to instrument the board (voltage rails to monitor, sampling rate, averaging window, idle-vs-active boundary detection) so that energy-per-inference numbers from different silicon vendors are actually comparable — something that was previously ad hoc and inconsistent across datasheets.

eAI: Embedded AI Benchmarking Beyond the Reference Models

"eAI" (embedded AI) benchmarking, as used by groups like EEMBC (Embedded Microprocessor Benchmark Consortium, the same body behind CoreMark), extends the idea to workloads and platforms MLPerf Tiny doesn't fully cover — production-representative embedded neural network kernels, sensor fusion pipelines, and vision workloads run on the actual RTOS/bare-metal stack a product ships with, rather than a bare reference harness.

Key differences from MLPerf Tiny in practice:

• Broader kernel coverage: individual operator-level benchmarks (convolution, depthwise conv, fully-connected, LSTM cell) in addition to full-model benchmarks, useful for library-level optimization work (e.g., validating a CMSIS-NN kernel improvement).
• Score normalization: results are often expressed as a composite score (analogous to CoreMark/MHz) rather than raw ms, intended for quick cross-part comparison during part selection rather than deep analysis.
• Vendor self-certification with EEMBC's review process, which is lighter-weight than MLPerf's peer-reviewed submission process — useful for faster vendor turnaround, at the cost of somewhat less rigorous cross-validation.

MLCommons: The Governance Layer

MLCommons is the nonprofit consortium that governs MLPerf (both datacenter/training/inference and Tiny). Its role is process, not silicon:

• Defines the reference model, dataset, and pre/post-processing exactly (down to normalization constants and file hashes) so no ambiguity exists about "what was run."
• Runs a peer-review process on submissions — other member companies can challenge a result before it's published.
• Maintains versioned benchmark suites (e.g., MLPerf Tiny v1.0, v1.1...) so that comparisons stay valid within a version and rules changes are tracked, rather than results silently drifting as reference implementations evolve.

This matters for engineers because a published MLPerf Tiny result is traceable back to an exact model checkpoint and quantization recipe — you can, in principle, reproduce it on your own board and check whether your integration is leaving performance on the table.

Worked Example: Comparing Two Parts on Keyword Spotting

Suppose two Cortex-M–class parts both publish MLPerf Tiny closed-division keyword-spotting results:

Part A: DS-CNN INT8, 172 KB model
  Latency:  12.4 ms
  Energy:   187 µJ/inference
  Accuracy: 90.2%

Part B: DS-CNN INT8, 172 KB model
  Latency:  6.1 ms
  Energy:   410 µJ/inference
  Accuracy: 90.1%

Same model, same accuracy (within noise) — this is exactly the case the closed division is designed to produce. Part B is ~2× faster but consumes ~2.2× more energy per inference. Which is "better" depends entirely on the product constraint:

• Battery-life-dominated product (e.g., always-on wake-word listening on a coin cell): energy/inference is the deciding metric → Part A wins. Estimate battery life: at 1 inference/second continuous listening, Part A draws 187 µJ × 1 Hz = 187 µW average from inference alone; Part B draws 410 µW — over 2× the inference-related drain, which can dominate a low-power budget where the rest of the system idles at tens of µW.
• Latency-dominated product (e.g., voice UI where response time is user-visible): Part B's 6.1 ms leaves more headroom in a real-time budget, even at higher energy cost.

Check: energy ≈ average power × latency. For Part A: if average active power during inference is P_A, then P_A × 12.4 ms = 187 µJ → P_A ≈ 15.1 mW. For Part B: P_B × 6.1 ms = 410 µJ → P_B ≈ 67.2 mW. Part B's silicon draws over 4× the active power of Part A to achieve 2× the speed — consistent with a higher-clock, higher-leakage design point rather than a pure architectural win. This is the kind of insight a single "TOPS" number would have hidden completely, and it only becomes visible because the benchmark reports latency and energy together on an identical, fixed model.

Practical Guidance for Part Selection

• Match the benchmark task to your workload. A part that leads on MLPerf Tiny's visual wake words benchmark says nothing about anomaly-detection latency — check the specific task closest to your model type.
• Read closed-division numbers for hardware comparison; open-division for optimization ceiling. Don't compare a closed-division number from one vendor against an open-division number from another — they answer different questions.
• Verify the measurement boundary. Confirm whether reported latency/energy includes DMA transfer and preprocessing, or only the NPU/CPU compute kernel — ask the vendor or check the submission's code repository (MLPerf Tiny submissions must include reproducible reference code).
• Re-benchmark your actual model on shortlisted hardware before committing. Reference-model benchmarks narrow the field; they don't replace running your quantized model, at your target frame rate, on your target board.

Key Takeaways

• Vendor "TOPS" and single-point latency claims are not comparable across parts because model, quantization, and measurement boundary are usually unstated — TinyML benchmarks fix these variables.
• MLPerf Tiny (MLCommons) defines four reference tasks — keyword spotting, visual wake words, image classification, anomaly detection — with closed (fixed model) and open (optimized model) divisions.
• Valid MLPerf Tiny results report accuracy, latency, and energy per inference together; any one number alone can be gamed.
• EEMBC's eAI benchmarking complements MLPerf Tiny with broader operator-level and production-representative workload coverage, often expressed as composite scores for quick part screening.
• MLCommons provides the governance (fixed reference implementations, peer review, versioning) that makes published results reproducible and genuinely comparable.
• Use standardized benchmarks to shortlist hardware, then always re-benchmark your actual model and target frame rate on the candidate board before final selection.