CMSIS-NN: ARM-Optimized Kernel Library for Cortex-M

Running a quantized CNN on a Cortex-M4 or M7 without CMSIS-NN means relying on whatever inner loops your compiler generates from generic C — and that leaves 3-5x performance and a comparable amount of energy on the table. CMSIS-NN is ARM's hand-optimized kernel library for exactly this gap: it implements the primitive operators (convolution, depthwise convolution, fully-connected, pooling, activation, softmax) that TFLite Micro and other inference engines call into, tuned per-core to use SIMD, MAC instructions, and the memory system efficiently. Understanding what it actually does under the hood — not just linking against it — is what separates "it runs" from "it runs in budget."

Where CMSIS-NN Sits in the Stack

Model (TFLite / ONNX, INT8) 
   → TFLite Micro interpreter (graph, memory planning)
       → CMSIS-NN kernels (arm_convolve_s8, arm_fully_connected_s8, ...)
           → CMSIS-DSP intrinsics / core instructions (SMLAD, MVE, etc.)
               → Cortex-M core (M4/M33/M55/M7...)

CMSIS-NN is not a training tool or a converter — it's the bottom layer that executes operators once a quantized model has already been mapped onto a graph. TFLite Micro's "reference kernels" are portable but unoptimized C; swapping in the CMSIS-NN kernel implementations for the same ops (via the CMSIS_NN build flag / kernel registration) is usually a drop-in replacement that changes nothing about model accuracy but changes latency dramatically.

What Makes the Kernels Fast

Three architectural features are exploited, and which ones apply depends on the specific Cortex-M core:

The core computational pattern in a quantized conv layer is an accumulation of int8 × int8 products into an int32 accumulator, followed by requantization. CMSIS-NN's arm_nn_mat_mult_s8 and friends pack pairs of (sign-extended) int8 values into 16-bit lanes and issue SMLAD (signed multiply-accumulate, dual 16-bit) for 2 MACs per instruction, or use MVE's VMLADAV to get four-lane int8 throughput per cycle, instead of the compiler emitting one multiply and one add per element in a scalar loop.

Quantized Arithmetic: The Actual Math

CMSIS-NN kernels operate on the standard TFLite int8 quantization scheme. For a tensor with real value r, quantized value q, scale S, and zero point Z:

r = S · (q − Z)

A convolution's output accumulator (int32) for quantized inputs is:

acc = Σ (q_input − Z_input) · (q_weight − Z_weight)

This is then rescaled back to int8 output range using a fixed-point multiplier and shift (output_multiplier, output_shift) computed offline by the quantization tool, avoiding floating-point division at runtime:

out_q8 = saturate( ((acc * output_multiplier) >> 31) >> (-output_shift) ) + Z_output

This multiply-and-shift is itself an ARM DSP intrinsic (arm_nn_requantize, built on SMMUL/SMULL), so the entire conv-plus-requantization pipeline stays in integer arithmetic with no soft-float calls — critical on cores without an FPU or with a single-precision-only FPU.

Worked Example: 3×3 Depthwise Convolution on M4

Consider a depthwise conv layer typical of a MobileNet-style vision model: input 32×32×32 (HWC, int8), 3×3 depthwise kernel, stride 1, same padding, output 32×32×32.

Reference C kernel (naive triple loop):

• MACs per output pixel per channel: 9
• Total MACs: 32·32·32·9 = 294,912
• On a scalar core doing ~1 MAC every 2-3 cycles (load, multiply, accumulate, address update) → roughly 700,000-900,000 cycles.

CMSIS-NN arm_depthwise_conv_s8:

• Uses SMLAD to process 2 int8 MACs per instruction where operands can be packed, and restructures the loop to keep weights resident in registers across the row, cutting load traffic.
• Practical measured throughput on M4 for this class of layer is commonly 3-4 MACs/cycle effective, i.e. ≈75,000-100,000 cycles.

Check: at 100 MHz, naive ≈ 7-9 ms vs. CMSIS-NN ≈ 0.75-1.0 ms for this single layer. Across a full network with a dozen such layers, this is the difference between ~100 ms and ~10 ms inference — the difference between "unusable for real-time" and "runs at 30+ fps-equivalent control loop rate." Based on the article's own figures the computed speedup here is roughly 7-9x (e.g. 800,000/87,500 ≈ 9x), consistent with published ARM benchmarks showing larger gains for pointwise/1x1 convs where matrix-multiply-style packing helps more.

Memory Layout and the Im2col / Reordering Trade-off

For standard (non-depthwise) convolutions, arm_convolve_s8 optionally uses a partial im2col transform: it unrolls patches of the input into a matrix so the convolution becomes a matrix multiply, which vectorizes better but costs extra RAM for the column buffer.

• Without im2col: lower peak RAM, somewhat lower throughput.
• With im2col: higher throughput (better SIMD utilization, fewer address computations), but requires a scratch buffer sized roughly kernel_h · kernel_w · in_channels bytes per output pixel batch.

CMSIS-NN exposes this via arm_convolve_s8 needing a caller-provided buffer_size from arm_convolve_s8_get_buffer_size() — a detail that matters directly for SRAM budgeting on parts with 128-256 KB total RAM, where a few missed KB of scratch space is the difference between fitting the model and a linker overflow.

Practical Integration Notes

• Kernel selection is automatic but architecture-gated. CMSIS-NN detects __ARM_FEATURE_MVE, __ARM_FEATURE_DSP, etc. at compile time via CMSIS-Core headers; building for the wrong -mcpu/-march silently falls back to slower generic paths, so always verify the target triple matches the actual silicon (e.g. -mcpu=cortex-m55 with -mfloat-abi=hard (and an appropriate -march if needed) to get MVE-accelerated kernels, not just -mcpu=cortex-m4-level codegen; there is no -mfpu=auto flag in GCC/Clang Arm toolchains).
• Per-channel vs. per-tensor quantization. CMSIS-NN supports per-channel scale/zero-point (one multiplier/shift pair per output channel), which is what TFLite's default int8 post-training quantization produces for conv weights — using per-tensor quantization here silently degrades accuracy, not speed, so this is a correctness issue, not just a performance one.
• Activation memory reuse. CMSIS-NN kernels are written to support in-place operation where the op allows it (e.g. some activations), and TFLite Micro's memory planner overlaps tensor lifetimes; profiling actual peak RSS with the built-in memory profiler is more reliable than summing tensor sizes by hand.
• Benchmarking on the actual target. Cycle counts vary meaningfully between M4 (no MVE), M33 (TrustZone, optional DSP), M55 (MVE), and M7 (higher clock, TCM); always benchmark on silicon or a cycle-accurate model — Cortex-M0/M0+ have no DSP extension at all and see far smaller CMSIS-NN gains, sometimes none, because the SIMD instructions don't exist there.

Key Takeaways

• CMSIS-NN is the optimized kernel layer under inference engines like TFLite Micro — it replaces generic reference C loops with SIMD/DSP-instruction-based implementations of conv, FC, pooling, and activation ops.
• Speed comes from packing multiple int8 operands into 32-bit words and issuing single instructions (SMLAD, MVE VMLADAV) per multiple MACs, instead of one multiply-add per scalar element.
• Quantized math stays entirely in integer/fixed-point (acc = Σ(q_in−Z_in)(q_w−Z_w), rescaled via integer multiply-and-shift), avoiding floating-point entirely at inference time.
• Measured speedups are typically 3-5x over naive C on M4/M7, and larger with MVE on M55/M85, but the benefit depends on whether the target actually has the DSP/MVE extensions the compiler is told to target.
• Correct per-channel quantization parameters and adequate scratch buffer sizing (for im2col-based conv paths) matter as much as raw kernel speed for a working, accurate deployment.