Person Detection: MobileNet-v2 on CMSIS-NN

Person detection is the workhorse task of embedded vision: smart doorbells, occupancy sensors, retail analytics, and battery-powered security cameras all need a binary or few-class answer — "is there a person in this frame?" — running at tens to hundreds of milliwatts, often with no cloud connection. MobileNet-v2, compiled down through CMSIS-NN kernels onto a Cortex-M MCU, is the reference architecture for this job because it was designed from the ground up for parameter and MAC efficiency, and CMSIS-NN gives it a hand-optimized execution path on ARM's own instruction set.

Why MobileNet-v2 for This Task

MobileNet-v2's core building block is the inverted residual with linear bottleneck, built from depthwise separable convolutions:

1×1 expand conv (pointwise) → 3×3 depthwise conv → 1×1 project conv (pointwise, linear)
                                    + residual add (when stride=1, channels match)

A standard 3×3 convolution over Cin input and Cout output channels costs 9·Cin·Cout MACs per output pixel. Splitting it into a depthwise (3×3, per-channel) plus pointwise (1×1, channel-mixing) stage costs 9·Cin + Cin·Cout MACs — for typical Cin = Cout = 64, that is a ~7–8× reduction in multiply-accumulates for the spatial-filtering stage. The "inverted" part expands channels before the depthwise filter (e.g., 6× expansion ratio) so the depthwise stage operates in a richer feature space, then projects back down — this preserves accuracy despite the width bottleneck at the block's input/output.

For person detection specifically, the task tolerates a low input resolution (96×96 or 128×128 grayscale/RGB is common) and a binary or 3-class head (person / no-person, sometimes + "person-in-background"), which lets you shrink MobileNet-v2's width multiplier (α = 0.35–0.5) and still hold >90% accuracy on curated datasets like Visual Wake Words.

CMSIS-NN's Role

CMSIS-NN is ARM's optimized NN kernel library for Cortex-M cores. It does not change the model — it replaces the generic "reference C" implementation of each layer with kernels that exploit:

• SIMD via SXTB16/SMLAD: Cortex-M4/M7/M33/M55 support 16-bit packed multiply-accumulate instructions, letting INT8 kernels process two MACs per instruction cycle instead of one.
• Helium (MVE) on Cortex-M55/M85: vectorized 8-way INT8 MAC operations per cycle, giving another 4–8× over M4-class SIMD for the same clock.
• Optimized memory layout: arm_convolve_HWC_q7/arm_depthwise_conv_s8 family kernels expect NHWC-like layouts and pre-arranged weight order to maximize cache/line-buffer reuse and minimize pointer-chasing.
• Fused requantization: each layer's INT32 accumulator is rescaled to INT8 output in the same kernel call (per-channel scale + zero-point), avoiding a separate pass over the feature map.

The practical effect: the same INT8 MobileNet-v2 graph runs 3–5× faster on CMSIS-NN kernels than on a portable reference-C TFLite Micro build, with identical numerical output (CMSIS-NN's INT8 kernels are bit-exact with the TFLite reference quantized ops when configured correctly).

Quantization: the Precondition for CMSIS-NN Speed

CMSIS-NN's fast path is INT8 (per-channel symmetric weights, per-tensor asymmetric activations) — this is the same scheme TFLite's post-training integer quantization or QAT produces. Key relation:

real_value = scale · (quantized_value − zero_point)

Per-channel weight scales let each output channel of a convolution have its own scale factor, which recovers most of the accuracy lost to INT8 weights (typically <1% top-1 delta vs. FP32 for MobileNet-v2 with QAT, 1–3% with pure post-training quantization).

Worked Example: Sizing a 96×96 Person Detector

Assume MobileNet-v2 with α = 0.35, input 96×96×3, INT8, targeting a Cortex-M4 at 80 MHz.

Step 1 — MAC count. A width-0.35 MobileNet-v2 at 96×96 input runs roughly 11–12 million MACs per inference (scaled down from the ~300M MACs of the full α=1.0, 224×224 model by resolution² (96/224)² ≈ 0.184 and width² ≈ 0.35² ≈ 0.1225 relative to the depthwise-heavy compute, combined effect ≈ 0.023 → 300M × 0.023 ≈ 7M; empirically published TF Slim numbers for this configuration land near 11M MACs due to the fixed-cost stem and head layers not scaling with width). Use 11M MACs as the design figure.

Step 2 — Cycles. CMSIS-NN INT8 kernels on Cortex-M4 achieve roughly 4–6 MACs/cycle for depthwise-heavy graphs (SIMD MAC pairs plus loop overhead). Take 5 MACs/cycle:

cycles ≈ 11,000,000 / 5 = 2,200,000 cycles

Step 3 — Latency.

latency ≈ 2,200,000 cycles / 80,000,000 Hz ≈ 27.5 ms

Step 4 — Check against power/duty budget. For a battery camera sampling at 2 fps, 27.5 ms of active compute per 500 ms period is a 5.5% duty cycle — well inside typical energy budgets even before accounting for MCU sleep between inferences. If the target were 10 fps continuous (100 ms period), 27.5 ms leaves only 72.5 ms margin for capture, preprocessing, and I/O — tight but workable on an M4, and this is exactly the scenario where moving to an M55 with Helium (roughly 5–8× the MACs/cycle) or an offload accelerator becomes necessary.

Sanity check: published MobileNet-v2 α=0.35 CMSIS-NN benchmarks on 80 MHz M4-class parts report person-detection inference in the 20–40 ms range for comparable input sizes — our 27.5 ms estimate sits inside that band, confirming the MAC/cycle assumption was reasonable.

Memory Footprint

Scratch RAM is often the binding constraint on Cortex-M0+/M3-class parts with <64 KB SRAM — CMSIS-NN's depthwise kernels support in-place and reduced-scratch variants (arm_depthwise_conv_s8_opt) specifically to keep peak RAM under control, since the classic im2col-based approach can spike scratch usage several-fold on the largest intermediate tensor.

Practical Design Implications

• Choose α and resolution jointly. Halving resolution cuts MACs ~4×; halving width cuts MACs ~4× too, but width also shrinks weight memory linearly while resolution does not — pick width reduction first when flash is the constraint, resolution reduction first when latency/RAM is the constraint.
• Keep the residual-add layers stride-1 with matching channels — CMSIS-NN's fused add path only accelerates this exact shape; mismatched channel counts force a fallback path.
• Profile per-layer, not just end-to-end. The stem conv and the final 1×1 classifier head are fixed-cost and don't shrink with α — at very low width multipliers they can dominate total latency.
• Validate bit-exactness after switching from reference kernels to CMSIS-NN — a mismatched quantization parameter (e.g., wrong per-channel scale array) will silently produce a functionally different but plausible-looking network.

Key Takeaways

• MobileNet-v2's inverted residual + depthwise-separable blocks cut MAC count by roughly an order of magnitude versus standard convolutions, making it viable at MCU compute budgets.
• CMSIS-NN doesn't change the model graph — it accelerates INT8 execution via SIMD/Helium instructions and fused requantization, typically 3–5× faster than reference-C kernels.
• INT8 quantization (per-channel weight scales) is a precondition for CMSIS-NN's fast path and, with QAT, costs under 1% accuracy versus FP32.
• A 96×96, α=0.35 MobileNet-v2 person detector runs in the tens-of-milliseconds range on an 80 MHz Cortex-M4 — always verify MAC/cycle assumptions against a latency budget derived from the target frame rate.
• Scratch RAM for activations, not flash for weights, is usually the binding memory constraint on smaller Cortex-M parts; CMSIS-NN's reduced-scratch depthwise kernels exist specifically to address this.