Face Detection on MCU: Haar Cascade, MobileNet-SSD

Face detection is often the first vision task pushed onto a microcontroller — a doorbell camera, an occupancy sensor, a battery-powered access panel — because it gates everything downstream (recognition, tracking, exposure control) and must run continuously at low power. On a Cortex-M with no GPU and tens to hundreds of KB of RAM, the choice between a classical Haar cascade and a quantized MobileNet-SSD is not academic: it determines whether the product hits its power and latency budget at all.

Two Different Detection Paradigms

Haar cascades (Viola-Jones, 2001) detect faces by sliding a window over the image and evaluating a cascade of weak classifiers, each built from Haar-like features (rectangular sums of pixel intensities) computed via an integral image. Early cascade stages are cheap and reject most non-face windows immediately; only windows that pass all stages are declared faces. It's hand-engineered, not learned end-to-end, and needs no matrix multiply hardware — just additions and a few comparisons per stage.

MobileNet-SSD is a learned CNN detector: a MobileNet backbone (depthwise-separable convolutions) extracts features, and an SSD (Single Shot Detector) head regresses bounding boxes and class scores from multiple feature-map scales in one forward pass. It's a general object detector re-trained/fine-tuned for the single "face" class, quantized to INT8 for MCU deployment.

Haar Cascade: The Math That Makes It Cheap

The integral image ii(x,y) = Σ(x'≤x, y'≤y) I(x',y') lets you compute the sum of pixels in any rectangle with exactly 4 array lookups and 3 additions, independent of rectangle size:

sum(rect) = ii(x2,y2) − ii(x1,y2) − ii(x2,y1) + ii(x1,y1)

A Haar feature (e.g., a 2-rectangle edge feature) is then just a difference of two such rectangle sums, compared against a learned threshold. A weak classifier is one feature + threshold + polarity. Stages combine 1–200 weak classifiers with an AdaBoost-derived weight; a window must pass every stage to reach the next.

Why this scales on an MCU: for a 320×240 frame, the integral image is a one-time O(WH) pass (≈77K additions). Each classifier stage evaluation is O(1) per rectangle regardless of window size, and roughly 50% of windows are rejected in the first stage alone — so amortized cost per window is far below the worst case of evaluating all ~20 stages.

Worked example — window count: scanning a 320×240 frame with a 24×24 base detector, step size 1 px, across 12 scales (scale factor 1.2, from 24×24 up to ~215×215):

scale s=1.2^k, k=0..11
windows per scale ≈ (320 − 24·1.2^k)·(240 − 24·1.2^k)

Summing across scales gives on the order of 1–2 million candidate windows for the full unoptimized scan. With early-stage rejection cutting ~80% of windows after stage 1 (1–2 feature evals) and ~95% after stage 3, the effective work is roughly 100–300K full-cost-equivalent window evaluations — feasible in real time (5–15 fps) on a 200 MHz Cortex-M7 with the integral image computed once per frame and cached.

MobileNet-SSD: Cost Breakdown

For a MobileNet-SSD at 160×160 input (typical MCU-friendly resolution), depthwise-separable convolution cost per layer is:

MACs_dw = Dk² · M · Hout · Wout        (depthwise, kernel Dk, M channels)
MACs_pw = M · N · Hout · Wout          (pointwise 1×1, M→N channels)

vs. a standard conv Dk²·M·N·Hout·Wout — the separable form cuts compute by roughly 1/N + 1/Dk², which is why MobileNet is the default backbone for MCU vision. A full MobileNet-SSD-Face at 160×160 typically lands at 150–300M MACs per inference. At INT8 on CMSIS-NN (Cortex-M4/M7 with DSP extensions), realistic throughput is on the order of 1–4 MACs/cycle, i.e., 40–200M MACs/s. That gives:

latency ≈ 200M MACs / 100M MACs/s ≈ 2 s   (M4 without accelerator, worst case)
latency ≈ 200M MACs / (Ethos-U55 ~ 1–4 GMAC/s) ≈ 50–200 ms

Check: this matches field reports of plain CMSIS-NN MobileNet-SSD face detectors running at 1–5 fps on Cortex-M7 @ 400 MHz, and 10–30 fps once an NPU (Ethos-U55, MAX78000) is added — a 10–50× speedup consistent with the GMAC/s ratio above. Without hardware acceleration, MobileNet-SSD at useful frame rates on a bare Cortex-M4 is not realistic; this is the practical reason Haar cascades remain common on the cheapest parts.

RAM and Flash Budget

• Haar: the cascade XML/binary (stages + thresholds) is typically 30 KB (simple frontal-face, few hundred stages/features) to a few hundred KB for high-accuracy variants. Working RAM is the frame buffer plus one integral image buffer of the same pixel count (as uint32_t, so 4× the byte count of an 8-bit grayscale frame) — for 320×240, that's 300 KB, which forces most Haar implementations onto tiled/strip processing on RAM-constrained M0/M4 parts rather than a full-frame integral image.
• MobileNet-SSD: INT8 weights for a compact 0.25–0.5-width-multiplier MobileNet backbone with an SSD head run 300 KB–1.5 MB in flash. Activation scratch (largest intermediate feature map, reusable via CMSIS-NN's in-place buffers) is typically 100–300 KB — this is the harder constraint and usually forces input resolution down to 96×96–160×160 on sub-512 KB-RAM MCUs.

Practical Design Implications

• Choose Haar when: RAM/flash is under ~256 KB total, no NPU is available, faces are roughly frontal and well-lit, and false positives can be filtered downstream (e.g., by a lightweight verification stage or temporal consistency across frames).
• Choose MobileNet-SSD when: an NPU or DSP-accelerated CMSIS-NN path exists, pose/lighting variation is significant, and a labeled dataset is available for fine-tuning — accuracy (especially recall on non-frontal faces) is substantially better and false-positive rate is lower, reducing downstream compute wasted on spurious detections.
• Hybrid pipelines are common: run Haar (or a even simpler motion/skin-color gate) as a cheap always-on trigger, and wake a MobileNet-SSD pass (possibly on a companion NPU or higher-power core) only when Haar flags a candidate region — this is the standard pattern for battery-powered smart cameras balancing standby current against detection latency.
• Quantization matters more than architecture choice once you commit to MobileNet-SSD: INT8 post-training quantization typically costs 1–3% mAP versus FP32; going to INT4 without quantization-aware training can cost far more on a task as fine-grained as small-face detection.
• Frame rate vs. detection window: Haar's per-frame cost is roughly independent of scene content once rejection cascades are tuned, but scales with the number of scales searched — reducing scale steps (e.g., factor 1.25 instead of 1.1) trades detection of small/distant faces for 2–3× speedup.

Key Takeaways

• Haar cascades use hand-crafted rectangle features and an integral image for O(1)-per-feature evaluation, requiring no NPU and modest RAM — the right default on sub-256 KB Cortex-M0+/M4 designs.
• MobileNet-SSD uses learned depthwise-separable convolutions and an SSD head; it delivers better accuracy and pose robustness but needs 150–300M MACs per frame, making an NPU or DSP-accelerated CMSIS-NN path essential for real-time operation.
• Integral image cost is O(W·H) once per frame; Haar's early-stage rejection is what keeps effective per-window cost low, not the theoretical worst case of evaluating every stage.
• Depthwise-separable convolution cuts MobileNet's MAC count by roughly 1/N + 1/Dk² versus a standard convolution — the core reason it's the default MCU CNN backbone.
• Without hardware acceleration, MobileNet-SSD face detection on a bare Cortex-M4 is impractically slow (seconds per frame); with an NPU like Ethos-U55 it reaches 10–30 fps.
• Hybrid Haar-gate + CNN-verify pipelines are the standard low-power pattern: cheap always-on detection triggers an expensive, accurate CNN pass only on demand.