Keyword Spotting (KWS): DS-CNN on Cortex-M for Wake-Word

Wake-word detection is the always-on gatekeeper in front of every voice assistant: a task that must run continuously on a power budget of a few hundred microwatts to a few milliwatts, on an MCU with tens to a few hundred KB of RAM, while keeping false accepts low enough that a device doesn't wake up every time someone says "hey" to their dog. The dominant architecture for this on Cortex-M class hardware is the Depthwise-Separable CNN (DS-CNN), because it hits the accuracy/compute/memory sweet spot that plain CNNs and RNNs don't.

The Pipeline: Audio to Decision

Mic (PDM/I2S) → Frame buffer → Mel-Spectrogram (MFCC/log-mel) → DS-CNN → Softmax → Posterior smoothing → Trigger

1. Acquisition: a MEMS microphone (PDM output) is decimated to PCM, typically 16 kHz, 16-bit.
2. Framing: windows of ~30–40 ms with 10–20 ms hop (e.g., 25 ms window, 10 ms stride → 100 frames/s).
3. Feature extraction: each frame is converted to 10–40 MFCC or log-mel filterbank coefficients. A 1-second utterance becomes a 2D "image" of roughly time × frequency = 98×40 (25 ms window, 10 ms stride) or 49×10 if a coarser 40 ms window with 20 ms stride and fewer filterbank channels is used instead.
4. DS-CNN inference: the spectrogram is fed as a single-channel image into a small convolutional network.
5. Decision logic: softmax outputs are smoothed across a sliding window of frames (to reject one-off spikes) before declaring a detection, since a single frame's classification is unreliable on noisy audio.

The feature extraction step is not incidental — MFCC/FFT computation on a Cortex-M0/M4 can consume as much CPU as the neural network itself, so it's usually done with a fixed-point FFT (CMSIS-DSP arm_rfft_q15) rather than floating point.

Why Depthwise-Separable Convolution

A standard convolution with Cin input channels, Cout output channels, and a k×k kernel costs:

MACs_standard = k² · Cin · Cout · Hout · Wout

Depthwise-separable convolution splits this into two cheaper stages:

• Depthwise: one k×k filter per input channel (no cross-channel mixing) — k² · Cin · Hout · Wout MACs
• Pointwise: a 1×1 convolution across channels to mix them — Cin · Cout · Hout · Wout MACs

MACs_DS = (k² · Cin + Cin·Cout) · Hout · Wout
Reduction ratio ≈ 1/Cout + 1/k²

For a typical k=3, Cout=64 layer, the reduction factor is roughly 1/64 + 1/9 ≈ 0.126, i.e. about 8× fewer MACs than a standard convolution for the same channel counts. This is the same trick MobileNet uses, applied to a 2D spectrogram instead of an RGB image.

DS-CNN Architecture for KWS

The reference architecture (from the widely cited Google/ARM keyword-spotting benchmark work) is:

Input: 49×10 (or 98×40) log-mel spectrogram, 1 channel
 └─ Conv2D (standard, e.g. 4×10, stride 2×2)   → extracts initial time-freq features
 └─ N × [Depthwise Conv2D 3×3 → BatchNorm → ReLU
         Pointwise Conv2D 1×1 → BatchNorm → ReLU]
 └─ Global Average Pooling
 └─ Fully Connected → Softmax (num_keywords + 1 for "unknown"/"silence")

Model sizes scale by width multiplier and depth, giving a family that trades accuracy for footprint:

*On a 12-keyword classification task (Google Speech Commands-style benchmark); actual numbers vary with training data and augmentation.

For comparison, a similarly-sized plain CNN or an LSTM/GRU-based model typically needs 2–5× more parameters or MACs to match the DS-CNN-S accuracy — the reason DS-CNN became the reference architecture in MLPerf Tiny's keyword-spotting benchmark.

Sizing for a Cortex-M4/M7 Budget

Take DS-CNN-S (~24K parameters) quantized to INT8:

• Weight memory (Flash): 24K params × 1 byte ≈ 24 KB, plus activation quantization scale tables (negligible).
• Activation RAM (peak): dominated by the input feature map and the first depthwise layer's intermediate buffer — typically 8–20 KB for this size of network, well within a 64–128 KB SRAM budget on an STM32F4/L4 or nRF52840.
• Inference latency: on a Cortex-M4 at 64–80 MHz with CMSIS-NN INT8 kernels, DS-CNN-S runs in roughly 10–20 ms per 1-second window — comfortably under the 100 ms decision cadence needed for streaming detection when only every 3rd–5th frame window is re-evaluated.
• Duty cycle: because the wake-word engine must run continuously, the real system-level cost is (inference time / inference interval) × active current + idle-mode current. Running inference once every 200–300 ms instead of every 10 ms frame is what makes multi-year coin-cell operation realistic on Cortex-M0+/M4 designs.

Worked power check: assume active current during inference is 15 mA at 3.3 V, inference takes 15 ms, and it's run once every 250 ms; deep-sleep between inferences draws 50 µA.

Duty cycle = 15 ms / 250 ms = 6%
Average current ≈ 0.06 × 15 mA + 0.94 × 0.05 mA
             ≈ 0.9 mA + 0.047 mA ≈ 0.95 mA

At 3.3 V that's ≈3.1 mW average — an order of magnitude below a continuously-running MCU (which would sit near 15 mA / ~50 mW), and consistent with published always-on KWS power figures in the 1–5 mW range. This is the number that makes DS-CNN-class wake-word detection viable on battery hardware rather than an always-plugged-in device.

Quantization and CMSIS-NN Mapping

DS-CNN for MCU deployment is almost always trained in FP32, then converted with post-training INT8 quantization (or QAT if the accuracy drop is unacceptable — see the dedicated PTQ/QAT articles in this series). The resulting int8 depthwise and pointwise layers map directly onto CMSIS-NN kernels:

• arm_depthwise_conv_s8 — optimized depthwise convolution using SIMD MAC instructions on Cortex-M4/M7 (DSP extension) or the MVE (Helium) pipeline on Cortex-M55.
• arm_convolve_1x1_s8_fast — pointwise convolution, memory-layout optimized for the 1×1 kernel case.
• arm_fully_connected_s8 and arm_softmax_s8 for the classifier head.

On a Cortex-M55 with Helium, or when paired with an Ethos-U55 NPU, the same DS-CNN graph is offloaded almost entirely to the NPU, cutting inference time and energy further — but the DS-CNN-S/M architecture itself is designed to be small enough that this hardware isn't a hard requirement.

Decision Logic: Beyond Argmax

A raw per-frame softmax is noisy. Production KWS systems apply:

• Posterior smoothing: average softmax probabilities over a sliding window (e.g., last 20–30 frames) to reduce spurious triggers from transient noise.
• Confidence threshold + hangover: require the smoothed keyword probability to exceed a threshold (e.g., 0.8) for several consecutive windows before firing, then apply a refractory period to avoid re-triggering on the same utterance.
• Two-stage cascade: a tiny always-on stage-1 detector (even simpler than DS-CNN-S, sometimes a small dense or DS-CNN variant with <10K parameters) filters out silence/noise cheaply; only on a positive stage-1 result does the full DS-CNN run, cutting average power further since most of the time is spent in near-silence.

Key Takeaways

• DS-CNN factors standard convolution into depthwise + pointwise stages, cutting MACs by roughly 1/Cout + 1/k² (~8× for typical 3×3, 64-channel layers) with minimal accuracy loss.
• The DS-CNN-S/M/L family spans ~24K–420K parameters and ~2.6M–26M MACs, letting designers trade accuracy for Flash/RAM/latency budget on Cortex-M0+ through M7/M55.
• Feature extraction (MFCC/log-mel via fixed-point FFT) is a non-trivial part of the compute budget and should be profiled alongside the network itself.
• INT8 post-training quantization plus CMSIS-NN kernels (arm_depthwise_conv_s8, arm_convolve_1x1_s8_fast) are the standard path from a trained model to real-time Cortex-M inference.
• System-level power is dominated by duty cycle, not raw inference speed — running a 15 ms inference every 250 ms instead of continuously is what enables multi-milliwatt, battery-friendly always-on listening.
• Posterior smoothing and two-stage cascades (cheap detector gating the full DS-CNN) are essential to control false-accept rate and further reduce average power in real deployments.