Model Pruning: Weight Magnitude, Structured vs. Unstructured

On an MCU with 256–512 KB of SRAM and no floating-point NPU, every superfluous weight costs flash, RAM bandwidth, and cycles. Pruning removes parameters that contribute little to model output, shrinking a network before quantization and compilation into CMSIS-NN or a vendor NPU graph. Done carelessly it wrecks accuracy; done with the right granularity and schedule it can cut model size 50–90% with sub-1% accuracy loss. The catch is that the pruning method must match what the target hardware can actually exploit — an unstructured 90%-sparse model that a Cortex-M kernel can't skip over buys you nothing at inference time.

Why Prune at All

Neural networks are typically over-parameterized relative to the task: many weights end up near zero after training, or contribute redundant/correlated features. Pruning exploits this by zeroing (and ideally removing) low-importance weights, filters, or channels, then optionally fine-tuning to recover accuracy.

Benefits chase three separate budgets:

• Storage — fewer nonzero weights means smaller flash footprint, especially after sparse encoding or compression.
• Compute — fewer MACs if the runtime/hardware can actually skip zeros.
• Memory bandwidth — for MCUs, weight fetch from flash often dominates latency more than the MACs themselves, so smaller weight tensors help even without sparse compute support.

Note the qualifier "if the runtime/hardware can actually skip zeros" — this is the crux of the structured vs. unstructured decision.

Weight Magnitude Pruning: The Baseline Criterion

The simplest and most common importance metric is magnitude: weights (or filters) with small absolute value are assumed to contribute least to the output, since y = Σ wᵢxᵢ and a near-zero wᵢ barely moves y for typical input ranges.

Basic algorithm (magnitude pruning, iterative):

1. Train (or start from) a converged model.
2. Rank weights by |w| (globally across the network, or per-layer).
3. Zero out the bottom p% by magnitude, producing a binary mask.
4. Fine-tune the remaining weights (mask stays fixed) for several epochs to recover accuracy.
5. Repeat steps 2–4, increasing sparsity gradually (e.g., 10% → 30% → 50% → 70%) rather than pruning to target sparsity in one shot.

Gradual, iterative pruning consistently beats one-shot pruning to the same final sparsity, because the network gets a chance to redistribute importance to surviving weights at each step rather than absorbing one large shock.

Global vs. per-layer thresholds: a single global magnitude threshold across all layers tends to over-prune layers with naturally small weight magnitudes (e.g., early conv layers) and under-prune others. Per-layer sparsity budgets, or normalizing magnitudes per layer before ranking, usually give a better accuracy/sparsity trade-off.

Limitation: magnitude is a proxy, not a direct measure of loss impact. A small weight feeding a large downstream activation can still matter. Costlier alternatives (gradient/Taylor-based or Hessian-based importance) exist, but for MCU-scale models plain magnitude pruning with iterative fine-tuning is usually good enough and far cheaper to implement.

Structured vs. Unstructured Pruning

Unstructured pruning zeroes individual weights anywhere in a weight tensor. It preserves accuracy best at a given sparsity ratio because it has maximum freedom to choose which weights to remove. But the resulting weight matrix is sparse in an irregular pattern — to get a speed benefit you need a sparse matmul kernel that skips zero entries efficiently (compressed sparse row/column formats, sparse tensor cores). Cortex-M CMSIS-NN kernels are dense: they do not check for zeros, so an unstructured-pruned model runs at the same cycle count as the dense original unless you add custom sparse inference code, which on an MCU is rarely worth the complexity and control-flow overhead.

Structured pruning removes entire structural units — a convolutional filter (output channel), an input channel, an attention head, or a fully-connected neuron — so the resulting tensor has genuinely smaller dimensions. A layer with 64 output filters pruned to 40 is simply a layer with 40 filters; every downstream toolchain (CMSIS-NN, TFLite Micro, vendor NPU compilers) treats it as an ordinary smaller dense layer with no special support required. The cost is coarser granularity: removing a whole filter removes all its weights regardless of their individual magnitudes, so structured pruning generally tolerates less sparsity before accuracy degrades.

Structured importance criteria commonly used instead of raw magnitude: L1/L2 norm of the filter's weights; average percentage of zeros (APoZ) after ReLU, where mostly-zero activations flag a low-value filter; and the BatchNorm scaling factor γ, where small γ directly attenuates a channel's contribution and L1-regularizing γ during training (network slimming) makes ranking nearly free.

Worked Example: Pruning a Person-Detection CNN for Cortex-M

Assume a MobileNet-style depthwise-separable CNN, 250 KB in INT8, targeting a Cortex-M4 with 320 KB SRAM / 1 MB flash for person detection. Baseline accuracy: 91.2%.

Attempt 1 — unstructured magnitude pruning to 80% sparsity:

• Fine-tuned accuracy: 90.9% (acceptable drop).
• Deployment: CMSIS-NN dense kernels still compute all weights (zero or not) → inference latency unchanged at ~48 ms/frame.
• Storage: with sparse index+value encoding, flash footprint drops to roughly 250 KB × (0.2 nonzero fraction × (8-bit value + ~4-bit index overhead) / 8-bit dense) ≈ 75 KB — useful if flash, not latency, is the binding constraint.

Attempt 2 — structured channel pruning to 45% of channels removed (guided by BN γ ranking), then fine-tune:

• Fine-tuned accuracy: 90.3% (larger drop than unstructured at similar "effective" sparsity — expected, since granularity is coarser).
• Resulting model: genuinely smaller dense tensors, ~80 KB in INT8 — (1 − 0.45)² ≈ 0.30 of the original pointwise-layer parameters once the Cout reduction compounds with the next layer's matching Cin reduction.
• Deployment: CMSIS-NN runs the smaller filter counts directly → latency drops to ~29 ms/frame (measured MAC reduction ≈ 42%, roughly tracking the channel reduction, since depthwise-separable layers scale near-linearly with channel count).

Check: MAC count for a conv layer scales as Cin × Cout × K² × Hout × Wout. Removing 45% of both Cin and Cout compounds to (1 − 0.45)² ≈ 0.30 of the original MACs for the pointwise layers — a ~70% MAC reduction there — but the network also contains depthwise layers (which scale with Cout only, ~45% reduction) and fixed-cost stages (activation quantization, DMA setup) that don't shrink at all. Blending these across the whole network is consistent with the measured ~42% latency drop, even though the pointwise layers alone shrink much more.

Conclusion for this target: structured pruning is the correct choice when the deployment kernel is dense (true for essentially all Cortex-M CMSIS-NN and most commercial MCU NPU compilers today). Unstructured pruning only pays off if paired with a runtime that has sparse kernel support, or if flash size — not latency or RAM — is the binding constraint.

Practical Design Implications

• Match the method to the compiler/runtime. Check whether your inference engine (CMSIS-NN, TFLite Micro, Ethos-U55 driver, vendor NPU SDK) has any sparse-kernel support before investing in unstructured pruning for a speed goal.
• Prune before quantization, then fine-tune again after. Pruning followed by post-training INT8 quantization (or better, quantization-aware training on the pruned model) compounds compression without compounding error — but always fine-tune after each transformation.
• Use iterative, gradual sparsity schedules, not one-shot pruning to target ratio; polynomial/linear sparsity ramps (as in TensorFlow Model Optimization Toolkit) are a reasonable default.
• Rank structured units with BN γ or L1 norm, not raw magnitude sums alone, when batch normalization is present — it's a nearly-free, well-correlated importance signal.
• Re-validate downstream tensor shapes. Structured pruning changes Cout of one layer and must correctly propagate to Cin of the next; toolchains that don't track this dependency graph will silently produce a broken graph or require manual shape surgery.
• Combine with knowledge distillation when accuracy loss is unacceptable — using the unpruned model as teacher for the pruned student's fine-tuning often recovers 0.3–1% accuracy over fine-tuning against ground-truth labels alone.

Key Takeaways

• Pruning removes low-importance weights or structures; weight magnitude is the simplest and most common importance criterion, best applied iteratively with fine-tuning between steps rather than in one shot.
• Unstructured pruning zeroes individual weights and tolerates the highest sparsity for a given accuracy, but needs sparse-kernel hardware/runtime support to convert into real speedup — CMSIS-NN and most MCU NPU compilers don't have it.
• Structured pruning removes whole filters/channels, producing genuinely smaller dense tensors that any standard toolchain accelerates automatically, at the cost of tolerating less sparsity before accuracy drops.
• For most MCU targets running dense kernels (CMSIS-NN, TFLite Micro, typical NPU compilers), structured pruning is the practical default for latency and RAM wins; unstructured pruning mainly helps when flash storage is the binding constraint.
• Always re-quantize and fine-tune after pruning, and consider knowledge distillation from the unpruned model to recover accuracy at aggressive sparsity levels.