Operator Fusion: Reducing Memory and Compute Overhead

On a Cortex-M with 256–512 KB of RAM, the difference between a model that runs and one that OOMs during inference is often not the weight count but how many intermediate activation tensors get materialized. A naive graph executor that runs each layer as a separate kernel call — conv, then batchnorm, then ReLU, then add — writes and re-reads the full activation tensor to memory three or four times when one pass would do. Operator fusion collapses these chains into single kernels, cutting both memory traffic and the per-op dispatch overhead that dominates on small, cache-poor MCUs. It's one of the highest-leverage optimizations a graph compiler (TFLite Micro's converter, Ethos-U's Vela, TVM, ONNX Runtime) applies, and understanding it explains a lot of the gap between a raw FLOP count and observed latency.

Why Unfused Execution Is Expensive

Each operator in an unfused graph is its own kernel invocation with its own read/write pass over memory. For a tensor of N elements at B bytes each, a single elementwise op costs roughly 2·N·B bytes of memory traffic (one read, one write). Chain four elementwise ops and you pay 8·N·B, even though the actual arithmetic is trivial.

On an MCU, this matters for two compounding reasons:

• Memory bandwidth, not compute, is the bottleneck for most conv/elementwise sequences. A Cortex-M7 at 400 MHz with tightly-coupled memory might sustain a few GB/s, but DMA or flash-backed weight fetches are far slower — every extra pass over an activation tensor is real, measurable latency.
• Per-kernel dispatch overhead is fixed and non-trivial. Each kernel call in TFLite Micro involves interpreter dispatch, tensor shape/type checks, and pointer setup — tens to hundreds of cycles of overhead independent of tensor size. For small tensors (common late in a CNN, e.g. 7×7×256), this overhead can exceed the actual compute time.

Operator fusion attacks both: fewer kernel calls (less dispatch overhead) and fewer full passes over memory (less bandwidth pressure), because intermediate results are kept in registers or a small local buffer instead of being written back to the activation arena.

What Gets Fused

Conv+BN+ReLU is the canonical case. Batch normalization at inference time is just an affine transform: y = γ·(x − μ)/√(σ²+ε) + β, which is algebraically a scale and shift, y = a·x + b. Since convolution is linear, this scale/shift can be pre-multiplied into the convolution's weights and bias during model conversion:

W' = W · a
b' = b_orig · a + b

At runtime there is no BN op at all — just a conv with adjusted weights, immediately followed by clamping for ReLU inside the same kernel's output stage. This is why a frozen/converted inference graph shows far fewer nodes than the training graph: BN literally disappears.

Worked Example: Memory Traffic on a Depthwise Separable Block

Consider a MobileNet-style block: depthwise conv → BN → ReLU6 → pointwise conv → BN → ReLU6, operating on a 56×56×32 activation (int8).

Tensor size: 56 · 56 · 32 · 1 byte = 100,352 bytes ≈ 98 KB per activation tensor (using the depthwise output size; pointwise output at 56×56×64 would be 196 KB — assume 64 output channels for the pointwise stage).

Unfused execution (6 ops, each a full read+write of its own tensor):

DW conv:   read input (98 KB)  + write output (98 KB)   = 196 KB
BN:        read (98 KB)  + write (98 KB)                = 196 KB
ReLU6:     read (98 KB)  + write (98 KB)                = 196 KB
PW conv:   read (98 KB)  + write (196 KB)                = 294 KB
BN:        read (196 KB) + write (196 KB)                = 392 KB
ReLU6:     read (196 KB) + write (196 KB)                = 392 KB
                                                Total   ≈ 1,666 KB

Fused execution (BN folded into each conv, ReLU6 applied in the conv's output stage → 2 kernel calls):

DW conv (+BN+ReLU6 fused): read input (98 KB) + write output (98 KB)  = 196 KB
PW conv (+BN+ReLU6 fused): read input (98 KB) + write output (196 KB) = 294 KB
                                                Total   ≈ 490 KB

Check: 490 / 1666 ≈ 0.29 — fusion here removes about 71% of the memory traffic for this block, and reduces kernel dispatch count from 6 to 2 (a 3× cut in fixed per-call overhead). Both numbers are consistent with the rule that each fused-away elementwise op saves one full read+write pass, and the BN/ReLU6 stages disappear entirely as separate calls.

Fusion in Practice Across Toolchains

• TFLite Micro / TFLite converter: fuses Conv+BN at conversion time (BN folding), and fuses activation functions (ReLU, ReLU6, tanh) into the preceding conv/FC op's "fused activation" field — a single enum checked in the kernel's output stage, no separate op.
• CMSIS-NN: kernels like arm_convolve_s8 accept a fused activation range (min/max clamp) applied to the int32 accumulator before requantization, avoiding a separate clamp pass over the output tensor.
• Ethos-U Vela compiler: performs more aggressive fusion including combining consecutive elementwise ops and mapping fused patterns onto the NPU's hardware pipeline stages so data never leaves local SRAM between fused steps.
• TVM / Relay, ONNX Runtime graph optimizers: general-purpose fusion passes (FuseConvBN, FuseElementwise) operate on the computation graph independent of target, followed by target-specific codegen that emits a single kernel per fused subgraph.

Practical Implications

• Always convert/quantize before benchmarking. A raw Keras/PyTorch graph timed op-by-op will look far worse than the converted, fused inference graph — comparing the two is comparing different workloads.
• Fusion changes memory planning, not just speed. Fewer live intermediate tensors means the scratch/arena allocator (see memory optimization techniques like in-place ops) needs less peak buffer space — fusion and buffer reuse compound.
• Not all fusions are free at all precisions. Folding BN into int8 conv weights can shift the required quantization scale/zero-point; verify accuracy after fusion, not just before.
• Custom ops break fusion. A custom or unsupported activation inserted mid-graph forces the compiler to fall back to unfused execution around it — check the converted model's op list, not just the source model, to confirm fusion actually happened.
• Hardware accelerators reward fusion more than CPUs do, because on an NPU like Ethos-U, keeping data resident in local SRAM across a fused subgraph avoids expensive round-trips to external DRAM — the same fusion that saves 71% of traffic in software can save even more energy on dedicated silicon.

Key Takeaways

• Operator fusion merges multiple graph nodes (conv, batchnorm, activation, elementwise) into a single kernel call, eliminating redundant memory read/write passes and per-op dispatch overhead.
• BatchNorm is algebraically an affine transform and can always be folded into a preceding linear layer's weights/bias at conversion time — it should never appear as a runtime op in a deployed model.
• Fusion savings are dominated by memory traffic reduction (each eliminated elementwise op saves a full read+write pass), which matters more than raw FLOPs on bandwidth-limited MCUs.
• A worked depthwise-separable block example shows ~71% memory traffic reduction and 3× fewer kernel calls from fusing BN+ReLU6 into the surrounding convs — typical of real MobileNet-class blocks.
• Fusion is applied by graph compilers (TFLite converter, Vela, TVM, ONNX Runtime), not hand-written by the model author — verify it actually occurred by inspecting the converted graph's op list, especially around custom ops that can block it.