Memory Optimization: Scratch Buffer, In-Place Operations

Running a CNN on a Cortex-M with 256–512 KB of SRAM is fundamentally a memory-allocation problem before it's a compute problem. A quantized MobileNet-v2 needs less than 1 MB of weights but its largest intermediate activation tensor can exceed 400 KB in fp32 or 100 KB in int8 — and that single tensor often has to coexist with the buffers for adjacent layers. Without deliberate scratch-buffer planning and in-place execution, the naive "allocate every tensor its own buffer" approach blows the memory budget on trivial models, forcing either a bigger (costlier, hungrier) MCU or a smaller, less accurate network. Memory optimization at this level is not a nice-to-have; it's what decides whether a model fits at all.

Where the Memory Actually Goes

For an inference graph, total RAM breaks into three categories:

Weights are usually placed in flash and read directly (or copied once to a fast SRAM/TCM region for cycle-critical kernels). The RAM battle is almost entirely about activations and per-operator scratch — and this is where the two techniques in this article, scratch-buffer reuse and in-place operations, do the real work.

A naive interpreter allocates a fresh buffer for every tensor in the graph and keeps all of them alive for the process lifetime. For an N-layer network, peak RAM would be the sum of every activation tensor. TFLite Micro, CMSIS-NN based runtimes, and most production inference engines instead compute a much smaller number: the peak concurrently-live set.

Liveness Analysis and the Scratch Arena

A tensor is "live" from the layer that produces it to the last layer that consumes it. Once its last consumer has executed, its memory can be reused by any tensor whose lifetime starts afterward. This is the same liveness-interval problem compilers solve for register allocation, applied to a static dataflow graph instead of a basic block.

Given the graph's execution order, the allocator:

1. Computes, for every tensor, [birth, death] in terms of op index.
2. Sorts tensors and greedily (or via graph-coloring) assigns them offsets into one shared byte arena such that no two live tensors overlap.
3. Reports the single number that matters: peak arena size = the high-water mark across the whole schedule.

Op:        0    1    2    3    4    5
T0 (in)    ██████
T1         ░░░░████████
T2              ░░░░░░████████
T3                        ░░░░░░████
T4 (out)                            ████████

T0 dies after op 1 starts consuming it into T1; its bytes can be reused for T2 once T0's last read completes. A linear-scan allocator packs T2 into T0's freed slot instead of appending it after T1.

Worked example — 3-layer MLP-style block:

Assume int8 activations, sizes: T0=4 KB (input), T1=6 KB, T2=6 KB, T3=2 KB (output). Dependency chain: T0→T1→T2→T3, strictly sequential, each op reads only its immediate predecessor.

• Naive sum: 4+6+6+2 = 18 KB
• Liveness-aware: at any point only two tensors are alive (producer's output + consumer still reading the previous one, briefly). Peak = T0+T1 while T1 is being produced (10 KB), then T1+T2 (12 KB), then T2+T3 (8 KB). Peak = 12 KB.
• Check: 12 KB ≤ 18 KB ✓, and 12 KB ≥ the largest single required pairing (T1+T2), so the schedule is tight — no further reduction possible without in-place tricks below.

That's a 33% RAM reduction from scheduling alone, with zero change to the math.

In-Place Operations: Reusing a Buffer Within Itself

Beyond reusing a tensor's slot after it dies, some operators can write their output directly over their input's memory while executing, because the algorithm never needs an already-overwritten input element again. This eliminates a separate output buffer entirely — not just reordering the arena, but shrinking the peak further.

Operators that are safe (or can be made safe) in-place:

• Element-wise activations (ReLU, ReLU6, sigmoid, tanh, clip): output[i] depends only on input[i]. Trivially in-place.
• Quantize/dequantize/requantize: per-element affine transform, same story.
• Reshape/flatten/squeeze: no data movement needed at all if the underlying byte layout is already correct — the "buffer" is just reinterpreted, zero-copy.
• Add/concat in some layouts: safe if the accumulation order matches memory order and no element is read after being overwritten.

Operators that are not safe in-place without care:

• Convolution / matrix multiply: each output element is a weighted sum over a window of input elements, several of which are still needed to compute neighboring outputs. Overwriting input[i] before a later output that also needs input[i] corrupts the result.
• Pooling with overlapping windows: same hazard — stride < kernel size means inputs are read multiple times.
• Softmax: needs a full pass for the max and the sum-of-exp before the final divide; if implemented carefully (read whole row into registers/accumulators, write back after), it can still be in-place at the tensor level even though it's not element-independent — the safety comes from doing the multi-pass reduction before any write.

Rule of thumb: an operator can safely write into its input buffer if and only if, at the time each output element is written, no input element still required by a not-yet-computed output has already been overwritten. Strided/windowed ops need either a temporary row buffer or a provably safe write order (e.g., processing in reverse when the write index always trails the read index).

im2col and the Scratch Buffer Problem in Convolution

Convolution is usually the layer that dominates RAM, not because of its own input/output tensors, but because of the im2col transform many kernels use to turn a convolution into a matrix multiply (im2col + GEMM). im2col explodes a Cin × K × K receptive field per output pixel into a dense row, duplicating overlapping input pixels across rows.

For a 3×3 conv, Cin=32, output spatial size 28×28:

im2col buffer size = (K·K·Cin) × (Hout·Wout) × elem_size
                    = (3·3·32) × (28·28) × 1 byte      (int8)
                    = 288 × 784
                    = 225,792 bytes ≈ 220.5 KB

That single scratch buffer can dwarf every activation tensor in the network. Two mitigations are standard:

• Tiled/partial im2col: materialize only enough rows for one output tile (e.g., one row of output pixels) at a time, cutting the scratch buffer by the tiling factor — CMSIS-NN's arm_convolve_* variants do this, trading a bit of instruction overhead for an order-of-magnitude smaller workspace.
• Direct convolution kernels: skip im2col altogether, accumulating directly from the input tensor with register-blocked loops (common on Cortex-M with CMSIS-NN's 1x1 and depthwise paths) — zero extra scratch at the cost of a less GEMM-friendly inner loop.

Either way, the workspace is itself pooled: the same scratch region is reused by every conv layer in the network (only one conv executes at a time), sized to the single largest per-layer requirement rather than summed across layers.

Practical Allocation Strategy

1. Profile the graph offline. Tools like TFLite Micro's memory planner or Edge Impulse's EON compiler compute the exact peak arena size and print it — don't guess; measure per-model.
2. Separate persistent from scratch. Weights (flash or a dedicated read-only region) never enter the reuse pool; only transient activations and workspace do.
3. Static allocation over the heap. Embedded inference runtimes almost universally use a single static/pre-allocated arena (uint8_t arena[N]) sized at build time — this avoids fragmentation and non-determinism from malloc/free at inference time, which matters for both RAM budget and real-time guarantees.
4. Align scratch buffers to word/cache-line boundaries for the DMA/SIMD paths (Helium/MVE on Cortex-M55, or NPU DMA descriptors) — unaligned scratch silently kills throughput even when it fits in RAM.
5. Watch for false savings from in-place ops that break parallel/pipelined execution — if a dual-core or NPU/CPU split pipeline expects to read layer N's output while computing layer N+1, in-place reuse of that buffer for layer N+1's output creates a data hazard. In-place is a single-threaded, in-order optimization by default; pipelined or NPU-offloaded designs need explicit double-buffering instead.

Key Takeaways

• RAM for inference is dominated by activations and per-op scratch, not by weights (which usually live in flash); peak RAM equals the peak concurrently-live tensor set, not the sum of all tensors.
• Liveness analysis (compiler-style interval scheduling) lets a memory planner pack tensors into one shared arena, cutting RAM by 30–50%+ versus naive per-tensor allocation with zero change to model math.
• In-place execution is safe for element-wise ops (ReLU, quantize, reshape) but unsafe by default for windowed ops (conv, pooling) unless the algorithm provably never overwrites an input before its last read.
• im2col-based convolution can require scratch buffers far larger than any activation tensor (hundreds of KB); tiling or switching to direct convolution kernels (as in CMSIS-NN) controls this.
• Use static, pre-sized arenas rather than dynamic allocation for deterministic, fragmentation-free embedded inference, and re-verify in-place assumptions whenever the execution model becomes pipelined or multi-core.