MCU AI Frameworks: TensorFlow Lite Micro (TFLite Micro)

Deploying a trained neural network on a microcontroller with 256 KB of RAM and no OS is a fundamentally different problem from serving it on a GPU. TensorFlow Lite Micro (TFLM) is the runtime most teams reach for because it runs with zero dynamic memory allocation, zero OS dependencies, and a C++11 core small enough to fit alongside application code on a Cortex-M0 to M55/M85. Understanding its memory model and interpreter internals is what separates "it compiles" from "it actually runs within budget in production."

Why TFLM Is Different from Desktop TensorFlow Lite

Mobile TFLite (Android/iOS) relies on a filesystem, dynamic heap, multithreading, and often a delegate (GPU/NNAPI/Hexagon). None of that exists reliably on an MCU. TFLM strips these away:

The model itself is unchanged — it's still a FlatBuffer produced by the standard TFLite converter. What changes is everything around it.

The Memory Model: Arenas, Not malloc

TFLM never calls malloc/free during inference. Instead, the application allocates one contiguous byte buffer — the tensor arena — and the interpreter carves it up internally using a static planner that runs once at model-load time.

uint8_t tensor_arena[64 * 1024];   // sized by trial or by profiling

tflite::MicroInterpreter interpreter(
    model, resolver, tensor_arena, sizeof(tensor_arena));
interpreter.AllocateTensors();     // arena planning happens here

AllocateTensors() walks the model graph and determines the liveness of every intermediate tensor: at what op index it's first written and last read. Tensors whose lifetimes don't overlap are placed at the same offset in the arena — this is the same in-place/scratch-buffer reuse strategy used across memory-constrained inference engines. Only weights (stored in flash, read-only, not in the arena) and tensors with overlapping lifetimes need distinct space simultaneously.

Sizing the arena in practice: there is no closed-form formula that accounts for every op's temporary buffers, so the standard workflow is:

1. Start with a generous guess (e.g., 100 KB).
2. Call interpreter.arena_used_bytes() after AllocateTensors() to get the actual high-water mark.
3. Shrink the declared arena to that value plus ~5–10% margin for future model revisions.

If the arena is undersized, AllocateTensors() returns an error at init time — it fails loudly, not silently, which is the correct failure mode for a safety-relevant embedded build.

The Interpreter Loop

TFLM's execution is a straight, allocation-free walk over a static graph:

1. Load — the FlatBuffer (compiled into flash as const unsigned char g_model[]) is mapped directly, zero-copy; no parsing/deserialization step.
2. Resolve ops — the MicroMutableOpResolver is populated only with the kernels the model actually uses (e.g., AddFullyConnected(), AddConv2D(), AddDepthwiseConv2D()). This is a deliberate design choice: unlike mobile TFLite's BuiltinOpResolver which registers everything, TFLM makes you opt in per-op, directly shrinking flash footprint.
3. Allocate — AllocateTensors() runs the static planner described above.
4. Invoke — interpreter.Invoke() executes each op's kernel in graph order, reading/writing tensors in the pre-planned arena locations, no allocation, no branching on shape at runtime (shapes are fixed post-allocation).

Because step 4 has no allocation and no OS calls, its execution time is deterministic run-to-run — a property that matters for real-time budgets in KWS or gesture pipelines where inference must complete inside a fixed audio/sensor frame period.

Worked Example: Sizing a Person-Detection Model

Assume an int8-quantized MobileNet-style model with:

• Weights: 250 KB (flash, read-only, not counted against arena)
• Largest single activation tensor: 96×96×8 int8 = 73,728 bytes at the first conv layer
• Reported arena_used_bytes() after a dry run: 118,000 bytes

Target MCU: Cortex-M7 with 512 KB flash, 320 KB RAM.

Flash check: 250 KB weights + ~40 KB TFLM runtime + application code (~60 KB) = 350 KB. Fits in 512 KB with margin.

RAM check: 118 KB arena + ~20 KB stack/globals + RTOS overhead (~10 KB) = 148 KB. Fits in 320 KB with over 50% headroom for double-buffered camera frames.

Verify against the naive worst case: if TFLM allocated every tensor without reuse, the sum of all activation tensors in a MobileNet-scale graph typically runs 3–5× higher than the planner's output — here that would be roughly 350–590 KB, which would not fit in 320 KB RAM. The arena planner's tensor-lifetime reuse is what makes this model viable on this part at all; without it, you'd need to drop to a smaller input resolution or a lighter backbone.

This is the calculation every deployment must redo per model revision — a single added layer or increased channel count can shift the high-water mark non-linearly, since it depends on which tensors overlap in lifetime, not just total tensor count.

Practical Implications for Design

• Op coverage gaps: TFLM supports a subset of TFLite ops. Check the reference-kernel list before finalizing a model architecture — an unsupported op (e.g., certain TRANSPOSE_CONV variants) forces either a custom kernel or a model redesign late in the project.
• Quantization is mandatory in practice: float32 TFLM models run, but int8 (post-training quantization or QAT) typically gives 2–4× speedup on Cortex-M via CMSIS-NN kernels and halves both flash and arena footprint.
• CMSIS-NN as the optimized backend: registering CMSIS-NN kernels instead of the pure reference kernels (tflite::ops::micro::AddConv2D variants) accelerates conv/depthwise/FC ops by 2–5× on Cortex-M4/M7 via SIMD (SMLAD) instructions — same graph, same arena math, faster Invoke().
• Static graphs mean static I/O shapes: dynamic batch size or variable input resolution isn't supported; the model must be exported with fixed input dimensions.
• No streaming allocation for RNNs/LSTMs: stateful ops need their state tensors accounted for in the arena across calls — TFLM keeps state buffers persistent between Invoke() calls rather than reallocating.
• Debugging arena overflows: enable MicroErrorReporter output; it prints the specific op/tensor index that failed to allocate, rather than a generic OOM.

Key Takeaways

• TFLM replaces dynamic heap allocation with a single static tensor arena, sized once via a build-time liveness/reuse planner — this is the core enabler of running CNNs on <1 MB RAM parts.
• The model format (FlatBuffer, quantized or float) is identical to mobile TFLite; only the runtime and its allocation strategy differ.
• Op resolution is opt-in (MicroMutableOpResolver) to keep flash footprint minimal — register only the kernels the specific model needs.
• Always measure arena_used_bytes() empirically rather than estimating; tensor lifetime overlap makes worst-case sizing non-obvious.
• Swapping reference kernels for CMSIS-NN kernels gives a substantial latency win with no change to the model or the arena-sizing math.
• Inference execution is deterministic (no runtime allocation, no branching on shape), which is essential for meeting fixed real-time frame budgets.