TFLite Micro Workflow: Train → Convert → Quantize → Deploy

Getting a trained model onto a Cortex-M MCU is not a one-shot export — it's a pipeline where each stage can silently break accuracy or blow the memory budget. TensorFlow Lite Micro (TFLite Micro, or TFLM) is the de facto runtime for this last mile, and the Train → Convert → Quantize → Deploy sequence is the workflow every production edge-AI project ends up implementing, whether through Edge Impulse, STM32Cube.AI, or a hand-rolled Makefile. Understanding what happens — and what can go wrong — at each stage is what separates a demo that works on your desk from a shipped product.

Stage 1: Train (in float32, on full hardware)

Training happens on a GPU/TPU host using standard TensorFlow/Keras, in float32, with no awareness of the target MCU's constraints. The only embedded-relevant decisions made here are architectural:

• Model size vs. SRAM/Flash budget. A Cortex-M4 with 256 KB SRAM cannot host a model whose largest intermediate activation tensor exceeds a few tens of KB, regardless of how small the weights are.
• Operator selection. Stick to ops TFLM's reference kernels or CMSIS-NN actually implement (CONV_2D, DEPTHWISE_CONV_2D, FULLY_CONNECTED, RELU, SOFTMAX, etc.). Exotic ops (dynamic shapes, custom TF ops) force a SELECT_TF_OPS fallback that doesn't exist in TFLM — the model simply won't convert or run.
• Batch norm folding readiness. Keep BatchNormalization directly after Conv2D/Dense so the converter can fold it into the preceding layer's weights; unfused BN adds inference-time ops and hurts INT8 accuracy.

Output of this stage: a Keras .h5/SavedModel, float32, validated against a held-out test set. Record the float32 baseline accuracy — every later stage will be measured as a delta from this number.

Stage 2: Convert (SavedModel → .tflite FlatBuffer)

The TFLite Converter (tf.lite.TFLiteConverter) traces the graph and emits a .tflite FlatBuffer — a flat, mmap-friendly binary format with no dynamic allocation, no Python runtime, and a fixed operator set. This is also where graph-level fusions happen: Conv+BN+ReLU sequences collapse into a single fused op, which is both faster and a prerequisite for correct per-channel quantization later.

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]   # enables quantization path
tflite_model = converter.convert()

At this point, without a quantization dataset supplied, the converter can only do "dynamic range" quantization (weights int8, activations float) — useful as a sanity check, but not the INT8-everywhere target most MCUs need for CMSIS-NN/Ethos-U acceleration.

Stage 3: Quantize (float32 → INT8)

This is the stage with the largest accuracy and footprint impact, and the one worth understanding at the math level.

Why quantize: INT8 gives a 4× reduction in weight storage vs. float32 and lets Cortex-M4/M7 use SIMD MAC instructions (via CMSIS-NN) that don't exist for float ops on those cores, typically a 3–5× inference speedup.

Affine quantization formula, applied per tensor (or per output channel for weights):

q = round(r / scale) + zero_point
r ≈ scale · (q − zero_point)

• r = real (float) value, q = quantized int8 value (range −128..127)
• scale = (r_max − r_min) / (q_max − q_min)
• zero_point maps real 0 exactly to an integer (critical for zero-padding correctness in conv layers)

Post-Training Quantization (PTQ): run 100–500 representative input samples through the float model to record min/max activation ranges, then compute scale/zero_point per tensor. Fast (minutes), no retraining, but can cost 1–5 percentage points of accuracy on sensitive models (especially with small dynamic ranges or heavy weight distributions).

def representative_dataset():
    for x in calibration_data[:300]:
        yield [x.astype(np.float32)]

converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
tflite_int8_model = converter.convert()

Quantization-Aware Training (QAT): insert fake-quant nodes during training so the network learns weights that are robust to the rounding error, recovering most or all of the PTQ accuracy loss at the cost of a retraining run. Use QAT when PTQ drops accuracy more than ~1–2 points, which is common for models with narrow-margin classification boundaries.

Always verify quantized accuracy on the same test set used for the float32 baseline — comparing against a different split hides regressions.

Stage 4: Deploy (interpreter, memory arena, kernels)

The .tflite file is converted to a C byte array (xxd -i model.tflite > model_data.cc) and linked into firmware. At runtime, TFLM uses:

• MicroInterpreter — parses the FlatBuffer and executes ops one by one; no dynamic memory allocation after setup.
• Tensor arena — a single pre-allocated static buffer (uint8_t tensor_arena[N]) TFLM carves up for all intermediate activations. Sizing this is trial-and-error: start generous (e.g. 64 KB), use interpreter.arena_used_bytes() to shrink it to the real requirement plus headroom.
• MicroMutableOpResolver — registers only the ops the model actually uses, keeping Flash footprint minimal instead of linking the full op table.
• Kernel backend — reference C kernels (portable, slow) vs. CMSIS-NN (Cortex-M SIMD-optimized) vs. Ethos-U55 (offloaded to the NPU via vendor driver). Same .tflite file, different backend registration — this is the main lever for latency without touching the model.

constexpr int kTensorArenaSize = 40 * 1024;
static uint8_t tensor_arena[kTensorArenaSize];

const tflite::Model* model = tflite::GetModel(g_model_data);
static tflite::MicroMutableOpResolver<6> resolver;
resolver.AddConv2D();
resolver.AddDepthwiseConv2D();
resolver.AddFullyConnected();
resolver.AddSoftmax();
resolver.AddReshape();
resolver.AddRelu();

tflite::MicroInterpreter interpreter(model, resolver, tensor_arena, kTensorArenaSize);
interpreter.AllocateTensors();

Worked Example: Person-Detection Model Sizing

A MobileNet-v2-style person detector, float32: 250 KB weights, largest activation tensor 96 KB, float32 baseline accuracy 94.2%.

1. Convert + PTQ INT8: weights shrink to 250/4 ≈ 62.5 KB Flash. Measured accuracy on test set: 91.8% (−2.4 pts) — above the 2-point tolerance for this product, so QAT is triggered.
2. QAT retrain (10 epochs, fake-quant inserted): accuracy recovers to 93.6% (−0.6 pts from float32) — acceptable.
3. Tensor arena sizing: largest activation was 96 KB in float32; in INT8 the same tensor is 24 KB, but TFLM needs working space for im2col-style buffers in conv layers too. Start arena at 48 KB, run on target, check arena_used_bytes() reports 34 KB — set final arena to 40 KB for headroom.
4. Check: total static Flash = 62.5 KB (weights) + ~30 KB (TFLM runtime + kernels) ≈ 92.5 KB; total static RAM = 40 KB (arena) + ~5 KB (interpreter/tensors overhead) ≈ 45 KB. Both fit comfortably inside a 512 KB Flash / 256 KB RAM Cortex-M4 part with margin for the rest of the application — confirms the design closes.

Practical Pitfalls

• Unsupported op at convert time. Converter throws an error naming the op — check TFLM's supported-ops list before designing the architecture, not after training for three days.
• Per-tensor vs. per-channel quantization. Per-channel (default for weights in modern converters) is materially more accurate for depthwise conv layers; verify it's enabled rather than assuming.
• Calibration set too small or unrepresentative. 20 samples from one class will produce clipped, biased scale/zero_point values — use at least a few hundred samples spanning the real input distribution, including edge cases (over/under-exposed images, quiet/loud audio).
• Arena undersized silently truncates. AllocateTensors() returns an error code on overflow — always check it; a missed check manifests as garbage inference output, not a crash.

Key Takeaways

• The workflow is Train (float32, host) → Convert (.tflite FlatBuffer, op fusion) → Quantize (INT8 via PTQ or QAT) → Deploy (MicroInterpreter + static tensor arena + kernel backend).
• Quantization is affine: q = round(r/scale) + zero_point; INT8 gives ~4× Flash reduction and 3–5× speedup on Cortex-M via CMSIS-NN SIMD.
• Try PTQ first (minutes, needs only calibration data); escalate to QAT only if accuracy loss exceeds product tolerance (~1–2 points is a common threshold).
• Tensor arena size is empirical — measure with arena_used_bytes() on target rather than guessing from activation tensor sizes alone.
• The same .tflite model can run on reference kernels, CMSIS-NN, or an Ethos-U NPU backend without reconversion — kernel backend choice is a separate, independent optimization lever from model design.