Image Preprocessing on MCU: Resize, Crop, Normalize

Every embedded vision pipeline spends cycles before the neural network ever runs. A camera delivers frames in whatever resolution, aspect ratio, and pixel format its sensor produces; the model expects a fixed-size tensor, typically 96×96 to 224×224, in a specific channel order and numeric range. The transform between those two — resize, crop, normalize — is often 30–60% of total inference latency on a Cortex-M class device, because unlike the quantized INT8 matrix multiplies inside CMSIS-NN, preprocessing is usually done in floating point on raw pixel data with poor cache/memory locality. Getting this stage wrong silently degrades accuracy even when the model itself is correct, because training-time preprocessing and deployment-time preprocessing must match bit-for-bit in behavior, not just in intent.

The Pipeline and Why Order Matters

A typical MCU vision preprocessing chain looks like:

Sensor frame (e.g., 640×480 YUV422/Bayer)
   → Color space conversion (→ RGB or grayscale)
   → Crop (region of interest / aspect-ratio fix)
   → Resize (→ model input dims, e.g., 96×96)
   → Normalize (uint8 → float32 or int8, scale + zero-point)
   → Tensor layout (HWC → CHW if required by the runtime)

The order is not arbitrary. Crop before resize whenever possible — cropping first reduces the pixel count the resize filter has to process, and it avoids resizing regions of the image you're about to discard, which wastes cycles and can introduce interpolation artifacts at the eventual crop boundary. Color conversion is usually cheapest done first (or fused into the sensor DMA/hardware ISP if available) because it's a per-pixel operation with no spatial dependency, so doing it while data is still contiguous in DMA-friendly order avoids a second full-frame pass later.

Cropping

Cropping serves two purposes: matching aspect ratio (a 4:3 sensor frame feeding a square model input) and focusing on a region of interest (e.g., a previously detected face bounding box for a second-stage classifier).

Center crop for aspect-ratio correction: given a source W×H and target aspect ratio Wt:Ht, compute the largest centered rectangle matching that ratio, then resize that rectangle to the model's input size. For a 640×480 (4:3) frame feeding a 96×96 (1:1) model:

crop_h = W_src * Ht / Wt = 640 * 1/1 = 640  → exceeds H_src, so constrain by height instead
crop_w = H_src * Wt / Ht = 480 * 1/1 = 480
x0 = (640 - 480) / 2 = 80
y0 = (480 - 480) / 2 = 0
→ crop region: [80, 0, 560, 480] (480×480)

This discards 160 columns of the original frame — acceptable for a centered subject, lossy if the object of interest is near the left/right edge. This is a design decision, not just an implementation detail: if your application (e.g., a wide-FOV security camera) can't tolerate losing the edges, resize to a non-square intermediate and pad instead of cropping.

ROI crop (from a detector's bounding box) needs boundary clamping — a box near the frame edge must be clamped to [0, W-1] × [0, H-1] before any pointer arithmetic, or you get an out-of-bounds read. Always clamp in integer coordinates before converting to the resize stage's fixed-point or float coordinates.

Resizing: Nearest, Bilinear, and the Aliasing Trap

Resize is where the real compute and the real accuracy risk live.

Bilinear interpolation for a destination pixel at fractional source coordinate (x, y):

x0 = floor(x); x1 = x0 + 1
y0 = floor(y); y1 = y0 + 1
fx = x - x0;   fy = y - y0

top    = P(x0,y0)*(1-fx) + P(x1,y0)*fx
bottom = P(x0,y1)*(1-fx) + P(x1,y1)*fx
out    = top*(1-fy) + bottom*fy

The aliasing trap: bilinear interpolation only samples a 2×2 neighborhood. If you're downscaling by more than ~2×, e.g., a 640×480 sensor frame straight to 96×96 (~6.67× reduction horizontally, 5× vertically), bilinear skips most source pixels and high-frequency detail aliases into noise, which then feeds the CNN as if it were signal. Training pipelines (OpenCV, PIL, TF) typically use area-averaging or apply a low-pass filter before large downscales — if your MCU pipeline uses plain bilinear at the same ratio, you introduce a train/deploy mismatch that can cost several points of accuracy even though nothing "crashes." The fix is either a two-stage resize (downscale by 2× repeatedly, each stage using proper averaging, until within 2× of target, then bilinear) or an explicit box-filter pass before the final bilinear step.

Fixed-point implementation matters on Cortex-M0/M3 without an FPU: represent fx, fy as Q8 (0–255 fixed point) instead of float, and do the interpolation in Q8 integer arithmetic — this is what CMSIS-DSP-style image resize routines do, and it avoids float promotion/demotion overhead entirely.

Normalization: Matching the Model's Training Statistics

Normalization converts pixel intensities into the numeric range and distribution the model was trained on. The three common conventions:

For a quantized model, the normalization is the quantization — you must use the exact scale and zero_point embedded in the .tflite file's input tensor, not a generic INT8 cast. Getting this wrong (e.g., assuming zero_point = 0 when the model was trained with asymmetric quantization and zero_point = -128) produces a systematic offset that degrades every inference without any obvious failure signature.

Worked check: input model expects int8 with scale = 0.0078125 (= 1/128) and zero_point = -128 (i.e., mapping [0,255] uint8 → [-128,127] int8 representing [0.0, 1.0] float). For a pixel value of 200:

float equivalent = 200 / 255 ≈ 0.784
x_q = round(0.784 / 0.0078125) + (-128) = round(100.392) - 128 = 100 - 128 = -28

Verify by dequantizing: (-28 - (-128)) * 0.0078125 = 100 * 0.0078125 = 0.78125 ≈ 0.784. ✓ Matches within quantization step size (1/128 ≈ 0.0078), confirming the round-trip is consistent.

Memory and Layout Considerations

• HWC vs. CHW: most MCU camera pipelines produce interleaved HWC (RGB, RGB, RGB...); most quantized TFLite Micro models expect HWC too, but frameworks ported from PyTorch (CHW-native) may require an explicit transpose — check the model's input tensor shape, don't assume.
• In-place vs. scratch buffer: resize cannot be done fully in-place (each output pixel depends on multiple input pixels that may already be overwritten), so budget a separate output buffer; crop can be done via pointer/stride tricks with zero copy if the downstream stage can read strided rows.
• DMA and hardware ISP offload: on STM32 parts with a DCMI/hardware JPEG or resize block, do color conversion and even resize in the DMA path where possible — this frees the CPU/NPU entirely for inference and removes a full-frame memory pass from the critical path.
• Working precision: do resize math in Q8 fixed-point or int16 accumulators, not float, on FPU-less cores; reserve float only for the final normalize step if the model input is float32.

Key Takeaways

• Preprocess in this order: color convert → crop → resize → normalize, to minimize wasted work and keep boundary handling simple.
• Center-crop for aspect ratio, ROI-crop with explicit boundary clamping for detector outputs; cropping first reduces resize workload.
• Bilinear interpolation is the standard choice, but plain bilinear aliases badly on downscale ratios beyond ~2× — match the training pipeline's use of area averaging/low-pass filtering or accuracy will silently drop.
• Normalization must exactly reproduce training-time statistics; for quantized models, use the .tflite input tensor's actual scale/zero_point, never an assumed default.
• Use fixed-point (Q8/int16) arithmetic for resize and normalize on FPU-less Cortex-M cores, and offload color conversion/resize to DMA or hardware ISP blocks where the silicon supports it.
• Preprocessing routinely consumes as much MCU time as INT8 inference itself — treat it as a first-class optimization target, not a throwaway glue step.