JPEG Decode on MCU: libjpeg-turbo, STM32 JPEG HW IP

Camera-equipped MCU products almost always land on JPEG at some point — either a sensor with a built-in JPEG encoder streams compressed frames to save bandwidth, or a cloud/mobile app expects JPEG for storage and transmission. Decoding that stream back into a raw framebuffer for on-device inference, or encoding a raw frame before it leaves the device, is CPU- and memory-heavy work that can dominate a Cortex-M budget if done naively. Understanding what JPEG actually costs, and how a hardware codec changes the equation, is the difference between a vision pipeline that hits 30 fps and one that stalls at 3.

What JPEG Decode Actually Does

Baseline JPEG decoding is a fixed pipeline, and every stage has a real computational cost on an MCU:

1. Entropy decoding (Huffman) — variable-length bitstream → quantized DCT coefficients. Inherently serial, branch-heavy, poor fit for SIMD.
2. Dequantization — multiply each coefficient by its quantization table entry.
3. Inverse DCT (IDCT) — 8×8 block transform back to spatial domain. This is the classic compute hot spot: a naive 2D IDCT is O(N²) per block (N=8): 8 row passes + 8 column passes, each costing 64 multiply-adds (8 outputs × 8 taps), for ~1024 multiply-adds per 8×8 block; fast algorithms (AAN, Loeffler) cut this to ~5 multiplies + 29 adds per 1D pass.
4. Upsampling — chroma subsampled formats (4:2:0, 4:2:2) need chroma blocks stretched back to full resolution.
5. Color conversion — YCbCr → RGB, a 3×3 matrix multiply per pixel with fixed coefficients (ITU-R BT.601).

For a 640×480 (VGA) 4:2:0 frame: VGA at 4:2:0 has 40×30 = 1200 MCUs (16×16 each), each containing 6 blocks (4 Y + 1 Cb + 1 Cr) = 7200 IDCT blocks total. At even 200 cycles/block for a fast IDCT plus entropy decode overhead, that's over a million cycles per frame — before color conversion and upsampling.

Software Decode: libjpeg-turbo on Cortex-M

libjpeg-turbo is the standard open-source JPEG codec, distinguished from stock libjpeg by SIMD-accelerated IDCT and color conversion (originally MMX/SSE2 on x86, later NEON on Cortex-A). On Cortex-M there is no NEON — the relevant win is Cortex-M4/M7's DSP extension (SIMD8/16, single-cycle MAC via SMLAD/SMLSD), which a Cortex-M-tuned build (or a hand-optimized libjpeg fork) can exploit for the IDCT's multiply-accumulate-heavy inner loop.

Practical constraints of running any full-featured libjpeg-turbo build on an MCU:

• Code size: the full library (all sampling factors, all DCT accuracy modes, progressive support) is typically 50–250 KB of flash — significant on a part with 512 KB–1 MB total. Trimming to baseline-only, fixed 4:2:0 sequential decode with the integer IDCT can shrink this substantially.
• RAM/working set: libjpeg-turbo's default decompressor allocates multiple scanline buffers and a full MCU row buffer; for VGA 4:2:0 this is tens of KB, on top of the output framebuffer itself (VGA RGB565 = 600 KB — already larger than most Cortex-M SRAM, forcing tiled/streamed decode instead of whole-frame buffering).
• Cycle budget: on a 400 MHz Cortex-M7 with the DSP-optimized IDCT, decoding a VGA frame typically costs on the order of 15–40 ms depending on quality/subsampling — adequate for a snapshot pipeline, marginal for continuous video at the sensor's native frame rate.

Hardware JPEG Codec: the STM32 JPEG IP

Several STM32H7-family parts (and similarly equipped Cortex-M7 devices) integrate a dedicated JPEG codec peripheral that implements the full baseline pipeline — Huffman codec, dequantization, IDCT/DCT, up/downsampling — in silicon, driven by DMA with minimal CPU involvement.

Typical operating model:

• The CPU configures the peripheral (quantization tables, Huffman tables, image dimensions, color format) and starts a job.
• Input compressed data streams in via DMA from flash/SDRAM/camera buffer; decoded MCUs stream out via DMA to a destination buffer (often directly into a display framebuffer or an SRAM region for further processing).
• An interrupt (or DMA complete flag) signals job completion; the CPU is free to do other work — including running the next inference pass on the previous frame — while decode is in flight.
• Throughput is on the order of tens of megabytes per second of compressed input, translating to full-frame JPEG decode/encode in roughly single-digit milliseconds for VGA-class images — 5–10× faster than an optimized software path, at near-zero CPU load.

This changes the system-level trade entirely: instead of budgeting tens of milliseconds of Cortex-M7 cycles per frame for decode, the CPU budget is freed almost entirely for the downstream vision/AI workload (preprocessing, CNN inference), and decode becomes a pipeline stage overlapped with everything else via DMA and double buffering.

Software vs. Hardware: Comparison

Worked Example: Pipeline Budget for a 15 fps Vision Application

Assume a person-detection pipeline on an STM32H7 at 480 MHz: camera delivers MJPEG frames at VGA resolution, target is 15 fps (66.7 ms/frame budget), and a MobileNet-style CNN inference takes 35 ms per frame via CMSIS-NN.

Software JPEG path:

• Decode: ~25 ms (software, DSP-optimized IDCT)
• Resize/preprocess: ~5 ms
• Inference: 35 ms
• Total: 65 ms → just barely fits 66.7 ms, zero margin for anything else (display update, sensor control, housekeeping).

Hardware JPEG path:

• Decode: ~3 ms (HW IP, DMA, overlapped with prior frame's inference)
• Resize/preprocess: ~5 ms
• Inference: 35 ms
• Total active CPU time: ~43 ms, well under the 66.7 ms budget, leaving ~23.7 ms margin.

Check: 25 + 5 + 35 = 65 ms ≤ 66.7 ms (software case holds, no slack). 3 + 5 + 35 = 43 ms ≤ 66.7 ms (hardware case, 23.7 ms margin), consistent with the figures above. The hardware path not only meets the deadline but leaves headroom to raise frame rate toward ~23 fps (1000/43 ≈ 23.3) or add a second processing stage (e.g., tracking, telemetry) without touching the frame budget.

Design Implications

• Choose HW IP when available and format matches — baseline sequential JPEG at fixed subsampling is the common case for camera modules; if the part has the peripheral, use it rather than spending CPU cycles reproducing it in software.
• Fall back to libjpeg-turbo (trimmed) when portability matters — cross-platform code, non-baseline JPEG (progressive), or MCUs without the IP block still need a software path; strip unused format support aggressively to control flash/RAM.
• Buffer sizing drives SRAM planning — full-frame RGB output buffers often exceed on-chip SRAM; plan for tiled/line-based decode output or place large buffers in external SDRAM/QSPI-mapped memory with DMA.
• Overlap decode with compute — with HW IP and DMA, decode one frame while inference runs on the previous one (double buffering) to hide the entire decode latency behind compute time.
• Verify color conversion coefficients match — HW IP and software libraries may default to slightly different YCbCr↔RGB rounding; mismatches show up as subtle color/contrast shifts that can affect downstream CNN accuracy if training data was preprocessed differently.

Key Takeaways

• Baseline JPEG decode involves Huffman entropy decoding, IDCT, chroma upsampling, and color conversion — the IDCT and entropy stage dominate CPU cycles on an MCU.
• libjpeg-turbo brings SIMD-style optimization to JPEG, but on Cortex-M its benefit depends on DSP extension support (SMLAD-class instructions); flash and RAM footprint must be trimmed for MCU deployment.
• STM32H7-class parts with a dedicated JPEG codec IP offload the entire pipeline to DMA-driven hardware, cutting VGA decode latency from tens of milliseconds to single digits and freeing the CPU almost entirely.
• The system-level payoff of hardware decode is not just speed — it's CPU/power headroom that can be reallocated to CNN inference, higher frame rate, or additional pipeline stages.
• Always validate that the frame budget (decode + preprocess + inference ≤ 1/fps) closes with real measured numbers, not just IP throughput datasheet figures, since DMA contention and memory bus sharing affect actual latency.