Post-Training Quantization: FP32 → INT8 Weight Conversion

Every TFLite Micro or CMSIS-NN deployment on a Cortex-M part eventually hits the same wall: an FP32 model that trained cleanly on GPU refuses to fit in 256–512 KB of SRAM/flash, and even if it fit, the MCU has no hardware FPU throughput to run it in real time. Post-training quantization (PTQ) converts the trained FP32 weights (and optionally activations) to INT8 without retraining, typically shrinking the model 4× and speeding up MAC throughput 2–4× on cores with SIMD INT8 support (e.g., Cortex-M4/M7 DSP extensions, Cortex-M55/M85 with Helium, or an attached NPU). The cost is accuracy — usually 0.5–2% on well-behaved CNNs, sometimes catastrophic if the weight distribution is pathological. Understanding the arithmetic is what separates "quantize and hope" from a defensible engineering decision.

The Core Mapping: Affine Quantization

INT8 quantization maps a real-valued range to the 256 integer codes using a scale and zero-point:

q = round(r / S) + Z        (quantize)
r ≈ S · (q − Z)              (dequantize)

• r — real (FP32) value
• q — quantized integer, clamped to [−128, 127] (signed INT8) or [0, 255] (unsigned)
• S — scale factor (FP32), S = (r_max − r_min) / (q_max − q_min)
• Z — zero-point (integer), the quantized code that represents real value 0

Symmetric vs. asymmetric:

Weights are almost always quantized symmetric per-channel because Z = 0 eliminates a cross-term in the integer matmul, simplifying the accelerator's dot-product hardware. Activations are usually asymmetric because their range is skewed (ReLU outputs are ≥ 0, so wasting half the codebook on negative values is wasteful).

Per-Tensor vs. Per-Channel Scale

A single scale for an entire weight tensor is simplest but forces every output channel to share one dynamic range. If one filter has weights in [−0.02, 0.02] and another in [−2, 2], per-tensor quantization wastes resolution on the narrow filter.

Per-channel (per-axis) quantization assigns one scale Sᵢ per output channel i of a conv/FC layer:

qᵢⱼ = round(wᵢⱼ / Sᵢ)     for channel i, weight index j

This is now the default in TFLite's converter for weights, because it costs nothing extra in the integer matmul (the per-channel scale is folded into the per-channel output rescaling that already exists to convert INT32 accumulator back to INT8) but recovers most of the accuracy lost to per-tensor quantization — often the difference between 1% and 5% top-1 drop on MobileNet-class networks.

The Integer Matmul and Requantization

A quantized conv/FC layer computes in INT32 accumulation, then rescales:

acc[m] = Σₖ (q_x[k] − Z_x) · (q_w[m,k] − Z_w)      // INT32 accumulator
y[m]   = round(acc[m] · (S_x · S_w[m]) / S_y) + Z_y // requantize to INT8

The multiplier (S_x · S_w[m]) / S_y is a real number but must be applied with integer-only arithmetic on an MCU with no FPU. This is done with a fixed-point multiplier + right-shift — CMSIS-NN and TFLite Micro both express it as M0 · 2^(-n), where M0 is a 32-bit fixed-point value in [0.5, 1) and n is a shift count, computed once at conversion time and stored per-channel alongside the weights.

Worked Example: Quantizing a Weight Tensor

Take one output channel with FP32 weights w = [−1.5, 0.3, 0.9, −0.05].

Step 1 — find range: w_max = 0.9, w_min = −1.5. Symmetric quantization uses the larger magnitude: |w|_max = 1.5.

Step 2 — compute scale (signed INT8, symmetric, range −127..127):

S = 1.5 / 127 = 0.011811

Step 3 — quantize each weight: q = round(w / S)

q(−1.5)  = round(−1.5  / 0.011811) = round(−127.0) = −127
q(0.3)   = round( 0.3  / 0.011811) = round(  25.4)  =  25
q(0.9)   = round( 0.9  / 0.011811) = round(  76.2)  =  76
q(−0.05) = round(−0.05 / 0.011811) = round(  −4.23) =  −4

Step 4 — verify by dequantizing:

r(−127) = −127 · 0.011811 = −1.4999  ✓ (target −1.5)
r(25)   =   25 · 0.011811 =  0.2953  (target 0.3, error 0.0047)
r(76)   =   76 · 0.011811 =  0.8976  (target 0.9, error 0.0024)
r(−4)   =   −4 · 0.011811 = −0.0472  (target −0.05, error 0.0028)

Max quantization error here is ≈0.005, consistent with the theoretical bound S/2 ≈ 0.0059. This confirms the scale and rounding are correct — every reconstructed value falls within half an LSB of the original, as expected for round-to-nearest quantization.

Calibration for Activations

Weights are quantized directly from their known static values. Activations require calibration: running a few hundred representative input samples through the FP32 model and recording the observed min/max (or a percentile/histogram, e.g., KL-divergence minimization as used in TensorRT) at each activation tensor. Poor calibration data (unrepresentative of deployment inputs, or too few samples) is the single most common cause of PTQ accuracy collapse — outliers in a small calibration set can blow up the range and crush resolution for the bulk of the activation distribution.

When PTQ Breaks Down

• Depthwise conv layers with wide per-channel weight variance are the classic MobileNet failure mode — per-channel quantization mitigates this but doesn't always fully fix it.
• Layers with large outlier activations (e.g., after batch-norm folding produces large scale factors) can force a wide range that starves resolution elsewhere.
• Small networks with few output classes are more sensitive per-parameter — a 1% absolute accuracy drop matters more on a 3-class keyword spotter than a 1000-class classifier.

When PTQ accuracy loss exceeds ~1–2%, the next step is quantization-aware training (QAT), which simulates quantization noise during fine-tuning so the network learns weights that are robust to it — at the cost of needing the original training pipeline and labeled data, which PTQ does not require.

Practical Deployment Flow

1. Train and validate the FP32 model normally.
2. Fold batch-norm into preceding conv/FC weights (BN folding) — quantizing BN parameters separately is almost never done in production.
3. Collect a calibration set (100–1000 samples spanning the real input distribution).
4. Run the converter (e.g., TFLite's TFLiteConverter with representative_dataset) to produce per-channel symmetric INT8 weights and asymmetric INT8 activations.
5. Validate accuracy on a held-out test set — not the calibration set — before flashing.
6. Profile actual MCU cycle count/RAM with CMSIS-NN or the vendor NPU driver; INT8 kernels use different scratch buffer sizing than FP32.

Key Takeaways

• PTQ maps FP32 values to INT8 via q = round(r/S) + Z, with symmetric (Z=0) quantization for weights and asymmetric quantization for activations.
• Per-channel weight scales are effectively free on integer hardware and recover most of the accuracy lost versus per-tensor scales — always prefer per-channel for conv/FC weights.
• The INT32 accumulator from an integer matmul is rescaled to INT8 using a fixed-point multiplier and shift, computed once at conversion time — no floating-point math at inference.
• Activation quantization requires calibration on representative data; a small or unrepresentative calibration set is the most common cause of unexpected accuracy loss.
• Always verify dequantized values against the original FP32 weights (error should be ≤ S/2) and validate accuracy on held-out data before committing to PTQ over QAT.