Model Deployment Pipeline: CI/CD for Edge AI Models

Shipping a trained model to a fleet of MCUs is not a one-time export step — it's a pipeline problem. A model that changes its input distribution assumptions, drifts in accuracy after retraining, or gets miscompiled for a target's NPU can silently degrade a product in the field with no crash to report. Treating model builds like firmware — versioned, tested, gated, and rolled out with the same rigor as a bootloader update — is what separates a research prototype from a maintainable edge AI product.

Why Edge AI Needs Its Own CI/CD

Standard software CI/CD (build → test → deploy) doesn't map cleanly onto ML artifacts for three reasons:

• Two independently versioned inputs: code (inference firmware, pre/post-processing) and data (training set, labels) both change the output binary. A pipeline must track both, not just git commits.
• Non-deterministic quality gates: a firmware build either compiles or doesn't. A model build produces a distribution of accuracy outcomes that must be measured statistically against a held-out test set, not just checked for build success.
• Target-specific compilation: the same .tflite or ONNX graph produces different latency, memory footprint, and even numerical results depending on the compiler (CMSIS-NN, Ethos-U vendor toolchain, TVM) and target silicon. A pipeline has to build and validate per target, not once.

Pipeline Stages

[Data versioning] → [Train] → [Convert/Quantize] → [Target compile]
      → [HW-in-loop validation] → [Model registry] → [Staged OTA rollout] → [Field telemetry]
                                          ^________________feedback_______________|

1. Data and experiment versioning

Every training run must be traceable to an exact dataset snapshot and hyperparameter set. Tools like DVC or a data lake with content-addressed hashes give each dataset version a hash; the training job records (dataset_hash, code_commit, hyperparams) → model_hash. Without this, a model that regresses in the field cannot be debugged — you can't reproduce what produced it.

2. Train and evaluate

Standard training loop, but the CI gate is a fixed, versioned evaluation set — never the live training split — scored on the metrics that matter for the product (accuracy, F1 per class, false-positive rate for a wake word, not just top-1). A model only proceeds if it beats the currently deployed model on this set by a defined margin (e.g., ≥0.5% absolute accuracy, to avoid promoting noise-level "improvements").

3. Convert and quantize

Export to the interchange format (ONNX, SavedModel) then to the deployment format: TFLite Micro flatbuffer, or vendor-specific for an NPU (Ethos-U, MAX78000, STM32N6 Neural-ART). Post-training INT8 quantization or QAT happens here. This step must run in CI, not on an engineer's laptop — quantization is sensitive to the calibration dataset, and an untracked local conversion is a silent source of train/deploy skew.

4. Target compilation

Run the vendor compiler/optimizer (CMSIS-NN codegen, Ethos-U Vela compiler, TVM microTVM) to produce the final target binary or linkable object. Capture as build artifacts:

A build that regresses any of these against the previous release's numbers fails automatically — this is the ML equivalent of a static size/timing regression check in firmware CI.

5. Hardware-in-the-loop (HIL) validation

Compiling for Cortex-M or an NPU does not guarantee correctness — toolchain bugs, alignment issues, and operator support gaps happen. A HIL rig flashes the candidate binary to actual target boards, feeds them a fixed suite of recorded input vectors (audio clips, images, IMU traces), and compares outputs against the golden reference. This catches divergence that pure simulation misses and is the single most important gate before promoting a model beyond internal testing.

6. Model registry and provenance

Every model that passes gates is stored in a registry (could be as simple as a tagged artifact store) with a manifest:

model_id: kws_dscnn_v14
dataset_hash: 9f3a2c...
code_commit: a1b2c3d
target: cortex-m4 / cmsis-nn 4.1
quant: int8, per-channel
eval_accuracy: 96.8%  (Δ +0.6% vs v13)
flash_kb: 84   ram_kb: 22   latency_ms: 11.4
signed_by: build-server-key-03

The manifest plus a cryptographic signature is what the OTA system checks before accepting a model — never trust an unsigned blob shipped to devices in the field.

7. Staged rollout and rollback

Mirror standard OTA canary practice, applied to model weights specifically:

• Canary (1–5% of fleet): deploy, monitor for crashes, latency regressions, and — where ground truth is available — accuracy proxies (e.g., user correction rate for a wake-word false accept).
• Ring rollout: expand to larger cohorts (25% → 100%) only if canary telemetry stays within bounds over a defined soak period.
• Automatic rollback: keep the previous model resident (A/B partition, same pattern as firmware A/B bootloading) so a bad rollout reverts without a truck roll. This is especially important for models — a subtle accuracy regression may not trip a watchdog but will trip a monitored KPI.

8. Field telemetry feeding back

Aggregate lightweight signals from deployed devices — inference confidence histograms, class distribution drift, rejected/low-confidence input rate — back to the training pipeline. This closes the loop for detecting data drift: if the field input distribution diverges from the training set (new accents for a KWS model, new lighting conditions for a vision model), it shows up as a shift in confidence statistics well before accuracy visibly collapses, and can trigger a retraining job automatically.

Worked Example: Keyword-Spotting Model Update

A DS-CNN wake-word model (see the KWS article for the architecture) needs a quarterly refresh with new accent data.

1. New audio batch tagged, hashed, added to dataset v9. Training job kicked off referencing dataset_v9 + commit e5f6....
2. Trained model scores 96.8% on the fixed eval set vs. 96.2% for the currently deployed v13 → passes the ≥0.5% gate.
3. Quantized INT8 (per-channel), converted to TFLite Micro flatbuffer, compiled with CMSIS-NN for Cortex-M4.
4. Footprint check: flash 84 KB (budget 128 KB, OK), RAM 22 KB (budget 32 KB, OK), latency 11.4 ms per 1-second frame (deadline 20 ms, OK).
5. HIL rig runs 500 recorded utterances through actual silicon: output matches float reference within expected INT8 tolerance (max logit delta 0.03), no divergent classifications.
6. Registered as kws_dscnn_v14, signed, pushed to canary ring (2% of fleet) for 72 hours.
7. Telemetry: false-accept rate stable, no crash reports, confidence histogram shifted positively (expected, matches accent improvement) → promoted to 100% over the next week.
8. Previous model v13 retained on the inactive OTA partition for one release cycle in case rollback is needed.

Every step here is auditable after the fact — if a false-accept spike appears three weeks later, the manifest ties the deployed binary back to the exact dataset, commit, and toolchain version that produced it.

Practical Design Implications

• Treat the quantized target binary as the artifact under test, not the float model — accuracy measured before quantization does not guarantee accuracy after.
• Automate the footprint/latency regression gate the same way you'd gate a firmware image size — a model that silently grows past the flash budget breaks the build the same way an oversized .bin does.
• Never skip HIL for NPU targets — Ethos-U and similar accelerators have operator support matrices that differ from the CPU reference kernel; a graph that "converts fine" can still silently fall back to slow CPU execution for an unsupported op if not caught in CI.
• Keep an A/B model partition in the OTA scheme, mirroring firmware A/B updates, so rollback is instantaneous rather than requiring a re-flash cycle.
• Version data with the same discipline as code — the most common cause of "it worked in the notebook, not on device" is an untracked change to the calibration or eval dataset between runs.

Key Takeaways

• Edge AI CI/CD must version both code and data, and gate on statistical accuracy metrics in addition to build success.
• The pipeline has extra stages beyond standard CI/CD: quantization/conversion, target-specific compilation, and hardware-in-the-loop validation against golden reference outputs.
• Footprint (flash/RAM), latency, and quantized-vs-float numerical delta are regression-gated the same way firmware size and timing are.
• A signed model registry with full provenance (dataset hash, commit, target toolchain, eval score) is required to make field issues debuggable.
• Staged canary rollout with an A/B rollback partition, plus field telemetry feeding drift detection back into retraining, closes the loop for a maintainable long-term deployment.