Continual / Incremental Learning on Embedded Devices

Most deployed edge models are frozen at flash time: train in the cloud, quantize, deploy, done. But sensor drift, new user behaviors, seasonal variation, and deployment-site-specific data distributions mean the fixed model degrades over months. Continual (incremental) learning is the set of techniques that let a device update its own model in the field — without shipping raw data back to a server and without forgetting what it already learned. On a Cortex-M or NPU-class device with kilobytes of SRAM and no backprop-friendly runtime, this is one of the hardest problems in embedded AI, which is why it sits at the top of the vision/AI skill tree.

Why On-Device Learning Is Different From Training in the Cloud

Cloud training assumes: large shuffled datasets, unlimited passes (epochs), full-precision gradients, and no constraint on RAM for activations or optimizer state. On-device incremental learning inverts every one of these assumptions:

The central problem is catastrophic forgetting: gradient descent on new data overwrites the weights that encoded old knowledge, because nothing in the loss function protects them. A keyword-spotting model retrained on a new user's voice can lose accuracy on the original wake word entirely within a few dozen updates if trained naively.

Taxonomy of Continual Learning Strategies

Regularization-based — penalize changes to weights that were important for old tasks.

• Elastic Weight Consolidation (EWC): adds a quadratic penalty L = L_new + Σᵢ (λ/2)·Fᵢ·(θᵢ − θᵢ*)², where Fᵢ is the Fisher information (importance) of parameter i and θᵢ* is its old value. Cheap to compute incrementally, but Fisher matrix estimation and storage still cost memory proportional to the number of parameters — a concern at MCU scale.
• Synaptic Intelligence (SI): similar idea, tracks per-parameter importance online during training rather than via a separate Fisher pass, slightly cheaper.

Replay-based — keep a small buffer of past exemplars (raw or compressed) and interleave them with new samples during updates so the loss surface still "remembers" old data.

• Exemplar replay: store a handful of raw feature vectors per class (not full images) — e.g., 5–20 embeddings of a few hundred floats each is often feasible in tens of KB.
• Generative replay: train a tiny generative model to synthesize pseudo-old-samples instead of storing them — trades storage for extra compute, rarely worth it below Cortex-A.

Architecture-based — grow or partition the network so new tasks get new capacity rather than overwriting shared weights (e.g., adding a new output head, or masking a subset of channels per task). Effective for stability but increases model size and requires deciding task boundaries at inference time, which the device may not know a priori.

Parameter-efficient / last-layer adaptation — freeze the feature extractor (backbone), retrain only the final classifier layer (or a small adapter). This is the dominant practical pattern on MCUs because it reduces the trainable parameter count from millions to hundreds, fitting comfortably in available RAM and avoiding backprop through the whole network.

The Practical Embedded Pattern: Frozen Backbone + Adaptive Head

Given the memory and compute ceiling, almost all shipping "on-device learning" systems use this structure:

Input → [Frozen CNN/backbone, INT8, no gradient]
              ↓ feature embedding (e.g., 128 floats)
        [Small trainable classifier: linear layer or small MLP]
              ↓
         class scores

Only the classifier head is updated in the field, typically with one of:

• Nearest-class-mean / prototype update: maintain a running mean embedding per class; classify new samples by nearest prototype (cosine or L2 distance). Update is μ_c ← μ_c + (1/n_c)·(x − μ_c) — an O(1) running average, no gradient descent needed at all. This is the lowest-cost form of incremental learning and is naturally forgetting-resistant since each class's prototype is independent.
• Online linear/logistic regression on the head: a few SGD steps on the last layer's weights per new labeled sample, with a small learning rate and optional L2 regularization toward the shipped weights (a lightweight EWC).
• k-NN over stored embeddings: store a handful of embeddings per class and classify by majority vote of nearest neighbors — trivially incremental (append), but memory grows with the number of stored exemplars unless capped.

Worked Example: Memory Budget for Prototype-Based Incremental Learning

Assume a person/gesture recognition model with a frozen backbone producing a 128-dim float embedding, and the device needs to support incremental learning of up to 10 new classes with 5 samples averaged into each prototype.

Storage for prototypes:

10 classes × 128 floats × 4 bytes = 5,120 bytes ≈ 5 KB

Storage for the running count per class (needed for the running mean):

10 classes × 4 bytes (uint32) = 40 bytes

Compute per update (one new labeled sample):

1 forward pass through frozen backbone (dominant cost, e.g., ~2M MACs for a small MobileNet-style embedding network)
+ update: 128 multiply-adds + 128 adds for the running mean ≈ 256 ops (negligible)

Check: total extra RAM ≈ 5.16 KB, well inside a 128–256 KB SRAM Cortex-M budget alongside model weights and activation buffers. The dominant cost is still the frozen-backbone inference (same as normal inference), so incremental learning adds essentially zero inference-time overhead and only a few hundred bytes of update logic — this is why prototype methods dominate in shipped products (e.g., on-device "add a new gesture" or "enroll a new keyword" features).

Compare this to naively fine-tuning the whole backbone: even a small 250k-parameter CNN needs gradient buffers and (for Adam) two additional momentum tensors per parameter — roughly 250k × 4 bytes × 3 ≈ 3 MB, which exceeds SRAM on most Cortex-M parts by 10–20×. This is precisely why full backprop-based continual learning is rare below Cortex-A/NPU tiers.

Evaluating Forgetting: Metrics That Matter

• Average accuracy (ACC): mean accuracy across all tasks/classes learned so far, after the last update.
• Backward transfer (BWT): change in accuracy on old tasks after learning a new one; negative BWT quantifies forgetting directly: BWT = (1/(T-1))·Σ (Aᵢ,T − Aᵢ,i) for tasks 1..T-1.
• Forward transfer (FWT): whether prior learning helps new-task accuracy versus training from scratch — relevant when the backbone itself was pretrained for transferability.
• Update latency and energy per sample: the embedded-specific metric absent from typical academic continual-learning papers, but the one that determines whether the feature is shippable on a coin-cell device at all.

Design Implications for Real Products

• Decide task/class boundaries at design time if possible. Fixed-slot prototype tables (e.g., "up to 10 enrollable gestures") are far simpler and more memory-predictable than open-ended class growth.
• Cap replay buffers explicitly and use reservoir sampling or class-balanced eviction — unbounded buffers are the most common cause of field memory exhaustion.
• Freeze normalization statistics (BatchNorm running mean/var) or fold them into INT8 scale/zero-point before deployment; updating BN stats during field learning reintroduces exactly the instability continual learning is trying to avoid.
• Gate updates by confidence/novelty detection — only trigger an incremental update on samples the model is uncertain about or that a human explicitly labels (enrollment flow), not on every inference, to control both drift and energy cost.
• Validate forgetting on a held-out "golden set" before accepting an on-device update as a checkpoint rollback trigger if BWT drops below a threshold.

Key Takeaways

• The core embedded constraint is catastrophic forgetting under extreme memory/compute limits, not accuracy in the abstract.
• Regularization (EWC/SI), replay, architecture-growth, and frozen-backbone/adaptive-head are the four main strategy families; the last is by far the most common in shipped MCU/NPU products.
• Prototype/nearest-class-mean updates give O(1)-per-sample, gradient-free incremental learning that costs only kilobytes of RAM and zero extra inference latency.
• Full backbone fine-tuning needs gradient and optimizer-state memory that typically exceeds Cortex-M SRAM by an order of magnitude — reserve it for Cortex-A/NPU-class devices.
• Backward transfer (BWT) is the metric to track explicitly; a shippable feature needs a defined forgetting budget, not just a good average accuracy number.