Edge AI Overview: Inference at the Device vs. Cloud

Every camera module, wake-word chip, and predictive-maintenance sensor shipping today faces the same architectural fork: run the model on-device, ship raw data to the cloud, or split the work between them. That decision shapes power budget, BOM cost, latency, privacy exposure, and the entire software lifecycle — and it's usually locked in early, before anyone has profiled a real model on real silicon. Getting it wrong means a re-spin or a product that can't hit its power/latency spec.

The Three Architectures

Cloud inference: the device captures raw data (audio, image, sensor stream) and streams it over Wi-Fi/cellular/LoRa to a server, which runs the model and returns a result.

Edge (on-device) inference: the model runs entirely on the MCU/MPU/NPU in the device. No raw data leaves the board except the final decision (and optionally a compressed log).

Hybrid / split inference: a small model on-device does cheap filtering (e.g., voice activity detection, motion trigger) and only forwards data — or intermediate feature tensors — to the cloud when a more expensive model is needed.

Latency Budget: Where the Milliseconds Go

For cloud inference, total latency is roughly:

T_total = T_capture + T_encode + T_uplink + T_queue + T_inference + T_downlink

On a typical LTE link, T_uplink + T_downlink alone is 40–150 ms round trip even with no congestion, and jitter can push tail latency (p99) into the seconds. For a keyword-spotting wake word or a collision-avoidance trigger, that's disqualifying — the response must be deterministic and sub-50 ms.

For edge inference on a Cortex-M with a CMSIS-NN or Ethos-U kernel:

T_total ≈ T_capture + T_preprocess + T_inference

A quantized INT8 DS-CNN keyword spotter on a Cortex-M4 at 80 MHz typically runs in 10–20 ms per inference window; the same model on an Ethos-U55-equipped Cortex-M55 drops to under 2 ms because the NPU offloads the MAC-heavy convolution layers. There's no network term at all, and jitter is bounded by the RTOS scheduler, not a cellular stack.

Power: The Real Differentiator

Radio power dominates cloud-dependent designs. A rough comparison for a battery-powered sensor node reporting once per second:

• Cloud path: capture (≈1 mA-s) + Wi-Fi/cellular TX burst (100–250 mA for tens to hundreds of ms) ≈ several mA-s to tens of mA-s per report.
• Edge path: capture + local INT8 inference on a low-power NPU (e.g., sub-1 mA active draw at a few hundred µA/MHz-equivalent efficiency) ≈ hundreds of µA-s per inference, with the radio only waking for an occasional summary or alert.

This is why battery-powered always-on audio/vision products (wake-word mics, PIR+vision triggers) are edge-first almost by necessity: keeping a cellular or Wi-Fi radio active for streaming would exhaust a coin cell in hours instead of years.

Worked example — wake-word device power budget:

Assume a device samples audio continuously and must detect a wake word with <200 ms latency, running on a coin cell rated 220 mAh.

• Cloud approach: streaming 16 kHz/16-bit audio continuously requires ~256 kbps sustained uplink. A BLE/Wi-Fi radio sustaining that draws ~40–80 mA continuously. Battery life ≈ 220 mAh / 60 mA ≈ 3.7 hours. Unusable for a battery product.
• Edge approach: a DS-CNN KWS model runs inference every 20 ms window at ~2 mA average (MCU + occasional NPU burst), with the radio asleep except for a rare positive detection. Average current ≈ 2–3 mA. Battery life ≈ 220 mAh / 2.5 mA ≈ 88 hours continuous, extendable to weeks with duty-cycled sampling.

Check: 60 mA / 2.5 mA ≈ 24×, roughly one order of magnitude — consistent with radio TX being in the 10–50× range of the current draw of an INT8 MCU inference, exactly why continuous-audio products never stream raw PCM to the cloud.

Model Size and Compute Ceiling

Edge deployment is bounded by hard memory and compute limits that cloud inference never sees:

• Flash/RAM: a Cortex-M0+/M4 class MCU offers 64 KB–2 MB flash and 32–512 KB RAM. A quantized INT8 model (post-training quantization, FP32→INT8) must fit its weights, activation buffers, and scratch memory inside that envelope — typically capping models at a few hundred KB to low single-digit MB.
• Compute: without an NPU, a Cortex-M4 delivers roughly tens of MHz-equivalent MACs/s effective for CNN kernels; an Ethos-U55 NPU adds hundreds of GOP/s at similar power, which is what makes MobileNet-SSD-class object detection viable on-device at all.
• Cloud: effectively unbounded — full FP32 transformer or large CNN models, ensemble methods, no quantization required, constrained only by cost per inference.

This ceiling is why techniques like quantization-aware training, structured pruning, knowledge distillation, and operator fusion exist specifically for the edge path — they're the tools that shrink a cloud-scale model into an edge-compatible footprint without destroying accuracy.

Privacy and Regulatory Pressure

Sending raw audio or video off-device creates a data-protection liability (GDPR, biometric data laws in several jurisdictions) that on-device inference avoids by construction: if only a classification label or bounding box leaves the chip, there's no raw personal data in transit or at rest on a server. This is a primary driver for edge-first designs in smart cameras, hearables, and health-adjacent wearables, independent of the latency/power argument.

Connectivity and Reliability

Cloud inference has a hard dependency on network availability. For industrial predictive-maintenance sensors, agricultural equipment, or anything operating in a basement, tunnel, or remote field, intermittent or absent connectivity makes cloud-only inference a non-starter — the device must at minimum have a local fallback model, even if a richer cloud model is used opportunistically when connectivity exists (the hybrid pattern).

Choosing an Architecture: Decision Checklist

• Latency requirement <100 ms and deterministic → edge.
• Continuous sensing with battery constraint → edge, gated by a cheap always-on trigger.
• Raw data is privacy-sensitive (audio, video, biometric) → edge, transmit only derived results.
• Model needs frequent retraining on large, centrally-aggregated datasets, and connectivity is reliable → cloud, or hybrid with cloud-side retraining and periodic OTA model updates to the edge.
• Compute requirement exceeds MCU/NPU ceiling even after pruning/quantization (e.g., large-vocabulary ASR, high-res multi-object tracking) → cloud or hybrid, offloading only the expensive stage.
• Connectivity is intermittent or absent by design (remote/industrial) → edge with store-and-forward for non-critical logging.

Key Takeaways

• Edge inference wins on latency (single-digit to low tens of ms, no network term), power (avoids radio TX, the dominant power cost in streaming designs), privacy (raw data never leaves the device), and offline reliability.
• Cloud inference wins on model capacity (no quantization or size ceiling) and centralized retraining/update simplicity, at the cost of network-dependent latency and continuous connectivity/power draw.
• Hybrid designs — a cheap always-on edge trigger gating an expensive cloud or larger on-device model — capture most of the power/latency benefit of edge while retaining cloud-scale accuracy when needed.
• The edge compute ceiling (KB–MB models, MCU/NPU MAC throughput) is why quantization, pruning, and distillation exist; they're not optional extras, they're the enabling techniques for the architecture choice.
• The right answer is workload-specific: always profile actual latency, power, and memory footprint on target silicon before committing to an architecture — back-of-envelope radio vs. compute power gaps (often 10–50×) usually settle the decision early.