ONNX Runtime for Embedded: Inference on Cortex-A

Cortex-A application processors running embedded Linux (i.MX8, RK3588, Jetson via Cortex-A cores, TI AM6x) are now common inference targets for vision and sensor-fusion workloads that need more than a Cortex-M can deliver but don't warrant a full server GPU. ONNX Runtime (ORT) is the most portable way to deploy a trained model onto that hardware: one intermediate representation, one runtime API, and a set of execution providers (EPs) that map ops onto CPU NEON kernels, vendor NPUs, or GPUs without touching the model source. Understanding how ORT actually schedules and executes a graph on Cortex-A — not just how to call Run() — is what separates a demo from a production inference pipeline meeting a latency budget.

Why ONNX Runtime on Cortex-A (vs. Cortex-M Frameworks)

Cortex-A cores run a full OS (Linux, sometimes with an MMU-backed RTOS), have hundreds of MB to several GB of RAM, and typically implement NEON SIMD and sometimes SVE. This changes the deployment calculus compared to TFLite Micro on a Cortex-M:

ORT's execution-provider abstraction is the key architectural feature: the same .onnx file can run purely on ARM NEON kernels during bring-up, then be repointed at a vendor NPU EP once silicon-specific acceleration is validated — without changing the model or the calling code.

ONNX Runtime Architecture on ARM

At a high level, ORT partitions the computation graph into subgraphs, assigns each to the best-available execution provider, and falls back to the default CPU EP for any op an accelerator EP doesn't support.

.onnx graph
     │
     ▼
Graph partitioner ──► EP capability query (per node)
     │
     ├─ Supported by NPU/GPU EP → subgraph compiled/offloaded
     └─ Not supported          → CPU EP (MLAS + NEON kernels)
     │
     ▼
Session::Run() executes partitioned subgraphs in topological order

Key EPs relevant to Cortex-A:

• Default CPU EP (MLAS): Microsoft's own math library, with NEON-optimized GEMM and convolution kernels for ARMv8-A. This is the fallback that guarantees correctness for any op.
• XNNPACK EP: optimized quantized/float kernels for mobile ARM CPUs, often faster than the default MLAS path for INT8 depthwise-heavy models (MobileNet-style).
• NNAPI EP (Android only): delegates supported ops to Android's Neural Networks API, which in turn may route to a vendor NPU/DSP driver.
• ACL EP (Arm Compute Library): targets Arm Mali GPUs and Cortex-A CPUs with hand-tuned NEON/OpenCL kernels; common on embedded Linux boards without NNAPI.
• Vendor EPs (e.g., TensorRT for Jetson's GPU, or a manufacturer's NPU EP): highest performance but tie the deployment to that SoC's SDK.

Partitioning granularity matters: if an accelerator EP only supports a subset of ops (e.g., no support for a particular reshape or a custom activation), ORT inserts a CPU subgraph around it, adding a data-copy/sync cost at each boundary. Profiling per-node EP assignment (via session.get_profiling() or the onnxruntime_perf_test tool) is the first debugging step when latency is worse than expected.

Quantization and Data Layout on Cortex-A

NEON's integer SIMD units process INT8 efficiently via SDOT/UDOT instructions (ARMv8.2-A dot-product extension) or SMLAL pairs on older cores. Practical implications:

• Post-training INT8 quantization (see the companion article on PTQ) typically gives 2–4× throughput over FP32 on Cortex-A cores with dot-product support, and roughly 1.5–2× without it (falling back to widening multiply-accumulate).
• Per-channel weight quantization is well supported by ORT's quantization tool (onnxruntime.quantization) and recovers most of the accuracy lost to per-tensor quantization, at negligible runtime cost since scales are baked in at conversion time, not computed at inference.
• NHWC vs. NCHW: ORT's CPU EP and most ARM kernels prefer NHWC (channel-last) for convolutions because it matches NEON's natural vectorization along the channel dimension. Models exported from PyTorch default to NCHW; running the ORT graph optimizer (GraphOptimizationLevel::ORT_ENABLE_ALL) inserts the necessary transposes, but for a performance-critical path it's better to export or convert the model to NHWC ahead of time and confirm no layout-conversion nodes remain in the optimized graph.

Worked Example: Latency Budget for a MobileNetV2 Classifier

Target: classify 224×224 RGB frames at 15 FPS (66.7 ms budget) on a quad-core Cortex-A53 @ 1.5 GHz, no NPU, using ORT's default CPU EP with INT8 quantization.

Step 1 — Compute estimate. MobileNetV2 at 224×224 requires ≈300 MFLOPs (≈150 MMACs) for one forward pass in FP32; INT8 execution is compute-bound on MACs, not FLOPs, so use ≈150 M MAC operations as the reference.

Step 2 — Peak throughput estimate. A Cortex-A53 core with NEON SDOT can retire roughly 16 INT8 MACs/cycle in well-vectorized GEMM-heavy kernels (this varies by microarchitecture and memory bandwidth; treat as an optimistic ceiling). At 1.5 GHz, one core:

16 MAC/cycle × 1.5×10⁹ cycle/s = 24×10⁹ MAC/s (24 GMAC/s) peak

Real convolution kernels achieve 20–40% of this peak due to memory bandwidth, tiling overhead, and non-GEMM layers (depthwise convs are notoriously bandwidth-bound, not compute-bound). Assume 30% efficiency → effective 7.2 GMAC/s per core.

Step 3 — Single-core latency.

t = 150×10⁶ MAC / 7.2×10⁹ MAC/s ≈ 20.8 ms

Step 4 — Multi-core scaling. ORT's CPU EP parallelizes across intra-op threads for large GEMMs. Assume 2.5× effective speedup on 4 cores (sub-linear due to depthwise layers that don't parallelize well and thread-sync overhead):

t₄-core ≈ 20.8 ms / 2.5 ≈ 8.3 ms

Step 5 — Add preprocessing and post-processing. Resize/normalize (~2 ms on NEON) + softmax/argmax (~0.1 ms) → total ≈ 10.4 ms.

Check against budget: 10.4 ms ≪ 66.7 ms budget → comfortable margin, confirming that even a modest quad-A53 cluster can hit 15 FPS on MobileNetV2-class models without an NPU, leaving headroom for a second concurrent model or higher input resolution. If the measured latency on real hardware comes in well above this (say >40 ms), the likely causes are: FP32 fallback on unsupported quantized ops, NHWC/NCHW transpose insertion, or single-threaded session configuration (SessionOptions.intra_op_num_threads left at 1).

Deployment and Integration Checklist

• Build a minimal ORT. The full ONNX Runtime shared library is tens of MB; for embedded Linux images, build with only the required EPs and disable training/ops-not-used to cut binary size and startup time.
• Pin thread count and affinity. Set intra_op_num_threads explicitly and consider CPU affinity (taskset) to keep inference threads off cores handling interrupts/DMA for camera or sensor I/O, avoiding jitter.
• Pre-load and mmap the model. Use ORT's Env with memory-mapped model loading so weight pages are shared across processes and paged in lazily rather than fully copied into heap at startup.
• Validate the optimized graph, not just the original. Always inspect the graph after ORT_ENABLE_ALL optimization (session dump or Netron) to confirm operator fusion (Conv+BN+ReLU) happened and no unexpected CPU-fallback subgraphs remain around the intended accelerator EP.
• Measure end-to-end, not just Run(). Camera capture, color conversion, and buffer copies often dominate wall-clock latency more than the model itself on Cortex-A vision pipelines.

Key Takeaways

• ONNX Runtime's execution-provider model lets the same .onnx file run on plain NEON CPU kernels, Arm Compute Library, NNAPI, or a vendor NPU without changing the model or application code — critical for portability across a Cortex-A SoC lineup.
• Cortex-A's ARMv8.2 dot-product NEON instructions make INT8 quantization the single biggest lever for throughput; per-channel quantization recovers most of the accuracy loss cheaply.
• Data layout (NHWC vs. NCHW) and operator fusion determine whether the graph optimizer eliminates costly transpose/copy nodes — always verify the optimized graph, not the exported one.
• A back-of-envelope MAC-count / peak-throughput / efficiency-factor calculation, checked against the actual FPS budget, catches most latency surprises before profiling on hardware.
• Real-world latency is usually dominated by thread configuration, EP-boundary fallbacks, and non-model I/O (capture, resize, color conversion) rather than the inference kernel itself.