For most of the last decade, the default IoT architecture sent raw telemetry to the cloud, where analytics and machine learning turned it into decisions. That default is breaking. Cameras generate more data than networks can economically carry, industrial control loops cannot wait for a round trip to a distant region, and data-sovereignty rules make some organizations reluctant to ship sensitive streams off site. At the same time, compact models and capable edge hardware have made local inference practical.
The result is a shift that analysts and vendors increasingly call edge AI or physical AI: models that run on gateways, industrial PCs and devices, acting on the physical world close to where data is produced. This guide explains what is driving the shift in 2026, the architectural patterns that work, how to decide where each workload should run, and the operating model you need to keep models healthy across a fleet.
Why Edge AI Is Accelerating Now
Several trends have converged over 2025 and 2026:
- Smaller, task-specific models. Gartner predicted in April 2025 that by 2027 organizations will use small, task-specific AI models three times more than general-purpose large language models. Compact language and vision models in the sub-billion to single-digit-billion parameter range now run on embedded accelerators.
- Edge-native runtimes from cloud providers. Microsoft’s Azure IoT Operations 2603 release (April 2026) lets custom WebAssembly modules run ONNX models inside edge dataflow graphs and adds governed cloud-to-edge management actions. AWS released a Greengrass component SDK for C, C++ and Rust in April 2026 with a footprint under 0.5 MB, aimed explicitly at heavier AI and ML workloads on constrained devices.
- Computer vision moving to production. Vendors such as Dell describe edge computer vision moving from proof of concept to production in manufacturing, retail, healthcare and cities during 2026.
- Multimodal sensing. The 2026 Edge AI Technology Report published by Wevolver highlights systems combining vision, audio, radar, LiDAR and inertial data, which is only feasible when fusion happens locally.
None of this makes the cloud less important. Training, fleet-wide analytics, model governance and long-term storage still belong there. What changes is the placement of inference and first-line decisions.
Where Should Inference Run? A Four-Tier Model
A useful way to reason about placement is to think in four tiers, each with different constraints:
- Device or sensor (microcontroller class). Milliwatt power budgets, kilobytes to a few megabytes of memory. Suited to TinyML: anomaly detection on vibration, keyword spotting, simple classification. Decisions are local and immediate.
- Gateway or embedded computer. Watts of power, gigabytes of memory, sometimes an NPU or GPU. Suited to multi-sensor fusion, protocol translation plus inference, and compact vision or language models.
- On-premises edge cluster. Industrial PCs or small servers running Kubernetes. Suited to site-wide vision analytics, digital twin synchronization, local data historians and hosting multiple models per site.
- Cloud region. Effectively unlimited compute. Suited to training, retraining, cross-site analytics, heavy generative AI and anything without strict latency or locality needs.
Decision criteria
For each AI use case, score it against five questions. The more “yes” answers, the stronger the case for running inference closer to the device:
- Latency: Does the decision need to happen in less time than a reliable cloud round trip, for example stopping a conveyor or adjusting a valve?
- Bandwidth: Would sending raw data (video, high-frequency vibration) be expensive or impossible over the available link?
- Autonomy: Must the site keep operating safely when connectivity is lost?
- Data locality: Are there regulatory, contractual or privacy reasons to keep raw data on site, such as video of people?
- Model size and update frequency: Can the model fit the hardware, and is it stable enough that pushing updates weekly rather than hourly is acceptable?
Use cases that fail the last question, because they need large models or constant retraining, usually belong in the cloud with only lightweight pre-processing at the edge.
Reference Architecture Patterns
Most production systems combine a handful of recurring patterns. Naming them explicitly helps teams agree on responsibilities.
Pattern 1: Filter and forward
The edge runs simple models or rules to discard noise and forward only meaningful events or summaries. This is the cheapest pattern to adopt and often pays for itself in bandwidth and storage savings. It suits high-frequency sensor data where most readings are uninteresting.
Pattern 2: Infer and act locally
The edge runs a model and triggers a local action: rejecting a defective part, raising an alarm, adjusting a setpoint. The cloud receives the decision, the inputs that justified it (or a sample of them) and model metadata. Governance matters here; Azure IoT Operations’ cloud-to-edge management actions, for instance, wrap configuration changes in role-based access control and activity logs.
Pattern 3: Edge inference, cloud learning
Models run locally, but the edge selectively uploads hard cases (low-confidence predictions, operator overrides, novel inputs) to the cloud for labeling and retraining. New model versions are pushed back through the device management channel. This closed loop is what turns a pilot into a system that improves over time.
Pattern 4: Hierarchical fusion
Devices run tiny models, gateways fuse their outputs with other sensors, and a site cluster aggregates gateway outputs into a site-level view or digital twin. Each tier reduces data volume and increases context. Industrial and smart-building deployments often converge on this pattern.
Edge AI is not a choice between cloud and device. It is a placement decision made per use case, backed by a single operating model that can deploy, observe and roll back models wherever they run.
Building Blocks on AWS and Azure
Both major cloud providers now offer an edge runtime, a device management plane and a path to their data platforms. The names differ; the responsibilities are similar.
AWS
AWS IoT Greengrass deploys and manages components, including ML runtimes and models, on Linux devices, with a lighter “nucleus lite” for constrained hardware and the new native SDK for C, C++ and Rust. AWS IoT Core remains the connectivity and identity layer, and telemetry and inference results typically flow into Kinesis, S3 and analytics services. Note that AWS IoT Events reached end of support in May 2026, so cloud-side alarm logic should be built on streaming and serverless services rather than that service.
Azure
Azure IoT Operations runs on Arc-enabled Kubernetes at the edge, providing an MQTT broker, connectors (OPC UA, ONVIF, media, REST and MQTT), dataflow graphs with WebAssembly-based transforms that can host ONNX models, and unified health reporting. Azure IoT Hub and Device Provisioning Service handle device connectivity, while Microsoft Fabric provides real-time analytics in the cloud. With Azure IoT Central scheduled to retire in September 2029, new edge AI programs should target these services directly.
Portable choices that reduce lock-in
- Package models in portable formats such as ONNX where the target hardware supports it.
- Use MQTT and OPC UA as the edge messaging and industrial data contracts.
- Containerize inference services where hardware allows, so the same artifact runs in a lab, on a gateway and in a site cluster.
- Keep model metadata (version, training data snapshot, evaluation metrics) in a registry independent of any single runtime.
The Operating Model: Edge MLOps
Shipping one model to one device is a demo. Keeping dozens of model versions healthy across thousands of heterogeneous devices is an operations discipline. The Wevolver report identifies edge MLOps and orchestration as a core deployment challenge, and that matches what teams experience in practice.
- Model registry and lineage. Every deployed model has a version, a link to its training data and code, evaluation results and an approval record.
- Staged rollout. Deploy to a canary group of devices, compare metrics against the previous version, then expand in rings. Always keep the prior version on device for fast rollback.
- Hardware-aware testing. Validate latency, memory and thermal behavior on each hardware SKU you support, not just accuracy on a GPU workstation.
- Runtime telemetry. Collect inference latency, confidence distributions, input statistics and error rates, so you can detect drift and failures remotely.
- Drift and feedback loops. Define thresholds that trigger data capture and retraining, and route operator overrides back as labels.
- Security and update hygiene. Sign model artifacts and containers, enforce secure boot where possible, and treat models as part of the product’s software bill of materials.
Regulation is now part of the design
In the EU, the Cyber Resilience Act requires manufacturers of products with digital elements to report actively exploited vulnerabilities and severe incidents from September 11, 2026, with broader security obligations from December 11, 2027. Where edge AI makes decisions affecting people, such as safety components or workplace monitoring, the EU AI Act may also apply. Explainability, logging and update mechanisms are no longer nice-to-haves.
A Worked Example: Visual Inspection Across Several Plants
Consider a generic manufacturer with several plants that wants to detect surface defects on a packaging line. Sending full-resolution video to the cloud is impractical, and rejects must happen within the time a product takes to pass the ejector.
The team places a compact vision model on an industrial PC at each line, fed by two cameras. The model runs in a container managed by the edge runtime and triggers the ejector directly (Pattern 2). Every prediction below a confidence threshold, plus a random sample of confident ones, is uploaded with metadata (Pattern 3). Quality engineers label those images weekly in the cloud, a retraining pipeline produces a candidate model, and it is evaluated against a fixed test set plus the previous week’s hard cases.
Approved models roll out first to one line in one plant, where the team compares reject rates and operator overrides with the previous version for several shifts. Only then does the model move to other lines and plants. Line-level metrics (throughput, inference latency, confidence distribution) flow into the central data platform, where operations and quality teams share one dashboard. When a new packaging material causes a spike in low-confidence predictions at one plant, the drift monitor flags it within a day and the hard cases become next week’s training data.
Selecting Edge Hardware and Estimating Cost
Hardware decisions lock in constraints for years, so they deserve the same rigor as model selection. Before procurement commits to a gateway or industrial PC, work through a short checklist with engineering, operations and security in the room:
- Accelerator and toolchain support: Confirm that your model formats and operators are supported by the NPU or GPU vendor’s compiler, and that the toolchain is maintained for the product’s expected life.
- Headroom: Size memory and compute for the next model generation, not the current one; a model that uses 90 percent of RAM today leaves no room for the update that fixes drift.
- Environmental envelope: Check operating temperature, dust and vibration ratings, and fanless options for plant-floor or outdoor locations.
- Security features: Prefer hardware with a secure element or TPM, secure boot and signed updates, which simplify compliance with product security obligations.
- Lifecycle and supply: Ask for long-term availability commitments and a documented end-of-life process, so replacements remain compatible.
For cost, compare total cost of ownership rather than unit price. Include hardware, installation, connectivity, edge software licenses, the engineering effort to optimize models per hardware SKU and the ongoing operations work of monitoring and updating the fleet. Then set that against avoided costs: bandwidth and cloud storage for raw data you no longer ship, downtime or scrap that local decisions prevent, and cloud inference spend that moves to the edge. In many programs the operations line is the one that is most underestimated, which is another argument for standardizing on a small number of hardware profiles.
Common Pitfalls
- Treating the pilot hardware as the fleet. A model tuned on one gateway may not fit the cheaper SKU procurement chooses later. Fix the hardware envelope early.
- No rollback path. Pushing a model without keeping the previous version on device turns a bad release into an outage.
- Ignoring the network. Large model updates over constrained cellular or LPWAN links need delta updates and scheduled windows.
- Blind spots in monitoring. If you only log decisions, you cannot diagnose why accuracy dropped. Capture input statistics and confidence.
- Skipping the business metric. Accuracy is not the goal; fewer escapes, less downtime or lower energy use is. Instrument the outcome from day one.
Outlook: Agents and Physical AI at the Edge
The next step is already visible. Industry predictions for 2026 point to agentic AI making local, closed-loop decisions with humans as guardrails, and to physical AI in autonomous equipment for mining, construction and agriculture. Standard bodies and alliances are also working on batteryless ambient IoT devices that will add vast numbers of simple sensors to the edge. All of this increases the importance of governed actions, robust fleet management and clear accountability for automated decisions.
Conclusion
Edge AI is becoming a normal part of IoT platform design. The winning approach is pragmatic: score each use case for latency, bandwidth, autonomy, locality and model fit; choose from a small set of proven patterns; build on the edge runtimes your cloud provider now supports; and invest early in edge MLOps so models can be deployed, observed and rolled back safely. Organizations that do this turn edge AI from a series of pilots into a durable capability.
How Hanoi Technology Can Help
Hanoi Technology designs and builds edge AI architectures on AWS and Azure IoT platforms, from model packaging and fleet deployment to monitoring and managed operations.
Further Reading
- Azure IoT Operations 2603 is now available – Microsoft Tech Community
- AWS IoT Greengrass component SDK for C, C++, and Rust applications – AWS What’s New
- The Power of Small: Edge AI Predictions for 2026 – Dell Technologies
- The 2026 Edge AI Technology Report – Wevolver
- Cyber Resilience Act reporting obligations – European Commission