On-device / edge inference (NPUs, GGUF)
Phone/PC NPUs and llama.cpp/MLX local inference
On-device inference runs on smartphone and PC NPUs (Apple Neural Engine, Qualcomm Hexagon, Copilot+ PCs) via stacks like llama.cpp/GGUF, Apple MLX, ONNX Runtime and ExecuTorch. This edge tier offloads inference cost, enables privacy-preserving and offline AI, and is a distinct hardware/software axis from datacenter compute.
Silicon
Apple ANE, Qualcomm Hexagon
Stacks
llama.cpp/GGUF, MLX, ExecuTorch
How it fits the stack
On-device / edge inference (NPUs, GGUF) with what it depends on (above) and what it feeds (below). The figure renders as a crawlable diagram and upgrades to an interactive 3D graph as it scrolls into view.
On-device / edge inference (NPUs, GGUF) in the AI stack. On-device / edge inference (NPUs, GGUF) with its immediate upstream dependencies (top) and downstream dependents (bottom) in the AI value chain. Hover a node in 3D, or read the full relationships below.
Graph data (text) — 4 entities, 3 relationships
- On-device / edge inference (NPUs, GGUF) —depends on→ Arm Holdings
- On-device / edge inference (NPUs, GGUF) —uses→ Llama
- On-device / edge inference (NPUs, GGUF) —used by→ ONNX