The numbers
On a Cortex-M85 running at 480 MHz with 512 KB SRAM and 8 MB PSRAM, the INT4 runtime achieves 11 tokens/second on a 1.3B-parameter GGUF model. Peak memory footprint is 312 MB flash + 4.2 MB SRAM for the KV cache. This is the first time interactive-speed LLM inference has been demonstrated on a microcontroller without an NPU.
Block-streamed inference
The key insight is block-streamed weight loading: instead of mapping the entire model into PSRAM, the runtime streams 4 KB weight blocks from flash into SRAM just-in-time for each attention layer. This reduces peak SRAM usage by 94% compared to a naive full-model load, at the cost of 12% throughput reduction from flash read latency.
Quantization quality
INT4 quantization uses per-channel symmetric quantization with outlier clamping. On the MMLU benchmark, the INT4 model scores 58.2% vs 61.4% for the FP16 reference — a 3.2 percentage point drop. For on-device use cases (intent classification, command parsing, sensor data summarization), this quality level is acceptable.
What this enables
With 11 tok/s on a $4 MCU, natural language device control (eBot) becomes viable without cloud connectivity. A device can parse free-form commands, generate sensor summaries, and respond to queries entirely on-chip. This is the foundation for the eAI Edge stack's offline-first design.