I went through the Edge0 repo after the clip of a 35B model on an iPhone, and I want to check my reading with people who actually run local models. No affiliation, just curious.

Edge0 (Edge0-AI, Apache 2.0, released the other day) is a streaming inference engine with two preview MoE models, Edge0-35B-A3B on a Qwen3.5-MoE base and Edge0-8B-A1B on Ling 3.0. The mechanism is expert offload: the 4-bit checkpoint stays on the SSD, expert weights are mmapped, and only the routed experts (4 of 256 per token, about 3B active of 35B) get read per token. The project reports 2.9 GB peak for the 35B and 1.0 GB for the 8B at short context on a Mac mini M4 Pro, with decode at 14.9-17.7 and 23.9-25.3 tok/s. The 35B checkpoint is 19.6 GB on disk.

My reading, which is the part I want checked: 3B active at 4 bits is about 1.5 GB, and at 15 to 18 tok/s a cold read of that per token would need over 20 GB/s, so most expert reads must be hitting the OS file cache. That makes "peak active memory" the process's resident set, with the cached pages outside it, and a 24 GB mini has room to hide a lot of the checkpoint there.

What shipped is MLX on Apple Silicon Macs only, the iPhone is the demo video, and I could not find the "iPhone 15 Pro and newer" floor on any primary page 🧐 Repo: https://github.com/Edge0-AI/edge0/

For anyone who has tried it on a Mac with less memory than that mini: does the 2.9 GB peak hold once the KV cache grows past a few thousand tokens, or does it start swapping?