在 1.9 GB 内存电视盒子上跑通 16B 参数模型
smolmaxxing 项目展示了在小米电视盒子（Amlogic S7D、约 1 GB 可用内存、无 root）上运行 260K 至 16B 参数语言模型的实测结果，通过自定义内核、内存优化与权重流式加载实现。其中 Qwen3.5-0.8B 达到 16.8 tok/s（为原版 2.2 倍），8B MoE 模型以专家按需分页方式达到 7.2 tok/s。所有数据均在真实硬件上背靠背测量，为边缘端本地推理提供了可复现的工程范式。
报道正文
本文目录2 个章节
🤏 smolmaxxing
Every device around you is more capable than it looks. TV boxes, old phones, single-board computers and routers all ship with CPUs, RAM and flash that sit idle most of the day. smolmaxxing is about unlocking that capability: finding the biggest language model each device can really run, then pushing it further with custom kernels, smarter memory use and weight streaming. Every claim is measured on the hardware itself.
Device #1 is a Xiaomi TV Box S (3rd Gen): Amlogic S7D, 4× Cortex-A55 @ 2.5 GHz, 1.9 GB of RAM (about 1 GB usable), eMMC flash, Android 14, no root. On it, this repo runs models from 260 K to 16 B parameters:
- Qwen3.5-0.8B: 16.8 tok/s (2.2× the stock 7.5; measured in the evening of 11-05, see §17).
- Qwen3.5-2B: 7.7 tok/s in real chat. The box's speed drifts by ~10 % over a day, so every step is measured back to back: §16 gave +11 %, §17 another +7 %.
- An 8B mixture-of-experts model (Moon-Beam-8B-A1B): 7.2 tok/s in real chat, its experts paged from flash on demand and prefetched one layer ahead (§19).
- A dense 1-bit 8B model (Bonsai-8B): 5.2 tok/s in real chat, up from 1.2: fully resident in a "headless" mode plus a lookup-table kernel (§19).
- A 16B mixture-of-experts model (Ling-mini-2.0): 5.1 tok/s in real chat, coherent over 600-word answers, the same expert pager applied unchanged (§20). Twice Moon-Beam's size, but not better per byte: bigger only helps on this box when the model is also better.
- A dense 9B model: 0.27 tok/s, streaming most of its weights from flash.
Device #2 is a first-generation iPad Air (2013): Apple A7, 2 cores, NEON without dotprod, 1 GB of RAM, iOS 12.5.7, jailbroken, CPU only. A 13-year-old tablet runs a 1.7B-parameter model at 24-26 tok/s (18-22 in long chats), a 4B one at about 10 tok/s, an 8B mixture of experts at about 3.5 tok/s (5 with half its experts, at a quality cost), and a 16B one at 2.6-4.5 tok/s, with their experts paged from flash, from a chat page in its own Safari. See Results on device #2.
Every step is written up experiment by experiment in EXPERIMENT.md (E1-E160).
Devices
| Device | Status |
|---|---|
| Xiaomi TV Box S (3rd Gen): Amlogic S7D, Cortex-A55, 2 GB | ✅ done: 260K → 16B parameters, 8B at 5-7 tok/s, 16B at 5 tok/s |
iPad Air (2013): Apple A7, 2 cores, 1 GB, iOS 12.5.7, jailbroken (ipad-air/) | ✅ running: Bonsai-1.7B (1-bit) at 24-26 tok/s, 18-22 in long chats; Bonsai-4B at about 10 tok/s; Moon-Beam-8B-A1B (MoE) at about 3.5 tok/s; Ling-mini-2.0 (16B MoE) at 2.6-4.5 tok/s; CPU only |
| More TV boxes, phones, single-board computers, routers… | 🔜 coming soon |
The techniques aren't tied to one box. The 1-bit lookup-table kernel needs only base NEON, as the iPad shows; any ARM chip with dotprod gets the other kernels; and any Linux or Android device with more model than RAM can use the streaming engine.
Scoreboard
| Class | Best model | First run → now (tok/s) | Gain | Bits/byte | What got it there | Download |
|---|---|---|---|---|---|---|
| 0.8B | Qwen3.5-0.8B | 7.5 → 16.8 bench | ×2.2 | ≈1.00 | all-Q4_K, padded 65K head, fp16 DeltaNet state | HF |
| 2B | Qwen3.5-2B | 0.1 → 7.7 chat | ×77 | 0.909 | all-Q4_K, padded 65K head, fused DeltaNet | HF |
| 4B | Ternary-Bonsai-4B | 2.2 → 4.3 chat | ×2.0 | ≈0.965 | ternary weights in 2-bit fields, merged scales | HF |
| 8B dense | Bonsai-8B (1-bit) | 1.21 → 5.2 chat | ×4.3 | 0.974 | 1-bit LUT kernel, fully resident in headless mode | HF |
| 8B MoE | Moon-Beam-8B-A1B | 1.87 → 7.2 chat | ×3.9 | 0.996 | Q2_K experts paged from flash, cache-aware routing, prefetch | HF |
| 9B | Qwen3.5-9B | 0.16 → 0.27 chat | ×1.7 | 0.885 | weight streaming plus its own MTP head | rebuild (q9) |
| 16B MoE | Ling-mini-2.0 | 4.1 → 5.1 chat | ×1.2 | 1.054 | the Moon-Beam recipe, unchanged | HF |
"First run" is the first time each model ran on the box:
- 0.8B: the stock Q4_K_M file.
- 2B: the stock file, streamed from flash because it didn't fit in RAM. Its first build that fit ran at 4.8, so from there the gain is ×1.6.
- 4B: the stock file, which thrashed.
- 8B dense: the last 0.1 GB streamed from flash.
- 8B MoE: the first expert-pager run, without routing bias.
- 9B: streamed.
- 16B: started on the Moon-Beam recipe, so its gain is small.
On this box, the 2B is the best speed-quality balance. Ling-mini-2.0 shows that bigger isn't automatically better: it is twice Moon-Beam's size, but scores worse per byte and runs slower.
What we built, grouped by the bottleneck it attacks (details and numbers in What's in here):
| Bottleneck | Innovations | Biggest wins |
|---|---|---|
| Compute (in-order A55, 10 items) | Q2_K and ternary repacks for vdotq_laneq, 1-bit bit planes, a 1-bit lookup-table kernel, ternary in 2-bit fields, Q8_0 attention with a skip threshold, fused Gated-DeltaNet, DeltaNet state read by id / fp16, the LM head on the fast kernel, guided matmul scheduling | Q2_K 7.8 → 13.9 G weights/s; Bonsai-8B 3.77 → 5.1-5.4; 4B 3.2 → 4.3; long answers no longer halve the speed |
| RAM (~1 GB, 5 items) | pruned LM head, lossless 1.875-bit ternary, headless mode, a fork-free memory guard, a repack cache on flash | −0.2 to −0.5 GB per model; free RAM ~1,000 → ~1,300 MB; Bonsai-8B 1.2 → 3.6 before any kernel work |
| Flash (278 MB/s, 5 items) | weight streaming with O_DIRECT, MTP speculation when flash-bound, expert pager, cache-aware routing, pre-gated prefetch | 9B +69 %; expert misses 13 % → ~2 %; Moon-Beam 4.4 → 7.2 |
| Output quality (2 items) | a long-answer degeneration test, a repeat penalty of 1.15 over 256 tokens | degenerate long answers 12/13 → 1/13 |
（注：更多技术实现细节与完整 API 文档请查阅原项目 README）