llama-modchip 让 284B DeepSeek 在 64GB 内存纯 CPU 运行Hacker News AIT2·开源与评测

llama-modchip 让 284B DeepSeek 在 64GB 内存纯 CPU 运行

原标题：Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU

74AI 研判分

开源项目 llama-modchip v2.4.0 宣称可在 64GB 内存的纯 CPU 工作站上运行 284B 参数的 DeepSeek-V4-Flash 混合专家模型，峰值内存占用 57.61 GiB，256K 上下文下解码约 9–15 tok/s。项目通过 SwiGLU-H2 等量化与内存优化压缩权重，并提供 Fractal-64/MXFP4 转码与状态检查工具链。

报道正文

本文目录4 个章节

♟️🐋 llama-modchip (v2.4.0) ### Break the 64 GB RAM Wall: Run 284B DeepSeek-V4-Flash ( 284.33B Frontier MoE params 56.56 GiB checkpoint on disk 57.61 GiB peak VmRSS @ 256K ctx 44.36 tok/s 284B CPU prefill 60.4–111.6 tok/s 30B MoE CPU decode 13.1 ms Claude Code restore 0 GPUs required Real-time Frontier & Agentic MoE inference on the CPU workstation you already own — 100% in RAM, zero NVMe swap:

<table> <tr> <th align="left">the machine you have</th> <th align="right">RAM</th> <th align="left">recommended model</th> <th align="right">speed (0 GPUs)</th> <th align="left">what is going on</th> </tr> <tr> <td align="left">a developer laptop / mini-PC</td> <td align="right">16–32 GB</td> <td align="left">Nemotron-3.5-30B-A3B (<code>14.35 GiB</code>)</td> <td align="right">25.0–60.4 tok/s</td> <td>46 Mamba-2 SSM layers + 1.64 MB L3 Stockfish-PVS drafter; 256K ctx in 6 GiB KV</td> </tr> <tr> <td align="left">a 64 GB desktop / server</td> <td align="right">64 GB</td> <td align="left">DeepSeek-V4-Flash 284B (<code>56.56 GiB</code>)</td> <td align="right">44.4 tok/s pp · 9.1–14.7 tok/s tg</td> <td>SwiGLU-H2 shaves 20.31 GiB; entire 284B model + 256K ctx sits in RAM (57.61 GiB VmRSS)</td> </tr> <tr> <td align="left">a hybrid desktop (CPU + 16 GB GPU)</td> <td align="right">32–64 GB</td> <td align="left">Nemotron-30B / Qwen3-30B</td> <td align="right">80.80 tok/s</td> <td>hot attention trunk on GPU (<code>-ngl 99</code>), cold routed MoE experts in CPU DDR5</td> </tr> <tr> <td align="left">an 8-channel workstation (Threadripper)</td> <td align="right">128 GB+</td> <td align="left">DeepSeek-284B / Nemotron-30B</td> <td align="right">111.62 tok/s (30B)</td> <td>8-CCD AVX-512BW <code>_1x8</code> register tiling saturates 8-channel DDR5 memory bandwidth</td> </tr> </table> </div>

TL;DR — The 3 Paradigm-Breaking Pillars of llama-modchip:

🐋 284.33B Frontier MoE in a 64 GB CPU Box: Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB on disk / 57.61–58.9 GiB VmRSS at full 256K context (44.36 tok/s prefill, 9.06–10.74 tok/s pure CPU decode, 11.5–14.7 tok/s with PVS) using closed-form Analytical SwiGLU-H2 Second-Order Hessian Compression + Fractal-64 (Q2_0, 3.75 bpw) MLA transcoding + 4-Level Register (ZMM AVX-512BW) + L1D (32 KB) + L2 (1 MB) + L3 (64 MB) Whole-Model Tiling + $L_2$ -Quadrature MoE Pruning.

⚡ 30B MoE at GPU Speeds on Pure CPU: Runs NVIDIA-Nemotron-3.5-30B-A3B in 14.35 GiB at 25.0–60.4 tok/s on 16-core Zen 5 (111.6 tok/s on Threadripper, 80.8 tok/s on AMD Vulkan Hybrid, 44.97 tok/s on AWS Graviton3 ARM64) via a 1.64 MB Stockfish-PVS v2 + RouteCrack L2/L3 Drafter (96.4% acceptance) and _MM_HINT_NTA L3 cache isolation (HARPCache).

🤖 Zero-API Claude Code (./modchip claude): Eliminates the 45-second CPU agent prompt re-evaluation bottleneck via /dev/shm Slot-0 KV Warm Restore (18.9 ms on Nemotron-30B / 13.1–32.6 ms on DeepSeek-284B) with all 40 Claude Code tools active.

⚡ Quickstart

🐧 Linux & 🍎 macOS (1-Command Automated Setup & Chat)

BASH

git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chat

⚡ Zero-Compile Direct Run (Pre-Compiled Linux `x86_64` Binary + Hugging Face `.gguf`)

⚠️ Important Compatibility Note (GGML_TYPE_Q2_0 = 40): Our Fractal-64 models use custom block-64 BF16-scale Gray-coded tensors (GGML_TYPE_Q2_0, type ID 40) and custom deepseek4 / Stockfish-PVS kernels. Opening them in unpatched llama.cpp, Ollama, or LM Studio will fail with unknown tensor type 40. Use either ./modchip above or our standalone single-file pre-compiled binary (llama-server-modchip-linux-x86_64, zero .so dependencies) bundled directly inside the Hugging Face repositories and GitHub Release v2.4.0:

BASH

# 1. For 30B Nemotron-3.5-A3B (14.35 GiB — runs in 16–32 GB RAM at 25–60.4 tok/s):
hf download Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64-CPU-GGUF \
  llama-server-modchip-linux-x86_64 NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf --local-dir .
chmod +x llama-server-modchip-linux-x86_64
./llama-server-modchip-linux-x86_64 -m NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf -c 262144 -t 16 --port 8080

# 2. For 284B DeepSeek-V4-Flash (56.56 GiB — fits in 64 GB RAM at 9.1–14.7 tok/s):
hf download Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF --local-dir ./dsv4
chmod +x ./dsv4/llama-server-modchip-linux-x86_64
./dsv4/llama-server-modchip-linux-x86_64 -m ./dsv4/DeepSeek-V4-Flash-0731-Fractal-00001-of-00003.gguf -c 32768 -t 16 -fa on --port 8080

🪟 Windows 10/11 (1-Click Hybrid GPU + CPU MoE Pack)

Open windows/ and double-click windows/AVVIA_MODCHIP.bat (or run ./modchip start in Git Bash). It automatically detects your CPU cores, free RAM, and GPU VRAM (nvidia-smi), downloads official llama.cpp CUDA 12.4 / Vulkan binaries, computes the optimal --n-cpu-moe split, and opens the Web Chat UI (http://127.0.0.1:8080).

`./modchip` Command	What It Does
`./modchip chat`	💬 Interactive terminal chat with live `tok/s`, TTFT, and `Stockfish-PVS` acceptance telemetry
`./modchip web`	🌐 Opens the Dark-Mode Web Chat & Live Telemetry HUD (`http://127.0.0.1:8787`)
`./modchip bench`	📊 Runs a 30-second 2-stage speed test and prints a shareable CPU Benchmark Scorecard
`./modchip claude`	🤖 Launches Claude Code CLI (`40 tools`) wired to the local 256K engine (`18.9–32.6 ms` warm restore)
`./modchip eval`	🧪 Runs the reproducible 100-Point Quality, Tool-Calling, Coding & 8.5K Needle Gate
`./modchip quantize`	🗜️ Transcodes `BF16`/`Q4_0`/`IQ2_XXS` GGUFs into `Fractal-64` (`3.75 bpw`) or `Fractal-MXFP4` (`4.25 bpw`)
`./modchip status` / `stop`	🔍 Inspects CPU readiness, `2MB HugePages`, and `/dev/shm` IPC, or stops background services

（注：更多技术实现细节与完整 API 文档请查阅原项目 README）