llama-modchip Runs 284B DeepSeek on 64GB RAM With No GPU

♟️🐋 llama-modchip (v2.4.0) ### Break the 64 GB RAM Wall: Run 284B DeepSeek-V4-Flash ( 284.33B Frontier MoE params 56.56 GiB checkpoint on disk 57.61 GiB peak VmRSS @ 256K ctx 44.36 tok/s 284B CPU prefill 60.4–111.6 tok/s 30B MoE CPU decode 13.1 ms Claude Code restore 0 GPUs required Real-time Frontier & Agentic MoE inference on the CPU workstation you already own — 100% in RAM, zero NVMe swap:

TL;DR — The 3 Paradigm-Breaking Pillars of llama-modchip:

• 🐋 284.33B Frontier MoE in a 64 GB CPU Box: Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB on disk / 57.61–58.9 GiB VmRSS at full 256K context (44.36 tok/s prefill, 9.06–10.74 tok/s pure CPU decode, 11.5–14.7 tok/s with PVS) using closed-form Analytical SwiGLU-H2 Second-Order Hessian Compression + Fractal-64 (Q2_0, 3.75 bpw) MLA transcoding + 4-Level Register (ZMM AVX-512BW) + L1D (32 KB) + L2 (1 MB) + L3 (64 MB) Whole-Model Tiling + $L_2$-Quadrature MoE Pruning.
• ⚡ 30B MoE at GPU Speeds on Pure CPU: Runs NVIDIA-Nemotron-3.5-30B-A3B in 14.35 GiB at 25.0–60.4 tok/s on 16-core Zen 5 (111.6 tok/s on Threadripper, 80.8 tok/s on AMD Vulkan Hybrid, 44.97 tok/s on AWS Graviton3 ARM64) via a 1.64 MB Stockfish-PVS v2 + RouteCrack L2/L3 Drafter (96.4% acceptance) and _MM_HINT_NTA L3 cache isolation (HARPCache).
• 🤖 Zero-API Claude Code (./modchip claude): Eliminates the 45-second CPU agent prompt re-evaluation bottleneck via /dev/shm Slot-0 KV Warm Restore (18.9 ms on Nemotron-30B / 13.1–32.6 ms on DeepSeek-284B) with all 40 Claude Code tools active.

⚡ Quickstart

🐧 Linux & 🍎 macOS (1-Command Automated Setup & Chat)

git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chat

⚡ Zero-Compile Direct Run (Pre-Compiled Linux x86_64 Binary + Hugging Face .gguf)

⚠️ Important Compatibility Note (GGML_TYPE_Q2_0 = 40): Our Fractal-64 models use custom block-64 BF16-scale Gray-coded tensors (GGML_TYPE_Q2_0, type ID 40) and custom deepseek4 / Stockfish-PVS kernels. Opening them in unpatched llama.cpp, Ollama, or LM Studio will fail with unknown tensor type 40. Use either ./modchip above or our standalone single-file pre-compiled binary (llama-server-modchip-linux-x86_64, zero .so dependencies) bundled directly inside the Hugging Face repositories and GitHub Release v2.4.0:

# 1. For 30B Nemotron-3.5-A3B (14.35 GiB — runs in 16–32 GB RAM at 25–60.4 tok/s):
hf download Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64-CPU-GGUF \
  llama-server-modchip-linux-x86_64 NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf --local-dir .
chmod +x llama-server-modchip-linux-x86_64
./llama-server-modchip-linux-x86_64 -m NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf -c 262144 -t 16 --port 8080

# 2. For 284B DeepSeek-V4-Flash (56.56 GiB — fits in 64 GB RAM at 9.1–14.7 tok/s):
hf download Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF --local-dir ./dsv4
chmod +x ./dsv4/llama-server-modchip-linux-x86_64
./dsv4/llama-server-modchip-linux-x86_64 -m ./dsv4/DeepSeek-V4-Flash-0731-Fractal-00001-of-00003.gguf -c 32768 -t 16 -fa on --port 8080

🪟 Windows 10/11 (1-Click Hybrid GPU + CPU MoE Pack)

Open windows/ and double-click windows/AVVIA_MODCHIP.bat (or run ./modchip start in Git Bash). It automatically detects your CPU cores, free RAM, and GPU VRAM (nvidia-smi), downloads official llama.cpp CUDA 12.4 / Vulkan binaries, computes the optimal --n-cpu-moe split, and opens the Web Chat UI (http://127.0.0.1:8080).

（注：更多技术实现细节与完整 API 文档请查阅原项目 README）