开源量化编译器将 Qwen3.5-4B 压至 1.2 GB
开源项目 LookAtMyQuant 提出一种基于 Fisher 信息与 trellis 编码的后训练量化编译器,按存储单元对输出损伤分配比特预算。其第二版将 Qwen3.5-4B 压缩至 1.23 GB(约 2.3 bit/权重),在 540 道精确答案数学题上解出 330 道,优于同尺寸的 Bonsai 2 Ternary 4B(328 道),KL 散度 0.45 nats 亦低于对方的 0.59。完整配方与约 34 美元 GPU 成本已公开,代码采用 MIT 许可。
报道正文
本文目录4 个章节
LookAtMyQuant
LookAtMyQuant is a quantization compiler. It measures how much each stored piece of a model hurts the output when compressed. It then picks a code for each piece so the whole model fits a size budget with the least damage.
On Gemma 4 E2B, its GGUF has 12× lower KL than Q4_K_M at the same file size. The method and every result are on the site.
The code is under the MIT license.
LookAtMyQuant 2: Qwen3.5-4B in 1.2 GB
LookAtMyQuant 2 takes Qwen3.5-4B to 1.23 GB, 2.3 bits per weight. The file solves 330 of 540 exact-answer math problems. Prism ML's Bonsai 2 Ternary 4B solves 328 at 1.20 GB, and the original solves 359. Its KL from the original on held-out text is 0.45 nats, against 0.59 for Bonsai.
The recipe is post-training quantization with trellis codes, calibrated on the model's own worked math solutions with digit tokens weighted 10×, and with bits allocated per row by Fisher information. 4,000 steps of distillation follow, in which every layer is re-quantized once from the original weights. The page has the findings, the full recipe and its cost, about $34 of GPU time.
The recipe's code is here, and the next section runs it. The 1.23 GB file itself is the lm2-qwen35-4b-1.2gb release, packed for the Apple Silicon runtime.
Qwen3.8-27B in 7.2 GB
Beside it sits a 4-bit DSpine drafter (1.02 GB, converted from tencent/Qwen3.8-27B-DSpine-b16 with runtime_mlx/convert_dspine.py and packed with pack_drafter.py), which roughly doubles decode speed on math and code on Apple silicon.
Inkling-2-Mini in 5.4 GB and 8.9 GB
The same compiler and proximal fine-tuning (SQYFR), extended to a 26.4B physical / 5.4B active 3×-looped MoE (L6–17 looped 3× = 48 logical visits, 128 experts top-8), compresses thinkingmachines/Inkling-2-Mini from 18.89 GB of text weights to 5.38 GB (1.63 bits/weight) and 8.87 GB (2.69 bits/weight) using 4 KB SRAM hashed block-vector trellis codes (ht2/ht3), row trellis codes (rtr1/rtr4), and a tr2 floor on the tail lm_head bucket (head.7).
Evaluated on the same BF16 teacher, openai/gsm8k (test[:200], greedy, reasoning_effort="none"), all 164 executable openai/openai_humaneval problems, 4 Vision VQA tests (with the untouched 637 MB BF16 vision tower), and 16+13-language TextMix windows:
| Model | Text GB | free_KL (en/chat/code) | all13_KL | GSM8K (200) | HumanEval (164) | Pooled vs Teacher |
|---|---|---|---|---|---|---|
| Original BF16 Teacher | 18.89 | 0.0000 | 0.0000 | 147 / 200 (73.5%) | 66 / 164 (40.2%) | 100.0% |
LookAtMyQuant 5.4 GB (SQYFR) | 5.38 | 0.0895 | 0.2871 | 134 / 200 (67.0%) | 60 / 164 (36.6%) | 91.2% |
Unsloth UD-IQ1_S | 7.76 | 0.0808 | 0.1236 | 127 / 200 (63.5%) | 46 / 164 (28.0%) | 81.6% |
LookAtMyQuant 8.9 GB (SQYFR) | 8.87 | 0.0208 | 0.0571 | 148 / 200 (74.0%) | 58 / 164 (35.4%) | 96.8% |
Unsloth UD-Q2_K_XL | 9.14 | 0.0486 | 0.0811 | 132 / 200 (66.0%) | 49 / 164 (29.9%) | 85.3% |
Unsloth UD-IQ3_XXS | 11.66 | 0.0189 | 0.0278 | 148 / 200 (74.0%) | 68 / 164 (41.5%) | 101.4% |
Install and use the Apple Silicon runtime
On Apple silicon, install the runtime package directly (this installs mlx, mlx-lm, numpy, and huggingface-hub without the PyTorch compiler dependencies):
pip install git+https://github.com/ShinMegamiBoson/lookatmyquant.gitFrom a local clone, pip install -e . installs the lamq command, or run python -m runtime_mlx without installing.
Command line. Pass either a local model folder or a Hugging Face repo ID:
lamq generate shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \
-p "What is 17 times 23? Show the steps."
lamq generate shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \
--drafter shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter \
-p "Write a Python function that merges overlapping intervals." --stats
lamq chat shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \
--drafter shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter
lamq generate shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-5.4GB \
--drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter \
-p "What is 17 times 23? Show the steps." --statslamq generate reads stdin when -p is omitted, supports --raw, --system, --thinking, --temp, --mode linear|chat-fork|copy-wide, --no-adaptive, and --json. lamq info <model> inspects a packed model's format and dimensions without loading weights.
Python integration. runtime_mlx.load returns an (model, tokenizer) pair compatible with mlx_lm.generate and mlx_lm.stream_generate (auto-detecting opgraph-packed-1 models such as Qwen3.5-4B / Qwen3.8-27B and lamq.json exports such as Gemma 4 E2B):
from mlx_lm import generate
from runtime_mlx import load
model, tok = load("shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB")
prompt = tok.apply_chat_template(
[{"role": "user", "content": "What is 17 times 23? Show the steps."}],
add_generation_prompt=True,
enable_thinking=False,
)
print(generate(model, tok, prompt, max_tokens=200))For streaming, multi-turn chat, or speculative decoding through one interface, use Runtime:
from runtime_mlx import Runtime
rt = Runtime.load(
"shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB",
drafter="shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter",
)
for chunk in rt.stream("Solve 3x + 4y = 31 and 5x - 2y = 17.", max_tokens=256):
print(chunk.text, end="", flush=True)(注:更多技术实现细节与完整 API 文档请查阅原项目 README)