Open Source Quantization Compiler Shrinks Qwen3.5-4B to 1.2 GB

Original title: Inkling-2-Mini in 5.4 GB Memory
71AI Score

The open source LookAtMyQuant project introduces a post-training quantization compiler that allocates bit budgets per stored unit based on measured output damage, using Fisher information and trellis codes. Its second version compresses Qwen3.5-4B to 1.23 GB (~2.3 bits per weight), solving 330 of 540 exact-answer math problems versus 328 for the same-size Bonsai 2 Ternary 4B, with lower KL divergence (0.45 vs 0.59 nats). The full recipe and roughly $34 of GPU cost are published under the MIT license.

Full Report

Contents4 sections

LookAtMyQuant

LookAtMyQuant is a quantization compiler. It measures how much each stored piece of a model hurts the output when compressed. It then picks a code for each piece so the whole model fits a size budget with the least damage.

On Gemma 4 E2B, its GGUF has 12× lower KL than Q4_K_M at the same file size. The method and every result are on the site.

The code is under the MIT license.

LookAtMyQuant 2: Qwen3.5-4B in 1.2 GB

LookAtMyQuant 2 takes Qwen3.5-4B to 1.23 GB, 2.3 bits per weight. The file solves 330 of 540 exact-answer math problems. Prism ML's Bonsai 2 Ternary 4B solves 328 at 1.20 GB, and the original solves 359. Its KL from the original on held-out text is 0.45 nats, against 0.59 for Bonsai.

The recipe is post-training quantization with trellis codes, calibrated on the model's own worked math solutions with digit tokens weighted 10×, and with bits allocated per row by Fisher information. 4,000 steps of distillation follow, in which every layer is re-quantized once from the original weights. The page has the findings, the full recipe and its cost, about $34 of GPU time.

The recipe's code is here, and the next section runs it. The 1.23 GB file itself is the lm2-qwen35-4b-1.2gb release, packed for the Apple Silicon runtime.

Qwen3.8-27B in 7.2 GB

Beside it sits a 4-bit DSpine drafter (1.02 GB, converted from tencent/Qwen3.8-27B-DSpine-b16 with runtime_mlx/convert_dspine.py and packed with pack_drafter.py), which roughly doubles decode speed on math and code on Apple silicon.

Inkling-2-Mini in 5.4 GB and 8.9 GB

The same compiler and proximal fine-tuning (SQYFR), extended to a 26.4B physical / 5.4B active 3×-looped MoE (L6–17 looped 3× = 48 logical visits, 128 experts top-8), compresses thinkingmachines/Inkling-2-Mini from 18.89 GB of text weights to 5.38 GB (1.63 bits/weight) and 8.87 GB (2.69 bits/weight) using 4 KB SRAM hashed block-vector trellis codes (ht2/ht3), row trellis codes (rtr1/rtr4), and a tr2 floor on the tail lm_head bucket (head.7).

Evaluated on the same BF16 teacher, openai/gsm8k (test[:200], greedy, reasoning_effort="none"), all 164 executable openai/openai_humaneval problems, 4 Vision VQA tests (with the untouched 637 MB BF16 vision tower), and 16+13-language TextMix windows:

ModelText GBfree_KL (en/chat/code)all13_KLGSM8K (200)HumanEval (164)Pooled vs Teacher
Original BF16 Teacher18.890.00000.0000147 / 200 (73.5%)66 / 164 (40.2%)100.0%
LookAtMyQuant 5.4 GB (SQYFR)5.380.08950.2871134 / 200 (67.0%)60 / 164 (36.6%)91.2%
Unsloth UD-IQ1_S7.760.08080.1236127 / 200 (63.5%)46 / 164 (28.0%)81.6%
LookAtMyQuant 8.9 GB (SQYFR)8.870.02080.0571148 / 200 (74.0%)58 / 164 (35.4%)96.8%
Unsloth UD-Q2_K_XL9.140.04860.0811132 / 200 (66.0%)49 / 164 (29.9%)85.3%
Unsloth UD-IQ3_XXS11.660.01890.0278148 / 200 (74.0%)68 / 164 (41.5%)101.4%

Install and use the Apple Silicon runtime

On Apple silicon, install the runtime package directly (this installs mlx, mlx-lm, numpy, and huggingface-hub without the PyTorch compiler dependencies):

BASH
pip install git+https://github.com/ShinMegamiBoson/lookatmyquant.git

From a local clone, pip install -e . installs the lamq command, or run python -m runtime_mlx without installing.

Command line. Pass either a local model folder or a Hugging Face repo ID:

BASH
lamq generate shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \
    -p "What is 17 times 23? Show the steps."

lamq generate shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \
    --drafter shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter \
    -p "Write a Python function that merges overlapping intervals." --stats

lamq chat shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \
    --drafter shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter

lamq generate shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-5.4GB \
    --drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter \
    -p "What is 17 times 23? Show the steps." --stats

lamq generate reads stdin when -p is omitted, supports --raw, --system, --thinking, --temp, --mode linear|chat-fork|copy-wide, --no-adaptive, and --json. lamq info <model> inspects a packed model's format and dimensions without loading weights.

Python integration. runtime_mlx.load returns an (model, tokenizer) pair compatible with mlx_lm.generate and mlx_lm.stream_generate (auto-detecting opgraph-packed-1 models such as Qwen3.5-4B / Qwen3.8-27B and lamq.json exports such as Gemma 4 E2B):

PYTHON
from mlx_lm import generate
from runtime_mlx import load

model, tok = load("shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "What is 17 times 23? Show the steps."}],
    add_generation_prompt=True,
    enable_thinking=False,
)
print(generate(model, tok, prompt, max_tokens=200))

For streaming, multi-turn chat, or speculative decoding through one interface, use Runtime:

PYTHON
from runtime_mlx import Runtime

rt = Runtime.load(
    "shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB",
    drafter="shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter",
)
for chunk in rt.stream("Solve 3x + 4y = 31 and 5x - 2y = 17.", max_tokens=256):
    print(chunk.text, end="", flush=True)

(注:更多技术实现细节与完整 API 文档请查阅原项目 README)