Termsslug: quantization

Quantization

量化

Technique that represents model weights at lower bit-width to shrink size and memory footprint.

Curated Intelligence

31 items · Page 1/4
Hugging Face BlogT1·Open Source
78 pts
llama.cpp Ships v1.0.0 Marking a Milestone for Local AI Inference

llama.cpp has officially released v1.0.0, establishing a stable baseline 3.5 years after inception. Built on quantization, optimized backends and a portable C/C++ implementation, it runs multimodal LLMs from consumer hardware to enterprise servers, serving as a key foundation for local and private AI inference.

IT之家 · 智能时代T2·Infra
58 pts
Hobbyist Runs 2.8T-Parameter Kimi K3 on Discontinued Intel Optane Memory

A Reddit hobbyist ran the 2.8-trillion-parameter Moonshot Kimi K3 model in a GPU-free home lab using discontinued Intel Optane Persistent Memory 200 series and llama.cpp. With 1-bit quantization, disabling DAX so hot data benefits from DRAM caching lifted throughput from 0.85 to 1.5 tokens per second, while a 3-bit test reached only 0.75 tokens/s, taking about 18.5 hours for 50,000 tokens. The experiment shows trillion-scale models can run entirely from a memory hierarchy instead of GPU VRAM, though at far-from-practical speeds.

Hacker News AIT2·Tools
58 pts
New MIPS Indexing Speeds Up Retrieval for Efficient RAG

The article presents a new approach to maximum inner product search (MIPS), arguing that conventional approximate retrieval methods assume query vectors resemble key vectors, an assumption that breaks down in RAG where short user questions are matched against long document keys. The proposed method targets this geometric mismatch and outperforms prior systems notably when query volume is large.

Hacker News AIT2·Open Source
74 pts
llama-modchip Runs 284B DeepSeek on 64GB RAM With No GPU

The open-source llama-modchip v2.4.0 claims it can run a 284B-parameter DeepSeek-V4-Flash MoE model entirely in RAM on a 64GB CPU-only workstation, peaking at 57.61 GiB with roughly 9-15 tok/s decode at 256K context. It relies on quantization and memory optimizations such as SwiGLU-H2, plus Fractal-64/MXFP4 transcoding and status tooling.

Rohan Paul (@rohanpaul_ai)T2·Agents
78 pts
Meta Opens Muse AI Agent Live Video Calls to All US Users

Meta has opened live video calls with its Muse AI agent to all US users, letting them talk to a customizable animated avatar with roughly 870ms response latency. A distilled student model reproduces in two passes what a teacher model needed 120 passes for, and with 4-bit quantization, caching and dynamic batching a single Nvidia GB200 serves 12 concurrent video streams—though serving cost at million-user scale remains the key constraint.

Hacker News AIT2·Tools
58 pts
Offline Survival Assistant Runs Local LLM and Kiwix from a USB Stick

The project delivers a fully offline survival-knowledge assistant that runs local models via llama.cpp or Ollama on Windows, Linux and macOS, retrieving from Kiwix offline libraries (medical Wikipedia, Where There Is No Doctor, iFixit and more) and answering with citations. It bundles multilingual practical guides and a LAN-shareable web UI, with a Python-only retrieval mode when no model is installed.

量子位 (微信公众号)T2·Models
78 pts
Alibaba's MNN Team Open-Sources 0.3B On-Device Model Hitting 350 Tokens/s

Alibaba's MNN team released MNN-Pocket-0.3B, an open-source ~0.3B-parameter model co-designed with its vocabulary, training data, post-training, quantization and inference engine for native on-device use. The team claims 350 tokens/s offline decoding on iPhone 18 Pro with peak memory under 300MB, targeting 8B-class task quality in business scenarios.

Apple Machine Learning ResearchT1·Tools
62 pts
New Method Speeds Up RAG Retrieval by Exploiting Query-Key Asymmetry

Targeting the computational bottleneck of maximum inner product search (MIPS) in RAG, this work argues that existing approximate methods model only key-vector geometry and ignore the mismatch between short user queries and long documents. The authors propose a retrieval scheme that exploits query distribution properties to cut cost while preserving accuracy, extending it to private search settings.

Marktechpost (@Marktechpost)T2·Models
78 pts
Deep Principle Releases Hydrogen-1.0 Open-Weight Science Model

Deep Principle released Hydrogen-1.0, an open-weight science model post-trained on GLM-5.3 using full lab notebook records and discovery trajectories, including failed attempts. It scores 38.3% on FrontierScience-Research, surpassing GPT-6-Astra's 33.3%, though it trails in NMR structure elucidation. The 753B parameter model features a 1M context window, is licensed under MIT, and supports BF16/NVFP4/MXFP4 quantization.