llama.cpp has officially released v1.0.0, establishing a stable baseline 3.5 years after inception. Built on quantization, optimized backends and a portable C/C++ implementation, it runs multimodal LLMs from consumer hardware to enterprise servers, serving as a key foundation for local and private AI inference.
Quantization
量化
Technique that represents model weights at lower bit-width to shrink size and memory footprint.
Curated Intelligence
A Reddit hobbyist ran the 2.8-trillion-parameter Moonshot Kimi K3 model in a GPU-free home lab using discontinued Intel Optane Persistent Memory 200 series and llama.cpp. With 1-bit quantization, disabling DAX so hot data benefits from DRAM caching lifted throughput from 0.85 to 1.5 tokens per second, while a 3-bit test reached only 0.75 tokens/s, taking about 18.5 hours for 50,000 tokens. The experiment shows trillion-scale models can run entirely from a memory hierarchy instead of GPU VRAM, though at far-from-practical speeds.
The article presents a new approach to maximum inner product search (MIPS), arguing that conventional approximate retrieval methods assume query vectors resemble key vectors, an assumption that breaks down in RAG where short user questions are matched against long document keys. The proposed method targets this geometric mismatch and outperforms prior systems notably when query volume is large.
The open-source llama-modchip v2.4.0 claims it can run a 284B-parameter DeepSeek-V4-Flash MoE model entirely in RAM on a 64GB CPU-only workstation, peaking at 57.61 GiB with roughly 9-15 tok/s decode at 256K context. It relies on quantization and memory optimizations such as SwiGLU-H2, plus Fractal-64/MXFP4 transcoding and status tooling.
Meta has opened live video calls with its Muse AI agent to all US users, letting them talk to a customizable animated avatar with roughly 870ms response latency. A distilled student model reproduces in two passes what a teacher model needed 120 passes for, and with 4-bit quantization, caching and dynamic batching a single Nvidia GB200 serves 12 concurrent video streams—though serving cost at million-user scale remains the key constraint.
webAI has open-sourced TwIL-8B, an 8.8B-parameter model built on IBM's Granite 4.2 8B and tuned for math, coding and instruction following. It reportedly fits in a 4.98 GiB quantized file that runs on CPUs or modest GPUs, scoring 81.6% on LiveCodeBench and reaching roughly 92-93% of frontier-model performance on coding and instruction following.
The project delivers a fully offline survival-knowledge assistant that runs local models via llama.cpp or Ollama on Windows, Linux and macOS, retrieving from Kiwix offline libraries (medical Wikipedia, Where There Is No Doctor, iFixit and more) and answering with citations. It bundles multilingual practical guides and a LAN-shareable web UI, with a Python-only retrieval mode when no model is installed.
Alibaba's MNN team released MNN-Pocket-0.3B, an open-source ~0.3B-parameter model co-designed with its vocabulary, training data, post-training, quantization and inference engine for native on-device use. The team claims 350 tokens/s offline decoding on iPhone 18 Pro with peak memory under 300MB, targeting 8B-class task quality in business scenarios.
Targeting the computational bottleneck of maximum inner product search (MIPS) in RAG, this work argues that existing approximate methods model only key-vector geometry and ignore the mismatch between short user queries and long documents. The authors propose a retrieval scheme that exploits query distribution properties to cut cost while preserving accuracy, extending it to private search settings.
Deep Principle released Hydrogen-1.0, an open-weight science model post-trained on GLM-5.3 using full lab notebook records and discovery trajectories, including failed attempts. It scores 38.3% on FrontierScience-Research, surpassing GPT-6-Astra's 33.3%, though it trails in NMR structure elucidation. The 753B parameter model features a 1M context window, is licensed under MIT, and supports BF16/NVFP4/MXFP4 quantization.