Termsslug: on-device-inference

On-Device Inference

端侧推理

Running model inference locally on user devices to avoid cloud latency and data egress.

Curated Intelligence

56 items · Page 1/6
Rohan Paul (@rohanpaul_ai)T2·Models
72 pts
Reson8 Releases Rumble-2-Flash 1GB On-Device Speech Model

Amsterdam-based startup Reson8 has released Rumble-2-Flash, a 0.5B-parameter, roughly 1GB speech-to-text model that runs offline on a phone, covers 45 languages and ships free under Apache 2.0. It targets the audio that breaks most transcription tools, including thick accents, overlapping speakers, poor phone lines and street noise, and is claimed to beat models 20x its size.

IT之家 · 智能时代T2·Models
62 pts
Tencent WeChat Xiaowei Hits 30M DAU with Lower Inference Costs

Tencent President Martin Lau said at an earnings call that WeChat's AI assistant Xiaowei has reached 30 million daily active users in its test pool, and its quarterly cost after full rollout will be lower than Yuanbao's earlier this year. Xiaowei runs on a custom WeLM model with 80B total and 3B activated parameters, with Tencent citing extremely low and falling inference costs and expecting some inference to move on-device.

IT之家 · 智能时代T2·Industry
78 pts
WeChat AI Assistant Xiaowei Hits 30 Million Daily Active Users

WeChat announced that its personal AI assistant Xiaowei has expanded its user testing and reached 30 million daily active users, gradually integrating into chats, Moments, official accounts and Channels. Built on a customized WeLM model optimized for cost efficiency, Xiaowei may offload some inference tasks to user devices in the future.

Hacker News AIT2·Models
88 pts
AutoArk Open-Sources 145B On-Device Model Edge0-145B

AutoArk has open-sourced Edge0-145B and Edge0-120B alongside a unified inference framework and deployment tooling. The 145B variant is billed as the first open-source 145-billion-parameter model that runs locally on a personal computer, with reported peak memory of about 29GB and decoding above 80 tokens/s. The company claims coding, reasoning and agent capabilities competitive with select frontier cloud models, with core capabilities working offline and without accounts or usage quotas.

Hugging Face BlogT1·Agents
62 pts
BotPsy Debuts On-Device Agentic Assistant Under One Billion Parameters

A Hugging Face team released BotPsy, a sub-billion-parameter agentic language model built to run as an on-device personal assistant, replacing costly and volatile cloud LLM calls with local inference. The post details evaluation protocols across ToolSandbox, GPQA Diamond, MMLU-Pro, IFEval and related benchmarks, arguing that tool-use and general capability can coexist at a very small parameter budget.

IT之家 · 智能时代T2·Agents
62 pts
Apple Siri Routes About 2% of English Requests to On-Device AI

Apple has introduced a front-end module called Skimmer in the first developer beta of iOS/macOS 27.3, routing roughly 2% of English Siri AI requests to an on-device model for the first time, mainly covering high-frequency fixed commands such as timers, music playback and calls. The module matches preset tool flows via regular expressions and executes directly, bypassing full AI inference to cut latency and protect privacy.

量子位 (微信公众号)T2·Models
78 pts
Alibaba's MNN Team Open-Sources 0.3B On-Device Model Hitting 350 Tokens/s

Alibaba's MNN team released MNN-Pocket-0.3B, an open-source ~0.3B-parameter model co-designed with its vocabulary, training data, post-training, quantization and inference engine for native on-device use. The team claims 350 tokens/s offline decoding on iPhone 18 Pro with peak memory under 300MB, targeting 8B-class task quality in business scenarios.