Taobao MNN Team Open-Sources 0.3B On-Device Model, Hits 350 Tokens/s Offline on iPhone 18 Pro
Taobao's MNN team trained from scratch and open-sourced MNN-Pocket-0.3B, a ~300M-parameter on-device language model co-optimized across architecture, tokenizer, training data, post-training and quantized inference, shipping as a 173MB INT4 package. Running fully offline via the MNN engine, it reaches roughly 350 tokens/s decode on iPhone 18 Pro and adds about 228MB peak Metal residency on iPad Pro M5, with a speculative-decoding draft model delivering up to 2.06x speedup.
Related Entities
Full Report
IT之家 November 6 news: Taobao's MNN team today released the MNN-Pocket-0.3B model, targeting the task requirements and resource budgets of mobile applications, balancing task capability, response speed, and sustained runtime experience under limited compute, memory, and energy consumption.
Focusing on comprehensive mobile applications such as Taobao, the team translated "mobile application-native large models" into four dimensions of design goals:
- On speed, under specified devices and context configurations, decode throughput exceeds 200 token/s. This goal references the generation throughput of high-speed cloud model services, striving to provide a smooth generation experience locally.
- On memory, under specified test conditions, peak memory is below 300MB. Even though phones have 8–16GB of memory, it must be shared by the system and multiple applications. Taking a model module budget of about 500MB as an example, keeping inference peaks below 300MB reserves headroom for context growth and integration overhead.
- On power consumption, under typical inference loads (input 512–1024 tokens, output 128 tokens), the incremental energy consumption per inference is targeted at no more than one ten-thousandth of the reference phone's battery energy, i.e., 100 inferences consume no more than 1% of total battery capacity, controlling the battery life overhead of high-frequency calls.
- On capability, the goal is to achieve the effect of an 8B-class model in specified business scenarios, balancing Chinese and English understanding, world knowledge, and mathematical and code reasoning, with emphasis on strengthening instruction following, while reserving expansion space for long contexts and multi-step tasks.
The MNN team designed and trained from scratch this lightweight language model with about 300 million parameters, with a focus on supporting Chinese and English, incorporating model architecture, vocabulary, training data, pre-training and post-training methods, as well as quantization and the inference engine into coordinated optimization, providing Taobao business and open-source community developers with a model foundation that has task understanding capability, responds quickly, and is easy to adapt and integrate.
In real-device tests of MNN-Pocket-0.3B on Apple iPhone 18 Pro, the model runs fully offline through the MNN inference engine, with an average decode speed of about 350 token/s.
The INT4 MNN deployment file of MNN-Pocket-0.3B is only 173MB in size, including model weights, tokenizer, computation graph, and inference configuration.
On iPad Pro M5 with the Metal backend and a 512-token test configuration, the peak Metal resident memory during model loading and inference increases by 228MB relative to the baseline before loading.
In the same test, Qwen3-0.6B has a deployment file size of 340MB and a peak memory increment of 427MB. By comparison, MNN-Pocket-0.3B reduces deployment size by about 49% and peak memory increment by about 47%.
For mobile application-native large models, a smaller deployment size can reduce model download and update costs, while lower runtime memory reserves more space for business modules such as pages, images, and videos, reducing the resource burden of application integration.
The team evaluated inference throughput on Apple and Android devices using the Metal, OpenCL, and CPU backends. The tests used MNN 3.6.2, INT4 quantization, and unified runtime settings, with each configuration warmed up once and measured five times, taking the median throughput, with Prefill and Decode measured separately.
On iPad Pro M5/Metal with a 512-token test configuration, MNN-Pocket-0.3B achieves a Prefill throughput of 8,721 token/s and a Decode throughput of 466 token/s, about 1.63 times that of Qwen3-0.6B (286 token/s) under the same conditions. On the CPU backend of Snapdragon 8 Gen 5, Decode throughput reaches 251 token/s.
For mobile application-native large models, the team set peak memory below 300MB and decode throughput above 200 token/s as architecture selection goals, preserving as much model capacity as possible within the resource budget. Around this goal, the team built dense model configurations ranging from about 50M to 1.3B parameters and tested them on iPad Pro M5 under unified inference settings. About 300M was the largest candidate scale in this set of experiments that simultaneously satisfied both constraints.
After determining the scale, the team further jointly adjusted model depth, hidden layer width, feed-forward network, and attention configuration within a similar parameter budget. Increasing from 16 layers to 28 layers brought capability improvements; continuing to increase to 36 layers did not further improve the average evaluation score, while decode throughput dropped from 355 token/s to 323 token/s. The throughput here is used for architecture selection comparison and differs from the test configuration of the final deployment version mentioned above.
Ultimately, MNN-Pocket-0.3B adopts a 28-layer structure, combined with configurations such as grouped query attention (GQA), achieving a balance among model capability, runtime memory, and inference efficiency.
Targeting Chinese mobile internet applications, MNN-Pocket-0.3B prioritizes Chinese and English as its key supported languages, lowering the priority of tokenization efficiency for other languages. Based on the Qwen3 tokenizer, the team reduced the vocabulary from approximately 151K to 64K, cutting the number of entries by about 57%, thereby reducing the parameters and storage requirements of the embedding and output layers, and lowering the output projection overhead during token-by-token generation.
In terms of energy consumption, the team also conducted power tests on the 4-bit model of MNN-Pocket-0.3B using MNN on a OnePlus 13 (Snapdragon 8 Elite). Under a baseline workload of 4 CPU threads, 1024 input tokens, and 128 output tokens, by collecting battery current and voltage and subtracting idle baseline power consumption, the average incremental energy consumption across three rounds of testing was approximately 2.7J per round. Based on the phone's 6000mAh battery capacity, executing 100 tasks of the same workload cumulatively accounts for about 0.3% of the total battery charge**.**
In terms of model capabilities, under the evaluation settings adopted in the technical report, MNN-Pocket-0.3B achieved an English IFEval score of 73.20, and a score of 74.59 on the Chinese instruction-following IFEval-zh constructed by the team, achieving the highest scores among the compared models.
Across 11 evaluations covering Chinese and English knowledge, reasoning, mathematics, code, and instruction following, the model achieved an average score of 50.00, higher than the same-scale models listed in the report, as well as larger comparison models such as MiniCPM4-0.5B and Qwen3-0.6B. Among them, the Chinese mathematics MGSM-zh score was 51.20, and the code generation HumanEval score was 53.66.
The value of native large models for mobile applications still needs to be validated through task adaptation and application integration. To this end, MNN-Pocket-0.3B provides multiple versions spanning pre-training, post-training, and on-device deployment, enabling researchers and developers to choose an appropriate starting point to continue training, evaluation, and application building.
From research to deployment, the MNN team open-sources model entry points at different stages:
| Model Version | Primary Use |
|---|---|
| MNN-Pocket-0.3B-Base | Continued pre-training, domain adaptation, and foundational capability research |
| MNN-Pocket-0.3B-SFT | Post-training research, reinforcement learning, distillation, and ablation experiments |
| MNN-Pocket-0.3B | General instruction following and business task adaptation |
| MNN-Pocket-0.3B-MNN | W4A16 QAT on-device deployment version, combined with the MNN inference engine's low-bit operators and hardware backend optimizations, achieving decoding throughput of up to 466 token/s on specific devices |
| MNN-Pocket-0.3B-DFlash2 | DFlash2 speculative decoding draft model, combined with the MNN inference engine's block-level generation and verification implementation, achieving decoding throughput up to 2.06x that of conventional autoregressive decoding on specific tasks |
| MNN-Pocket-0.3B-Dec | Completes candidate selection tasks via a decision head |
The open-source address is provided below:
https://github.com/alibaba/mnn
Advertising statement: This article contains external redirect links (including but not limited to hyperlinks, QR codes, passcodes, etc.), which are used to convey more information and save selection time. The results are for reference only. All articles on IT之家 that contain external links include this statement.