Alibaba's MNN Team Open-Sources 0.3B On-Device Model Hitting 350 Tokens/s

Intelligence Within Reach

Contributed by the MNN Team

QuantumBit | WeChat Official Account QbitAI

Running large models on mobile phones has already seen considerable exploration.

From dense models in the 1B, 4B, and 27B ranges to MoE models with roughly 30B total parameters and about 3B activated parameters, different scales and architectures are expanding the possibilities of on-device deployment.

But for mobile app developers, successfully loading a model and completing a single conversation is only the starting point.

For frequently used apps like Taobao and Xianyu, a large model must share compute, memory, and bandwidth with the app's own business logic. At the same time, it must balance response latency, power consumption, and temperature rise under continuous invocation and multi-task concurrency.

The MNN Team calls this design direction "mobile-app-native large models." They have released MNN-Pocket-0.3B, which targets the task requirements and resource budgets of mobile apps, balancing task capability, response speed, and sustained runtime experience under limited compute, memory, and energy.

Background on Mobile-App-Native Large Models

Focusing on comprehensive mobile apps such as Taobao, the team translates "mobile-app-native large models" into design goals across four dimensions:

• Speed: Under specified devices and context configurations, decoding throughput exceeds 200 tokens/s. This goal references the generation throughput of high-speed cloud model services, aiming to provide a smooth local generation experience.
• Memory: Under specified test conditions, peak memory is below 300MB. Even though phones have 8–16GB of memory, it must be shared by the system and multiple apps. Taking a model module budget of about 500MB as an example, keeping inference peaks below 300MB leaves headroom for context growth and integration overhead.
• Power: Under typical inference loads (512–1024 input tokens, 128 output tokens), the incremental energy per inference is targeted at no more than one ten-thousandth of the reference phone's battery energy—that is, 100 inferences consume no more than 1% of total battery capacity—controlling the battery cost of high-frequency invocation.
• Capability: The goal is to achieve the effect of an 8B-class model in specified business scenarios, covering Chinese and English understanding, world knowledge, and mathematical and code reasoning, with a focus on strengthening instruction following and reserving room for extension to long contexts and multi-step tasks.

These goals must also be validated together under continuous invocation: meeting task requirements while controlling power consumption and temperature rise and keeping the whole device smooth is what allows the model to fit into daily app usage.

MNN-Pocket-0.3B is a concrete practice of this design direction. The MNN team designed and trained from scratch this lightweight language model with about 300 million parameters, with a focus on Chinese and English, incorporating model architecture, vocabulary, training data, pre-training and post-training methods, as well as quantization and the inference engine into coordinated optimization, providing Taobao business and open-source community developers with a model foundation that has task understanding capability, responds quickly, and is easy to adapt and integrate.

△ Mobile-app-native large model quadrant: MNN-Pocket-0.3B balances speed, memory* and capability (*Note: Peak memory depends not only on total parameter count but also on vocabulary size, quantization parameters, and loading buffers. See the technical report for detailed metrics and test settings.)

With 0.3B Parameters, How Is Lightweight and High Speed Achieved?

First, look at the real-device performance of MNN-Pocket-0.3B on iPhone 18 Pro. The model runs fully offline through the MNN inference engine, with an average decode speed of about 350 tokens/s. The screen recording simultaneously shows the speed of input processing (Prefill), token-by-token generation (Decode), and peak memory usage, intuitively presenting the model's response efficiency and resource overhead.

Deployment size and runtime memory: The INT4 MNN deployment file of MNN-Pocket-0.3B is only 173MB, including model weights, tokenizer, computation graph, and inference configuration. Under the iPad Pro M5, Metal backend, and 512-token test configuration, the peak Metal resident memory during model loading and inference increases by 228MB relative to the pre-loading baseline. In the same test, Qwen3-0.6B has a deployment file size of 340MB and a peak memory increment of 427MB. By comparison, MNN-Pocket-0.3B reduces deployment size by about 49% and peak memory increment by about 47%. For mobile-app-native large models, a smaller deployment size can reduce model download and update costs, while lower runtime memory reserves more space for business modules such as pages, images, and videos, reducing the resource burden of app integration.

△ Memory usage and deployment size of on-device large models on iPad Pro (M5) (context length 512 tokens)

Cross-Platform Inference Speed: The team evaluated inference throughput on Apple and Android devices using the Metal, OpenCL, and CPU backends. Tests used MNN 3.6.2, INT4 quantization, and unified runtime settings, with each configuration warmed up once and measured five times, taking the median throughput, with Prefill and Decode measured separately. Under the iPad Pro M5/Metal, 512-token test configuration, MNN-Pocket-0.3B achieved a Prefill throughput of 8,721 tokens/s and a Decode throughput of 466 tokens/s, approximately 1.63× that of Qwen3-0.6B (286 tokens/s) under the same conditions. On the CPU backend of the Snapdragon 8 Gen 5, Decode throughput reached 251 tokens/s.

△ Inference speed of on-device large models on different platforms (context length 512 tokens, unit: tokens/s)

Prefill affects input processing time, while Decode affects subsequent generation speed. The above results are based on the quantization, context, and measurement conditions specified in the report; actual performance will vary with device, backend, and input/output length. The complete configuration can be found in Section 5 of the technical report.

These results come from the synergy of three levels: parameter scale and structure control overall overhead, vocabulary design reduces storage and output computation, and the inference engine improves hardware execution efficiency.

Determining Model Structure from the Resource Budget of Mobile Applications

For a native large model for mobile applications, the team set peak memory below 300MB and decode throughput above 200 tokens/s as the architecture selection targets, preserving as much model capacity as possible within the resource budget. Around this goal, the team built Dense model configurations covering approximately 50M to 1.3B parameters and tested them on the iPad Pro M5 under unified inference settings. Approximately 300M was the largest candidate scale in this set of experiments that simultaneously satisfied both constraints.

After determining the scale, the team further jointly adjusted model depth, hidden layer width, feed-forward network, and attention configuration within a similar parameter budget. Increasing from 16 layers to 28 layers brought capability improvements; continuing to increase to 36 layers did not further improve the average evaluation score, while decode throughput dropped from 355 tokens/s to 323 tokens/s. The throughput here is used for architecture selection comparison and differs from the test configuration of the final deployed version mentioned above.

Ultimately, MNN-Pocket-0.3B adopts a 28-layer structure, combined with configurations such as grouped-query attention (GQA), achieving a balance among model capability, runtime memory, and inference efficiency.

△ MNN-Pocket text model architecture and parameter scale selection

Streamlining the Vocabulary to Reduce Storage and Output Computation Overhead

For a 0.3B model, the token embedding and output layers occupy a considerable portion of the parameters. A larger vocabulary usually helps compress text length but also increases storage, memory access, and output projection overhead, so it needs to be designed in conjunction with the target language and tasks.

Targeting Chinese internet mobile applications, MNN-Pocket-0.3B focuses on Chinese and English as key supported languages, lowering the priority of tokenization efficiency for other languages. Based on the Qwen3 tokenizer, the team reduced the vocabulary from approximately 151K to 64K, cutting the number of entries by about 57%, reducing the parameters and storage requirements of the token embedding and output layers, and lowering the output projection overhead during token-by-token generation.

During the pruning process, the team used a corpus covering Chinese and English, code, mathematics, and instruction formats to filter entries, while preserving special tokens, byte fallback tokens, and necessary BPE merge dependencies. By balancing vocabulary size and encoding efficiency, the model can retain efficient encoding of common expressions and avoid excessive pruning that lengthens sequences and offsets deployment gains.

Model-Engine Synergy to Accelerate On-Device Inference

The on-device efficiency of MNN-Pocket-0.3B comes from the synergy between model design and the inference engine. On the CPU side, MNN customizes low-bit kernels for Arm dot-product and matrix instructions; on the GPU side, it optimizes cache access, computation tiling, and thread configuration for different architectures, and provides a dedicated backend for Qualcomm Hexagon. At the same time, it selects appropriate execution paths for Prefill and Decode, reducing memory access and scheduling overhead through QKV and FFN operator fusion and GPU command reuse. Beyond conventional autoregressive decoding, MNN also supports speculative decoding and implements dedicated GPU kernels for the 8-draft configuration, optimizing parallel computation of multiple tokens.

On devices equipped with the Snapdragon 8 Gen 5, the team evaluated the inference performance of MNN-Pocket-0.3B on CPU, GPU, and NPU respectively, and compared it with llama.cpp. Under 4-bit quantization, MNN's Prefill performance is 1.51–2.90× that of llama.cpp, and its Decode performance is 1.43–1.82× that of llama.cpp, showing a clear advantage.

In terms of energy consumption, the team also conducted a power test of the 4-bit MNN-Pocket-0.3B model using MNN on the OnePlus 13 (Snapdragon 8 Elite). Under a baseline workload of 4 CPU threads, 1024 input tokens, and 128 output tokens, by collecting battery current and voltage and subtracting idle baseline power consumption, the average incremental energy consumption across three rounds of testing was approximately 2.7J per round. Based on the phone's 6000mAh battery capacity, executing 100 tasks of the same workload accumulates incremental energy consumption of about 0.3% of the total battery capacity**.**

MNN-LLM lowers the integration cost across chips and operating systems through a unified model conversion and inference interface. MNN has accumulated 16K GitHub Stars, MNN-LLM supports 200+ large models, and related models have been downloaded over a million times cumulatively, providing an engineering foundation for MNN-Pocket-0.3B to enter practical applications.

Deployment efficiency determines the cost of bringing a model into mobile applications. What needs to be verified next is whether it can accurately understand input and complete tasks as required under limited parameters.

0.3B Parameters, Equally Impressive Task Performance

The native large model capabilities of mobile applications need to be reflected in results that business processes can actually use. Comment extraction requires faithful reproduction of the original text, label judgment must be confined to the candidate set, and structured output must comply with the specified fields and formats. Content omissions, unsupported additions, or category overreach may all affect downstream processing.

Therefore, while building Chinese-English understanding, mathematical reasoning, and coding capabilities, MNN-Pocket-0.3B focuses on strengthening instruction following and constrained output, providing a foundation for adaptation to different business tasks.

Model performance: Under the evaluation settings adopted in the technical report, MNN-Pocket-0.3B achieves a score of 73.20 on English IFEval and 74.59 on the Chinese instruction-following IFEval-zh benchmark constructed by the team, the highest scores among the compared models. Across 11 evaluations covering Chinese-English knowledge, reasoning, mathematics, coding, and instruction following, the model achieves an average score of 50.00, higher than the same-scale models listed in the report as well as larger comparison models such as MiniCPM4-0.5B and Qwen3-0.6B. Among these, the Chinese mathematical MGSM-zh score is 51.20, and the code generation HumanEval score is 53.66. These results provide a baseline capability reference for application adaptation; specific business scenarios still need to be validated through task data and acceptance criteria. Knowledge-intensive tasks and complex reasoning also remain areas where the model needs continued improvement.

△ Accuracy comparison of MNN-Pocket-0.3B and other baseline models across 11 benchmarks

At a fixed parameter scale, the team establishes the foundation through multi-stage pre-training, fills capability gaps through iterative SFT, integrates capabilities through multi-teacher distillation, and then preserves deployment performance through quantization-aware training.

△ Complete training pipeline of MNN-Pocket-0.3B

Multi-stage pre-training: making data more suitable for small-model learning

Small models are more sensitive to data quality and mixing ratios. MNN-Pocket-0.3B annotates and grades pre-training data across five dimensions—domain, text form, knowledge density, difficulty, and quality—to create more refined recipes for different training stages.

Is the data worth learning? On top of cleaning and deduplication, quality and knowledge density are evaluated separately: quality focuses on the accuracy, coherence, and completeness of content, while knowledge density focuses on the amount of effective information and the degree of redundancy. Each is divided into low, medium, and high levels, corresponding to Q1–Q3 and D1–D3, distinguishing "clean content" from "information-rich content" to improve the precision of filtering and mixing.

Is the data suitable for learning? Difficulty is divided into three levels, H1–H3, and combined with domain and text form to control the distribution of samples of different complexity. At the same time, Q&A, tables, and lists are structurally organized, and concepts, reasoning steps, or explanations are added to some mathematical and code samples to enhance learnability.

When should the data be learned? The above dimensions are combined into data buckets, and through candidate recipe trials and actual mixed training, the sampling ratios at each stage are jointly adjusted. For example, when increasing the proportion of mathematical data, the quality, knowledge density, and difficulty composition within it are also considered simultaneously, so that the domain mix matches the model's current learning stage.

Based on this grading system, MNN-Pocket-0.3B completes approximately 4.1T tokens of four-stage pre-training: establishing language and knowledge foundations, strengthening Chinese, mathematics, and code, consolidating capabilities through high-quality data annealing, and finally extending context support to 32K.

△ The first three pre-training stages of MNN-Pocket-0.3B; finer data grading improves learning efficiency

SFT Loop Engine: autonomous iterative training around capability gaps

Building on pre-training, the team constructed an SFT data recipe of approximately 3.16 million samples, covering knowledge Q&A, mathematics, programming, Chinese tasks, and open-ended generation, transforming foundational capabilities into the ability to complete tasks according to instructions.

To improve iteration efficiency, the team developed the SFT Loop Engine, which orchestrates training, evaluation, and data improvement through an Agent to form a closed loop:

Training → Evaluation → Failure attribution → Data quota allocation → Synthesis and screening → Retraining.

The process maps failed responses to fine-grained task categories and, combined with the historical training data distribution, determines the categories, languages, and sample counts that need supplementation. The synthesis stage does not directly read the original evaluation questions, and performs deduplication and leakage checks before using the filtered data for the next round of training.

This mechanism aligns training improvements with specific problems, while regression evaluation checks the gains from newly added data and changes in other capabilities. For a mobile application native large model that needs to cover multiple business tasks, this helps continuously fill capability gaps and maintain capability balance.

△ Flowchart of the MNN-Pocket-0.3B SFT Loop Engine

MOPD: integrating multi-domain capabilities into a single model

After completing SFT, the team further addressed the challenge of coordinating multi-domain capabilities under limited capacity. Mathematics, code, knowledge, and instruction following each have their own optimization objectives; mixed training needs to handle interactions among objectives, and sequential training may also cause regression in existing capabilities.

MNN-Pocket-0.3B adopts multi-teacher On-Policy distillation (MOPD), building four teacher routes optimized through domain-specific reinforcement learning, selecting teachers according to the domain of each sample, and jointly supervising a single student model.

Among these, the key to On-Policy is that the student generates first. The teacher reads the context actually produced by the student and provides token distribution supervision at these positions, so that training covers the generation states the student itself is likely to encounter. The team adopts a distribution matching objective to provide denser learning signals for the small model and improve training stability.

After MOPD, the model's average score across 11 evaluations increased from 35.79 to 50.00, with improvements on 10 tasks and one tie. Notable gains were achieved in Chinese math, Chinese instruction following, and mathematical reasoning.

Multiple teachers provide capability guidance during the training phase, while only a single 0.3B student model is retained at deployment, enabling capability improvements without adding storage and inference overhead for teacher models on the device.

QAT: Preserving Training Gains in the Deployed Version

The low-precision model ultimately used in mobile applications must also undergo capability validation. The team incorporated quantization into the post-training pipeline, using Self-OPD and quantization-aware training (QAT) to align the W4A16 student with the frozen BF16 model on its own generated context, mitigating the capability loss caused by quantization.

On the same set of 11 evaluations, the INT4 version achieved an average score of 49.59, a decrease of 0.41 points from the BF16 version's 50.00. This allows the deployment efficiency demonstrated in the previous section and the task capabilities presented in this section to ultimately be delivered on the same INT4 version.

△ MNN-Pocket-0.3B Model MOPD and QAT Pipeline Diagram

From base training to low-precision deployment, the goal of MNN-Pocket-0.3B has always been to build capabilities that applications can use. Next, through model releases at different stages and validation on specific tasks, researchers and developers can continue to adapt, deploy, and evaluate the model.

0.3B Model Releases Multi-Stage Weights, Moving from Task Fine-Tuning to Real-World Applications

The value of a native large model for mobile applications also needs to be validated through task adaptation and application integration. To this end, MNN-Pocket-0.3B provides multiple versions spanning pre-training, post-training, and on-device deployment, allowing researchers and developers to choose an appropriate starting point for continued training, evaluation, and application building.

From research to deployment, the MNN team open-sources model entry points at different stages:

For students and researchers, the Base and SFT checkpoints provide experimental entry points for base training and post-training respectively, facilitating exploration of data recipes, reinforcement learning, and distillation methods. The 0.3B parameter scale helps reduce the cost of individual fine-tuning and evaluation runs, supporting more efficient multi-round experiments.

For application developers, the final instruction model can be used for business adaptation, and the MNN version provides deployment assets for the corresponding inference ecosystem. In addition, the Dec version targets classification and candidate selection tasks, completing candidate scoring in a single forward pass through a decision head; the DFlash2 draft model targets generation acceleration, proposing 8 candidate tokens at a time that are jointly verified by the target model, reducing the serial overhead of token-by-token generation.

Around business adaptation and interaction experience, the team further selected two pipelines—e-commerce review opinion mining and fully on-device voice dialogue—to validate the model's performance on specific tasks and in complete applications.

Vertical Task Adaptation: E-Commerce Review Understanding Capabilities of the 0.3B Model

"Very good, but the concealing effect is a bit weak"—a short review that contains both an overall positive evaluation and a negative evaluation of a specific function. The model needs to identify the correspondence between the two and distinguish explicit expressions from implicit information. In the Tianchi e-commerce review opinion mining task, the output consists of four fields: attribute term, opinion term, category, and sentiment polarity. Attribute terms and opinion terms must come from the original text, categories and sentiments must be limited to predefined sets, and fields not explicitly expressed are left empty according to the rules. Multiple opinions within a single review also need to be fully extracted.

The team used the 3,229 reviews provided by the Tianchi task to apply LoRA fine-tuning with the same recipe to each model, then submitted to the platform for scoring against hidden test labels. The evaluation requires all four fields of the quadruple to match to count as correct, thus simultaneously testing original-text extraction, fine-grained classification, and format-following capabilities. In the experiment, MNN-Pocket-0.3B achieved an F1 of 80.15, higher than Qwen3-0.6B (77.47) and Qwen3-1.7B (78.59) under the same settings. With less than one-fifth of the latter's parameter scale, it achieved a task score 1.56 percentage points higher.

△ Capability Validation of the MNN-Pocket-0.3B Model on E-Commerce Review Understanding

This result validates the model's fine-tuning adaptability on this task: through domain data and LoRA training, the 0.3B model can transform general understanding and instruction-following capabilities into structured outputs that meet business requirements, providing a lightweight implementation path for text analysis tasks within mobile applications.

Fully On-Device Voice Dialogue: Putting Speed and Capability into a Complete Interaction

Voice interaction further tests the model's response efficiency and collaboration capabilities in a complete application pipeline. The team built a fully on-device streaming voice system that connects speech recognition, MNN-Pocket text generation, and speech synthesis, and integrates voice activity detection, semantic sentence segmentation, and interruption control. All neural network modules run locally on the device via MNN. The system uses streaming processing to shorten waiting time: after speech is converted to text, MNN-Pocket begins generating a response, and text increments are sent to TTS in segments and then played segment by segment, without waiting for the complete response to be generated. When the user speaks again, the system stops the current generation, synthesis, and playback, and enters a new round of interaction. For the detailed workflow, please refer to Section 7.2 of the technical report.

Under the warm-up test conditions in the report, the system's measured first-audio latency P50 values are:

• iPad Pro M5: 600ms;
• Snapdragon 8 Gen 5 device: 935ms;
• iPhone 13 Pro: 1,055ms.

Under a unified ASR frontend and evaluation settings, the system using MNN-Pocket achieved a total score of 45.44 across nine VoiceBench tasks, the highest among the language backbones compared in the report, demonstrating the model's comprehensive task capabilities under voice input conditions.

△ Capability Validation of the MNN-Pocket-0.3B Model in Voice Conversation Scenarios

This pipeline validates the feasibility of building on-device applications with MNN-Pocket as the language processing foundation, working in concert with voice perception, synthesis, and interaction control. The complete system includes additional modules such as ASR and TTS, and its overall storage and memory requirements need to be evaluated separately based on the specific application.

Letting Real Applications Continue to Define the Model's Next Steps

The value of native large models for mobile applications must ultimately be demonstrated in real business scenarios, real devices, and everyday interactions.

MNN-Pocket-0.3B is designed around application requirements, co-designing model architecture, data and training methods, quantization, and the inference engine. At the 0.3B parameter scale, it balances task capability, response speed, and deployment cost, providing application developers with a model foundation that is trainable, adaptable, and integrable.

Going forward, the team will continue advancing on-device model research, focusing on requirement understanding, search and recommendation, and intelligent interaction in businesses such as Taobao, to improve model reliability, device coverage, and end-to-end efficiency, while also exploring applications on devices such as AI glasses.

By open-sourcing models and deployment versions at different stages, the team looks forward to working with researchers and developers to validate and improve the model across more specific tasks, so that intelligence naturally becomes part of everyday applications.

GitHub: https://github.com/alibaba/mnn Hugging Face: https://huggingface.co/taobao-mnn/MNN-Pocket-0.3B ModelScope: https://modelscope.cn/models/MNN/MNN-Pocket-0.3B Technical Report: https://modelscope.cn/models/MNN/MNN-Pocket-0.3B/resolve/master/MNN-Pocket-0.3B-Technical-Report.pdf

Feel free to share your thoughts in the comments!

— End —

【Academic Submissions】Please send an email to [email protected] on weekdays with the subject line 【Submission】, and tell us: who you are, where you are from, attach a project/homepage link to your submission content, and provide your contact information.

🎓 We will (try our best to) reply to you promptly :)

🌟 Light up the star 🌟

Daily updates on the latest in science and technology

Read the original article

Open in WeChat