IT之家 · 智能时代 · 10/15/2026, 11:22:04
StepFun Releases Step 5 Preview: 600B MoE Model with 1M Context for Long-Horizon Agents
StepFun has open-sourced the weights of Step 5 Preview, a 600B-parameter sparse MoE model (27B active) supporting 1M-token context and multimodal inputs. Key innovations include a 92-layer narrow-deep architecture, Sparse GQA with block-level indexing to reduce computational overhead, and RL strategies optimized for long-horizon agent tasks, positioning it as a top performer on OpenRouter.
SOURCE COVERAGEFull Original Report
According to official announcements, Step 5 Preview is designed for long-horizon Agent tasks. It employs a sparse Mixture-of-Experts (MoE) architecture with a total of 600 billion parameters, activating only 27 billion parameters per token. The model supports a context window of up to 1 million tokens and natively handles both text and visual inputs. It demonstrates exceptional performance in scenarios such as AI programming, software engineering, professional knowledge work, and finance. Recently, Step 5 Preview ranked second globally on OpenRouter’s weekly and daily call volume leaderboards.
The model weights are now available on Hugging Face and ModelScope, offering compatibility support for various chip platforms and inference frameworks. Developers can deploy API services compatible with the OpenAI interface using vLLM or SGLang, or perform offline inference via Transformers. This allows users to integrate the model into their own data, tools, and systems to build agents, applications, workflows, and services.
IT Home noted that the accompanying technical blog post details how Step 5 Preview achieves support for longer and more complex Agent tasks through co-optimization of architecture and training. Key design features include:
- 92-Layer Narrow-Deep Architecture: Provides deeper computational paths for implicit multi-hop reasoning while controlling parameter scale and computational costs.
- Sparse GQA: Reduces computational overhead for million-token contexts by merging block-level indexing with sparse computation. Specifically, 8-token block-level indexing reduces the overhead of indexing and top-k selection to 1/8th of the original cost.
- More Stable Training: Maintains FP32 precision for critical states and parameters, combined with fine-grained gradient clipping, to mitigate abnormal activation growth and gradient spikes in deep networks.
- Million-Scale Verifiable Environments: Covers software engineering, scientific reasoning, machine learning R&D, and professional knowledge work, with continuous calibration of task difficulty, diversity, and evaluation reliability.
- Reinforcement Learning for Long-Horizon Tasks: Incorporates context compression into end-to-end policy training, enabling Agents to learn to organize task states and continue execution. Combined with more refined reward attribution, this improves learning effectiveness for long trajectories.