TII 开源 Falcon 3 系列:含 Mamba 架构小模型

原标题:Welcome to the Falcon 3 Family of Open Models!
78AI 研判分

阿布扎比技术创新研究所(TII)正式开源 Falcon 3 系列大语言模型,涵盖 1B 至 10B 参数规模。该系列重点强化了科学、数学与代码能力,并创新性地引入了基于 Mamba 架构的 7B 基础模型,旨在提升训练效率与推理性能。

报道正文

本文目录9 个章节

返回文章列表

欢迎加入 Falcon 3 开源模型家族!

发布于 2024 年 12 月 17 日 在 GitHub 上更新 点赞 133

Falcon LLM TII UAE FalconLLM tiiuae

我们隆重推出 Falcon3,这是一系列由阿布扎比 技术创新研究所 (TII) 开发的、参数量低于 100 亿的仅解码器(decoder-only)大型语言模型。通过不断突破性能与训练效率的极限,此次发布体现了我们致力于推进开放且易于获取的大型基础模型的持续承诺。

Falcon3 是此前版本的自然演进,重点增强了模型在科学、数学和代码方面的能力。

本次迭代包含五个基座模型:

  1. Falcon3-1B-Base
  2. Falcon3-3B-Base
  3. Falcon3-Mamba-7B-Base
  4. Falcon3-7B-Base
  5. Falcon3-10B-Base

在开发这些模型的过程中,我们融入了多项关键创新,旨在提升模型性能的同时降低训练成本:

  • Transformer 模型的一次性预训练: 我们在 7B 模型上进行了单次大规模预训练,使用 1024 块 H100 GPU,利用包含网页、代码、STEM(科学、技术、工程、数学)以及精选高质量多语言数据的 14 万亿 token。
  • 通过深度扩展提升推理能力: 基于近期关于模型深度影响的研究,我们通过复制冗余层并将 7B 模型扩展至 10B 参数规模,并使用 2 万亿 token 的高质量数据继续预训练。由此产生的 Falcon3-10B-Base 在参数量低于 13B 的模型中实现了零样本(zero-shot)和少样本(few-shot)性能的业界领先(SOTA)。
  • 通过知识蒸馏优化微型模型: 为了提供紧凑高效的替代方案,我们利用剪枝和知识蒸馏技术开发了 Falcon3-1B-Base 和 Falcon3-3B-Base,仅使用少于 100GT 的精选高质量数据,从而重新定义了预训练效率。
  • 纯 SSM 架构: 我们通过在额外 1.5 万亿 token 的高质量数据上进行训练,进一步增强了 Falcon Mamba 7B,得到了 Falcon3-Mamba-7B-Base。值得注意的是,更新后的模型在推理和数学能力方面有了显著提升。
  • 其他变体: Falcon3 系列的所有模型均提供 Instruct、GGUF、GPTQ-Int4、GPTQ-Int8、AWQ 和 1.58-bit 等变体,为广泛的应用场景提供了灵活性。

核心亮点

Falcon3 展示了在大语言模型中小规模和中等规模范围内的性能极限,在常见基准测试中表现卓越:

  • Falcon3-1B-Base surpasses SmolLM2-1.7B and is on par with gemma-2-2b.
  • Falcon3-3B-Base outperforms larger models like Llama-3.1-8B and Minitron-4B-Base, highlighting the benefits of pre-training with knowledge distillation.
  • Falcon3-7B-Base demonstrates top performance, on par with Qwen2.5-7B, among models under the 9B scale.
  • Falcon3-10B-Base stands as the state-of-the-art achieving strong results in the under-13B category.
  • All the transformer-based Falcon3 models are compatible with Llama architecture allowing better integration in the AI ecosystem.
  • Falcon3-Mamba-7B continues to lead as the top-performing State Space Language Model (SSLM), matching or even surpassing leading transformer-based LLMs at the 7B scale, along with support for a longer 32K context length. Having the same architecture as the original Falcon Mamba 7B, users can integrate Falcon3-Mamba-7B seamlessly without any additional effort.
  • The instruct versions of our collection of base models further show remarkable performance across various benchmarks with Falcon3-7B-Instruct and Falcon3-10B-Instruct outperforming all instruct models under the 13B scale on the open leaderboard.

Enhanced Capabilities

We evaluated models with our internal evaluation pipeline (based on lm-evaluation-harness) and we report raw scores. Our evaluations highlight key areas where the Falcon3 family of models excel, reflecting the emphasis on enhancing performance in scientific domains, reasoning, and general knowledge capabilities:

  • Math Capabilities: Falcon3-10B-Base achieves 22.9 on MATH-Lvl5 and 83.0 on GSM8K, showcasing enhanced reasoning in complex math-focused tasks.
  • Coding Capabilities: Falcon3-10B-Base achieves 73.8 on MBPP, while Falcon3-10B-Instruct scores 45.8 on Multipl-E, reflecting their abilities to generalize across programming-related tasks.
  • Extended Context Length: Models in the Falcon3 family support up to 32k tokens (except the 1B supporting up to 8k context), with functional improvements such as scoring 86.3 on BFCL (Falcon3-10B-Instruct).
  • Improved Reasoning: Falcon3-7B-Base and Falcon3-10B-Base achieve 51.0 and 59.7 on BBH, reflecting enhanced reasoning capabilities, with the 10B model showing improved reasoning performance over the 7B.
  • Scientific Knowledge Expansion: Performance on MMLU benchmarks demonstrates advances in specialized knowledge, with scores of 67.4/39.2 (MMLU/MMLU-PRO) for Falcon3-7B-Base and 73.1/42.5 (MMLU/MMLU-PRO) for Falcon3-10B-Base respectively.

Models' Specs and Benchmark Results

Detailed specifications of the Falcon3 family of models are summarized in the following table. The architecture of Falcon3-7B-Base is characterized by a head dimension of 256 which yields high throughput when using FlashAttention-3 as it is optimized for this dimension. These decoder-only models span 18 to 40 layers for the transformer-based ones, and 64 layers for the mamba one, all models share the SwiGLU activation function, with vocabulary size of 131K tokens (65Kfor Mamba-7B). The Falcon3-7B-Base is trained on the largest amount of data ensuring comprehensive coverage of concepts and knowledge, the other variants require way less data.

Training efficiency

The table below highlights the performances of Falcon3-7B-Base and Falcon3-10B-Base on key benchmarks showing competitive performances in general, math, reasoning, and common sense understanding domains. Feel free to take a look at models' cards where we provide additional evaluation results (e.g. MT-Bench, Alpaca, etc).

Training efficiency

[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]

The instruct models also demonstrate competitive and super performances with equivalent and small-size models as highlighted in the tables below.

Instruct models

Falcon3-1B-Instruct and Falcon3-3B-Instruct achieve robust performance across the evaluated benchmarks. Specifically, Falcon3-1B attains competitive results in IFEval (54.4), MUSR (40.7), and SciQ (86.8), while Falcon3-3B exhibits further gains—particularly in MMLU-PRO (29.7) and MATH (19.9)—demonstrating clear scaling effects. Although they do not surpass all competing models on every metric, Falcon models show strong performances in reasoning and common-sense understanding relative to both Qwen and Llama. In our internal evaluation pipeline:

  • We use lm-evaluation harness.
  • We report raw scores obtained by applying chat template without fewshot_as_multiturn (unlike Llama3.1).
  • We use same batch-size across all models.

Training efficiency

Furthermore, Falcon3-7B and Falcon3-10B show robust performance across the evaluated benchmarks. Falcon3-7B achieves competitive scores on reasoning (Arc Challenge: 65.9, MUSR: 46.4) and math (GSM8K: 79.1), while Falcon3-10B demonstrates further improvements, notably in GSM8K (83.1) and IFEval (78), indicating clear scaling benefits.

Training efficiency

Open Source Commitment

In line with our mission to foster AI accessibility and collaboration, all models in the Falcon3 family are released under the Falcon LLM license. We hope the AI community finds these models valuable for research, application development, and further experimentation. Falcon3 is not a culmination but a continuation of our efforts to create more capable, efficient, specialized foundation models. In January 2025, we will further release other models of the Falcon3 family featuring enhanced multi-modal capabilities including image, video, and audio support, as well as a full technical report covering our methodologies. We welcome feedback and collaboration from the community as we continue to refine and advance these technologies.

Acknowledgments

We warmly thank the following people for their smooth support and integration within the ecosystem.

Citation

If the Falcon3 family of models were helpful to your work, feel free to give us a cite.

文本
@misc{Falcon3,
    title = {The Falcon 3 Family of Open Models},
    url = {https://huggingface.co/blog/falcon3},
    author = {Falcon-LLM Team},
    month = {December},
    year = {2024}
}