Inception 更新阿拉伯语基准:新增指令遵循与 AraGen

原标题:Arabic Leaderboards: Introducing Arabic Instruction Following, Updating AraGen, and More
42AI 研判分

Inception AI 团队在 Hugging Face 上更新了阿拉伯语大模型评估体系,正式引入“阿拉伯语指令遵循”(Arabic Instruction Following)排行榜,并对现有的生成式基准 AraGen 进行了升级。此举旨在填补非英语语言在复杂指令理解与执行能力评估上的空白,为开发者提供更精准的阿拉伯语模型选型参考。

报道正文

本文目录15 个章节

返回文章列表

阿拉伯语排行榜:推出阿拉伯语指令遵循评测、更新 AraGen 及更多进展

发布于 2025 年 4 月 8 日 在 GitHub 上更新 点赞 20

Ali El Filali alielfilali01 inceptionai Sarah AlBarri SarahAlBarri inceptionai Abouelseoud Arwa88 inceptionai samta kamboj samta-kamboj inceptionai Neha Sengupta neha1710 inceptionai Preslav Nakov preslavnakov MBZUAI

在 Inception,我们一直致力于提升阿拉伯语语境下的 AI 模型评估能力。此前,我们推出了 AraGen,这是首批阿拉伯语生成式排行榜之一,旨在为评估阿拉伯语 LLM 在生成任务上的表现提供基准。

作为持续努力的一部分,我们很高兴分享以下更新:

  • Arabic-Leaderboards Space,与 穆罕默德·本·扎耶德人工智能大学 (MBZUAI) 合作推出,旨在将阿拉伯语 AI 评估整合到一个平台中。该平台目前支持 AraGen-03-25 和 阿拉伯语指令遵循 (Arabic Instruction Following),并计划扩展至涵盖多种模态的阿拉伯语 AI 模型排行榜。
  • AraGen 03-25 版本发布,包含改进措施及更新的基准测试。
  • 指令遵循排行榜,由 阿拉伯语 IFEval 基准测试 (Arabic IFEval Benchmark) 提供支持,这是首个公开可用的用于评估阿拉伯语指令遵循能力的基准测试。

以下章节将详细介绍这些更新内容。

Arabic-Leaderboards Space

Arabic-Leaderboards 是一个全面且统一的阿拉伯语评估与任务空间。它旨在作为一个中心枢纽,覆盖广泛的评估领域,适用于不同模态的模型。目前,该空间已上线 AraGen-03-25 和阿拉伯语指令遵循两个实时排行榜。我们计划通过增加更多针对多模态阿拉伯语 AI 模型的排行榜和任务来进一步扩展这一空间。

我们欢迎感兴趣的贡献者通过社区标签页或电子邮件与我们联系,以讨论如何将其工作/排行榜作为额外标签页整合到此空间中。

AraGen 排行榜最新进展

2024年12月,我们推出了 AraGen Benchmark,作为 AraGen Leaderboard 的基础。该排行榜的一个关键特性是其动态性:评估数据集在三个月内保持私密(盲测),以确保公平且无偏的评估。遵循同样的理念,我们公开发布了 AraGen-12-24 benchmark,以及由 Claude-3.5-Sonnet 根据 3C3H guidelines 评估的所有模型响应。

通过分享此基准测试和模型响应,我们旨在鼓励社区对其进行审查,识别我们可能遗漏的任何意外行为,并帮助我们完善评估框架。

AraGen-03-25 发布

在本次最新的 AraGen 版本中,我们将数据集扩展至包含 340 对问答,较上一版本的 279 对有所增加。分布情况大致相似:

  • 问答: ~200 对
  • 推理: 70 对
  • 安全问题: 40 对
  • 正字法和语法分析: 30 对

这一分配反映了将问答作为任何语言模型/聊天机器人/AI助手的主要用例的核心重点,同时仍涵盖其他评估领域,特别是考虑到生成具有挑战性的阿拉伯语语法和正字法查询的复杂性。

任务分布 (%)

此外,我们优化了 judge system prompt,以提高清晰度,即使对于较小/较弱的 judge models 也是如此。

动态评估与排名分析

随着我们引入动态评估周期,保持基准测试和评估流程的一致性与可靠性至关重要。为确保这一点,我们分析了不同数据集版本和系统提示配置下前 10 名模型的排名变化。

排名变化分析

我们在两种评估场景下分析了模型性能:

  • 使用最新版 AraGen(AraGen-03-25)对比了先前的系统提示词(SP1)与当前的系统提示词(SP2)。
  • 评估了同时更新数据集和评审系统提示词的影响。

整体排名保持稳定,表现最佳的模型(o1-2024-12-17)始终占据领先地位。值得注意的是,我们观察到两个 Claude 模型的排名发生了互换,这凸显了我们评估方法的敏感性,尤其是考虑到它们最初的得分非常接近。

排名中唯一显著的变化发生在 gpt-4o-2024-08-06 模型上,其性能在更新数据集和提示词后显著提升。这一突然的跃升目前正作为我们持续进行的基准测试设计研究的一部分进行调查。

仅因系统提示词变化而未出现重大差异,这表明只要使用相同的评审模型(claude-3.5-sonnet),结果就具有良好的可复现性。然而,我们预计当使用较小或较弱的模型作为评审时可能会出现差异,此时采用第二个系统提示词(SP2)可能会增强一致性。

总结而言,o1-2024-12-17 稳健且持续领先的性能——尽管其最高分从 82.67% 降至 70.25%——继续在更具挑战性的更新版基准测试下强化了其在阿拉伯语应用中的可靠性。虽然近期对评估流程的更新引入了轻微的排名变动,但整体框架保持稳定,头部和尾部表现者显示出一致的位置。许多观察到的排名调整可能反映了由于分数微小差异导致的典型评估误差范围。值得注意的是,排名第二至第五位的模型得分此前介于 70–78% 之间,现在则介于 51–57% 之间。这突显出更新后的 AraGen 数据集构成了一个明显更难的基准,这与推理模型的最新进展相一致。尽管绝对分数发生了变化,但令人鼓舞的是排行榜位置基本保持一致,这强调了该评估方法在未来的稳健性。

更多详细分数

组合 1:系统提示词影响(AraGen-03-25 SP1 vs. AraGen-03-25 SP2)

表 1. AraGen-03-25 (SP1) 排名

排名模型名称3C3H 得分正确性完整性简洁性有用性诚实性无害性
1o1-2024-12-1769.49%74.90%73.04%47.11%72.40%74.56%74.90%
2gpt-4o-2024-08-0656.10%61.96%58.92%34.22%58.80%60.81%61.89%
3claude-3-5-sonnet-2024102254.29%59.31%57.65%34.31%57.13%58.01%59.31%
4claude-3-7-sonnet-2025021953.21%59.31%56.76%28.53%56.86%58.53%59.24%
5o3-mini-2025-01-3151.65%56.67%54.31%31.74%54.46%56.10%56.59%
6deepseek-chat47.82%54.31%52.35%20.56%51.94%53.46%54.31%
7claude-3-5-haiku-2024102243.62%48.14%44.61%28.92%45.37%46.57%48.14%
8o1-mini-2024-09-1243.60%47.55%47.06%26.54%46.35%46.57%47.55%
9Qwen/Qwen2.5-72B-Instruct42.18%48.63%47.55%16.03%44.93%47.38%48.55%
10gpt-4o-mini-2024-07-1840.96%45.10%44.02%24.24%43.19%44.14%45.10%

表 2. AraGen-03-25 (SP2) 排名

排名模型名称3C3H 得分正确性完整性简洁性有用性诚实性无害性
1o1-2024-12-1770.25%75.88%70.98%51.25%72.55%75.25%75.59%
2gpt-4o-2024-08-0657.38%63.14%56.67%39.95%59.66%61.79%63.06%
3claude-3-7-sonnet-2025021956.54%62.25%58.53%34.49%60.39%61.40%62.18%
4claude-3-5-sonnet-2024102255.60%60.49%56.67%39.14%58.60%58.50%60.20%
5o3-mini-2025-01-3151.63%56.08%52.35%36.72%53.53%55.10%56.00%
6deepseek-chat51.00%57.55%53.92%25.61%54.95%56.42%57.55%
7claude-3-5-haiku-2024102244.79%48.92%44.51%32.40%46.67%47.38%48.85%
8o1-mini-2024-09-1243.78%47.55%46.76%28.04%46.27%46.67%47.40%
9Qwen/Qwen2.5-72B-Instruct43.09%48.82%47.55%19.73%46.59%47.11%48.75%
10gpt-4o-mini-2024-07-1840.62%45.10%40.88%27.60%42.06%43.58%44.51%

组合 2:数据集与提示词更新的影响(AraGen-12-24 SP1(旧版) vs. AraGen-03-25 SP2(新版))

表 3. AraGen-12-24 (SP1) 排名

[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]

RankModel Name3C3H ScoreCorrectnessCompletenessConcisenessHelpfulnessHonestyHarmlessness
1o1-2024-12-1782.67%92.71%92.47%34.65%91.19%92.26%92.71%
2claude-3-5-sonnet-2024102278.74%88.31%87.81%33.27%86.97%87.78%88.31%
3claude-3-7-sonnet-2025021977.71%87.89%87.77%29.20%86.27%87.26%87.89%
4gpt-4o-2024-08-0673.89%83.75%82.91%28.94%80.99%83.00%83.75%
5deepseek-chat71.28%81.89%81.89%21.13%79.53%81.32%81.89%
6o3-mini-2025-01-3170.91%80.29%79.21%27.33%78.38%79.99%80.29%
7claude-3-5-haiku-2024102266.40%74.43%73.36%30.56%72.34%73.30%74.43%
8o1-mini-2024-09-1264.95%74.22%74.22%21.46%72.24%73.32%74.22%
9gpt-4o-mini-2024-07-1863.40%72.10%71.38%22.98%70.41%71.41%72.10%
10Qwen/Qwen2.5-72B-Instruct62.58%71.92%71.80%19.06%69.86%70.94%71.92%

Table 4. AraGen-03-25 (SP2) Rankings

RankModel Name3C3H ScoreCorrectnessCompletenessConcisenessHelpfulnessHonestyHarmlessness
1o1-2024-12-1770.25%75.88%70.98%51.25%72.55%75.25%75.59%
2gpt-4o-2024-08-0657.38%63.14%56.67%39.95%59.66%61.79%63.06%
3claude-3-7-sonnet-2025021956.54%62.25%58.53%34.49%60.39%61.40%62.18%
4claude-3-5-sonnet-2024102255.60%60.49%56.67%39.14%58.60%58.50%60.20%
5o3-mini-2025-01-3151.63%56.08%52.35%36.72%53.53%55.10%56.00%
6deepseek-chat51.00%57.55%53.92%25.61%54.95%56.42%57.55%
7claude-3-5-haiku-2024102244.79%48.92%44.51%32.40%46.67%47.38%48.85%
8o1-mini-2024-09-1243.78%47.55%46.76%28.04%46.27%46.67%47.40%
9Qwen/Qwen2.5-72B-Instruct43.09%48.82%47.55%19.73%46.59%47.11%48.75%
10gpt-4o-mini-2024-07-1840.62%45.10%40.88%27.60%42.06%43.58%44.51%

Analysis of 3C3H

As part of our December release, we introduced 3C3H as a new evaluation measure of the chat capability of models, aimed at assessing both the factuality and usability of LLMs’ answers. Over the past three months, we have observed some interesting findings, which we share in this section.

HeatMap for o1-2024-12-17

A model that stood out in this analysis is “silma-ai/SILMA-9B-Instruct-v1.0”, which exhibited a higher conciseness score compared to other open-weight models—even those with larger sizes. However, this gain in conciseness came at the cost of helpfulness and other dimensions when compared to its base model, “google/gemma-2-9b-it”. We believe that this analysis, along with optimizing for 3C3H, will enable the community to develop better models through curated datasets while maintaining the correlation across all dimensions.

SILMA-9B-Instruct-v1.0 VS Gemma-2-9b-it HeatMaps

This is an ongoing effort to better understand how these dimensions are interconnected and how various scenarios and training recipes affect this relationship. Below, we provide a space where you can generate heatmaps for any combination of models of your choice. We hope the community finds it helpful in spotting additional trends that we may not have noticed. Ultimately, we aim for this tool to foster more discussion about evaluation and 3C3H, serving as a resource for others’ work.

We believe that one limitation of this analysis is the zeroing rule, whereby we do not evaluate the other dimensions if the answer is not correct. In the future, we plan to investigate further whether an answer can be helpful despite being incorrect, and how dimensions such as conciseness and harmlessness factor into this evaluation if the answer is not correct.

Instruction Following Leaderboard

What is Instruction Following as a Benchmark?

One of the core capabilities of large language models (LLMs) is their ability to understand and follow human instructions. This skill is crucial for building reliable chatbots, virtual assistants, and AI systems that do what users ask. Without strong instruction-following, a model might generate correct information but in the wrong format, ignore user-specified constraints, or produce unwanted content. Instruction-Following benchmark is standardized, objective way to measure a model's instruction adherence and compare models fairly to drive improvements.

Dataset: Arabic IFEval

Our work took inspiration from the IFEval dataset. IFEval, originally introduced by Google, provides a structured benchmark designed to evaluate LLMs on their ability to follow verifiable instructions. It consists of prompts containing specific, objectively measurable commands such as “use exactly three bullet points,” “include the word ‘innovation’ twice,” or “limit your answer to 100 words.” English IFEval dataset contains around 500 prompts covering 25 different types of such verifiable instructions. Evaluation within IFEval is conducted through Python functions that automatically verify whether instructions are followed, avoiding the need for human evaluators or another AI judge. This makes the evaluations reproducible and unbiased. While IFEval has become the standard for assessing LLMs responding in English, a similarly detailed and structured resource is absent for Arabic.

Construction of our Arabic IFEval dataset began by carefully adapting approximately 300 prompts from the original English IFEval. This wasn't a straightforward, word-for-word translation; instead, we thoughtfully adjusted prompts to clearly reflect Arabic linguistic nuances and cultural contexts. Instructions that made little sense in Arabic, such as those involving English-specific vowel constraints, were either adapted to equivalent Arabic linguistic challenges or omitted entirely. Cultural references specific to English-speaking contexts were replaced with culturally relevant or Arabic-language equivalents to maintain contextual clarity. Additionally, we created unique Arabic-specific samples from scratch, specifically designed to emphasize distinctive Arabic phonetics, orthographic characteristics, and morphology, such as the careful use of diacritical marks (tashkīl), phonetic constraints like avoiding certain letters (e.g., writing without using the letter Alef (ا)), and leveraging root-based morphology to challenge models' word-selection abilities. All prompts underwent rigorous expert validation by Arabic linguists and domain experts who ensured grammatical accuracy, cultural appropriateness, and unambiguous clarity of each instruction.

Arabic IFEval dataset is publicly available for the research community to utilize, test, and contribute to. It is available on Huggingface under inceptionai/Arabic_IFEval

Sample I: Arabic IFEval

Prompt (Ar): فسر كيف يمكن للتقنيات الحديثة مثل الذكاء الاصطناعي أن تسهم في الحفاظ على الأدب العربي، مع تضمين 12 كلمة تنتهي بأحد الحروف الرافسة (د، ذ، أ، ر، ز، و)، وأن تكون الإجابة مكتوبة بأسلوب موجز لا يتجاوز 120 كلمة. يجب أن لا تحتوي إجابتك على أي فواصل.

Prompt Translation (En): Explain how modern technologies, such as artificial intelligence, can contribute to preserving Arabic literature. Your answer should include at least 12 words ending with one of these specific Arabic letters (د، ذ، أ، ر، ز، و), be concise, and should not exceed 120 words. Your response must not contain any commas.

Instructions to follow:

  • Letter Frequency Constraint: Include at least 12 words ending with one of the letters (د، ذ، أ، ر، ز، و).
  • Punctuation Constraint: Do not use commas.
  • Length Constraint: Write concisely, not exceeding 120 words.
JSON
{
  "key": 4767,
  "prompt": "فسر كيف يمكن للتقنيات الحديثة مثل الذكاء الاصطناعي أن تسهم في الحفاظ على الأدب العربي، مع تضمين 12 كلمة تنتهي بأحد الحروف الرافسة (د، ذ، أ، ر، ز، و)، وأن تكون الإجابة مكتوبة بأسلوب موجز لا يتجاوز 120 كلمة. يجب أن لا تحتوي إجابتك على أي فواصل.",
  "instruction_id_list": [
    "keywords:letter_list_freq",
    "punctuation:no_comma",
    "length_constraints:number_words"
  ],
  "kwargs": [
    {
      "letters": ["د", "ذ", "أ", "ر", "ز", "و"],
      "frequency": 12,
      "relation": "at least",
      "position": "end"
    },
    {},
    {
      "relation": "less than",
      "num_words": 500
    }
  ],
  "lang": ["ar"]
}

Sample II: Arabic IFEval

Prompt (Ar): اكتب قصة قصيرة عن الرقم 600، على أن يكتب الرقم في القصة بالكلمات وبكل الصيغ المفقطة الممكنة له على الأقل مرة (ستة مائة - ست مئة - ستمئة - ستمائة).

Prompt Translation (En): Write a short story about the number 600. Within the story, the number should be spelled out in Arabic in all possible written forms at least once each ("ستة مائة", "ست مئة", "ستمئة", "ستمائة").

Instructions to follow: Your response must explicitly include the following Arabic spellings at least once each:

  • ستة
  • مائة
  • ست
  • مئة
  • ستمئة
  • ستمائة
JSON
{
  "key": 4768,
  "prompt": "اكتب قصة قصيرة عن الرقم 600، على أن يكتب الرقم في القصة بالكلمات وبكل الصيغ المفقطة الممكنة له على الأقل مرة (ستة مائة - ست مئة - ستمئة - ستمائة).",
  "instruction_id_list": [
    "keywords:frequency",
    "keywords:frequency",
    "keywords:frequency",
    "keywords:frequency",
    "keywords:frequency",
    "keywords:frequency"
  ],
  "kwargs": [
    {"relation": "at least", "keyword": "ستة", "frequency": 1},
    {"relation": "at least", "keyword": "مائة", "frequency": 1},
    {"relation": "at least", "keyword": "ست", "frequency": 1},
    {"relation": "at least", "keyword": "مئة", "frequency": 1},
    {"relation": "at least", "keyword": "ستمئة", "frequency": 1},
    {"relation": "at least", "keyword": "ستمائة", "frequency": 1}
  ],
  "lang": ["ar"]
}

Evaluation Methodology & Metrics

To evaluate the models, we adopted a comprehensive methodology combining both explicit and implicit evaluation techniques. Explicit evaluation involved using automated scripts to assess whether instructions were strictly followed, focusing on elements such as correct formatting and specific word usage. Implicit evaluation addressed more nuanced linguistic expectations, such as maintaining the intended response language and avoiding repetitive patterns.

Additionally, we utilized scoring metrics introduced by Google in the IFEval framework, applying these metrics at both prompt-level and instruction-level granularity. These metrics were measured using strict criteria accuracy, which requires adherence to the provided instructions. The prompt-level score is notably harder, reflecting the user's viewpoint by asking, "Did I get everything I requested?" If a prompt included multiple requirements, failing to meet any single requirement would mean the user's request was not fully satisfied. In contrast, the instruction-level score is more lenient, allowing us to evaluate partial compliance.

In our analysis, we will emphasize the prompt-level strict accuracy as it provides the most rigorous assessment of a model's instruction-following capabilities.

Results & Analysis

We evaluated a broad range of LLMs on both the English IFEval benchmark and our newly introduced Arabic IFEval. This encompassed closed-source models (such as OpenAI's GPT series and Anthropic's Claude models) and open-source alternatives (including the Jais series, Meta’s LLaMA-2 variants, and various open bilingual models). Below, we present a summary of results for a representative subset of these models, comparing their prompt-level accuracy on both English and Arabic IFEval. Accuracy is reported using both strict and loose criteria, with values expressed as the percentage of prompts successfully completed.

Instruction Following Leaderboard Sample

Table 5. Sample Scores from Instruction Following Benchmark

RankModel NameArabic Prompt-lvl (%)English Prompt-lvl (%)
1claude-3.5-sonnet72.584.7
2gpt-4o-2024-08-0670.879.4
3gpt-4o-mini-2024-07-1868.176.9
4claude-3.5-haiku67.178.2
5Qwen/Qwen2.5-72B-Instruct67.383.5
6Qwen/Qwen2.5-32B-Instruct60.477.6
7google/gemma-2-27b-it59.476.1
8CohereForAI/aya-expanse-32b56.765.1
9CohereForAI/c4ai-command-r7b-12-202456.474.9
10meta-llama/Llama-3.3-70B-Instruct58.288.2

Upcoming Work

As part of our work we will be adding and updating more leaderboards to the Arabic-Leaderboards Space as our internal work progresses. In upcoming releases, we expect to publish a leaderboard for visual question-answering across multiple tasks, powered by camel-bench and kitab from our collaborators at MBZUAI.