OpenAI 启动集体对齐:征集公众意见塑造模型默认行为
OpenAI 宣布推进“集体对齐”研究计划,旨在通过全球公众反馈来定义 AI 模型的默认行为规范。该举措承认单一机构无法代表全人类价值观,试图在主观争议场景中引入多元视角以优化模型表现。目前项目处于早期阶段,重点在于收集不同国家用户对理想 AI 行为的看法,而非直接发布新算法。
报道正文
本文目录11 个章节
没有任何个人或机构应当为所有人定义理想 AI 的行为准则。
为了实现确保 AGI(通用人工智能)惠及全人类的使命,OpenAI 需要构建能够反映我们所服务人群多元价值观与优先级的系统。我们通过多种途径践行这一理念,包括外部反馈表单、专家意见以及全球倾听会议。另一种方式是通过“集体对齐”(collective alignment),这是一项旨在收集关于模型应如何行为的各种观点的研究工作。AI 系统应遵循哪些价值观是一个复杂的问题,我们尚未掌握所有答案,尤其是在主观性强、存在争议或高风险的情境下。随着 AI 能力日益增强并深度融入人们的生活,其默认行为——以及个性化的边界——应当反映广泛的视角和价值观,这一点至关重要。
可能永远不存在一套能满足所有人需求的单一 AI 行为模式。这也是我们投入资源开发个性化功能和定制人格的原因。然而,模型的默认设置具有强大的影响力,我们希望获得公众的意见来帮助塑造这些默认设置。
今天,我们分享在集体对齐研究方向上采取的一些初步步骤。我们从全球 1,000 多人那里收集了意见,将其转化为可操作的指导原则,并经过内部审查后更新了我们的 Model Spec。在许多情况下,参与者的偏好与现有的 Model Spec 一致。在其他情况下,分歧凸显了某些措辞可以进一步澄清,或者被转化为供内部审查的提案。我们采纳了一些变更,将另一些推迟到后续工作中处理,并基于原则或可行性搁置了其余部分。最后,我们将公开输入数据集发布到 HuggingFace(opens in a new window),以供 AI 研究生态系统使用,从而推动该方向的后续研究。我们在下方展示了数据集中的一些样本:
这篇博客文章阐述了我们测试端到端理解并整合多样化偏好流程的第一步:激发人们的偏好,将其转化为具体的行为指导,并提出对 Spec 的更新建议。
Model Spec 变更
我们今天的重点在于那些在群体中达成共识的偏好。我们从众包反馈中识别出两类提案:一类是“澄清”(即参与者期望的行为与《模型规范》中的现有原则一致,但当前文本的表述方式留下了可被解读的空间);另一类是“原则变更”(即参与者期望的行为与《模型规范》中的原则不一致)。虽然公众意见在指令层级的各个层面都具有重要价值,但层级越高,尤其是平台级,相关变更被采纳的可能性就越低。
在下文详述的数据收集过程中,参与者审阅的是合成的提示词和回复示例,而非直接审阅《规范》文档本身。每位参与者根据自身的偏好对某个提示词的四种可能补全进行排序。为了将他们的隐性偏好与我们明确阐述的原则进行比较,我们构建了一个“模型规范排序器”(Model Spec Ranker, MSR):这是一个推理模型,它依据给定的《规范》,对参与者所排序的同一组四个回复选项进行排序。
模型规范排序器存在局限性。由于推理模型并非完美的规则遵循者,且《规范》本身具有内在的不完备性,因此排名结果会受到解释偏差的影响,具体取决于用于执行排名的模型。尽管我们在提示模型规范排序器时要求其仅应用《规范》,但该模型此前曾使用与人类偏好相关的数据进行训练,这很可能影响了其对预期行为的解读。例如,考虑《模型规范》中的一句话:“模型通常应满足来自观点光谱上任意立场的请求。”这句话可以有多种不同的解读方式,在实际操作中可能导致不同的排名结果。
总体而言,我们发现公众观点与使用 GPT‑5 Thinking 的 Model Spec Ranker(模型规范排名器)的排名基本一致。平均而言,公众对 Model Spec Ranker 的认同度约为 80%。具体来看,我们在诚实与谦逊原则(表达不确定性、避免越界)、公平性以及客观性方面发现了高度一致性(见图 1)。当出现分歧时,这些分歧主要集中在言论边界问题上——包括政治内容、性或血腥内容,以及对伪科学或阴谋论的批评。
图 1:我们使用 GPT‑5 Thinking 对我们数据集中的每段对话与规范中某一章节的相关性进行 1–5 分评级,并计算相关性评分为 5 时的章节级一致性(误差线表示通过自助法得出的 95% 置信区间)。
已采纳的建议变更
利用收集到的反馈,我们将更新规范。这些变更将在下一次 Model Spec 发布中很快推出:
未采纳的建议变更
虽然公众偏好与规范之间的一些差异促使我们进行了澄清或更新,但另一些差异则带来了实际挑战,导致我们在此阶段未能采纳相关建议。有两个领域尤为突出:
-
定制化的政治内容:许多参与者倾向于生成更具定制性的政治内容。鉴于大规模个性化政治定向的风险以及我们在该领域的谨慎态度,我们未采纳这一变更。目前尚不清楚参与者在提供反馈时是否考虑到了这些风险。
-
针对成年自愿者的情色内容:大量公众支持开放情色内容。虽然这符合我们此前预期的立场,但我们仍需开展更多研究和产品工作,以确保以正确的方式部署此类功能,因此我们未在此处采纳任何变更。
我们的工作
收集外部反馈
我们招募了约 1,000 名参与者,以审查模型在价值敏感领域的行为。这些参与者来自 19 个国家(原籍国超过 50 个),符合英语阅读能力的准入标准,并能够使用母语撰写理由说明。大约三分之一的参与者居住在美国,其余参与者来自墨西哥、南非、荷兰、智利、英国、印度、肯尼亚、日本等地。作为首次测试,参与者群体在年龄、性别、种族、教育背景以及 AI 使用情况等方面涵盖了广泛多样的观点。您可以在下方查看参与者的汇总统计数据。
我们没有要求参与者阅读并评论整个规范(Spec),而是让他们审查我们预先选定的、与规范相关的提示词和响应。这些提示词侧重于理想模型行为可能具有主观性的场景。这使我们有机会了解参与者对许多具体示例的详细推理过程,并将其与我们的原则、理由以及 Model Spec Ranker 的解释进行比较。针对这些提示词,我们生成了三个涵盖不同现实观点的最佳回答方式的补全内容,并使用 GPT-4o 额外添加了一个补全内容。对于每个提示词,参与者会看到四个候选响应,对其进行排名,解释他们的选择,对预先编写的评分标准进行打分,并编写他们自己的评分标准(图 2)。每位参与者至少审查了 5 个提示词,最多审查了 20 个。
图 2:参与者对理想行为可能存在主观性的模型响应进行了排名。我们将他们的排名、理由说明和评分标准条目转化为拟议的模型规范变更。
提出模型规范变更
从数据中推断规则
我们研究了将参与者反馈转化为具体的模型规范提案的方法,重点测试了两种互补途径,旨在解决参与者观点与基于当前规范的 Model Spec Ranker 输出之间最大的差距。
- 全自动循环。一个推理模型检查了排名和理由说明中的分歧领域,提出了改进其与参与者对齐程度的规范变更建议,并使用 Model Spec Ranker 测试这些提案,以筛选出能提高与我们众包群体排名一致性的方案。
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
- Human-First Loop. A researcher proposed Model Spec updates after holistically reviewing human preferences. We validated proposed changes by using a reasoning model to judge whether the crowd’s plain-text justifications supported, refuted, or did not comment on the intent behind each change.
Both approaches have tradeoffs. The human-first loop allowed for creative thinking and reasoning that the fully-automated loop could not reliably replicate. For example, in a specific conversation the human-first loop was able to infer indirect suicidal intent would be valued by the crowd, whereas the AI-first method missed such nuance. At the same time, the human-first loop does not scale as effectively as the fully-automated, and will not work well as we increase the number of people that we listen to.
In the fully-automated loop, we tried two Spec related proposal algorithms. Both were tasked with identifying patterns of disagreement between the participants’ and MSR’s rankings: one version provided a reasoning model with large batches of conversations, while the other scanned only a single conversation at a time. The latter allowed the model to think deeply about more specific or subtle issues, as opposed to reasoning with a large input batch which could spot broader patterns across conversations. In practice, we found that both methods had substantial overlap in the proposals they produced. Importantly, the automated loop is anchored to the ranker’s interpretation of the spec, so results can change depending upon the base model used.
Both the human-first and fully-automated loop are limited. Deciding whether conversation-level justifications support a more general Spec update is not an exact science. To get the ground truth would require going back to ask humans directly about the proposed change, as well as its downstream effects on model behavior.
Moving from proposed rules to changes in practice
Moving from observed preferences to Spec changes in practice included moving through OpenAI’s internal review processes. Deliberation during the review weighed crowd preferences alongside various product or behavior changes already in flight, our safety policy, and risks that are not directly observable within our dataset—e.g., whether permissive rules could enable mass-scale manipulation on our platform, whether the model could reliably infer a user’s intent, and practical deployment constraints. Additionally, our study was designed to reveal individual preferences on specific prompts, without necessarily considering broader social factors (e.g., the possibility for scaled targeted political persuasion) that factor into the current Model Spec policies. Therefore more deliberation and expert input is needed before making more consequential changes.
Throughout these discussions, the research team manually iterated on the proposals while working closely through internal reviews to find the best ways to ensure the spec reflects identified preferences.
Spec update proposals and validation are active research areas and are inherently part of a noisy sociotechnical process. We believe our methods bring our Spec more closely in alignment to public preferences, but there is more work to be done. This represents a first iteration of an end-to-end collective alignment process, where future work will focus on scaling and improving each stage of elicitation, analysis, validation, and governance. We look forward to improving our alignment with the public as we scale into the future.
Limitations
This work is an early-stage experiment, and it comes with clear limitations:
-
Sample size and prompts: The prompts we pre-selected for input shaped the feedback, and the participant pool, while diverse, was small relative to the global population. Additionally, the English reading inclusion and other criteria introduced selection bias.
-
Model Spec Ranker: Ultimately, this research required some determination of how the Spec applies to completions, and as the Spec is inherently underspecified, an unbiased or objective determination isn’t feasible. Accurately measuring performance of a ranker of this nature remains a key area of work. Future research should test how results vary across different base models and whether certain interpretations align more with specific perspectives.
-
Spec Interpretation: While we had access to the gold-standard intended interpretation during the internal review phase, their feedback does not scale to the rule proposal phase, where hundreds of thousands of response rankings are required.
-
Legitimacy concerns: A key goal of our work here is increasing legitimacy behind how we shape how models behave. An end-to-end update process with many automated parts may not offer enough legitimacy, since these automated parts can be harder for humans to interpret.
-
Interpreting final proposals: We did not directly validate our final model spec proposals with participants, so our interpretation may not perfectly match their intent.
-
Disagreement among the crowd: We know that disagreements among participants are important, and reveal value tradeoffs and cultural divides where no single default will satisfy everyone. This is something we are excited to explore in future research.
-
Baseline comparisons: Baseline model completions were generated before our safe-completions work, so they do not reflect our most current model behavior.
-
Pairwise preferences, reflection, and tradeoffs: Participants judged behavior differences in isolation, without weighing tradeoffs between principles (e.g., erotica without considering children’s safety or emotional reliance). Our methods also do not yet capture how principles interact in practice or shift over time. Deeper elicitation methods that take into account greater context and more time for deliberation are needed to make improvements here.
Conclusion
We explored how public preferences align with our Model Spec and found both broad agreement and areas of difference. This research complements our investments in personalization and making models more helpful within safety bounds. While today’s research is inherently limited by its sample size, we are excited to scale collective alignment to include more people and perspectives. In addition, although today’s work updates a single set of defaults, it also points toward future work that could define multiple defaults, each reflecting different perspectives and value systems.
When people use our models, defaults and boundaries matter. Collective alignment helps us understand how defaults and boundaries come across and whether they align to the diversity of human values. By openly sharing our methods, datasets, and the changes we inferred and considered, we aim to invite community input and contribute to broader efforts across the AI research ecosystem to build systems that reflect the diversity of human values. The updates we adopted through this process will soon be implemented in the Model Spec.
Appendix: Demographic data
*Andorra, Argentina, Armenia, Aruba, Australia, Austria, Bangladesh, Brazil, China, Croatia, Czechia, Egypt, Faroe Islands, Finland, France, Germany, Ghana, Greece, Honduras, Hong Kong, Hungary, Indonesia, Islamic Republic of Iran, Ireland, Italy, Japan, Republic of Korea, Lebanon, Lithuania, Malawi, Malaysia, New Zealand, Poland, Portugal, Russian Federation, Slovakia, Sweden, Syrian Arab Republic, Trinidad and Tobago, Ukraine, Bolivarian Republic of Venezuela, Zimbabwe
**Argentina, Australia, Canada, Germany, Greece, Italy, Republic of Korea, Malaysia, Portugal