Hugging Face 发布 AI 智能体伦理与社会影响综述
Hugging Face 团队发表文章,系统梳理了 AI 智能体(Agents)兴起背后的技术逻辑与伦理挑战。文章指出,随着 LLM 能力的快速提升,业界正转向具备自主行动能力的智能体,并探讨了其作为“多模态模型++”的训练范式及潜在社会风险。该综述旨在为开发者提供在构建智能体时所需的伦理框架与反思视角。
报道正文
本文目录25 个章节
AI Agent 已至。接下来该做什么?
发布于 2025 年 1 月 13 日 在 GitHub 上更新 点赞 88
Margaret Mitchell meg Avijit Ghosh evijit Sasha Luccioni sasha Giada Pistilli giadap
本文亦有简体中文版本 简体中文。
引言
LLM 能力的突然且快速提升——例如写出流畅的句子以及在 benchmark 上取得越来越高的分数——促使 AI 开发者和企业纷纷将目光投向下一步:哪项颠覆性技术即将登场?近期迅速兴起的一项技术便是“AI agent”,即能够根据部署者的目标在数字世界中执行动作的系统。当今大多数 AI agent 通过将大型语言模型(LLM)整合进具备多种功能的大型系统中构建而成。这一波新技术背后的核心理念在于:计算机程序不再需要作为人类控制的工具运行,局限于专门的任务;它们现在可以在无需人工干预的情况下组合完成多项任务。
这一转变标志着向能够在非确定性环境中创建特定情境计划的系统迈出了根本性的一步。许多现代 AI agent 不仅仅执行预定义的动作,而是被设计用于分析新情况、制定相关目标,并采取此前未定义的动作以实现目标。
在本文中,我们简要概述了 AI 智能体(AI agents)的定义,详细阐述了其中涉及的伦理价值观,并记录了 AI 智能体在收益与风险之间的权衡。随后,我们提出了实现未来的路径,旨在让 AI 智能体为社会带来尽可能大的益处。如需了解智能体的技术方面介绍,请参阅我们最近的开发者博客文章。若想了解在现代生成式 AI 出现之前撰写的关于智能体的介绍(其内容在很大程度上至今仍适用),请参阅 Wooldridge 和 Jennings, 1995。
我们的分析表明,随着系统自主性水平的提升,对人们构成的风险也随之增加:用户让渡的控制权越多,源自系统的风险就越大。尤其令人担忧的是,由推动 AI 智能体开发的同一类收益所引发的个人安全风险,例如使开发者免于预测系统可能采取的所有行动。这一问题进一步加剧,因为某些安全危害会为其他类型的危害打开大门——例如对隐私和安全性的危害——而对不安全系统的不当信任则会引发更多危害的滚雪球效应。因此,我们建议不要开发完全自主的 AI 智能体。例如,能够编写并执行自身代码的 AI 智能体(超出开发者控制的受限代码选项范围)将被赋予覆盖所有人类控制的能力。相比之下,半自主 AI 智能体可能具有利大于弊的优势,具体取决于自主性水平、系统可用的任务以及个人对其控制权的性质。接下来我们将深入探讨这些主题。
什么是 AI 智能体?
概述
我们为本通讯调研的所有近期推出的 AI 智能体均基于机器学习模型构建,更具体地说,它们利用大语言模型(LLMs)来驱动其行动,这对于计算机软件而言是一种全新的、新颖的方法。除了基于机器学习这一共同点外,当今的 AI 智能体与过去的智能体也存在相似之处,在某些情况下甚至实现了关于智能体可能形态的早期理论构想:具备自主行动能力、展现(被感知的)社交能力,以及在反应性行为和主动性行为之间进行适当平衡。
这些特征存在程度差异:不同的 AI 智能体拥有不同水平的能力,它们可以单独工作,也可以与其他智能体协同合作以达成目标。因此,我们可以说 AI 智能体的自主性(或智能体特性)有高低之分,而某事物在多大程度上属于“智能体”则可以在一个连续谱系上加以衡量。这种对 AI 智能体概念的流动性理解导致了近期关于 AI 智能体本质的混淆和误解,我们希望在此提供一些澄清。下表详细列出了 AI 智能体的不同层级。
| 智能体层级 | 描述 | 控制方 | 名称 | 示例代码 |
|---|---|---|---|---|
| ☆☆☆☆ | 模型对程序流程无影响 | 👤 开发者控制系统可执行的所有功能及其执行时机。 | 简单处理器 | print_llm_output(llm_response) |
| ★☆☆☆ | 模型决定基本控制流 | 👤 开发者控制系统可执行的所有功能;系统决定何时执行各项功能。 | 路由器 | if llm_decision(): path_a() else: path_b() |
| ★★☆☆ | 模型决定函数的执行方式 | 👤 💻 开发者控制系统可执行的所有功能及其执行时机;系统决定如何执行这些功能。 | 工具调用 | run_function(llm_chosen_tool, llm_chosen_args) |
| ★★★☆ | 模型控制迭代和程序延续 | 💻 👤 开发者定义系统可执行的高层功能;系统决定执行哪些功能、何时执行以及如何执行。 | 多步智能体 | while llm_should_continue(): execute_next_step() |
| ★★★★ | 模型编写并执行新代码 | 💻 开发者定义系统可执行的高层功能;系统控制所有可能的功能及其执行时机。 | 完全自主智能体 | create_and_run_code(user_request) |
Table 1. One example of how systems using machine-learned models, such as LLMs, can be more or less agentic. Systems can also be combined in "multiagent systems," where one agent workflow triggers another, or multiple agents work collectively toward a goal. Adapted from smolagent blog post, with changes tailored for this blog post.
From an ethics perspective, it is also useful to understand the continuum of autonomy in terms of how control is ceded from people and given to machines. The more autonomous the system, the more we cede human control.
Throughout this piece, we use some anthropomorphising language to describe AI agents, consistent with the language that is currently used to describe them. As was also noted in historic scholarship, describing AI agents using mentalistic language ordinarily applied to humans – such as having knowledge, beliefs, and intentions – can be an issue for appropriately informing users about system abilities. For better or worse, such language serves as an abstraction tool to gloss over more precise details of the technology. Understanding this is critical when grappling with the implications of what these systems are and the role they may play in peoples’ lives: The use of mentalistic language describing AI agents does not entail that these systems have a mind.
The Spectra of AI Agents
AI agents vary on a number of interrelated dimensions:
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
-
Autonomy: Recent “agents” can take at least one step without user input. The term “agent” is currently used to describe everything from single-step prompt-and-response systems (citation) to multi-step customer support systems (example).
-
Proactivity: Related to autonomy is proactivity, which refers to the amount of goal-directed behavior that a system can take without a user directly specifying the goal (citation). An example of a particularly “proactive” AI agent is a system that monitors your refrigerator to determine what food you are running out of, and then purchases what you need for you, without your knowledge. Smart thermostats are proactive AI agents that are being increasingly adopted in peoples’ homes, automatically adjusting temperature based on changes in the environment and patterns that they learn about their users’ behavior (example).
-
Personification: An AI agent may be designed to be more or less like a specific person or group of people. Recent work in this area (example1, example2, example3) has focused on designing systems after the Big Five personality traits – Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism as a “psychological framework” (citation.) for AI. At the end of this spectrum would be “digital twins” (example non-agentic digital twin). There are currently not agentic digital twins that we are aware of. Why creating agentic digital twins is particularly problematic has recently been discussed by the ethics group at Salesforce, among others (example).
-
Personalization: AI agents may use language or perform actions that are aligned to a user’s individual needs, for example, to make investment recommendations based on current market patterns and investments a user has made in the past.
-
Tooling: AI agents also have varying amounts of additional resources and tools they have access to. For example, the initial wave of AI agents accessed search engines to answer queries, and further tooling has since been added to allow them to manipulate other tech products, like documents and spreadsheets (example1, example2).
-
Versatility: Related to above is how diverse the actions that an agent can take are. This is a function of:
-
Domain specificity: How many different domains an agent can operate in. For example, just email, versus email alongside online calendars and documents.
-
Task specificity: How many different types of tasks the agent may perform. For example, scheduling a meeting by creating a calendar invite in participants’ calendars (example), versus additionally sending reminder emails about the meeting and providing a summary of what was said to all participants when it’s over (example).
-
Modality specificity: How many different modalities that an agent can operate in – text, speech, video, images, forms, code. Some of the most recent AI agents are created to be highly multimodal (example), and we predict that AI agent development will continue to increase multimodal functionality.
-
Software specificity: How many different types of software the agent can interact with, and at what level of depth.
-
Adaptibility: Similar to versatility is the extent to which a system can update its action sequences based on new information or changes in context. This is also described as being “dynamic” and “context-aware”.
-
Action surfaces: The places where an agent can do things. Traditional chatbots are limited to a chat interface; chat-based agents may additionally be able to surf the web and access spreadsheets and documents (example), and may even be able to do such tasks via controlling items on your computer’s graphical interface, such as by moving around the mouse (example1, example2, example2). There have also been physical applications, such as early-stage agents embodied in robots (example).
-
Request formats: A common theme across AI agents is that a user should be able to input a request for a task to be completed, without specifying fine-grained details on how to achieve it. This can be realized with low-code solutions (example), with human language in text, or with voiced human language (example). AI agents whose requests can be provided in human language are a natural progression from recent successes with LLM-based chatbots: A chat-based “AI agent” goes further than a chatbot because it can operate outside of the chat application.
-
Reactivity: This characteristic refers to how long it takes an AI agent to complete its action sequence: Mere moments, or a much longer span of time. A forerunner to this effect can be seen with modern chatbots. For example, ChatGPT responds in mere milliseconds, while Qwen QwQ takes several minutes, iterating through different steps labelled as “Reasoning”.
-
Number: Systems can be single-agent or multi-agent, meeting needs of users by working together, in sequence, or in parallel.
Risks, Benefits, and Uses: A Values-Based Analysis
To examine AI agents through an ethical lens, we break down their risks and benefits according to the different values espoused in recent AI agent research and marketing. These are not exhaustive, and are in addition to the risks, harms, and benefits that have been documented for the technology that AI agents are based on – such as LLMs. We intend this section to contribute to the understanding of how to develop AI agents, providing information on the benefits and risks in different development priorities. These values might also inform evaluation protocols (such as red-teaming).
Value: Accuracy
- 🙂 Potential Benefits: By grounding in trusted data, agents can be more accurate than when operating from pure model output alone. This may be done via rule-based approaches or machine learning approaches such as RAG, and time is ripe for novel contributions for ensuring accuracy.
- 😟 Risks: The backbone of modern AI agents is generative AI, which does not distinguish between real and unreal, fact and fiction. For example, large language models are designed to construct text that looks like fluent language – meaning they often produce content that sounds right, but is very wrong. Applied within an AI agent, LLM output could lead to incorrect social media posts, investment decisions, meeting summaries, etc.
Value: Assistiveness
- 🙂 Potential Benefits: Agents are ideally assistive for user needs, supplementing (not supplanting) people. Ideally, they can help increase a user’s speed in completing tasks and their efficiency in finishing multiple tasks simultaneously. Assistive agents may also augment capabilities to minimize negative outcomes, such as an AI agent that helps a blind user navigate busy staircases. AI agents that are well-developed to be assistive could offer their users more freedom and opportunity, help to improve their users’ positive impact within their organizations, or help users to increase their reach on public platforms.
- 😟 Risks: When agents replace people – such as when AI agents are used instead of people at work – this can create job loss and economic impacts that drive a further divide between the people creating technology and the people who have provided data for the technology (often without consent). Further, assistiveness that is poorly designed could lead to harms from overreliance or inappropriate trust.
Value: Consistency
One idea discussed for AI agents is that they can help with consistency, as they can be less affected than people by their surrounding environment. This can be good or bad. We are not aware of rigorous work on the nature of AI agent consistency, although related work has shown that the LLMs many AI agents are based on is highly inconsistent (citation1, citation2). Measuring AI agent consistency will require the development of new evaluation protocols, especially in sensitive domains.
- 🙂 Potential Benefits: AI agents are not “affected” by the world in a way that humans are, with inconsistencies caused by mood, hunger, sleep level, or biases in the perception of people (although AI agents perpetuate biases based on the human content they were trained on). Multiple companies have highlighted consistency as a key benefit of AI agents (example1, example2).
- 😟 Risks: The generative component of many AI agents introduces inherent variability in outcomes, even across similar situations. This might affect speed and efficiency, as people must uncover and address an AI agent’s inappropriate inconsistencies. Inconsistencies that go unnoticed may create safety issues. Consistency may also not always be desirable, as it can come in tension with equity. Maintaining consistency across different deployments and chains of actions will likely require an AI agent to record and compare its different interactions – which brings with it risks of surveillance and privacy.
Value: Efficiency
- 🙂 Potential Benefits: A selling point of AI agents is that they can help people to be more efficient – e.g., they’ll organize your documents for you, so you can focus on spending more time with your family or pursuing work you find rewarding.
- 😟 Risks: A potential drawback is that they may make people less efficient, as trying to identify and fix errors that agents introduce – which may be a complex cascade of issues due to agents’ ability to take multiple sequential steps – can be time-consuming, difficult, and stressful.
Value: Equity
AI agents may affect how equitable, fair, and inclusive situations are.
- 🙂 Potential Benefits: AI agents can potentially help “level the playing field”. For example, a meeting assistant might display how much time each person has had to speak. This could be used to promote more equal participation or highlight imbalances across gender or location (example).
- 😟 Risks: The machine learned models underlying modern AI agents are trained on human data; humans data can be inequitable, unfair, exclusionary and worse. Inequitable system outcomes may also emerge due to sample bias in data collection (for example, overrepresenting some countries).
Value: Humanlikeness
- 🙂 Potential Benefits: Systems capable of generating human-like behavior offer the opportunity to run simulations on how different subpopulations might respond to different stimuli. This can be particularly useful in situations where direct human experimentation might cause harm, or when a large volume of simulations help to better solve the experimental question at hand. For example, synthesizing human behavior could be used to predict dating compatibility, or forecast economic changes and political shifts. Another potential benefit currently being researched is that humanlikeness can be useful for ease of communication and even companionship (example).
- 😟 Risks: This benefit can be a double-edged sword: Humanlikeness can lead users to anthropomorphise the system, which may have negative psychological effects such as overreliance (citation), inappropriate trust, dependence, and emotional entanglement, leading to anti-social behavior or self-harm (example). There is concern that AI agent social interaction may contribute to loneliness, but see citation1, citation2 for nuances that may be gleaned from social media use. The phenomenon of uncanny valley adds another layer of complexity - as agents become more humanlike but fall short of perfect human simulation, they can trigger feelings of unease, revulsion, or cognitive dissonance in users.
Value: Interoperability
- 🙂 Potential Benefits: Systems that can operate with others can provide more flexibility and options in what an AI agent can do.
- 😟 Risks: However, this can compromise safety and security, as the more an agent is able to affect and be affected by systems outside of its more limited testing environment brings with it increased risk of malicious code and unintended problematic actions. For example, an agent that is connected to a bank account so that it can easily purchase items on behalf of someone would be in a position to drain the bank account. Because of this concern, tech companies have refrained from releasing AI agents that can make purchases autonomously (citation).
Value: Privacy
- 🙂 Potential Benefits: AI agents may offer some privacy in keeping transactions and tasks wholly confidential, aside from what is monitorable by the AI agent provider.
- 😟 Risks: For agents to work according to the user's expectations, the user may have to provide detailed personal information, such as where they are going, who they are meeting with, and what they are doing. For the agent to be able to act on behalf of the user in a personalized way, it may also have access to applications and information sources that can be used to extract further private information (for example, from contact lists, calendars, etc.). Users can easily give up control of their data - and private information about other people - for efficiency (and even more if there is trust in the agent); if there is a privacy breach, the interconnectivity of different content brought by the AI agent can make things worse. For example, an AI agent with access to phone conversations and social media posting could share highly intimate information to the world.
Value: Relevance
- 🙂 Potential Benefits: One motivation for creating systems that are personalized to individual users is to help ensure that their output is particularly relevant and coherent for the users.
- 😟 Risks: However, this personalization can amplify existing biases and create new ones: As systems adapt to individual users, they risk reinforcing and deepening existing prejudices, creating confirmation bias through selective information retrieval, and establishing echo-chambers that reify problematic viewpoints. The very mechanisms that make agents more relevant to users - their ability to learn from and adapt to user preferences - can inadvertently perpetuate and strengthen societal biases, making the challenge of balancing personalization with responsible AI development particularly difficult.
Value: Safety
- 🙂 Potential Benefits: Robotic AI agents may help save people from bodily harm, such as agents that are capable of diffusing bombs, removing poisons, or operating in manufacturing or industrial settings that are hazardous environments for humans.
- 😟 Risks: The unpredictable nature of agent actions means that seemingly safe individual operations could combine in potentially harmful ways, creating new risks that are difficult to prevent. (This is similar to Instrumental Convergence and the paperclip maximizer problem.) It can also be unclear whether an AI agent might design a process that overrides a given guardrail, or if the way a guardrail is specified inadvertently creates further problems. Therefore, the drive to make agents more capable and efficient - through broader system access, more sophisticated action chains, and reduced human oversight - conflicts with safety considerations. Further, access to broad interfaces (for example, GUIs, as discussed in “Action Surfaces” above) and humanlike behavior gives agents the ability to perform actions similar to a human user with their same level of control without setting off any warning systems – such as manipulating or deleting files, impersonating users on social media, or using stored credit card information to make purchases for whatever ads pop up. Still further safety risks emerge from AI agents’ ability to interact with multiple systems and the by-design lack of human oversight for each action they may take. AI agents may collectively create unsafe outcomes.
Value: Scientific Progress
There is currently debate about whether AI agents are a fundamental step forward in AI development at all, or a “rebranding” of technology that we have had for years – deep learning, heuristics, and pipeline systems. Re-introducing the term “agent” as an umbrella term for modern AI systems that share common traits of producing operations with minimal user input is a useful way to succinctly refer to recent AI applications. However, the term carries with it connotations of freedom and agency that suggest a more fundamental change in AI technology has occurred.
All of the listed values in this section are relevant for scientific progress; most of them are provided with details of potential benefits as well as risks.
Value: Security
- 🙂 Potential Benefits: Potential benefits are similar to those for Privacy.
- 😟 Risks: AI agents present serious security challenges due to their handling of often sensitive data (customer and user information) combined with their safety risks, such as ability to interact with multiple systems and the by-design lack of human oversight for each action they may take. They might share confidential information, even when their goals were set by users acting in good faith. Malicious actors could also potentially hijack or manipulate agents to gain unauthorized access to connected systems, steal sensitive information, or conduct automated attacks at scale. For instance, an agent with access to email systems could be exploited to share confidential data, or an agent integrated with home automation could be compromised to breach physical security.
Value: Speed
-
On speed for users:
-
🙂 Potential Benefits: AI agents may help users to get more tasks done more quickly, acting as an additional helping hand for tasks that must be done.
-
😟 Risks: Yet they may also cause more work due to issues in their actions (see Efficiency).
-
On speed of systems:
-
As with most systems, getting a result quickly can come at the expense of other desirable properties (such as accuracy, quality, low cost, etc.). If history sheds light on what will happen next, it may be the case in the future that slower systems will provide better results overall.
Value: Sustainability
- 🙂 Potential Benefits: AI agents may theoretically help address issues relevant to climate change, such as forecasting the growth of wildfires or flooding in urban areas alongside the analysis of traffic patterns, then suggesting optimal routes and methods of transportation in real-time. A future self-driving AI agent may make such routing decisions directly, and could coordinate with other systems for relevant updates.
- 😟 Risks: Currently, the machine learning models AI agents are based on bring with them negative environmental impacts, such as carbon emissions (citation) and the usage of drinking water (citation). Bigger is not always better (example), and efficient hardware and low-carbon data centers can help reduce this.
Value: Trust
- 🙂 Potential Benefits: We are not aware of any benefits of AI agents relevant to trust. Systems should be constructed to be worthy of our trust, meaning that they are shown to be safe, secure, reliable, etc.
- 😟 Risks: Inappropriate trust leads people to be manipulated, and other risks detailed for Efficiency, Humanlikeness, and Truthfulness. A further risk stems from LLMs’ tendency to create false information (called “hallucinations” or “confabulations”): A system that is right the majority of the time is more likely to be inappropriately trusted when it’s wrong.
Value: Truthfulness
- 🙂 Potential Benefits: We are not aware of any benefits of AI agents relevant to truthfulness.
- 😟 Risks: The deep learning technology AI agents are based off of is well-known to be a source of false information (citation), which can take shape in forms such as as deepfakes or misinformation. AI agents can be used to further entrench such false information, such as by gathering up-to-date information and posting on several platforms. This means that AI agents can be used to provide a false sense of what’s true and what’s false, manipulate people’s beliefs, and widen the impact of non-consensual intimate content. False information propagated by AI agents, personalized for specific people, can also be used to scam them.
AI Agents at HF
At Hugging Face, we have begun introducing the ability for people to build and use AI agents in a number of ways, grounding in values as discussed above. This includes:
-
Our recent release of smolagents, which provides tools, tutorials, guided tours, and conceptual guides;
-
The AI Cookbook, which contains “recipes” for many kinds of agents:
-
Build an agent with tool-calling superpowers 🦸 using Transformers Agents
-
Agentic RAG: turbocharge your RAG with query reformulation and self-query! 🚀
-
Data analyst agent: get your data’s insights in the blink of an eye ✨
-
Have several agents collaborate in a multi-agent hierarchy 🤖🤝🤖
-
Our gradio agent user interface, to provide the front-end of agents you build;
-
Our gradio code-writing agent, which allows you to try out code ideas in real-time in a coding playground.
-
Jupyter Agent, an agent to write and execute code inside a Jupyter notebook.
Recommendations & What Comes Next
The current state of the art of AI “agents” point forward in several clear directions:
-
Rigorous evaluation protocols for agents must be designed. An automated benchmark may be informed by the different dimensions of AI agents listed above. A sociotechnical evaluation may be informed by the values.
-
Effects of AI agents must be better understood. Individual, organizational, economic, and environmental effects of AI agents ought to be tracked and analyzed in order to inform how they should be further developed (or not). This should include analyses of the effects of AI agents on well-being, social cohesion, job opportunity, access to resources, and contributions to climate change.
-
Ripple effects must be better understood. As agents deployed by one user interact with other agents from other users, and they perform actions based on one another’s outputs, it is currently unclear how their ability to meet the user’s goals will be affected.
-
Transparency and disclosure must be improved. In order to achieve the positive effects of the values listed above, and minimize their negative effects, it needs to be clear to people when they are talking to an agent and how autonomous it is. Clear disclosure of AI agent interactions requires more than simple notifications – it demands an approach combining technical, design, and psychological considerations. Even when users are explicitly aware they're interacting with an AI agent, they may still experience anthropomorphization or develop unwarranted trust. This challenge calls for transparency mechanisms that operate on multiple levels: clear visual and interface cues that persist throughout interactions, carefully crafted conversation patterns that regularly reinforce the agent's artificial nature, and honest disclosure of the agent's capabilities and limitations in context.
-
Open source can make a positive difference. The open source movement could serve as a counterbalance to the concentration of AI agent development in the hands of a few powerful organizations. Consistent with the broader discussion on the values of openness, by democratizing access to agent architectures and evaluation protocols, open initiatives can enable broader participation in shaping how these systems are developed and deployed. This collaborative approach not only accelerates scientific progress through collective improvement but also helps establish community-driven standards for safety and trust. When agent development happens in the open, it becomes harder for any single entity to compromise on relevant and important values like privacy and truthfulness for commercial gain. The transparency inherent in open development also creates natural accountability, as the community can verify agent behavior and ensure that development remains aligned with public interest rather than narrow corporate objectives. This openness is particularly important as agents become more sophisticated and their societal impact grows.
-
Developers are likely to create more agentic “base models”. This is clearly foreseeable based on current trends and research patterns, not a recommendation we are providing relevant to ethics. Current agent technology utilizes a collection of recent and older techniques in computer science – near-term future research will likely attempt to train agent models as one monolithic general model, a kind of multimodal model++: Trained to perform actions jointly with learning to model text, images, etc.
Acknowledgements
We thank Bruna Trevelin, Orion Penner, and Aymeric Roucher for contributions to this piece.
