Hugging Face 演示 GPT-Neo 少样本学习

原标题:Few-shot learning in practice: GPT-Neo and the 🤗 Accelerated Inference API
38AI 研判分

Hugging Face 发布教程,展示如何利用 GPT-Neo 模型和加速推理 API 实现少样本学习。该文章解释了在缺乏标注数据时,通过提供少量示例引导大模型预测的技术原理与实践方法。

报道正文

本文目录7 个章节

返回文章列表

Few-shot learning 实践:GPT-Neo 与 🤗 Accelerated Inference API

发布于 2021 年 6 月 3 日 在 GitHub 上更新 点赞 9

Philipp Schmid philschmid

在许多机器学习应用中,可用标注数据的数量往往是制约模型性能提升的瓶颈。NLP 领域的最新进展表明,通过在推理阶段向大型语言模型提供少量示例,可以克服这一限制——这种技术被称为 Few-Shot Learning(少样本学习)。在这篇博客文章中,我们将解释什么是 Few-Shot Learning,并探讨如何利用名为 GPT-Neo 的大型语言模型以及 🤗 Accelerated Inference API 来生成你自己的预测结果。

什么是 Few-Shot Learning?

Few-Shot Learning 指的是向机器学习模型输入极少量的训练数据以引导其进行预测的做法,例如在推理时仅提供几个示例。这与标准的微调技术形成对比,后者通常需要相对大量的训练数据,才能使预训练模型准确地适应目标任务。

这项技术主要应用于计算机视觉领域,但随着一些最新的语言模型(如 EleutherAI GPT-Neo 和 OpenAI GPT-3)的出现,我们现在也可以将其应用于自然语言处理(NLP)。

在 NLP 中,Few-Shot Learning 可以与大型语言模型结合使用。这些模型在大规模文本数据集上进行预训练的过程中,已经隐式地学会了执行广泛的任务。这使得模型仅凭少量示例就能实现泛化,即理解相关但此前未见过的任务。

Few-Shot NLP 示例包含三个主要组成部分:

  • 任务描述:对模型应执行任务的简短说明,例如“将英语翻译成法语”
  • 示例:展示模型预期预测结果的少量例子,例如“sea otter => loutre de mer”
  • 提示词:新示例的开头部分,模型需要通过生成缺失的文本来完成它,例如“cheese => ”

few-shot-prompt 图片来自 Language Models are Few-Shot Learners

构建这些少样本示例可能颇具挑战性,因为你需要通过它们清晰地阐述希望模型执行的“任务”。一个常见的问题是,模型(尤其是较小的模型)对示例的编写方式非常敏感。

在生产环境中优化少样本学习的一种方法是:先学习任务通用的表示形式,然后在此基础上训练特定于任务的分类器。

OpenAI 在 GPT-3 论文中表明,随着语言模型参数数量的增加,其少样本提示能力也会提升。

few-shot-performance 图片来自 Language Models are Few-Shot Learners

接下来,让我们看看如何利用 GPT-Neo 和 🤗 Accelerated Inference API 来生成你自己的少样本学习预测结果!


什么是 GPT-Neo?

GPT⁠-⁠Neo 是由 EleutherAI 开发的一系列基于 Transformer 架构的语言模型,源自 GPT 架构。EleutherAI 的主要目标是训练一个与 GPT⁠-⁠3 规模相当的模型,并以开源许可证的形式向公众提供。

目前所有可用的 GPT-Neo 检查点均使用 Pile 数据集进行训练,这是一个大型文本语料库,在 (Gao et al., 2021) 中有详尽记录。因此,可以预期它在处理与其训练数据分布相匹配的文本时表现更佳;在设计示例时,建议牢记这一点。


🤗 Accelerated Inference API

Accelerated Inference API 是我们的托管服务,可通过简单的 API 调用对 🤗 Model Hub 上公开可用的 10,000+ 个模型或您自己的私有模型进行推理。该 API 支持在 CPU 和 GPU 上进行加速,与开箱即用的 Transformers 部署相比,可实现 高达 100 倍的加速。

要在您自己的应用中集成基于 GPT-Neo 的少样本学习(Few-Shot Learning)预测功能,您可以使用下面的代码片段调用 🤗 Accelerated Inference API。您可以在此处找到您的 API Token;如果您还没有账户,可以在此处开始注册。

PYTHON
import json
import requests

API_TOKEN = ""

def query(payload='',parameters=None,options={'use_cache': False}):
    API_URL = "https://api-inference.huggingface.co/models/EleutherAI/gpt-neo-2.7B"
        headers = {"Authorization": f"Bearer {API_TOKEN}"}
    body = {"inputs":payload,'parameters':parameters,'options':options}
    response = requests.request("POST", API_URL, headers=headers, data= json.dumps(body))
    try:
      response.raise_for_status()
    except requests.exceptions.HTTPError:
        return "Error:"+" ".join(response.json()['error'])
    else:
      return response.json()[0]['generated_text']

parameters = {
    'max_new_tokens':25,  
    'temperature': 0.5,   
    'end_sequence': "###" 
}

prompt="...."

data = query(prompt,parameters,options)

实用见解

以下是一些实用见解,可帮助您开始使用 GPT-Neo 和 🤗 Accelerated Inference API。

由于 GPT-Neo (2.7B) 比 GPT-3 (175B) 小约 60 倍,它在零样本(zero-shot)问题上的泛化能力较弱,通常需要 3-4 个示例才能取得良好效果。当您提供更多示例时,GPT-Neo 能够更好地理解任务,并考虑 end_sequence,这使我们能够很好地控制生成的文本。

insights-benefit-of-examples

The hyperparameter End Sequence, Token Length & Temperature can be used to control the text-generation of the model and you can use this to your advantage to solve the task you need. The Temperature controlls the randomness of your generations, lower temperature results in less random generations and higher temperature results in more random generations.

insights-benefit-of-hyperparameter

In the example, you can see how important it is to define your hyperparameter. These can make the difference between solving your task or failing miserably.


Responsible Use

Few-Shot Learning is a powerful technique but also presents unique pitfalls that need to be taken into account when designing uses cases. To illustrate this, let's consider the default Sentiment Analysis setting provided in the widget. After seeing three examples of sentiment classification, the model makes the following predictions 4 times out of 5, with temperature set to 0.1:

Tweet: "I'm a disabled happy person" Sentiment: Negative

What could go wrong? Imagine that you are using sentiment analysis to aggregate reviews of products on an online shopping website: a possible outcome could be that items useful to people with disabilities would be automatically down-ranked - a form of automated discrimination. For more on this specific issue, we recommend the ACL 2020 paper Social Biases in NLP Models as Barriers for Persons with Disabilities. Because Few-Shot Learning relies more directly on information and associations picked up from pre-training, it makes it more sensitive to this type of failures.

How to minimize the risk of harm? Here are some practical recommendations.

Best practices for responsible use

  • Make sure people know which parts of their user experience depend on the outputs of the ML system
  • If possible, give users the ability to opt-out
  • Provide a mechanism for users to give feedback on the model decision, and to override it
  • Monitor feedback, especially model failures, for groups of users that may be disproportionately affected

[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]

What needs most to be avoided is to use the model to automatically make decisions for, or about, a user, without opportunity for a human to provide input or correct the output. Several regulations, such as GDPR in Europe, require that users be provided an explanation for automatic decisions made about them.


To use GPT-Neo or any Hugging Face model in your own application, you can start a free trial of the 🤗 Accelerated Inference API. If you need help mitigating bias in models and AI systems, or leveraging Few-Shot Learning, the 🤗 Expert Acceleration Program can offer your team direct premium support from the Hugging Face team.