GCP Xeon C4 基准测试:嵌入吞吐提升 24 倍
Hugging Face 与 Intel 联合在 Google Cloud 上对第五代 Xeon (C4) 与上一代 (N2) 进行语言模型推理基准测试。结果显示,C4 在文本嵌入任务中吞吐量比 N2 高 10-24 倍,在文本生成中高 2.3-3.6 倍;考虑成本后,C4 在嵌入任务上的总拥有成本(TCO)优势达 7-19 倍。该数据支持在纯 CPU 环境中部署轻量级智能体 AI 工作负载的可行性。
报道正文
本文目录13 个章节
在 GCP 第五代 Xeon 处理器上对语言模型性能进行基准测试
发布于 2024 年 12 月 17 日 在 GitHub 上更新 点赞 7
Matrix Yao MatrixYao Intel Ke Ding kding1 Intel Ilyas Moutawwakil IlyasMoutawwakil
TL;DR: 我们在两款基于 Google Cloud Compute Engine Xeon CPU 的实例(N2 和 C4)上,对两个代表性的 Agentic AI 工作负载组件——文本嵌入(text embedding)和文本生成(text generation)进行了基准测试。结果显示,C4 在文本嵌入方面的吞吐量比 N2 高 10 倍至 24 倍,在文本生成方面的吞吐量比 N2 高 2.3 倍至 3.6 倍。考虑到价格因素,C4 的小时单价约为 N2 的 1.3 倍;因此,在文本嵌入方面,C4 相比 N2 拥有 7 倍至 19 倍的 TCO(总拥有成本)优势,在文本生成方面则拥有 1.7 倍至 2.9 倍的 TCO 优势。这些结果表明,完全在 CPU 上部署轻量级 Agentic AI 解决方案是可行的。
引言
人们普遍认为,人工智能的下一个前沿领域在于 Agentic AI(智能体 AI)。这种新范式采用 感知 - 推理 - 行动 流水线,将 LLM 复杂的推理能力和迭代规划能力与强大的上下文理解增强相结合。上下文理解能力由向量数据库和传感器输入等工具提供,从而创建更具情境感知能力的 AI 系统,使其能够自主解决复杂的多步骤问题。此外,LLM 的函数调用(function calling)能力使得 AI 智能体可以直接采取行动,远远超越了聊天机器人仅提供对话交互的功能。Agentic AI 为提升各行各业的生产和运营效率带来了令人兴奋的前景。

人们正在将越来越多的工具引入 agentic AI 系统,而这些工具大多在 CPU 上运行,这引发了一个担忧:在这种范式下,主机与加速器之间的流量开销可能不可忽视。与此同时,模型构建者和供应商正在开发更小但功能强大的小型语言模型(SLM),最新的例子是 Meta 的 1B 和 3B llama3.2 模型,它们具备先进的多语言文本生成和工具调用能力。此外,CPU 也在不断演进,开始提供更强的 AI 支持;Intel 高级矩阵扩展(AMX)作为一种新的 AI 张量加速器,已在其第四代 Xeon CPU 中引入。将这三条线索结合起来,探索 CPU 托管整个 agentic AI 系统的潜力——尤其是在使用 SLM 时——将非常有趣。
在这篇文章中,我们将对 agentic AI 的两个代表性组件进行基准测试:文本嵌入(text embedding)和文本生成(text generation),并比较 CPU 在这些组件上的代际性能提升。我们选择了 Google Cloud Compute Engine 的 C4 实例和 N2 实例进行对比。其背后的逻辑是:C4 由 第五代 Intel Xeon 处理器(代号 Emerald Rapids)驱动,这是 Google Cloud 上最新一代的 Xeon CPU,集成了 Intel AMX 以提升 AI 性能;而 N2 由 第三代 Intel Xeon 处理器(代号 Ice Lake)驱动,这是 Google Cloud 上一代的 Xeon CPU,仅具备 AVX-512 指令集而没有 AMX。我们将展示 AMX 带来的优势。
我们将使用 optimum-benchmark,即 Hugging Face 推出的用于多后端和多设备的统一基准测试库,来测量性能。该基准测试在 optimum-intel 后端上运行。optimum-intel 是 Hugging Face 的一个加速库,旨在加速 Intel 架构(CPU、GPU)上的端到端流水线。我们的基准测试用例如下:
- for text embedding, we use
WhereIsAI/UAE-Large-V1model with input sequence length 128, and we sweep batch size from 1 to 128 - for text generation, we use
meta-llama/Llama-3.2-3model with input sequence length 256 and output sequence length 32, and we sweep batch size from 1 to 64
Create instance
N2
Visit google cloud console and click on create a VM under your project. Then, follow the below steps to create a single 96-vcpu instance which corresponds to one Intel Ice Lake CPU socket.
- pick N2 in
Machine configurationtab and specifyMachine typeasn2-standard-96. Then you need setCPU platformas below image:
- configure
OS and storagetab as below:
- keep other configurations as default
- click
CREATEbutton
Now, you have one N2 instance.
C4
Follow the below steps to create a 96-vcpu instance which corresponds to one Intel Emerald Rapids socket. Please note that we use the same CPU core count between C4 and N2 in this post to ensure an iso-core-count benchmark.
- pick C4 in
Machine configurationtab and specifyMachine typeasc4-standard-96. You can also setCPU platformand turn on all-core turbo to make performance more stable:
- configure
OS and storageas N2 - keep other configurations as default
- click
CREATEbutton
Now, you have one C4 instance.
Set up environment
Follow below steps to set up the environment easily. For reproducibility, we list the version and commit we are using in the commands.
- SSH connect to instance
$ git clone https://github.com/huggingface/optimum-benchmark.git$ cd ./optimum-benchmark$ git checkout d58bb2582b872c25ab476fece19d4fa78e190673$ cd ./docker/cpu$ sudo docker build . -t <your_docker_image_tag>
We are in container now, do following steps:
$ pip install "optimum-intel[ipex]"@git+https://github.com/huggingface/optimum-intel.git@6a3b1ba5924b0b017b0b0f5de5b10adb77095b$ python -m pip install intel-extension-for-pytorch==2.3.10$ cd /workspace/optimum-benchmark$ pip install .[ipex]$ export OMP_NUM_THREADS=48$ export KMP_AFFINITY=granularity=fine,compact,1,0$ export KMP_BLOCKTIME=1$ pip install huggingface-hub$ huggingface-cli login, then input your Hugging Face token to access llama model
Benchmark
text embedding
You need update examples/ipex_bert.yaml in optimum-benchmark directory as below to benchmark WhereIsAI/UAE-Large-V1. We change numa binding to 0,1 because both N2 and C4 have 2 NUMA domains per socket, you can double check with lscpu.
--- a/examples/ipex_bert.yaml
+++ b/examples/ipex_bert.yaml
@@ -11,8 +11,8 @@ name: ipex_bert
launcher:
numactl: true
numactl_kwargs:
- cpunodebind: 0
- membind: 0
+ cpunodebind: 0,1
+ membind: 0,1
scenario:
latency: true
@@ -26,4 +26,4 @@ backend:
no_weights: false
export: true
torch_dtype: bfloat16
- model: bert-base-uncased
+ model: WhereIsAI/UAE-Large-V1Then, run benchmark: $ optimum-benchmark --config-dir examples/ --config-name ipex_bert
text generation
You can update examples/ipex_llama.yaml as below to benchmark meta-llama/Llama-3.2-3.
--- a/examples/ipex_llama.yaml
+++ b/examples/ipex_llama.yaml
@@ -11,8 +11,8 @@ name: ipex_llama
launcher:
numactl: true
numactl_kwargs:
- cpunodebind: 0
- membind: 0
+ cpunodebind: 0,1
+ membind: 0,1
scenario:
latency: true
@@ -34,4 +34,4 @@ backend:
export: true
no_weights: false
torch_dtype: bfloat16
- model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+ model: meta-llama/Llama-3.2-3BThen, run benchmark: $ optimum-benchmark --config-dir examples/ --config-name ipex_llama
Results and Conclusion
Text Embedding Results
The GCP C4 instance delivers approximately 10x to 24x higher throughput over N2 in the text embedding benchmark cases. 
Text Generation Results
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
Consistently, the C4 instance shows approximately 2.3x to 3.6x higher throughput over N2 in the text generation benchmark. Across batch sizes of 1 to 16, throughput is 13x better without significantly compromising latency. It enables concurrent query serving without needing to sacrifice user experience. 
Conclusion
In this post, we benchmarked 2 representative workloads of agentic AI on Google Cloud Compute Engine CPU instances: C4 and N2. The results show prominent performance boost thanks to AMX and memory capability improvements on Intel Xeon CPUs. Intel released Xeon 6 processors with P-cores (code-named Granite Rapids) one month ago, and it offers ~2x performance boost in Llama 3. We believe, with the new Granite Rapids CPU, we can explore to deploy light-weight Agentic AI solutions wholly on CPU, to avoid intensive host-accelerator traffic overheads. We will benchmark it once Google Cloud Compute Engine has Granite Rapids instance and report results.
Thanks for reading!