Hugging Face 推出远程 VAE 解码方案

原标题:Remote VAEs for decoding with Inference Endpoints 🤗
48AI 研判分

Hugging Face 提出将扩散模型的 VAE 解码步骤委托给远程推理端点,以缓解消费级 GPU 在高分辨率生成时的显存压力。该方案旨在替代传统的设备卸载或分块处理,从而在不牺牲图像质量的前提下降低本地硬件门槛。目前代码已开源且承诺不存储用户数据,处于社区试点阶段。

报道正文

本文目录14 个章节

返回文章列表

使用 Inference Endpoints 进行解码的远程 VAE 🤗

发布于 2025 年 2 月 24 日 在 GitHub 上更新 点赞 41

hlky hlky Sayak Paul sayakpaul

(本文由 hlky 和 Sayak 共同撰写)

在使用潜空间扩散模型进行高分辨率图像和视频合成时,VAE 解码器可能会占用大量内存。这使得用户难以在消费级 GPU 上运行这些模型,除非牺牲推理延迟或采取其他妥协措施。

例如,采用卸载(offloading)策略会引入设备间数据传输开销,从而导致整体推理延迟增加。分块处理(Tiling)是另一种解决方案,它允许我们对输入的“块”进行操作。然而,这可能会对最终图像的质量产生负面影响。

因此,我们希望与社区共同探索一种思路——将解码过程委托给远程端点。

不会存储或追踪任何数据,且代码完全开源。我们对 huggingface-inference-toolkit 进行了一些修改,并使用了自定义处理器。

这一实验性功能由 Diffusers 🧨 开发。

目录:

入门指南

下面,我们将介绍三个我们认为远程 VAE 推理具有优势的使用场景。

代码

首先,我们创建了一个用于与远程 VAE 交互的辅助方法。

请从 main 分支安装 diffusers 以运行代码。pip install git+https://github.com/huggingface/diffusers@main

代码

PYTHON
from diffusers.utils.remote_utils import remote_decode

基本示例

这里,我们展示如何在随机张量上使用远程 VAE。

代码

PYTHON
image = remote_decode(
    endpoint="https://q1bj3bpq6kzilnsu.us-east-1.aws.endpoints.huggingface.cloud/",
    tensor=torch.randn([1, 4, 64, 64], dtype=torch.float16),
    scaling_factor=0.18215,
)

Flux 的用法略有不同。由于 Flux 的 latent 是打包(packed)格式,因此我们需要传入 height 和 width。

代码

PYTHON
image = remote_decode(
    endpoint="https://whhx50ex1aryqvw6.us-east-1.aws.endpoints.huggingface.cloud/",
    tensor=torch.randn([1, 4096, 64], dtype=torch.float16),
    height=1024,
    width=1024,
    scaling_factor=0.3611,
    shift_factor=0.1159,
)

最后,这是一个 HunyuanVideo 的示例。

代码

PYTHON
video = remote_decode(
    endpoint="https://o7ywnmrahorts457.us-east-1.aws.endpoints.huggingface.cloud/",
    tensor=torch.randn([1, 16, 3, 40, 64], dtype=torch.float16),
    output_type="mp4",
)
with open("video.mp4", "wb") as f:
    f.write(video)

生成

但我们希望在实际 pipeline 中使用 VAE 来生成真实的图像,而不是随机噪声。下面的示例展示了如何在 SD v1.5 中实现这一点。

代码

PYTHON
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
    variant="fp16",
    vae=None,
).to("cuda")

prompt = "Strawberry ice cream, in a stylish modern glass, coconut, splashing milk cream and honey, in a gradient purple background, fluid motion, dynamic movement, cinematic lighting, Mysterious"

latent = pipe(
    prompt=prompt,
    output_type="latent",
).images
image = remote_decode(
    endpoint="https://q1bj3bpq6kzilnsu.us-east-1.aws.endpoints.huggingface.cloud/",
    tensor=latent,
    scaling_factor=0.18215,
)
image.save("test.jpg")

这里还有一个使用 Flux 的示例。

代码

PYTHON
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-schnell",
    torch_dtype=torch.bfloat16,
    vae=None,
).to("cuda")

prompt = "Strawberry ice cream, in a stylish modern glass, coconut, splashing milk cream and honey, in a gradient purple background, fluid motion, dynamic movement, cinematic lighting, Mysterious"

latent = pipe(
    prompt=prompt,
    guidance_scale=0.0,
    num_inference_steps=4,
    output_type="latent",
).images
image = remote_decode(
    endpoint="https://whhx50ex1aryqvw6.us-east-1.aws.endpoints.huggingface.cloud/",
    tensor=latent,
    height=1024,
    width=1024,
    scaling_factor=0.3611,
    shift_factor=0.1159,
)
image.save("test.jpg")

以下是使用 HunyuanVideo 的示例。

代码

PYTHON
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel

model_id = "hunyuanvideo-community/HunyuanVideo"
transformer = HunyuanVideoTransformer3DModel.from_pretrained(
    model_id, subfolder="transformer", torch_dtype=torch.bfloat16
)
pipe = HunyuanVideoPipeline.from_pretrained(
    model_id, transformer=transformer, vae=None, torch_dtype=torch.float16
).to("cuda")

latent = pipe(
    prompt="A cat walks on the grass, realistic",
    height=320,
    width=512,
    num_frames=61,
    num_inference_steps=30,
    output_type="latent",
).frames

video = remote_decode(
    endpoint="https://o7ywnmrahorts457.us-east-1.aws.endpoints.huggingface.cloud/",
    tensor=latent,
    output_type="mp4",
)

if isinstance(video, bytes):
    with open("video.mp4", "wb") as f:
        f.write(video)

队列处理

使用远程 VAE 的一大优势在于,我们可以对多个生成请求进行排队处理。当当前的 latent 正在被解码时,我们即可将下一个请求加入队列。这有助于提升并发性能。

代码

PYTHON
import queue
import threading
from IPython.display import display
from diffusers import StableDiffusionPipeline

def decode_worker(q: queue.Queue):
    while True:
        item = q.get()
        if item is None:
            break
        image = remote_decode(
            endpoint="https://q1bj3bpq6kzilnsu.us-east-1.aws.endpoints.huggingface.cloud/",
            tensor=item,
            scaling_factor=0.18215,
        )
        display(image)
        q.task_done()

q = queue.Queue()
thread = threading.Thread(target=decode_worker, args=(q,), daemon=True)
thread.start()

def decode(latent: torch.Tensor):
    q.put(latent)

prompts = [
    "Blueberry ice cream, in a stylish modern glass , ice cubes, nuts, mint leaves, splashing milk cream, in a gradient purple background, fluid motion, dynamic movement, cinematic lighting, Mysterious",
    "Lemonade in a glass, mint leaves, in an aqua and white background, flowers, ice cubes, halo, fluid motion, dynamic movement, soft lighting, digital painting, rule of thirds composition, Art by Greg rutkowski, Coby whitmore",
    "Comic book art, beautiful, vintage, pastel neon colors, extremely detailed pupils, delicate features, light on face, slight smile, Artgerm, Mary Blair, Edmund Dulac, long dark locks, bangs, glowing, fashionable style, fairytale ambience, hot pink.",
    "Masterpiece, vanilla cone ice cream garnished with chocolate syrup, crushed nuts, choco flakes, in a brown background, gold, cinematic lighting, Art by WLOP",
    "A bowl of milk, falling cornflakes, berries, blueberries, in a white background, soft lighting, intricate details, rule of thirds, octane render, volumetric lighting",
    "Cold Coffee with cream, crushed almonds, in a glass, choco flakes, ice cubes, wet, in a wooden background, cinematic lighting, hyper realistic painting, art by Carne Griffiths, octane render, volumetric lighting, fluid motion, dynamic movement, muted colors,",
]

pipe = StableDiffusionPipeline.from_pretrained(
    "Lykon/dreamshaper-8",
    torch_dtype=torch.float16,
    vae=None,
).to("cuda")

pipe.unet = pipe.unet.to(memory_format=torch.channels_last)
pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)

_ = pipe(
    prompt=prompts[0],
    output_type="latent",
)

for prompt in prompts:
    latent = pipe(
        prompt=prompt,
        output_type="latent",
    ).images
    decode(latent)

q.put(None)
thread.join()

Available VAEs

[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]

Advantages of using a remote VAE

These tables demonstrate the VRAM requirements with different GPUs. Memory usage % determines whether users of a certain GPU will need to offload. Offload times vary with CPU, RAM and HDD/NVMe. Tiled decoding increases inference time.

SD v1.5

GPUResolutionTime (seconds)Memory (%)Tiled Time (secs)Tiled Memory (%)
NVIDIA GeForce RTX 4090512x5120.0315.60%0.031 (0%)5.60%
NVIDIA GeForce RTX 40901024x10240.14820.00%0.301 (+103%)5.60%
NVIDIA GeForce RTX 4080512x5120.058.40%0.050 (0%)8.40%
NVIDIA GeForce RTX 40801024x10240.22430.00%0.356 (+59%)8.40%
NVIDIA GeForce RTX 4070 Ti512x5120.06611.30%0.066 (0%)11.30%
NVIDIA GeForce RTX 4070 Ti1024x10240.28440.50%0.454 (+60%)11.40%
NVIDIA GeForce RTX 3090512x5120.0625.20%0.062 (0%)5.20%
NVIDIA GeForce RTX 30901024x10240.25318.50%0.464 (+83%)5.20%
NVIDIA GeForce RTX 3080512x5120.0712.80%0.070 (0%)12.80%
NVIDIA GeForce RTX 30801024x10240.28645.30%0.466 (+63%)12.90%
NVIDIA GeForce RTX 3070512x5120.10215.90%0.102 (0%)15.90%
NVIDIA GeForce RTX 30701024x10240.42156.30%0.746 (+77%)16.00%

SDXL

GPUResolutionTime (seconds)Memory Consumed (%)Tiled Time (seconds)Tiled Memory (%)
NVIDIA GeForce RTX 4090512x5120.05710.00%0.057 (0%)10.00%
NVIDIA GeForce RTX 40901024x10240.25635.50%0.257 (+0.4%)35.50%
NVIDIA GeForce RTX 4080512x5120.09215.00%0.092 (0%)15.00%
NVIDIA GeForce RTX 40801024x10240.40653.30%0.406 (0%)53.30%
NVIDIA GeForce RTX 4070 Ti512x5120.12120.20%0.120 (-0.8%)20.20%
NVIDIA GeForce RTX 4070 Ti1024x10240.51972.00%0.519 (0%)72.00%
NVIDIA GeForce RTX 3090512x5120.10710.50%0.107 (0%)10.50%
NVIDIA GeForce RTX 30901024x10240.45938.00%0.460 (+0.2%)38.00%
NVIDIA GeForce RTX 3080512x5120.12125.60%0.121 (0%)25.60%
NVIDIA GeForce RTX 30801024x10240.52493.00%0.524 (0%)93.00%
NVIDIA GeForce RTX 3070512x5120.18331.80%0.183 (0%)31.80%
NVIDIA GeForce RTX 30701024x10240.79496.40%0.794 (0%)96.40%

Provide feedback

If you like the idea and feature, please help us with your feedback on how we can make this better and whether you’d be interested in having this kind of feature more natively integrated into the Hugging Face ecosystem. If this pilot goes well, we plan on creating optimized VAE endpoints for more models, including the ones that can generate high-resolution videos!

Steps:

  1. Open an issue on Diffusers through this link.
  2. Answer the questions and provide any extra info you want.
  3. Hit submit!

Models mentioned in this article 4

More Articles from our Blog

[

guidediffusersquantization

Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

70 July 23, 2026](https://huggingface.co/blog/nunchaku-diffusers)[![](https://huggingface.co/blog/assets/peft-beyond-lora/thumbnail.png)

peftloraguide

104 June 18, 2026](https://huggingface.co/blog/peft-beyond-lora)

Community

christopher5106 Feb 24, 2025

The article is not so clear about some points:

  • "time" does mean "inference time" or "offloading time"
  • what do you mean by tiled memory / tiled time ?
  • is vae tiling always happening ?

· sayakpaul Article author Feb 25, 2025

I am not sure how this is unclear but in the interest of completeness.

is vae tiling always happening ?

We never mention this to be happening.

what do you mean by tiled memory / tiled time ?

Tiled memory / time mean we are applying tiling..

"time" does mean "inference time" or "offloading time"

Total round-trip time. If it meant anything else, it would have been specified like others.

ryg81 Feb 24, 2025

Do we need Pro account for this?

sayakpaul Article author Feb 24, 2025

No.

freykun Feb 25, 2025 • edited Feb 25, 2025

Hello there. Very nice implementation, works flawlessly, 2-4 seconds to decode, depending on image size.

Simple question, is there a possibility to create a local endpoint on other machine (not HF) and use it as vae-decode machine?

So a Comfy Ui implementation for example.

sayakpaul Article author Feb 25, 2025

Comfy implementation: https://github.com/kijai/ComfyUI-HFRemoteVae

You can host the endpoint on a local machine and use it as is as shown in the blog post if the input and output schemas match.

· freykun Feb 25, 2025

Alright, I'll give it another read. Thank you.

ngxson Feb 25, 2025

📻 🎙️ Hey, I generated an AI podcast about this blog post, check it out!

This podcast is generated via ngxson/kokoro-podcast-generator, using DeepSeek-R1 and Kokoro-TTS.

· sayakpaul Article author Feb 25, 2025

No audio.

eramth Feb 26, 2025

Very nice feature.

DigitalSolomon Feb 27, 2025

Great work! Just posted a video short to LinkedIn showcasing this. Check it outhttps://www.linkedin.com/posts/activity-7301008441123164161-rTau?utm_source=social_share_send&utm_medium=member_desktop_web&rcm=ACoAAAI2QycBC7JU4jV0Vw61G4AWBGaE_MlKUtU

Semii Apr 15, 2025

Can we get also a Wan Video 2.1 VAE for remote Decode?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here. Tap or paste here to upload images

· Sign up or log in to comment