Hugging Face 评测 LoRA 替代方案:OFT 等 PEFT 技术对比
Hugging Face 团队发布深度评测,挑战 LoRA 在参数高效微调(PEFT)中的默认地位。文章通过代码示例与性能对比,验证了 OFT 等其他 PEFT 技术在特定场景下可能优于 LoRA,为开发者提供了更精细的选型依据。
报道正文
本文目录9 个章节
超越 LoRA:你能击败最流行的微调技术吗?
发布于 2026 年 6 月 18 日 在 GitHub 上更新 点赞 104
Benjamin Bossan BenjaminB Sayak Paul sayakpaul Marian Tietz hubnemo Kashif Rasul kashif

当你计划以参数高效的方式微调模型时,请跳出 LoRA 的思维定式
如果你想用自己的数据微调一个开源模型,你很可能对所谓的“参数高效微调”(Parameter-Efficient Fine-Tuning,简称 PEFT)感兴趣。这一术语指的是那些能显著降低模型微调内存需求的技术。尽管存在数十种此类技术,但几乎所有人都选择了其中一种名为“LoRA”的方法。在这篇博客文章中,我们将探讨 LoRA 是否真的是最佳选择,有哪些工具可以帮助你做出明智的决策,以及如何通过拓宽视野、超越 LoRA 来获益。
什么是 PEFT,以及何时需要它
目前有无数可用的开源模型,但它们往往无法完全满足你的特定用例需求。提示工程(Prompting)可能有所帮助,但通常还不够。与其从头训练一个新模型,不如考虑微调一个现有的模型。
然而,微调非常消耗内存:你通常需要足够的内存将完整模型加载多次。量化(Quantization)可以减少模型的内存占用,但量化后的模型无法直接进行微调。因此,出现了一组旨在减少微调所需内存的技术,被称为“参数高效微调”,即 PEFT。
使用 PEFT,你可以仅用一小部分内存来微调模型,甚至可以对量化模型进行微调。它还提供了其他优势,例如极小的检查点大小、更强的抗灾难性遗忘能力,以及能够从同一个基础模型服务多个微调版本的能力。
At Hugging Face, we develop the PEFT library, which implements many PEFT techniques behind a unified API and integrates well with the ecosystem, for example Transformers and Diffusers. It also supports multiple quantization methods, enabling further accessibility in parameter-efficient fine-tuning. PEFT provides a good starting point, whether you want to fine-tune on your own data or you're researching a new PEFT method.
LoRA: The queen of fine-tuning techniques 👑
One parameter-efficient fine-tuning technique that emerged early and proved to be quite effective is called “Low Rank Adaptation”, or short “LoRA”. It works by adding a handful of parameters on top of the base model, freezing the base model weights, and only training those few parameters.
Among all PEFT techniques, LoRA is by far the most popular. Here are a few estimates:
- Of a sample of 20,834 model cards on Hugging Face Hub that mention exactly one PEFT technique, 20,509 mention LoRA (98.4%).
- We checked which PEFT techniques are popular for image generation on an external site, too. Using a sample of 10,000 checkpoints, we found 7,111 to be LoRAs. The other identified PEFT techniques are LoCon (363) and DoRA (11, arguably a LoRA variant). That means 95.0% of PEFT checkpoints are LoRAs.
- Searching for the code snippet
from peft import <PEFT CONFIG>on GitHub (example GH query), 71.3% of results are for LoRA. The runners-up are LoHa (3.7%) and AdaLoRA (3.5%).
Although these estimates are not perfect, the conclusion is nonetheless that LoRA is almost certainly by far the most common PEFT technique.
This could just mean that LoRA works best for everyone, and this fact is reflected in its usage statistics. There is, however, another possibility: LoRA was one of the earlier, popular PEFT techniques. So maybe its usage became self-reinforcing: LoRA has the highest visibility, the highest number of tutorials/examples, and it has the best support in downstream packages. Thus LoRA's popularity feeds on itself.
This all leads to the question: Are we all leaving performance on the table by shunning better techniques? After all, there are countless researchers whose papers claim their technique beats LoRA. Isn't that sufficient proof that we should go beyond LoRA in favor of newer techniques?
Choosing the right PEFT technique based on paper results is problematic
There are dozens of papers that investigate fine-tuning techniques other than LoRA. Just in the PEFT library, there are more than 40 distinct PEFT techniques at the time of writing (and numerous more when counting variations of PEFT techniques). For almost all of them, you will find researchers claiming that their technique beats LoRA according to their benchmarks.
The trouble with these claims is that researchers are under pressure to provide results that beat the existing benchmark. Even without ill intent, this can bias the results, e.g. by spending less time tuning the alternative techniques compared to the one proposed by the researchers. One study found, for instance, that LoRA can match supposedly better PEFT techniques by tuning the learning rate.
Another complication is that each paper chooses a different set of PEFT techniques to compare to, and a different set of benchmarks to run. And even if the same technique is compared on the same benchmark, the code is often not available or not easy to run yourself, which makes results hard to reproduce.
Overall, it's difficult to figure out the PEFT technique that works best for you by only checking paper results. Therefore, you might be tempted to just go with the default, LoRA.
How we approach benchmarking in PEFT
At Hugging Face, we thought about how we can help users make informed decisions about which PEFT technique to use. With the PEFT library, we already provide a package that implements many PEFT techniques and exposes them with the same API. The next step is to provide benchmarks that can shed more light on the discussed issue.
We already had a benchmark that checks fine-tuning of LLMs on a math dataset for some time. This benchmark takes an LLM and fine-tunes it on chain-of-thought reasoning to produce the result to a mathematical question using a base model that is not instruction fine-tuned. The benchmark thus checks if the model can learn to perform mathematical reasoning and also to adjust the generated output to the expected format.
To extend our findings on another modality, we also added an image generation benchmark. This one tests whether the model can be fine-tuned to learn a new concept, a cat plushy, and generate it in new contexts without forgetting existing concepts.
![]() | ![]() |
|---|---|
| Left: Sample question and answer from the MetaMathQA dataset. Right: Sample image from the cat plushy dataset. |
All PEFT techniques are evaluated according to the exact same conditions: same base model, same dataset, same training and evaluation code, same hardware. As different users have different needs, we track more than just test performance. Besides VRAM usage, we track metrics like forgetting/drift, runtime, and checkpoint size. The results are designed to run on consumer hardware, and adding a new experiment only requires adding a new PEFT config and running a script.
Since we compare all PEFT techniques on equal footing and have no horse in the race, we believe that these benchmarks can draw an objective picture of how well different PEFT techniques work. We argue that if you have your own dataset, you can take a similar approach and take advantage of the PEFT library to evaluate multiple PEFT techniques.
Our findings: LoRA works well but is not necessarily the best choice
After finishing the benchmark runs, we found that although LoRA works well, other PEFT methods can beat it on one or multiple axes and should thus be considered. Check the image below that compares the performance of LoRA and five other PEFT techniques.
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
One way to interpret the results above is to think in terms of tradeoffs, for example: How well does the model perform on the test set vs how much memory is needed to train it? If a PEFT technique cannot be beaten on both of these metrics at the same time by any other technique, it is on the Pareto Frontier. In other words: If you want better test accuracy, you need more memory, and if you want more memory efficiency, you have to give up on accuracy.
Let's take a closer look at the results for the LLM Math dataset benchmark. When it comes to test accuracy vs memory, we find that LoRA is indeed on the Pareto frontier. It achieves 53.2% test accuracy and requires 22.6 GB of VRAM at the peak. There are, however, other PEFT techniques on the Pareto Frontier. For instance, BEFT achieves 32.9% test accuracy and requires only 20.2 GB of memory at max. On the other end, we have Lily, which achieves 54.9% test accuracy but requires 25.6 GB of memory. Depending on what's more important to you, you may conclude that LoRA does not present the best tradeoff for you.
It is also worth noting that even though LoRA does well on this task, we're not talking about vanilla LoRA. On one side, we have LoRA with rank stabilized initialization, which is a technique to scale the LoRA contribution differently from the default initialization and provides very good test accuracy (53.2%). On the other end, we have LoRA-FA, which uses an optimizer specialized for LoRA that freezes part of the LoRA weights and is thus more memory efficient (20.2 GB). Normal LoRA only achieves an accuracy of 48.1% at 22.5 GB memory and should thus be avoided in favor of the alternatives.
Next let's take a look at the image generation benchmark. In the Hugging Face Space, choose “image-gen” in the “Select Task” dropdown to show the results. The goal of the task is to learn a new concept, namely a cat plushy, and generalize it to new prompts.
![]() |
|---|
Cat plushy image created with LoRA fine-tuned on FLUX.2-klein-base-4B. |
For this task, the main metric is “dino similarity”, which measures how much a generated image resembles the picture from a holdout test dataset, with higher values being better. As always, we also want to keep an eye on memory usage. When plotting the Pareto Frontier of these two metrics, we find that LoRA is below that frontier. Let's get concrete numbers: LoRA achieves a similarity score of 0.697 whereas OFT achieves 0.708; in terms of memory, LoRA requires 9.97 GB, and OFT requires 9.01 GB. Therefore, OFT strictly dominates LoRA on these metrics.
Of course, you should also check the other PEFT methods that are close to the Pareto frontier, as metrics can be subject to small variations due to randomness. Also, you should explore other metrics: is runtime performance important to you or do you care about the size of the checkpoints? Choose the relevant metric from the dropdown and the picture can change considerably. For the image generation benchmark, do inspect the generated sample images to get a vibe of the fine-tuned model's capability.
Limitations
Objection: But the benchmarks favor one method over another!
One criticism that could be leveled at the PEFT benchmarks is that the choice of hyper-parameters may favor one technique over another. This is true, doing an exhaustive and fair hyper-parameter sweep with this many techniques is difficult. It is, however, very easy for everyone to contribute their own experiments to PEFT: If you believe that a specific PEFT technique can be improved by choosing different hyper-parameters, create a PR! We added instructions on how to do that. In a similar vein, if you want to contribute a completely new benchmark, reach out to us to discuss your idea.
Another problem with the benchmarks is that they may not fully reflect the capabilities of a specific PEFT technique. We make it possible to compare the techniques along many different dimensions and discover the best ones according to these tradeoffs. But it's impossible to capture all facets this way. For instance, one PEFT technique called Cartridges was developed to compress long prompts, which is not measured in the benchmarks. Other factors can also influence the choice, for instance:
- Depending on the PEFT technique, only certain layer types can be modified.
- Not all PEFT techniques support quantized base models (but we actively expand the support in
PEFT). - Some PEFT techniques allow merging of the adapter to reduce runtime overhead but others don't.
The benchmarks cannot fully lift the responsibility to do your research, but they can be reasonable pointers.
Objection: But llama.cpp/vLLM/... only supports LoRA
A limitation of using a PEFT technique other than LoRA is that they don't get the broad support in downstream packages that LoRA sees. For example, if you want to serve the model using vLLM, only LoRA checkpoints can be loaded. Thankfully, PEFT now supports converting other adapters into LoRA. That way, you can convert a non-LoRA checkpoint into LoRA and use it in vLLM or other downstream packages.
To test this, we converted an image adapter using the GraLoRA technique into a LoRA checkpoint. The test scores were virtually identical after conversion (similarity 0.702 → 0.694, 0.260 → 0.269). Below are test images for the prompt “sks cat at the beach”:
![]() | ![]() |
|---|---|
| Left: Image generated by GraLoRA. Right: Image generated by the same GraLoRA checkpoint converted to a LoRA checkpoint. The images quality is comparable. |
At the moment, we haven't implemented conversion for all PEFT techniques, but if there is demand, we will expand the support.
Conclusion and what you can do
While working on the PEFT package, we noticed that LoRA has a lot of momentum behind it, even though other PEFT techniques are potentially better. Therefore, we set out to add benchmarks to PEFT that could paint a more objective picture of how well different PEFT techniques perform on different metrics.
Given the results we found, we can confidently conclude that LoRA is not a bad choice at all, but there are potentially better choices. Especially when checking the image generation benchmark, LoRA is beaten by other techniques. We discussed that besides metrics, other considerations must be taken into account when choosing the right PEFT technique. However, even then, we are pushing PEFT further to achieve feature parity between LoRA and those other techniques.
Our journey is far from finished; we want to extend and improve the existing benchmarks, and we also plan to add more benchmarks in the future. We ensured that it is easy for the community to contribute, so if this is something you would like to do, please open an issue on the PEFT repository and let us know how you would like to contribute.
If you take away only one thing from this article, it is that LoRA should not be the automatic default when choosing a PEFT technique for your use case. Given the unified API provided by PEFT, changing from one PEFT technique to another is as easy as switching one config in your code. And even if you stick with LoRA, check out all the variants that are supported in PEFT: DoRA, rs-LoRA, LoRA-FA etc. Give these other techniques a try and you might be pleasantly surprised.
Example: Changing from LoRA to OFT using PEFT:
from transformers import AutoModelForCausalLM
-from peft import LoraConfig, get_peft_model
+from peft import OFTConfig, get_peft_model
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B", dtype="bfloat16")
-config = LoraConfig(target_modules=["q_proj", "v_proj"])
+config = OFTConfig(target_modules=["q_proj", "v_proj"])
model = get_peft_model(base_model, config)







