TRL + OpenEnv 训练代码模型绘制水彩画
Hugging Face 展示如何利用 TRL 和 OpenEnv 框架,通过强化学习微调代码生成模型,使其能编写 JavaScript 调用 p5.brush 库绘制高质量水彩画。该案例验证了将视觉反馈转化为奖励信号以优化代码生成能力的可行性,为多模态智能体训练提供了新范式。
报道正文
本文目录12 个章节
使用 TRL 和 OpenEnv 训练代码模型绘制水彩画
发布于 2026 年 9 月 3 日 在 GitHub 上更新 点赞 77
8 月 23 日,Surya Narreddi 发布了一段由语言模型绘制的水彩画精美视频。该模型通过 p5.brush 编写 JavaScript,这是一个“为 p5.js 添加自然绘图工具”的库。这段视频迅速走红,截至撰写本文时已获得超过 150 万次观看。
该视频附带了一篇博客文章,解释了该项目早期且范围较窄阶段的训练过程(专注于特写花卉而非视频中的完整构图),遗憾的是尚未公开相关产物。他的网站表示将发布完整的技术报告,因此请务必关注他。最初的创意来自他本人,源于艺术与设计领域,而他在这些方面的技能远超于我。我的尝试侧重于工程方面,旨在以完全开源的方式复现这一方案,并发布所有组件。
注意: 如需了解由 Surya 本人讲述的项目背景,请观看他的论文演示视频。
在本文中,我尝试使用 TRL 和 OpenEnv 复现他的想法。参考池数据集、强化学习环境、训练脚本以及训练好的模型均已开源。
整个流程端到端运行于 Hugging Face 平台:
- 在 Jobs 上进行训练
- 将 RL 环境和评分模型作为 Spaces 部署
- 通过 Inference Providers 调用成对评判器(pairwise judge)
- 并将所有产物汇集到 Hub 上的 一个集合 中
一旦这两个 Spaces 启动,整个流程只需一条命令。复制 环境 和 评分模型,设置两个用于奖励混合的环境变量,然后启动:
hf jobs uv run train/watercolour_grpo.py --flavor h200 --timeout 48h --secrets HF_TOKEN -- \
--env-url https://<you>-watercolour-env.hf.space \
--model Qwen/Qwen3.5-35B-A3B --lora --all-linear --bf16 --gradient-checkpointing \
--subject 'a peach hibiscus' --references 4 \
--top-p 0.95 --top-k 20 \
--lr 5e-5 --lr-scheduler constant_with_warmup --warmup-steps 5 \
--scale-rewards none \
--steps 110 --n-episodes 240 --num-generations 8 \
--per-device-batch-size 1 --gradient-accumulation-steps 8 \
--max-completion-length 8192 \
--run-tag my-run --out <you>/watercolour-grpo --push-to-hub本文其余部分讲述了达成这一目标的过程,所有组件都在仓库中。
我严格遵循了原始博客的步骤,仅在绝对必要时才进行修改。我自己的每一个想法都记录在一个列表中,而不是直接融入实验,这个列表最终成为了文末的“接下来我想尝试的内容”,旁边是已发布产物的完整清单。如果你已经读过他的文章,那么其中的框架和奖励设计对你来说会很熟悉。新增的内容包括开源实现、人工标注的样本池,以及三种经过训练和比较的奖励混合方案,这些内容从 你需要构建的 RL 环境 开始介绍。
三次运行,每次对应一种奖励混合方案,并行演进。每一帧展示的是某个步骤的中位数画作。目前无需区分它们,文章后续会解释哪个运行对应哪种方案。
为什么人们喜欢它
这些画作看起来松散、不完美、充满手工感,而当下图像模型生成的恰恰是那种完美的(统计意义上的平均)图片。我猜测,这种反差正是视频走红的重要原因之一。这让我想起了生成式 AI 艺术的早期阶段——那时人们探索的重点在于媒介本身。DeepDream(2015)原本是一个调试工具,却被人们转化成了艺术作品;像 Edmond de Belamy(2018)这样的作品,源于艺术家对 GAN 潜能的试探;而像 Mario Klingemann 这样的艺术家,则在那几年间利用神经网络创作了 梦幻般的人像作品。
这个项目感觉更接近那些早期时光。在他的论文中,Surya 描述了通往这一成果的历程。他最初通过提示词驱动文生图模型,但在这种模式下,提示词是你唯一能操控的杠杆,且增加细节带来的控制力存在上限。直接训练模型本身则能走得更远。这个想法的另一半核心在于“媒介”。模型会编写一段约 150 行的 JavaScript 程序来绘制图像。模型的输出是代码。你可以阅读它、编辑它并重新运行,每一笔触背后的决策逻辑都清晰可见。此外,其风格源自一种限制:模型仅被允许使用 该库中的十个方法。更多细节见下文。
在同一时期,Anna Ridler 拍摄了数千张郁金香照片,手动为每一张打上标签,将数据集本身作为艺术品展出,随后基于此数据训练了一个模型。我在构建本项目时,通过 AI 智能体检索到的参考文献发现了她的作品,并为之着迷,因为本项目也做了非常类似的事情:先人工策展一组图像,然后基于这些图像进行训练。
基于审美偏好的强化学习
近期大多数针对语言模型的强化学习(RL)工作都依赖于可验证的奖励信号。例如,有标准答案的数学题、能通过测试的代码,或者正确与否明确且运行成本低廉的评分器。而这个项目更接近早期的例外情况——RLHF(基于人类反馈的强化学习),即模型从人类偏好中学习一个奖励模型。
在这里,奖励机制基于审美偏好。不存在所谓的 正确 答案。该项目真正要探讨的问题是:能否在“品味”之上实施强化学习?
根据他的博客定义以及我所构建的 RL 环境实现方式,奖励机制如下:
| term | weight | what it measures |
|---|---|---|
gate | 0.05 | the sketch compiles, paints something, does not cheat |
length | 0.05 | a soft push towards longer code snippets |
| pairwise judge | 0.60 | style, compared against references drawn from a pool |
| HPSv3 | 0.30 | aesthetic preference on the render |
HPSv3 is an open 7B preference model. Give it an image and a text description, and it returns a score for how much a person would prefer that image. It was trained on a large set of human choices between pairs of images, so its score is an average of many people's taste. The pairwise judge is Qwen3-VL-30B-A3B-Instruct, a general vision model called through HF Inference Providers. The pairwise judge sees the candidate painting next to four references randomly selected from the pool, guided by a written description of what to weigh (bleeds, translucent washes, soft edges), each comparison in both presentation orders, and its score is the share of comparisons the candidate wins. Its only standard is the pool, so its score is my taste, as encoded in those ratings.

Two of the four terms in the reward function are models. Both are proxies for someone's taste.
Those are the weights Narreddi converged on. The pool defines taste here. That moves the work from tuning hyperparameters to building the set that decides what is beautiful.
I trained three runs with this reward. They differ only in how the weight splits between the two model judges:
| run | pairwise judge | HPSv3 | role |
|---|---|---|---|
judge-led | 0.60 | 0.30 | the original mix, stopped at step 110 |
hps-led | 0.30 | 0.60 | the middle point, stopped at step 110 |
hps-only | 0.00 | 0.90 | the validation run, stopped at step 60 |
[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]
I started with hps-only to validate that the pipeline could learn at all. Once the reward was going up and the metrics were healthy, there was no reason to run it longer, so I launched the two longer runs instead. The question that the longer runs ask is how much of HPSv3's power can you hand to the pairwise judge? The more weight the judge carries, the more the reward means my taste instead of everyone's, and the harder it should be to climb. Incidentally, if you push it far enough or your style is too far from the average, the model could stop entirely.
Fortunately, it did not stop and both runs with the pairwise judge on learned too. The hand-rated pool can steer the policy, at least as far as the metrics and the final paintings show. The numbers are below.
Disclaimer. If we use a frontier model, it can already generate the JavaScript code that paints watercolour from a prompt. That's the starting point. The work here is about teaching a smaller model to do it combined with a person's own artistic preferences.
The RL environment you need to build
The environment wraps everything that sits between the model and the reward, including the JavaScript library that the model uses for painting, the system prompt that restricts it, the headless Chromium that renders each sketch, and the gate that rejects cheats.
The library does more of the work than it seems. p5.brush, by @acamposuribe, simulates a medium rather than drawing shapes: pigment bleeds past the edges of a fill, paper has texture, strokes have mass, flow fields drag brushwork around. When the model calls brush.fillBleed(0.25) it is deciding how far the ink runs.
Note. The author of p5.brush had been trying to teach a machine to paint long before any of this. In 2022 he made a generative art series that hides a diary about teaching p5.js to draw like a child: "It is barely able to use the crayons [...] It cannot follow simple commands. I'm done for today, very infuriating." The series was meant to have three pieces, and he made two. When Surya's video went viral, he quoted it, sharing that diary and saying this work is the third piece arriving on its own.
p5.brush exposes 47 methods. The prompt allows 10: scaleBrushes, noStroke, fill, noFill, fillBleed, fillTexture, beginShape, vertex, endShape and circle. What the other thirty-seven add, lines, hatching, custom brushes, would break the watercolour look. With these ten the model can only paint filled shapes, and the library adds the bleed to every one of them.

Part of one rollout's draw() and what it renders to. The comments are the model's own. Reward 0.864, 129 lines, step 22. The full source of every painting is in the rollouts dataset.
His blog post saved me a lot of time I could have wasted iterating on the prompt. A long API reference makes the model invent methods that do not exist, and his 200 GEPA iterations converged on a strict allowlist with no documentation. I saw the same failures and wrote the allowlist by hand. My only addition to that recipe is one sentence: paint each petal two or three times, a big pass first and a smaller, more opaque one inside it. This small change made my outputs a lot more colorful.
Note. If it's the first time you hear about GEPA, it is an automatic prompt optimizer. A language model reflects in plain words on where the current prompt failed and proposes a better one, and the loop repeats.
The gate is the final piece. The sketch has to compile, use the library instead of direct p5 calls, put real pigment on the canvas, and not try to trick the scorer, for example by writing text on the canvas.
The pool is the reward function
The pool consists of 178 paintings divided into two tiers based on my personal preferences, love and okay. All of them are actually generated by a model. Four open-weight models, called through Inference Providers, wrote p5.brush sketches, each one working from a real, openly licensed photo of a hibiscus from iNaturalist. A vision model gave written feedback on every sketch, over three refinement iterations. Every final render was then rated one at a time, by me, and 178 made the cut.
| generator | number of paintings |
|---|---|
| GLM-5.2 | 64 |
| Kimi-K3 | 57 |
| Qwen3-Coder-Next | 35 |
| Qwen3.5-122B-A10B | 22 |
Here I chose four different families of models to test their different styles. These four were the open models that produced a valid sketch every time in a quick reliability check, and two other candidates were dropped for failing it. If you want to produce your own pool, you might choose others.

The two tiers, as they actually look. Disagreeing with the rating is reasonable, somebody's judgement is now the reward function.
The tiers do real work in the reward. When the pairwise judge draws four references, half come from love and half from okay, so the policy always faces some rivals it can sometimes beat, and a win pays the same against either tier. This is one of my few deliberate changes: the original compares against its top tier only, and I kept the easier tier in the draw so a weak early policy still gets signal.
No human-made painting is in there, which is a real limitation. p5.brush is a niche library, and the human work that exists in it with accessible code is a handful of pieces, nowhere near what a training corpus would need, as his blog also notes.
The interesting idea here, as I've already discussed previously, is that the model will learn to imitate what the pool contains. If we point the environment at a different dataset, the reward would automatically change without touching a single line of code. For the dataset I generated and openly share, I also include the source sketch.
If we look closer at the judges, the two of them answer different questions. HPSv3 decides whether it is a flower, and the pairwise judge decides whether it is well painted in the style I chose.
Just one more yolo run
Before anything worked, there was a long stretch of flat reward curves. If you've tried to reproduce a research paper/blog without open artifacts, you probably can relate. Every run tested what I thought was a reasonable theory about what was wrong. A run takes a lot of time, so I queued the next one while still going through the last one's results. As always, starting from something easier that works and then building on top of that was the answer. A simple control task, with no browser and no judges, was the first attempt that learned, and the reason was that my learning rate was just too low.

Three experiments on the reward, three flat lines. I swapped the pool, removed renderer noise and turned the pairwise judge off, and none of it moved the curve. The run that moves changed the trainer configuration, so the difference is the trainer, not the reward mix.
Another change that cost me time to find was adjusting the LoRA parameters correctly. The usual target_modules list assumes a dense model, and Qwen/Qwen3.5-35B-A3B is a mixture of experts that names most of its projections differently, so the adapter was training ten layers out of forty. I solved this by changing it to all-linear, which reaches every linear layer. The routed experts in this architecture are fused tensors that even all-linear leaves frozen, but everything else gets an adapter, and that was enough to learn.
The fix was four changes in TRL's GRPOTrainer:
| setting | from | to | why |
|---|---|---|---|
| learning rate | 2e-5 | 5e-5 | the ceiling LoRA Without Regret uses for GRPO |
| scheduler | linear | constant_with_warmup | linear decay had spent most of the learning rate by mid-run, so the reward never took off |
scale_rewards | group | none | one gate rejection was shrinking every other advantage in the group |
target_modules | hand list | all-linear | reach every linear layer |
These four changes unlocked the first successful run (hps-only), with the reward clearly improving.
With that configuration, all three runs learn. Both judge runs were launched for 200 steps and stopped at 110, with the reward still climbing slowly. A step takes fifteen to eighteen minutes, and the comparison between mixes was already stable, so I stopped both to save compute. Mean group reward over the first and final third of each run:
| run | steps | first third | final third | Δ |
|---|---|---|---|---|
hps-only | 60 | 0.58 | 0.71 | +0.13 |
judge-led | 110 | 0.45 | 0.72 | +0.27 |
hps-led | 110 | 0.57 | 0.82 | +0.24 |

The three curves line up with how much weight my taste carries. The more the judge weighs, the lower the start and the noisier the climb. judge-led spent its first thirty steps nearly flat before it moved. It is the same move that solved the debugging. Shrink the problem until something learns, then add the hard parts back one at a time.
The pairwise judge term itself climbed in the two runs that used it. The model wins more comparisons against the pool as training advances, which is the claim hps-only could not make. No group in any run collapsed to identical rewards, the GRPO failure mode that kills the gradient. For the curious, the per-metric curves (HPSv3, paint coverage, entropy) are in the repository as CSV.
The full launch command, the hardware and the two environment variables that turn this into the other two runs are in the recipe.
What it actually learned
In every run, the first thing the model learned was to stop producing bad paintings, the near-blank canvases and shapeless washes that score under 0.3 in total reward. In hps-only, three quarters of the rise in the group mean comes from bad paintings becoming rare. In the judge runs the collapse is even steeper: rollouts under 0.3 fall from 99 to 16 across judge-led's thirds, and from 37 to 4 in hps-led.

This is why the obvious visual, the best painting of each step, shows almost no difference in hps-only. It moves +0.034 across the run while the median moves +0.155. The learning is visible in the middle of the distribution.
What the pairwise judge changes is the top. In hps-only, paintings got more reliable without getting better. The quality of the good ones added just +0.03 to the group mean, and once HPSv3 saw petals around a centre and a stem, it stopped asking for more pigment. With the judge on, the other half of the story appears. Better here means closer to the pool, so closer to what I rated as "good" or something I liked more. That added +0.12 in judge-led and +0.16 in hps-led, the best of each step rose too, and paint coverage doubled in both runs (0.11 to 0.23, and 0.13 to 0.30) where hps-only barely moved it. With a reference left to beat, a good painting can still get better, and the model starts being rewarded for using more pigment.
One more finding. The model ignores an explicit instruction, and it is right to. The system prompt asks for fifteen to thirty filled shapes. If we look at the real mean, it is between 7 and 9, and n_shapes barely correlates with reward in any run (+0.000, −0.14, +0.07). The policy is not rewarded for obeying that sentence, so it does not obey it.
There is also a ceiling on the hps-only route. If every rollout matched its good ones, that run's mean would sit at 0.771. Whether more steps would break it is an open question.
The paintings also show something that the tables miss. Within each run, they all look similar. As training advances, the rewards inside each group get closer together, and the median paintings in the opening video look like takes of the same flower. That is GRPO doing what it is designed to do with a pool built from one subject. The pool decides what counts as variety, the same way it decides what counts as quality. If the reward only pays for matching one flower, the model learns to paint that one flower. More diverse output would need a more diverse pool, and building one is more curation work. Surya's newer compositions are an example of this. Alex Yango's animal paintings are the same recipe with different choices in the pool. This is the biggest difference between an aesthetic reward and a maths grader. Behind the number there is a very human job, deciding what belongs in the reward set. Jason Liu's essay on taste says the general version in one line. AI shifted the bottleneck from making to noticing.
Surya closes his blog with some of his favourites. Instead of picking mine, below is a wall with the 178 paintings the reward scored highest across the two judge runs, the same number the reference pool holds, in no particular order. Open it and pick your own.

The reward's 178 favourites from the two judge runs, shuffled. Now pick yours.
To choose, you looked at many and kept a few, and that is exactly the job that built the reward of this project. Every painting of every run, with its sketch and its reward, is in the rollouts datasets, and browsable in this gallery.

The last step's median painting of each run, in the same order as the opening video. Same base model, same pool, three reward mixes, three styles.
Since the reward was partly based on my taste, it is fair to close with my verdict as a viewer. To my eye, judge-led is the run that ends up the most diverse and the most artistically interesting. hps-led paints convincing watercolours, but its best ones share a soft, wet-on-wet look that is almost a style of its own. hps-only converges the hardest, and most of its paintings settle on the same colours. You can judge for yourself in the gallery, which has every painting of every run, sortable by step and by reward.
Infra is hard
This project is mostly infra. A run needs a trainer, two Spaces, an inference router and a websocket to stay healthy for hours straight, and every piece that fails quietly turns into a wrong number somewhere else. Half the work is checking that the number you read matches what actually happened.
Failures of the infrastructure were entering the reward as zeros. A render that timed out or a scorer that did not answer scored the same as a bad painting, 0.0 inside the group. Across all my runs that was about 1.5% of rollouts, and in the worst run it reached 5.2%. That trains the model on noise, so those paths now return None and the rollout is excluded from the group.
I also found a bug in OpenEnv, and sent the fix upstream. The client keeps one persistent websocket, and a socket closed by the far end stayed cached, so every later call failed even though the environment was healthy. It cost me two half-finished runs to find it. The fix is submitted upstream, and the runs launched with it have been running clean since.

Two consecutive steps of the judge-led run, best four of each. Whether step 12's paintings look half a point worse is for you to decide.
The reward of a step depends on which references it drew. The pairwise judge samples four references per step, so every step faces a different set of rivals, and some draws are simply harder. GRPO itself is mostly safe, because advantages are computed inside the group and a hard draw moves the whole group together. The curve I was reading was not safe, and some of what looked like a bad step was just a hard draw. The image above is one example. Step 12 scored half a point below step 11 mostly because it drew the hardest references of the run, while the paintings themselves look close.
What it costs
Rounded numbers, and only for the runs that finished.
| piece | what it needs |
|---|---|
| trainer | 1 H200. 18 hours for 60 steps, about 34 for 110 |
| HPSv3 | an a100-large Space, up for the whole run |
| the environment | a cpu-upgrade Space, which renders comfortably in time |
| the pairwise judge | Inference Providers quota for Qwen/Qwen3-VL-30B-A3B-Instruct |
| the pool, one-off | openly licensed photos from iNaturalist, Inference Providers quota for the four generators, and rating is your own hours |
A step is eight rollouts and takes fifteen to eighteen minutes, of which 70 to 80% is rendering. A single render takes 69 to 96 seconds against a 90 second deadline. Part of that is expected, the Space has no GPU, so Chromium renders the WEBGL canvas in software and p5.brush's bleeds and textures are heavy pixel work. Even so, I expected it to be faster, and I have not found the full cause.
A scorer can cost more than the training that uses it: HPSv3 has to be up for the whole run, so pause the Space, or set its sleep timer, when the run ends.

Four paid services have to stay healthy at once. Only the Hub and trackio outlive the run.
Everything runs on HF Jobs, with the environment as a Docker Space and metrics in trackio.
What I would try next
The rule of this project was to reproduce the recipe with every resource open, not to improve it, so a list of untried ideas piled up along the way. These are the ones I would actually try, in order of how much evidence there is.
Multi-step, and letting the model see what it paints. This is the first thing I would try. The original blog trains single turn, so I trained single turn, and in this setup the model paints with its eyes closed. No image ever goes in, and the only feedback it gets is one number. The evidence that a feedback loop works is the pool itself. The reference paintings came from models iterating three rounds under a vision critic, and the later rounds are better. The material that defines the reward was made with a loop the policy never gets.
Smaller models. There is evidence that 35B is more than needed. In my side experiments a 4B already wrote valid sketches that passed the gate. If a 4B can learn this, the cost of the experiment drops by an order of magnitude.
Other ideas on the list are SFT on the pool sources before starting RL, rewarding pigment explicitly, moving the judge's reference mix from easy to hard as the run advances until only love remains, widening the ten-method allowlist for more visual range (my attempts on this crashed more sketches and broke the watercolour look), and checking how consistent the pairwise judge really is by scoring the same image twice.
And the method is not specific to flowers. Alex Yango painted animals with the same mechanism, and Brendan Hogan trained canvas animations against a pool of hand-rated clips. I had also played with something similar before using Simon Willison's pelican benchmark, where code is rendered to an image and scored.
And underneath all of it sits the question this project cannot close. 178 paintings made by models define what this trained model considers beautiful. The pool is the bottleneck, and it is the part of the pipeline with no principled answer.
What I changed from the original
For anyone reproducing this, there are two deliberate divergences from Narreddi's recipe, both argued earlier in the article:
- The pairwise judge draws its references half from
loveand half fromokay, instead of comparing against the top tier only, so a weak early policy still gets signal. - One small sentence of craft in the prompt, paint each petal two or three times, a big pass first and a smaller, more opaque one inside it.
The rest, all-linear for the LoRA, infrastructure failures returning None instead of 0.0, stopping the judge runs at step 110, are decisions I had to make along the way because his blog does not specify them. His implementation is not published, so I cannot tell whether they match his choices or diverge from them.
Everything is published
| artifact | where |
|---|---|
| the recipe, and how to reproduce it | 02-watercolour/ |
| the reference pool, with every source sketch | watercolour-reference-pool |
| the environment, ready to duplicate | watercolour-env |
| the HPSv3 scorer, ready to duplicate | watercolour-hpsv3 |
| the hps-only adapter and rollouts | watercolour-grpo-hps-only · watercolour-rollouts-hps-only |
| the judge-led adapter and rollouts | watercolour-grpo-judge-led · watercolour-rollouts-judge-led |
| the hps-led adapter and rollouts | watercolour-grpo-hps-led · watercolour-rollouts-hps-led |
| the gallery, every painting browsable | watercolour-gallery |
| the training curves | live: judge-led · hps-led · hps-only, and the CSV files in results/ |
| all of it | Paint with Code |
The per-rollout numbers in this article can be recomputed from the published datasets. Nothing here depends on any Space staying switched on.
The method and original idea are Surya Narreddi's. The library is Alejandro Campos Uribe's.