Using TRL and OpenEnv to teach a code-generation model the art of watercolor painting

On 23 August, Surya Narreddi shared a gorgeous video of watercolour paintings created by a language model. The model doesn't output images directly — it writes JavaScript using p5.brush, a library that "adds natural drawing tools to p5.js". The clip spread quickly and had passed 1.5 million views by the time I wrote this.

Alongside the video, Narreddi published a blog post describing the training behind an earlier, smaller stage of the project — close-up flowers rather than the complete scenes in the video. Unfortunately, no open artifacts were released. His site promises a full technical report is on the way, so he's worth following. The original concept is entirely his, coming from the art and design world where his abilities far exceed mine. My contribution lives on the engineering side: reproducing his recipe in the open, with every component published.

Note: for the backstory in Narreddi's own words, watch this video of his thesis.

In this article, I recreate his idea using TRL and OpenEnv. Everything is open: the reference pool dataset, the RL environment, the training scripts, and the trained models.

The entire pipeline runs on Hugging Face, from start to finish:

• training on Jobs
• the RL environment and the scorer model as Spaces
• the pairwise judge via Inference Providers
• all artifacts on the Hub, collected in one collection

Once the two Spaces are running, the whole recipe is a single command. Duplicate the environment and the scorer model, set two environment variables that control the reward mix, and launch.

The rest of this article tells the story of how I got there, and every piece lives in the repo.

I followed the original blog step by step, deviating only when absolutely necessary. My own ideas went into a separate list rather than into the experiments, and that list became "What I would try next" at the end, alongside the complete list of published artifacts. If you've already read his post, the framing and reward design will feel familiar. The new material here is the open implementation, the hand-rated pool, and three reward mixes that I trained and compared — that starts at The RL environment you need to build.

Three runs, one per reward mix, evolving in parallel. Each frame shows the median painting of a step.

Why people loved it

The paintings feel loose, imperfect, handmade — at a moment when image models produce flawless, statistically average pictures. I suspect that contrast is a large part of why the video took off. It reminded me of the early days of generative AI art, when the point was exploring the medium itself. DeepDream (2015) was a debugging tool that people turned into art, works like Edmond de Belamy (2018) came from artists probing what a GAN could do, and artists such as Mario Klingemann spent those years creating dreamy neural portraits.

This project sits closer to that era. In his thesis, Narreddi describes the road that led here. He began by prompting text-to-image models, where the prompt is your only lever, and extra detail buys extra control only up to a point. Training the model itself goes further. The other half of the idea is the medium: the model writes a program of roughly 150 lines of JavaScript that paints the image. That output is code — you can read it, edit it, rerun it, and the reasoning behind each brushstroke is visible. And the style emerges from a constraint: the model may use only ten of the library's methods. More on that below.

In that same period, Anna Ridler photographed thousands of tulips, hand-labelled every one, exhibited the dataset itself as the artwork, and later trained a model on it. I discovered her work through references that AI agents brought back while building this project, and it resonated because this project does something similar: curating a set of images by hand, then training against them.

RL over taste

Most recent RL work on language models relies on verifiable rewards — math problems with known answers, code that passes tests, or graders that are cheap, fast, and unambiguous. This project belongs to the older exception, RLHF, where a model learns a reward model from human preferences.

Here, the reward is aesthetic preference. There is no correct answer. The real question the project asks is whether you can do RL over taste.

The reward, as his blog defines it and as my RL environment implements it:

HPSv3 is an open 7B preference model. Given an image and a text description, it returns a score for how much a person would prefer that image. It was trained on a large set of human choices between image pairs, so its score reflects an average of many people's taste. The pairwise judge is Qwen3-VL-30B-A3B-Instruct, a general vision model called through HF Inference Providers. It sees the candidate painting beside four references randomly drawn from the pool, guided by a written description of what to value (bleeds, translucent washes, soft edges), each comparison shown in both presentation orders. Its score is the share of comparisons the candidate wins. Its only standard is the pool, so its score encodes my taste as expressed in those ratings.

Two of the four reward terms are models, and both are proxies for someone's taste.

Those are the weights Narreddi settled on. The pool defines taste here, which shifts the work from tuning hyperparameters to building the set that decides what counts as beautiful.

I trained three runs with this reward, differing only in how the weight splits between the two model judges:

I began with hps-only to confirm the pipeline could learn at all. Once the reward was rising and the metrics looked healthy, there was no reason to keep it going, so I launched the two longer runs instead. Those runs ask: how much of HPSv3's authority can you hand to the pairwise judge? The more weight the judge carries, the more the reward reflects my taste rather than everyone's, and the harder the climb should be. Push it far enough, or make your style too far from average, and the model could stop learning entirely.

Fortunately, it didn't. Both runs with the pairwise judge learned too. The hand-rated pool can steer the policy, at least as far as the metrics and final paintings show. The numbers are below.

Disclaimer. A frontier model can already generate JavaScript that paints watercolour from a prompt — that's the starting point. The work here is about teaching a smaller model to do it, combined with a person's own artistic preferences.

The RL environment you need to build

The environment wraps everything between the model and the reward: the JavaScript library the model paints with, the system prompt that constrains it, the headless Chromium that renders each sketch, and the gate that rejects cheats.

The library does more work than it appears to. p5.brush, by @acamposuribe, simulates a medium rather than drawing shapes: pigment bleeds past fill edges, paper has texture, strokes have weight, flow fields tug the brushwork around. When the model calls brush.fillBleed(0.25), it is deciding how far the ink runs.

Note. The author of p5.brush had been trying to teach a machine to paint long before any of this. In 2022 he made a generative art series that hides a diary about teaching p5.js to draw like a child: "It is barely able to use the crayons [...] It cannot follow simple commands. I'm done for today, very infuriating." The series was planned as three pieces; he made two. When Surya's video went viral, he quoted it, sharing that diary and saying this work is the third piece arriving on its own.

p5.brush exposes 47 methods. The prompt allows 10: scaleBrushes, noStroke, fill, noFill, fillBleed, fillTexture, beginShape, vertex, endShape and circle. What the other thirty-seven add — lines, hatching, custom brushes — would break the watercolour look. With these ten, the model can only paint filled shapes, and the library adds bleed to every one of them.

Part of one rollout's draw() and what it renders to. The comments are the model's own. Reward 0.864, 129 lines, step 22. The full source of every painting is in the rollouts dataset.

His blog post saved me hours I might have wasted iterating on the prompt. A long API reference makes the model invent methods that don't exist, and his 200 GEPA iterations converged on a strict allowlist with no documentation. I saw the same failures and wrote the allowlist by hand. My only addition to that recipe is one sentence: paint each petal two or three times, a big pass first and a smaller, more opaque one inside it. This small change made my outputs noticeably more colorful.

Note. If GEPA is new to you, it's an automatic prompt optimizer. A language model reflects in plain words on where the current prompt failed, proposes a better one, and the loop repeats.

The gate is the final piece. The sketch must compile, use the library rather than direct p5 calls, put real pigment on the canvas, and not try to trick the scorer — for example, by writing text onto the canvas.

The pool is the reward function

The pool consists of 178 paintings split into two tiers based on my personal preferences: love and okay. All of them were actually generated by models. Four open-weight models, called through Inference Providers, wrote p5.brush sketches, each working from a real, openly licensed photo of a hibiscus from iNaturalist. A vision model gave written feedback on every sketch across three refinement iterations. I then rated every final render one at a time, and 178 made the cut.

I deliberately picked four different model families to test their different styles. These four were the open models that produced a valid sketch every time in a quick reliability check; two other candidates were dropped for failing it. If you build your own pool, you might choose others.

The two tiers, as they actually look. Disagreeing with the rating is reasonable — somebody's judgement is now the reward function.

The tiers do real work in the reward. When the pairwise judge draws four references, half come from love and half from okay, so the policy always faces some rivals it can sometimes beat, and a win pays the same against either tier. This is one of my few deliberate changes: the original compares against its top tier only, and I kept the easier tier in the draw so a weak early policy still gets signal.

No human-made painting is in there, which is a genuine limitation. p5.brush is a niche library, and the human work that exists in it with accessible code amounts to a handful of pieces — nowhere near what a training corpus would need, as his blog also notes.

The interesting idea here, as I've discussed before, is that the model learns to imitate whatever the pool contains. Point the environment at a different dataset, and the reward changes automatically without touching a single line of code. For the dataset I generated and openly share, I also include the source sketch.

Looking closer at the judges, the two of them answer different questions. HPSv3 decides whether it is a flower, and the pairwise judge decides whether it is well painted in the style I chose.

Just one more yolo run

Before anything worked, there was a long stretch of flat reward curves. If you've tried to reproduce a research paper or blog without open artifacts, you can probably relate. Every run tested what I believed was a reasonable theory about what was wrong. A run takes a lot of time, so I queued the next one while still reviewing the last one's results. As always, the answer was to start from something easier that works and build up from there. A simple control task, with no browser and no judges, was the first attempt that learned — and the reason was that my learning rate