A Hugging Face engineer has trained a coding language model to produce watercolour-style paintings — not by fine-tuning a diffusion model, but by using reinforcement learning to reward the model for writing better draw() calls in Python. The result is a working pipeline that any practitioner with a modest GPU can reproduce, and it reframes what "image generation" can even mean.
Key takeaways
- Hugging Face's experiment uses TRL (Transformer Reinforcement Learning) and OpenEnv to train a code-generating LLM to paint watercolours by writing Python drawing commands, not by predicting pixels.
- The reward function blends two model-based judges — aesthetic and style proxies — meaning someone's trained taste is literally baked into the optimisation target.
- Three separate ablations (swapping the image pool, removing renderer noise, disabling the pairwise judge) all produced flat learning curves; only changing the trainer itself moved the metric.
- Every rollout's source code is published in a companion dataset, so the brushstroke logic is fully auditable — unlike the latent space of a diffusion model.
- The pipeline runs on consumer hardware, making it a viable low-cost alternative to fine-tuning large image-generation models for a specific visual style.
How the reward function doubles as an aesthetic manifesto
The core mechanic is disarmingly simple: the model writes Python code that calls a renderer, the renderer produces an image, and a reward signal tells the model how good that image looks. According to the Hugging Face Blog, two of the four reward terms are themselves neural models — one judging aesthetic quality, one judging watercolour fidelity. That means the reward function is not a rule; it is a proxy for a person's taste, frozen into weights.
This is worth sitting with. When a photographer chooses Ilford over Kodak film, or a painter reaches for Prussian blue, that preference is an act of authorship. Here, authorship is distributed: the engineer chose the reward models, the reward models encode someone else's aesthetic training data, and the code model learns to satisfy both. The resulting images carry the fingerprints of at least three layers of human judgement, none of them fully visible in the final brushstroke.
For creators who have spent time generating images on Charmloop and wrestling with prompt language to hit a specific painterly style, this pipeline offers a different lever entirely. Instead of describing an aesthetic in words, you could in principle encode it in a reward model — and let RL find the code that produces it.
What the flat ablation curves actually reveal
The most instructive part of the writeup is what didn't work. Three separate experiments — swapping the training image pool, removing noise from the renderer, and disabling the pairwise judge — each produced a flat reward curve. The run that finally moved the needle changed the trainer configuration, not the reward signal.
That is a useful negative result for anyone building custom RL pipelines for visual output. Reward shaping is the obvious place to iterate, but the training dynamics — learning rate schedules, rollout batching, KL penalty — may be the actual bottleneck. It's the kind of hard-won detail that rarely surfaces in polished model releases, and it makes this blog post more practically valuable than most.
The auditable brushstroke
Diffusion models are opaque by design: a latent vector becomes pixels through a process that resists line-by-line inspection. Code-based rendering inverts that entirely. Every painting in this experiment has a corresponding Python source file in the published rollouts dataset — you can read exactly which curves were drawn, in which order, with which opacity.
For artists concerned about process documentation — provenance, reproducibility, the ability to explain how an image was made — that auditability is a genuine structural advantage over latent diffusion. It also opens a path toward style transfer that works at the algorithmic level rather than the pixel level: share the code, not the weights.
Creators exploring the broader range of AI art models and styles will find this approach sits in its own category: slower to iterate than a prompt, but far more inspectable than a LoRA. The Hugging Face team has published the full TRL and OpenEnv setup, so the barrier to experimenting is lower than it looks — and the guides on Charmloop cover the prompting fundamentals that remain relevant even when your "prompt" is a reward function.
