I Replaced a 730 MB Cartoon Model with a 78 MB One — and It Got Better
I Replaced a 730 MB Cartoon Model with a 78 MB One — and It Got Better
I wanted a small, fast model that turns a photo into one illustration style — without needing a massive diffusion model for every picture. I used FLUX.2-Klein as the teacher to generate training pairs, and trained pix2pixHD as the student. After 195 epochs the samples looked great, but the model was silently failing on photos it hadn't seen. This is the story of how I found that, what I rebuilt, and where this kind of model hits its ceiling.
The first version was a 730 MB pix2pixHD generator that took a photo plus a 28-class label map and produced an illustration. It looked good in my training dashboard — but that dashboard was showing me memory, not skill. When I finally scored it on 99 held-out photos, the numbers told the truth: between epoch 60 and 190, it got better on training data and worse on new photos. No learning-rate decay, barely any augmentation, a massive generator for only 2,394 pairs. I'd spent 130 epochs making the model worse.
The rebuild was a 35 MB U-Net with skip connections, random crops centred on people, proper LR decay, and held-out scoring every epoch. It beat the 730 MB model on new photos immediately — 20x smaller, trained in 3.5 hours. I later widened it to 78 MB (v7), added a small face refiner for distant faces that were coming out smeared, and shipped the whole thing: two Python files, three weight files, 149 MB total. It runs in about 5 seconds on a desktop CPU. No FLUX, no GPU, no segmentation model at runtime. Along the way I tried — and failed at — a dozen things: more face data that didn't help, a 2x resolution trick that made things worse, a deeper architecture that scored the same, an edge loss that did nothing. Each failure taught me something. The most useful lesson: a small model can repaint a photo, but it cannot redraw it the way FLUX does. That told me to stop tuning.