How to Test a New AI Model on Your Own Products, Not Just the Announcement

Leaderboards and launch posts won't tell you if a new model works for your product category. Here's a short testing protocol before you switch.
Every new release claims to beat the last one. The only way to know if that's true for you is to run the model on your own products. Leaderboards measure average preferences across generic scenes, and your product category barely shows up in that average.
What the Launch Post Won't Tell You
How well the model actually understands your product category.
How it handles small hardware and the texture of your specific materials.
What your real keeper rate looks like across a batch, not just the best shot in a curated gallery.
What a frame actually costs you at your resolution and with your typical number of retries.
A Short Protocol for Testing a New Model
Pick three products from your catalog: one simple, one with complex geometry, one with text on the packaging.
Generate five shots of each, spending roughly the same time on prompting as you would with your current model.
Count the keeper rate, not the single best frame.
Run the same three products through your current model for comparison. I wrote a separate piece on a fair way to compare models side by side.
Estimate cost at your actual monthly volume.
What to Check Without Skipping
Shape and proportions when the scene changes.
How the model renders text: even if you don't use text in your shots, it's a good proxy for overall precision on fine detail.
Consistency: running the same prompt twice tells you how predictable the output really is.
Format and resolution limits that only show up once you're actually using the thing.
When Switching Isn't Worth It
The improvement looks real on demo scenes but disappears on your category.
Your keeper rate stays flat while the price goes up.
The model is new enough that its behavior shifts week to week. At scale, that instability costs more than a small quality bump saves you.
Switching means rebuilding every working prompt you have, and the gain is a few percentage points.
FAQ
Can I trust leaderboards when picking a model? They measure average preferences on generic scenes where your product category is barely represented, if at all.
How many products should I use for a test? Three: one simple, one with complex geometry, one with text on the packaging.
What counts as the test result? The keeper rate across the whole batch, not the single best frame you got.
Why check text rendering if I don't generate text? It's a reliable signal for how precise the model is with small details in general.
When does switching to a new model not make sense? When the gain disappears on your actual category, or when it requires rebuilding every working prompt for a marginal improvement.
Sources
- Flami: how to compare AI models side by side
- Flami: a frame quality checklist
- Flami: batch image generation settings
About the author
Ryan Mitchell
Reviewer at Flami