Choosing an LLM for Yolo: accuracy vs. speed vs. cost

Gemini 3 Flash agreed with our most accurate model on 87.8% of meals, at a quarter of the cost and less than half the latency. Here's how we measured that, and what it taught us about models, photos and user trust.

Why "accurate" is harder than it sounds

Turning a meal photo into nutrition data takes three steps: identify the foods, estimate the portions, and calculate calories and macros. One slip cascades: mistake rice for pasta, or 50 g for 150 g, and the whole meal is off.

For people tracking their weight, blood sugar or training, that's not a cosmetic bug. One obviously wrong answer is enough to make them doubt every answer that follows.

LLMs add their own complications: the same photo can return different results on different calls, and new model versions ship almost monthly. There's no standard benchmark for food recognition, and every model choice trades accuracy against cost and speed. So while building Yolo, our nutrition-tracking app, we built our own evaluation framework.

Our evaluation lab

We built our own evaluation platform that runs any combination of model, prompt and input setting against a test dataset. It compares accuracy, cost and latency side by side, with 95% confidence intervals so small samples don't look more certain than they are.

Two design choices mattered most:

• Judge answers the way users do. A wrong food is obvious; a portion that's 10% off usually isn't. So food names are matched strictly (synonyms like "grilled chicken" and "chicken breast, grilled" both count), while weights get a ±15–20% tolerance.
• Re-score without re-running. When a model gives an unexpected but acceptable answer, we update the acceptance criteria. Past runs are re-scored against them with no new API calls, so the test set gets more realistic with every round.

Under the hood: Next.js, TypeScript and Supabase (Postgres), calling the model APIs for text and vision, with built-in support for a variety of providers (Google, Anthropic, OpenAI and others). It's also the gate for every AI feature we build: no model or prompt change ships without passing through it.

A strong model as the reference

Weighed, hand-labeled meals are the gold standard, but they're slow to make. Our first set of 43 meals, each item photographed and weighed on a kitchen scale, took real effort. That's far too much to repeat for every model or prompt change.

So we used it to pick a reference instead. Against those meals, Gemini 3 Pro was the most accurate at naming foods, estimating portions and avoiding invented items. It became our "oracle": it labels the test photos, and faster, cheaper models are scored on how often they agree with it.

That scales far beyond what we can weigh by hand and catches regressions whenever a model changes. The trade-offs: no candidate can score above the reference, and its blind spots carry over, possibly including non-Western dishes. So it still needs regular checks against hand-labeled meals.

The experiment

In early 2026, we tested three candidate models against the Gemini 3 Pro reference. The prompt stayed the same; only the image changed: full resolution, or one of four compressed versions ranging from 1 MP at JPEG quality 40 to 5 MP at quality 80.

Model	Agreement, full resolution	Agreement, compressed	Cost per photo	Latency
Gemini 3 Flash (preview)	87.8%	46–54%	$0.0006	16.1 s
Gemini 2.5 Flash	41.9%	14–23%	$0.0001	18.3 s
Gemini 2.5 Flash-Lite	31.4%	23–33%	<$0.0001	10.8 s

Agreement is the share of meals where a model's answer matched the reference; the compressed column shows the range across all four versions. The reference itself costs $0.0025 per photo and takes 36.4 s.

What we learned

1. Gemini 3 Flash is the sweet spot. Its 87.8% agreement means about one meal in eight still differs from the reference: for someone logging three meals a day, roughly one every two to three days. At 16 seconds, it's fast enough for logging, where people snap, review and confirm anyway.
2. Don't compress the photos. Every compressed version cut Gemini 3 Flash's agreement by 34 to 42 points, even 5 MP at quality 80. Compression saves a little bandwidth and time but no money, since billed tokens were the same at every resolution. Our rule: upload full resolution, always.
3. Model generations matter enormously. Gemini 2.5 Flash reached 41.9%, less than half of Gemini 3 Flash, and Flash-Lite hovered around 30% throughout. Jumps like this, and regressions, stay invisible unless you test every time a model changes.
4. Better models need better input. Flash-Lite barely reacted to image quality, while Gemini 3 Flash was highly sensitive to it. This suggests that the more a model can get out of a good photo, the more it loses when detail is stripped away.

From numbers to product decisions

Errors aren't equal: how they feel matters as much as how often they happen. Our estimates for Gemini 3 Flash:

Error	How often (est.)	How it feels	What helps
Invented food item	~2% of responses	Alarming: "where did that come from?"	A confirmation step before logging
Wrong food	~5% of items	Obviously wrong, frustrating	Confidence display and easy correction
Portion off by more than 20%	~8% of items	Noticed by careful trackers	Show the weight, adjustable with a slider
Missed food	~3% of items	Annoying but fixable	Ask "Anything else on the plate?"

Invented items are rare but do the most damage to trust. That's the case for a confirmation step: users review what the AI found before anything is logged, turning a trust-breaker into a moment of engagement.

Food photos are personal, too: they can reveal diets, health conditions, cultural background and daily routines. Our test images were collected with explicit consent, and the evaluation looks only at food and portions. On our servers, meal photos are deleted right after processing, and users should always be able to review, edit and delete AI-generated logs.

What's next

• Beyond Gemini. We are also testing Anthropic's Claude, OpenAI's GPT models and open-weight models like Kimi and GLM via OpenRouter. The question: stick with one provider, or route each task to whichever model handles it best?
• Sharper prompts, smaller vocabulary. We'll compare prompt strategies (concise vs. detailed, with or without examples) and a curated food database, for example only the 200 most-logged foods. The hypothesis: fewer options mean fewer invented items and better portions for everyday meals, even if rare dishes suffer. The goal is to push Gemini 3 Flash past 90%.
• Chat and voice. Users will also be able to describe a meal instead of photographing it: "a big bowl of pasta with some chicken and a side salad." Same framework, new challenge: mapping vague descriptions to the right foods and portions.

Lessons for anyone building with LLMs

1. Build your own evals early. Public benchmarks won't tell you whether your feature works for your users.
2. Define "correct" the way users would. Be strict where errors are obvious and tolerant where they aren't.
3. Use a strong model as reference, and keep humans in the loop. A small hand-labeled set keeps the oracle honest.
4. Test inputs, not just models and prompts. In our tests, image compression cost almost as much accuracy as dropping back a model generation.
5. Re-run on every model release. Generational jumps can be huge, and so can regressions.
6. Translate metrics into experience. "87.8%" is abstract; "one mismatched meal every two to three days" drives decisions.

Building an AI feature and wondering whether it's good enough to ship? We're happy to share what we've learned, so get in touch.