Your vision model can't weigh food — and that's fine if you architect for it

Description and use

Vision models accurately identify food density but systematically overestimate mass, causing inaccurate calorie estimates. By splitting the task into model identification, verified database lookup, and user‑confirmed portion adjustment, the team built a friction‑free, trustworthy food scanner.

Problem

Food tracking apps suffer from friction because users must manually search for foods; vision models can identify food but systematically overestimate portion mass, leading to inaccurate calorie estimates.

Outcome

By separating density, mass, and user‑provided portion, the app achieved accurate calorie tracking with minimal user effort, turning a systematic model bias into a trusted workflow.

Workflow

• 1. Capture photo of meal
• 2. Vision model identifies dish and returns per‑100g nutrition
• 3. System queries database for verified macros of identified dish
• 4. Model provides a mass estimate (biased)
• 5. UI pre‑fills grams with estimate and lets user adjust in one tap
• 6. App calculates total calories and macros from user‑confirmed grams and database values

Evidence

On mass, they were consistently, directionally wrong. A chicken portion we weighed at 162g came back around 180g. Across the models we tested, overestimates ranged from roughly +10% to well over +100% on identical images.

Source

We ran reference meals with known ground truth (weighed, with composition calculated from reference data) across several models and compared average error.

Source