A web app that turns a photo of a restaurant menu into a protein, carbohydrate and fat estimate per dish, using Google Vision for the text and Gemini for the nutrition reasoning.
Problem
Deciding what to eat when you are tracking macros is a data problem with no data. Restaurant menus almost never carry nutrition information, and looking up a dish in a database requires the dish to have a name a database recognises, which "grandma's slow-braised anything" does not.
The information is on the menu already, in the description. The task is to read it and estimate from ingredients and method.
What it does
Capture or upload a menu photo. The app extracts the dish names and descriptions, and returns a macro estimate for each item, ranked against a target you set. You can then narrow to a dish and see the reasoning behind the estimate rather than just the number.
How it's built
Two model stages, deliberately separated. Google Vision does the optical character recognition, because it is excellent at it, cheap, and deterministic enough to cache. Gemini does the nutrition estimation, because that is a reasoning task over unstructured descriptions and not a lookup.
Splitting them matters more than it sounds: OCR is the flaky step in a real-world photo, and keeping it in a separate stage means a bad crop produces visibly bad text rather than a confidently wrong calorie count. It also means the expensive stage only ever sees clean strings.
The client is plain JavaScript with no framework. The interaction is a single upload, one result view, and a filter - enough to justify avoiding a build step entirely, which kept the whole thing deployable as static files with a thin serverless function holding the API keys.
Status
Deployed at menumacro.vercel.app. Estimates are ranges, not clinical values, and the UI says so - the useful output is a ranking of the menu, not a food diary entry.
Credits
- Google Cloud Vision API
- Google DeepMind, Gemini API
- Every restaurant photographer whose terrible lighting this project has now seen
