Truvio, measuring how brands surface in LLM answers
Deployed publicly. More than 500 audits run, 600 unique visitors, and 200 waitlist sign ups.
problem
A growing share of the questions that used to go to a search engine now go to a language model. Ask one to recommend something in a category and it names a handful of options, but unlike a results page there is nothing to inspect: no ranking, no sources, no way to see why those names and not others.
I wanted to find out whether that was measurable, and what it would take to measure it in a way that survived comparison between models.
approach
It takes a URL and works out what the brand is before asking anything. The first stage reads the site itself for context: what is being sold, to whom, in what category.
The second stage turns that context into prompts. They are generated per brand rather than drawn from a fixed list, because the question somebody actually asks about a running shoe is not the question they ask about a payroll tool, and one list would be measuring the wrong thing for nearly every subject.
The third stage runs those prompts against Claude, ChatGPT, and Gemini, then scores each response for how the brand surfaces: whether it appears at all, where in the answer, and how it is framed. One audit engine sits behind all three APIs and normalizes their responses into a single shape. That normalizing step is most of the work, and it is the only reason a result from one model can be held next to a result from another.
The interface is Next.js, the audit engine is FastAPI, and Redis holds completed audits between runs, because a set of model calls is slow and expensive enough that repeating it for a page refresh is not defensible.
tradeoffs
- choice One audit engine behind all three APIs, not a separate integration per model.cost Normalizing into a single response shape throws away detail that individual providers return, and every new model means extending the shared shape rather than writing an isolated adapter.
- choice Redis holds completed audits between runs.cost A cached audit goes stale and there is no invalidation beyond a time limit, so a result can be quietly out of date.
- choice Prompts are generated per brand rather than drawn from one fixed list.cost Two audits are no longer strictly comparable, because they were not asked the same questions. The alternative is scoring every brand against phrasing that only suits some of them, which is a cleaner number and a less useful one.
what broke
Model output is not deterministic. The same audit run twice came back with different orderings, so the early results could not be trusted and could not be compared against each other.
The fix was to stop treating one response as an answer: each prompt runs several times and the audit reports the spread rather than a single result. Audits got slower and considerably more honest.
what I learned
I treat any single model response as one sample now, not as an output.
The interesting engineering was not in calling the models. It was in making three different response shapes mean the same thing.