Why AI answers vary — and what that means for measurement

Ask an AI assistant the same question on two different days and you may get two different answers. Different providers answer differently. Different model versions answer differently. Even the same model can vary its phrasing, its list order, and its citations from one response to the next.

This variability is not a reason to give up on measurement. It is the reason measurement has to be done carefully.

An answer is a sample, not a verdict

A single AI response is evidence about that response at that time — nothing more. It is not a claim about every user, every provider, or every model version. Treating one answer as "what AI says about you" is like polling one person and calling it public opinion.

Serious measurement therefore works with a defined sample: a pinned set of questions, run across the assistants buyers actually use, with every response stored. The score describes that sample, and the report says so.

What pinning buys you

Because questions, scoring rules, and methodology versions are pinned, a later run can be compared to an earlier one on equal footing — where the pins allow it. Without pinning, "movement" could come from the questions changing, the providers changing, or the scoring changing, and nobody could tell the difference.

Pinning also disciplines us. When a comparison is not valid — because the methodology moved — we say the comparison is not valid instead of presenting a trend that is not there.

What variability does not excuse

Variability is not a license to shrug. Patterns across a recorded sample are meaningful: a business absent from 17 of 17 recorded recommendation answers is not unlucky — it is absent. A competitor cited in most decision-support answers is not a coincidence — it is a pattern worth understanding.

The craft is in holding both truths at once: each response is a sample, and the sample as a whole is evidence. That is what a baseline is for.

Frequently asked questions

If answers change constantly, why measure at all?

For the same reason anyone samples anything variable: a defined, recorded sample reveals patterns a single observation cannot. You cannot manage what you have never measured, and you cannot measure what you have never pinned.

Do you track which provider said what?

Yes. Responses are stored per provider, so the evidence shows which assistants named, cited, or described a business — and how that differs across them.

Can a score from March be compared to a score from June?

Only when the prompt suite, scoring rules, and methodology version are pinned compatibly across both runs. When they are, the comparison is reported with its methodology context. When they are not, we do not present the two numbers as a trend.

This note is for information only. Logres measurements describe recorded AI responses for a defined set of questions at a point in time; they are not a guarantee that any AI system will name, cite, or recommend a business in the future, and they do not predict rankings, traffic, leads, or revenue.