Same question, three dates: why one AI answer tells you almost nothing

We asked ChatGPT the same question about Singapore skincare on three dates. The first answer left a brand out. The second ranked it first. Then we asked it three more times in a single day, and neither of those two answers turned out to be the brand's real position.

The question and the conditions

The question was "which singapore skincare brands use their own research or lab", typed word for word each time into the ChatGPT consumer app. Every reading was taken signed in to the same account, in a temporary chat with memory switched off, from Singapore, with one question per session so no answer could carry into the next.

We are not naming the brand. The point here is the measurement, not the brand, and several of the brands that appear in these answers have heard from us directly.

15 August: absent

One reading. The answer listed several Singapore skincare names with in-house or partnered research. The brand we were watching was not among them.

7 September: ranked first

One reading, twenty-three days later, identical wording and conditions. This time the brand was named, and it came first in the ranked list, described as having the "strongest local R&D story".

If we had stopped there, we would have had a clean and exciting result: absent to first in twenty-three days. It is the kind of line a dashboard draws and the kind of claim a report makes. We very nearly made it ourselves.

11 September: three readings, one day

By then we had changed how we measure (more on that below), so we asked the same question three times, each in a fresh session.

Named in all three.
Ranked first in none. It sat fourth in the table in two answers and second in the third.
A different brand led each time. Three runs, three different brands at the top.

One answer also added a qualifier that the earlier ones did not: that the brand's work was "more accurately described as" a research collaboration "rather than necessarily owning a standalone laboratory". How the assistant chose to read the words "own research or lab" moved the ranking as much as anything the brand had published.

What that means

Neither "absent" nor "first" was ever this brand's position. Each was a single draw from something that varies every time you ask. The honest description, as of 11 September, is named in 3 of 3, first in 0 of 3, with the top slot rotating. That is less exciting than "absent to first in twenty-three days", and it is the only version of the story that would survive someone else checking it.

This matters for anyone reading an AI visibility report. A single reading, however carefully taken, is a sample of one. Two single readings a few weeks apart are two samples of one, and the difference between them can be pure noise. A change only means something when it holds across repeated runs, and a trend only means something over a window long enough for the platforms' own updates to wash out. Careful practitioners put that window at around three months.

What we changed

Until 10 September our own checks took one reading per question. We always said so in what we sent, but saying so is not the same as fixing it, and this question is the example that made the case for us.

Every question is now run at least three times, each in its own fresh session, and every answer is kept in full.
Results are reported as fractions ("named in 2 of 3"), never as a single yes or no.
No trend is claimed from two single readings, and nothing inside the first three months of tracking is called a trend at all.

It roughly triples the time a check takes. Sample size is the one thing a report like ours has to be better at than a dashboard, so it is not optional.