Different questions measure different things.
“What’s the best mattress to buy?”
No brand supplied. Measures what the system introduces.“What do you think of [brand]?”
The name is supplied. Measures the response to a named entity.“What mattress suits a side sleeper?”
Purchase context is supplied. A separate condition.A protocol from discovery to choice.
The developing Brand Research Protocol separates unaided awareness, aided awareness, associations, differentiation, evaluation, concerns, consideration, comparison, and recommendation or stated choice. Each stage needs its own prompt and outcome definition.
The Mattress baseline covers unaided category discovery. It is not a completed full-protocol brand audit.
One answer is one observation.
Fresh contexts prevent a previous response from steering the next one. We freeze prompt wording, record each model and configuration, and interleave observations where feasible. Failed, refused, truncated, and missing responses remain visible.
Fifty observations is an initial design choice, not a universal threshold for reliability. A later dated replication is needed to study day-to-day variation.
A mention is not a recommendation.
We code any mention, an explicit first choice, and an explicit rank separately. First mention never automatically becomes first choice. Ambiguous answers, ties, and no recommendation are valid outcomes.
Raw responses stay attached to classifications. Proposals from alias matching require review, especially for ordinary words such as “Purple” and “Birch.” Dictionary changes trigger recoding; reviewer decisions retain a reason and raw-text hash.
Measure how much the sample can miss.
Per-model shares carry denominators and marginal 95% intervals. Reliability simulations for 1, 5, 10, 25, and 50 observations compare a new sample with a modeled latent leader. They retain ties and no-clear-choice cases.
Modeled leader error depends on assumptions, reviewed classifications, and the observed brand set. Unknown competitors and systematic model behavior can fall outside that calculation. We do not label it an observed error rate.
API models are proxies for consumer experiences.
We document the closest available family, exact API identifier, reasoning settings, search availability, and the limits of the match. Consumer apps may add routing, memory, retrieval, experiments, or personalization absent from the API.
Results are reported by exact model and condition first. An equal-weight model average describes the sampled systems, not the percentage of consumers using each system.
A publication gate protects the evidence.
- Freeze the protocol and review the cost.
- Approve and collect the observations.
- Review classifications against raw evidence.
- Analyze with the stated denominators and uncertainty.
- Review and release an explicit public artifact.
Private collection and draft analysis are never automatically exposed to public pages. Synthetic demonstrations remain labeled and separate.