It is easy to count brand mentions in LLM answers, but hard to measure them accurately.
WhiskeyJack tested 240 LLM answers and found six traps in brand mention measurement: inflated citations, prompt bias, noise, and ranking errors.
Key Takeaways
- The article analyzes 240 answers from four web-search LLM engines across 48 fixed prompts to show that brand mention counting is easy, but accurate measurement is difficult.
- Six major traps distort results: raw citation counts, prompts that name brands, single-run noise, apparent agreement caused by prompt wording, verbose answers inflating mention share, and pooled rankings overweighting engines with more rows.
- With only 40 answers per software category, small week-to-week changes are likely noise; moves under about 10 percentage points should be treated as leads to check rather than findings.
- Better practice includes tagging brand-name prompts, counting citations once per domain per answer, repeating runs, reporting engine-specific and position-aware metrics, and using explicit weighting or bootstrap intervals.
On 7 November 2026, WhiskeyJack published 240 answers from four web-search large language model engines, or LLM engines — OpenAI, Anthropic, Google and Perplexity — to show six traps in LLM brand recommendation measurement. The data comes from a free weekly newsletter, Mention Impossible, and no brand mentioned is a client or sponsor. The text was drafted with an AI writing assistant, then edited by the author and checked against raw result tables.
How the 240-answer measurement was built
The panel has 48 fixed English prompts, eight per category across six software categories: customer relationship management, email marketing, project management, ecommerce platforms, website builders and AI writing tools. The first four prompts in each category are shared by all four engines.
| Engine | Model / setting | Prompts | Runs per prompt |
|---|---|---|---|
| OpenAI | gpt-6.1-sol | 24 shared | 1 |
| Anthropic | claude-sonnet-5-5 | 24 shared | 1 |
| gemini-3.8-flash | all 48 | 1 | |
| Perplexity | Agent API "fast" preset | all 48 | 3 |
The run returned 240 answers with no failures. The processing is deliberately simple: mentions use a case-insensitive, whole-word match against a reviewed dictionary of brands and aliases; unmatched candidate names go to a weekly review file. Rank is order of first appearance. Citations come from each API’s structured citation objects, normalised to a domain name. A citation is “owned” if the domain belongs to a tracked brand.
Six traps that changed the numbers
Trap 1: raw citation counts let one engine dominate. Across 240 answers, APIs returned 3,775 citations. Counting each domain once per answer leaves 1,404. Average raw citations per answer: Gemini 52.3, range 9 to 86; Claude 18.9, 12 to 28; ChatGPT 7.2, 3 to 12; Perplexity 4.4, 2 to 11. Vendor-owned sources are 15.8% of raw citations but 20.7% of unique-per-answer citations. The unique version answers “did this answer rely on this site?” rather than “how many times did the API list it?”
Trap 2: “alternatives to a brand” prompts reward that brand. Six prompts ask for alternatives to HubSpot, Mailchimp, Asana, Shopify, WordPress and Jasper, producing 34 answers. The excluded brand was named in 33 answers and named first in 28, because answers often restate the question before listing alternatives. In email marketing, Mailchimp was named in 21 of 40 answers, the category’s highest share. Two of eight email prompts contain “Mailchimp”: the alternatives question and a comparison involving Klaviyo, Mailchimp and Omnisend. Those two prompts produced 10 of Mailchimp’s 21 mentions. In the other 30 answers, ActiveCampaign leads 12 to Mailchimp’s 11 and Klaviyo’s 10. The first-named excluded brand appears in 6 of 6 prompts for Mailchimp and Jasper, 5 of 6 for HubSpot and Shopify, 3 of 4 for WordPress, and 3 of 6 for Asana. The counts come from the author’s figure drawn from raw result tables dated 7 November 2026. The fix is to tag prompts that contain brand names and keep them out of rankings. For rank, “first brand recommended” needs a classifier or manual labels, not a regular expression.
Trap 3: one run per prompt hides noise. Perplexity ran three times per prompt. It named the same brand first in all three runs on 42 of 48 prompts, ranging from 8 of 8 in email marketing to 6 of 8 in AI writing tools. Across three runs, 185 question-brand pairs appeared at least once: 131 in all three runs, 24 in two runs, 30 in one run. So 54 of 185, or 29%, did not appear in every run. A brand at the bottom of a list can disappear without anything changing. ChatGPT, Claude and Gemini ran once, so their noise is unmeasured, not absent. If budget allows only one engine with repeats, those repeats still show roughly how much movement to ignore.
Trap 4: apparent agreement can be a prompt effect. On the 24 prompts every engine answered, all four named the same brand first on 10. But 8 of those shared prompts name a brand. On those eight, the engines agreed 4 times. On the other 16, they agreed 6 times. Without the brand-named prompts, agreement is 6 of 16, a bit under 40%, closer to “the engines usually disagree on the top pick.”
Trap 5: verbose engines inflate share of mention. Share of mention counts an answer if a brand appears anywhere. Distinct brands per answer: Claude 6.3, Gemini 4.7, ChatGPT 3.8, Perplexity 3.3. A brand is more likely to be “mentioned” by Claude because Claude lists nearly twice as many brands as Perplexity. Report share per engine, then add a position-aware metric such as presence in the top three.
Trap 6: pooling silently weights the engine with the most rows. With three runs per prompt and all 48 prompts, Perplexity supplies 24 of the 40 answers in each category. Gemini supplies 8, ChatGPT 4 and Claude 4. A pooled share of mention is mostly a Perplexity number. That may be fine if Perplexity represents buyers, but it should not happen by accident. Either weight engines explicitly, for example equally or by estimated user share, or report each engine with its own sample size.
What the 240 answers say about noise
With 40 answers per category, one answer moves a share by 2.5 percentage points. A rough 95% margin for a share around 50% is about plus or minus 15 percentage points at sample size 40, and about plus or minus 18 at sample size 30. That is a normal approximation, a rough statistical method, and repeated runs inside the panel probably make true uncertainty larger. Most week-to-week moves in a panel this size are noise. Moves under about 10 percentage points can be treated as noise; larger moves are leads to check, not findings. A better approach is to publish intervals by bootstrapping, a resampling method, over prompts and runs.
Checklist, limits and next steps
Freeze the prompt panel; add new prompts as a new version and report them separately until they have a baseline. Tag brand-name prompts and keep them out of rankings. Count citations once per domain per answer and say so. Repeat runs on at least one engine to estimate noise. Report per engine, with sample size, before pooling anything. Prefer “named in the top three” or “recommended” over first-appearance rank. Keep mentions and citations separate: a brand can be cited without being named, and named without being cited.
Limitations: scope is 48 prompts, six software categories and English only. Engines were called through APIs with web search on; consumer apps may answer differently because of personalisation, other models or other search settings. This is one snapshot week, 7 November 2026, compared with the same panel a week earlier, 1 November. Only Perplexity ran multiple times per prompt. Brand matching is dictionary-based, so it misses brands not yet in the dictionary and cannot tell a recommendation from a passing mention. There is no significance testing; the margins are rough arithmetic. Grok is being added from the next weekly run and is not in these numbers.
The main lesson is that most of the work in measuring LLM brand recommendations is in the counting rules, not the API calls. Prompt wording, citation de-duplication, repeat runs, answer length and pooling weights each moved numbers in this sample by as much as week-to-week changes people tend to report. Making those choices explicit and publishing them makes numbers comparable from week to week.
Reference: WhiskeyJack, Mention Impossible: Method and weekly tables (2026), https://data.mentionimpossible.com/method/; originally published in Towards AI on Medium.
Meta description: Six measurement traps from 240 AI answers show how citation counts, prompts and engine weights distort LLM brand recommendation rankings in a weekly panel.
Tags: LLM brand recommendation measurement, AI search visibility, citation counting, prompt engineering, model benchmarking
Featured image: Abstract, friendly illustration of four chat bubbles above a funnel sorting colored cards, no people, no logos, no text.

Comments (0)