The Personalization Gap: What 8,609 AI Responses Tell Us About Brand Visibility
AI search visibility has a measurement problem. When we measure whether a brand appears in ChatGPT, Gemini, Claude, or Perplexity, we’re usually measuring responses without the same account history and context that a real customer might have. But the actual people using these platforms aren’t starting from zero.
Their AI accounts may know where they live. They may know something about their interests, family, preferences, previous questions, or other characteristics. And platforms like ChatGPT and Gemini are actively building more personalized experiences around that context.
So Joao da Silva, co-founder of friction AI, and I wanted to answer a deceptively simple question: Does what an AI system knows about the user change which brands it recommends?
We analyzed 8,609 successful collections across ChatGPT, Gemini, Claude, and Perplexity to find out. The answer is yes, but the size and nature of that effect depends heavily on the AI system.
And that has implications for how marketers think about AI visibility, measurement, audience research, localization, and the increasingly common impulse to turn every missing brand mention into another piece of content.
What We Tested
The research examined four major AI systems:
- ChatGPT
- Gemini
- Claude
- Perplexity
We created six controlled personas across three consumer categories and tested 30 prompts.
The experiment was conducted in a UK market environment using UK IP addresses. We collected responses over nine days, resulting in 8,609 successful collections.
But the important part of the methodology was how we asked the questions. We compared three conditions.
1. API
The first used a web-enabled API without a user account or persistent history. This is particularly relevant to AI visibility measurement because APIs are commonly used to collect responses at the scale required for monitoring.
2. Temporary
The second used the normal logged-in AI product in a temporary session without retained user history. That gets us closer to the interface an actual user interacts with while removing persistent history.
3. History-Primed
The third used logged-in accounts with controlled histories. For two weeks before measurement began, we provided those accounts with controlled information about specific personas, including characteristics such as location, family situation, age, and income.
Then we asked the same questions across the three conditions. The goal wasn’t to determine whether every individual AI answer would be different. LLM outputs naturally vary from one run to another. Instead, we wanted to determine whether user history created a measurable difference in the brands that reliably appeared across repeated responses beyond that normal volatility.
The Personalization Gap
We call that difference the Personalization Gap, or the difference between the brands an AI system tends to recommend without knowing the user and the brands it tends to recommend when it has context about that person.
Importantly, personalization didn’t completely rewrite the recommendations. Across the experiment, the average history-primed core contained about 3.7 recurring brands. Approximately 2.8 of those were shared with the neutral session, while roughly 0.9 were different.
The takeaway isn’t that two people will necessarily receive completely different brand recommendations. It’s that part of the brand set can change based on what the AI knows about the person asking. And the size of that effect varied considerably by platform.
ChatGPT and Gemini Showed the Clearest Personalization Effects
After accounting for normal run-to-run volatility, we detected significant history-associated divergence in the recurring brand sets produced by ChatGPT and Gemini.
- For ChatGPT, the divergence above the study’s Volatility Floor was +16.0 percentage points.
- For Gemini, it was +33.3 percentage points.
Gemini therefore showed the largest history-associated brand-set divergence of the four systems we tested. That doesn’t mean Gemini “changes 33% of its recommendations,” and it doesn’t mean every Gemini response is personalized. The statistic measures divergence beyond the normal variation we observed under the study conditions.
But the bigger point for marketers is this: The brands appearing in AI recommendations aren’t necessarily determined by the prompt alone.
User context can become another variable. And that means the traditional idea that there is one set of results we’re all competing to appear within becomes much harder to apply to AI discovery.
Claude Did Something Different
Claude produced one of the more interesting results. Under our study conditions, we didn’t detect a history-associated difference in its recurring brand set beyond normal volatility. The measured difference was just +1.0 percentage point. But that doesn’t mean user context had no effect on Claude. Its search behavior changed.
During our Found in AI podcast conversation, Joao explained that Claude localized its searches and looked for more UK-relevant sources when it had contextual information about the user. But those changes didn’t ultimately produce a detectable change in the recurring brands being recommended.
That’s an important reminder that personalization can happen at different stages of an AI system’s process. The final brand recommendation isn’t necessarily the only thing worth studying. Retrieval behavior, queries, source selection, citations, and final answers can each tell us something different about how these systems are operating.
Perplexity Fell Somewhere in the Middle
Perplexity showed a smaller effect than ChatGPT or Gemini in our primary comparison. Its measured history-associated brand-set divergence was +7.2 percentage points, with a one-sided p-value of 0.034.
In our conversation, we also discussed changes in how focused Perplexity’s recommendations became. When the system had more context about the user, responses sometimes narrowed the number of brands being presented or changed which brand received the most prominent recommendation.
I wouldn’t take this study and conclude that Perplexity either “personalizes” or “doesn’t personalize.” The more useful takeaway is that personalization didn’t manifest identically across the four platforms. And that matters when brands are trying to optimize for “AI search” as though it’s one channel. (It isn’t.)
Geography Was the Clearest Personalization Dimension We Observed
One of the findings I think marketers should pay particularly close attention to was geography. Because this study was conducted in a UK environment, we were able to examine whether user context shifted recommendations toward brands and sources that were more relevant to that market.
In the history-primed condition, we observed:
- a 6.6 percentage-point increase in UK/EU-origin brand share
- a 4.9 percentage-point decrease in US-origin brand share
- a 6.0 percentage-point increase in .uk-domain citations
This doesn’t prove that publishing localized content will automatically increase your AI visibility. It does, however, suggest that market relevance can influence the recommendation environment. That creates an interesting opportunity for smaller and regional brands. Being the biggest global entity in a category may not be the only factor an AI system uses to determine which recommendation is most appropriate for a particular person.
For marketers operating across multiple countries or highly regional markets, localization shouldn’t be treated purely as a traditional SEO consideration. It may also become part of AI visibility strategy.
Your AI Visibility Tool Isn’t Showing You Every Possible Customer Experience
If real users receive some recommendations influenced by account history, what does that mean for platforms measuring AI visibility through history-free APIs? It would be easy to jump from this research to: “API-based AI visibility measurement doesn’t work.”
That’s not what we found.
Across 173 matched comparisons evaluated at an equal depth of up to 16 repeated responses per series, the cold API recovered roughly half of the history-primed recurring brand core from a single response.
- As sampling increased, recall increased.
- At the measured endpoint, the API reached 79% recall of the history-primed core.
- The temporary logged-in condition reached 95%.
So the API was capturing a substantial amount of what appeared in the history-primed experience. It just wasn’t capturing all of it. That’s a much more nuanced measurement problem than “the tools are wrong.” AI visibility platforms can give marketers useful measurements without those measurements representing every answer every customer might receive. The methodology matters here.
One AI Response Isn’t a Measurement System
There’s another problem hiding underneath the personalization question. AI responses are probabilistic. That means users can ask the same question repeatedly and aren’t guaranteed to receive the exact same answer. That’s why this research didn’t simply compare one API response with one personalized response and call every difference “personalization.”
We accounted for normal run-to-run variation through what the study calls the Volatility Floor. For marketers, the practical implication is straightforward: A screenshot is evidence that an answer happened. It isn’t a measurement system.
If your brand appears once, that doesn’t necessarily mean you “rank.” If it disappears once, that doesn’t necessarily mean you’ve lost visibility. And if a competitor appears in a response you don’t, immediately restructuring your content strategy around that single observation is probably premature.
AI visibility needs to be measured across repeated responses and over time. The questions become:
- How consistently does the brand appear?
- How often is it cited?
- Which competitors recur?
- Which sources recur?
- Are those patterns changing?
- Do those patterns differ across platforms?
Those are much more useful questions than, “What did ChatGPT say when I asked it this morning?”
Your Customer Persona Just Became Part of the GEO Conversation
One of my biggest takeaways from this research is about audience research. For years, marketers have talked about understanding the customer because it helps us create better positioning, messaging, content, and campaigns. AI personalization adds another reason.
If what an AI system knows about the person can influence which brands it recommends, then who you’re trying to reach can influence the AI discovery environment you’re trying to appear within.
That doesn’t mean we now need to build a visibility dashboard for every imaginable customer persona. It does mean that a list of 100 generic “high-value prompts” isn’t necessarily enough to represent the discovery journey.
AI search doesn’t make audience strategy obsolete. I’d argue that it makes the lack of one harder to ignore.
What Marketers Should Do With This Research
I wouldn’t walk away from this study and completely rebuild your GEO strategy tomorrow. But I would start asking harder questions about the data you’re using to make decisions.
For one, how are you measuring visibility? If your reporting is based on a single response to a prompt, that’s not enough. These systems are probabilistic, and some of what looks like a visibility change may simply be normal variation. You need repeated measurements before you can start separating a pattern from noise.
The same goes for the tools you’re using. An API-based measurement isn’t suddenly useless because personalized accounts can produce different recommendations. But you should know how your tool collects its data and what that data can reasonably tell you. There’s a difference between saying, “This is how my brand is performing across thousands of consistently collected responses,” and “This is exactly what my customers see.” Our research suggests the second claim is much harder to make.
I’d also be careful about lumping every AI platform into one big “AI search” bucket. ChatGPT, Gemini, Claude, and Perplexity didn’t behave the same way in this experiment. If one platform matters disproportionately to your audience, its behavior deserves to be looked at on its own.
And then there’s the audience itself. This may be the part marketers have been overlooking. If user context can change some of the brands being recommended, a generic list of prompts can only take you so far. Who is asking the question matters, too. Geography was the clearest example we saw in this study, but the broader point is that you need to know who you’re actually trying to reach before you can decide which AI discovery journeys are worth measuring.
What We Still Don’t Know
This research also created new questions. One of the biggest is whether deeper sampling can eventually close the remaining gap between history-free API measurement and the history-primed user experience.
We call this the Accuracy Ceiling hypothesis. At the endpoint we measured, API recall reached 79% while the temporary logged-in condition reached 95%. But we did not prove that 79% is a permanent ceiling.
We don’t yet know whether significantly deeper sampling would continue closing the gap or whether persistent user context creates some portion of the experience that history-free measurement can’t fully recover. That’s a question for future research.
There are other limitations, too. The experiment was conducted in a UK market environment and covered three consumer categories, six personas, 30 prompts, and four AI systems during a defined measurement window.
AI systems change quickly. The results shouldn’t be treated as permanent rules about how every model, category, market, or user will behave. What the research does give us is evidence that user context is another variable marketers need to consider when thinking about AI discovery.
The Bigger Marketing Implication
For me, the most important conclusion from this research is that the old instinct to reduce visibility to one answer is becoming increasingly difficult to defend. There isn’t necessarily one universal AI result we’re all trying to rank inside. And that doesn’t make AI visibility impossible to measure. It makes simplistic measurement harder to defend.
The job isn’t to know exactly what every AI system will say to every person. It’s to understand the patterns shaping whether your brand becomes part of the answer, and make smarter marketing decisions from there.
Read the full research:
The Personalization Gap: How a Model’s Knowledge of the User Reshapes Brand Recommendations in Generative AI
Cassie Wilson Clark & Joao da Silva
DOI: 10.5281/zenodo.22161044
Listen to the conversation:
On the September 22 episode of Found in AI, Joao da Silva and I break down the experiment, what surprised us, and what the findings mean for marketers measuring and improving AI search visibility.