Let’s talk

Marketing Insights

Why ChatGPT, Claude and Gemini recommend your competitor

Sam Hill
·
·
5
MIN READ
Three rows naming the crawlers to allow: OAI-SearchBot for ChatGPT, Claude-User and Claude-SearchBot for Claude, and Googlebot for Gemini

ChatGPT, Claude and Gemini tend to recommend the competitor they can find and verify more easily: pages that answer the question directly, the same facts across the web, and mentions on other sites. First check that OpenAI’s, Anthropic’s and Google’s crawlers can reach your website, then fix the pages and mentions they would cite.

ChatGPT, Claude and Gemini tend to recommend the competitor they can find and verify more easily: pages that answer the question directly, the same facts across the web, and mentions on other sites. First check that OpenAI’s, Anthropic’s and Google’s crawlers can reach your website, then fix the pages and mentions they would cite.

Can the assistants see your website at all?

Check this first, because it costs nothing to fix. OpenAI’s search crawler is called OAI-SearchBot. OpenAI says sites that opt out of it won’t be shown in ChatGPT search answers, though they can still appear as navigational links (OpenAI).

To be eligible, allow OAI-SearchBot in your robots.txt file and confirm your host or CDN allows traffic from OpenAI’s published IP addresses (OpenAI Help Centre). A firewall rule that blocks unknown bots can shut it out without anyone noticing. OpenAI says it can take about 24 hours for a robots.txt change to take effect.

Claude uses three robots. ClaudeBot collects content for training, Claude-User fetches pages when a person asks Claude a question, and Claude-SearchBot indexes the web for Claude’s search. Anthropic says blocking Claude-User or Claude-SearchBot may reduce your visibility in Claude’s answers, and that ClaudeBot can be blocked for training while the other two stay open (Anthropic).

Gemini draws on Google’s Search index, so if Googlebot can index you, Gemini can see you. Google-Extended is a separate robots.txt token that controls whether your content is used to train Gemini and for grounding, the step where Gemini pulls pages from the index into its answer. Google says it has no effect on your inclusion or ranking in Search (Google). A site that blocked Google-Extended to stay out of training has also opted out of being quoted in Gemini’s answers.

Can each assistant reach your website
Can each assistant reach your website: OAI-SearchBot for ChatGPT, Claude-User and Claude-SearchBot for Claude, and Googlebot for Gemini.

How do you test who the assistants recommend?

Write ten questions a real buyer would ask, and run each in a fresh chat in ChatGPT, Claude and Gemini. Include your city or suburb where a buyer would, because OpenAI advises adding a city, neighbourhood or postcode to improve local results. Search is also available to people who aren’t signed in, so test signed out as well.

For each answer, record which businesses are named and which sources are cited. Then open the cited pages. OpenAI warns that search results and citations can be incomplete, outdated or incorrect, so don’t treat one answer as a verdict.

Why does a competitor get picked over you?

Usually for one of three reasons. OpenAI says ChatGPT ranks results using multiple factors and doesn’t publish them, and we haven’t seen Anthropic or Google publish theirs either, so these are our view, not a formula.

  • Their page answers the exact question in its first paragraph, and yours describes services in general terms.
  • Their name, address and offer read the same on every site that mentions them, and yours vary.
  • Other sites mention them, and nothing outside your own website mentions you.

The second and third are the entity problem we describe in getting AI to recognise your brand. The third is why brand mentions beat backlinks in AI search. ChatGPT search also sometimes partners with other search providers, so it’s worth being findable beyond one engine.

Three reasons a competitor gets picked
Three reasons a competitor gets picked: their page answers the question first, their details match across sites, and other sites mention them.

What should you fix first?

Fix access, then pages, then mentions, in that order.

  1. Allow OAI-SearchBot, Claude-SearchBot and Claude-User, leave Google-Extended open unless you accept dropping out of Gemini’s answers, and check OpenAI’s IP addresses with your host.
  2. Rewrite your top three pages so each answers one buyer question in its first two sentences. Our guide to answer-shaped content shows the format.
  3. Make your business name, address and service descriptions identical on your site, your Google Business Profile and your directories.
  4. Pick two or three industry sites, partners or publications that could mention you, and ask.
  5. Re-run your ten questions and compare.

If you want us to run the test and send you the results, get in touch.

What to fix, in order
What to fix, in order: allow the crawlers, rewrite your top three pages, ask two or three sites to mention you, then re-run your ten questions.

How long until it changes?

You can’t be sure, and anyone who gives you a number is guessing. OpenAI says placement isn’t guaranteed, and the only timing it gives is about 24 hours for a robots.txt change to take effect.

We’d plan for weeks rather than days for page changes, and longer for mentions on other sites. Re-test monthly with the same questions, so you’re comparing like with like.

If your business sells in a niche, narrow the questions further. A buyer asking for a branding studio that works with hospitality in St Kilda is a different question from one asking for a branding studio in Melbourne, and your pages should answer the narrow one in as many words as the buyer would use.

FAQ

Will paying for ChatGPT ads get me recommended?

No. OpenAI says ads appear below ChatGPT’s responses and are separate from them (OpenAI Help Centre). They’re a paid placement, not a recommendation inside the answer.

Can I block AI training and still appear in the answers?

With OpenAI and Anthropic, yes: GPTBot and ClaudeBot are training crawlers with settings separate from the search robots. With Google, the Google-Extended token covers training and grounding together, so blocking it also takes you out of Gemini’s grounded answers.

Does ChatGPT use Google?

OpenAI says ChatGPT search sometimes partners with other search providers, and its help page names Microsoft and Shopify in that context. It doesn’t publish a full list, so we’d aim to be findable across several.