What GPTBot, ClaudeBot and PerplexityBot actually read on your site

AI assistants answer questions about your category every day. Whether your brand appears in those answers depends on something unglamorous: what the crawler could read from your raw HTML.

The crawlers that matter

• GPTBot — OpenAI's content crawler, feeds training and browsing.
• OAI-SearchBot — powers ChatGPT search results and citations.
• ClaudeBot — Anthropic's crawler.
• PerplexityBot — indexes for Perplexity's answer engine.
• Google-Extended — controls whether your content informs Gemini and AI Overviews.
• Bingbot — still the substrate for Copilot answers.

They fetch HTML. That is it.

None of these crawlers run a JavaScript engine on your page the way a browser does. They issue an HTTP request, take the HTML that comes back, and parse it. If your content is injected client-side after hydration, the crawler sees an empty shell — a title, a nav, and a div with a spinner.

curl -A "GPTBot" https://yoursite.com/pricing | wc -w

Run that against your most important page. If the word count is under a couple of hundred, an AI assistant cannot describe what you sell.

What they extract, in priority order

1. 1The <title> and meta description — the page's one-line identity.
2. 2The H1 and heading hierarchy — the structure of the argument.
3. 3Body text in the server-rendered HTML — the substance.
4. 4JSON-LD structured data — unambiguous facts about entities, prices, FAQs and authorship.
5. 5Link anchor text — how pages relate to one another.

The five failures we see most

• Client-side-only content: the page is real to humans and empty to crawlers.
• No structured data: the assistant has to guess what your product costs and does.
• Headings used for styling rather than meaning, so the outline is nonsense.
• Key claims locked inside images, carousels or accordions rendered on interaction.
• robots.txt blocking AI crawlers by accident, usually copied from a template.

Answer coverage beats keyword density

Assistants assemble answers to questions. A page that states a question and answers it in the following paragraph is directly quotable. A page that circles the topic is not. Write the question as a heading, answer it in the first two sentences underneath, then elaborate.

You can check any page against all of this in about ten seconds with our free AI visibility check — it fetches your URL without JavaScript, exactly like these crawlers do, and scores what is readable.