What an AI reads when it reads your website

Three kinds of automated visitor now read a business website on behalf of an AI system, and the companies behind them document each one.

The first gathers text to train models. OpenAI calls its one GPTBot, Anthropic calls its one ClaudeBot, and Google’s is controlled by a token called Google-Extended. Blocking these stops your pages being used for training. It brings no visits, by construction, and Google says blocking Google-Extended has no effect on Search.

The second builds a search index for the assistant to consult when someone asks it a question. OpenAI’s is OAI-SearchBot and Anthropic’s is Claude-SearchBot. Block these and your site cannot appear in that assistant’s answers. Google’s summaries and AI Mode use the ordinary Google index: there is no separate crawler to allow or refuse, and a page that is indexed and eligible for a snippet is eligible for them.

The third fetches a page when a user asks about it directly. OpenAI’s is ChatGPT-User and Anthropic’s is Claude-User. Both companies say these may not obey the robots file, because a person asked for the page.

Each is controlled separately in the robots file. The practical setting for a business that wants to be found is to allow the search and user-fetch crawlers, and to decide about the training ones on your own view, knowing the answer affects nothing you can measure.

The one technical difference that has been demonstrated is this. In December 2024 two hosting and search firms measured the AI crawlers on a large network and found that none of the crawlers from OpenAI, Anthropic or Perplexity ran JavaScript. They fetched the HTML and read what was in it. Google’s crawler, by contrast, renders pages as a browser does, and Gemini uses Google’s infrastructure.

So a page whose content only exists after a script has run, which describes many sites built on JavaScript frameworks and some page builders, is blank to the assistants. A page whose content is in the HTML the server sends is read in full. That study is nearly two years old and has not been repeated by the same team, so treat it as the best available rather than current. It is also the one point in this whole subject where how a site is built matters, and it happens to be a point on which a site built as static files is right by construction.

Nothing else about the build has been shown to matter. No study we found links page speed or Core Web Vitals to being quoted by an AI, though both matter for Google ranking and for visitors.

What is measured about which pages the systems cite is vendor data, and the vendors sell tracking tools, so read it for direction.

Ranking still matters, but less than it did and less for assistants than for Google. In July 2025, three quarters of the pages Google’s summaries cited were in the top ten ordinary results; by March 2026, after Google began running several related searches to build each answer, it was 38 per cent. For ChatGPT, Gemini and Copilot, only 12 per cent of cited pages ranked in Google’s top ten for the same question, and about four in five did not rank at all for it, because the assistant searched for something related rather than the question as typed.

Being mentioned elsewhere matters. One vendor found that ranking for the related searches the system runs was the strongest correlation with being cited in Google’s summaries, and that how often the brand was mentioned on other sites was the strongest with the brand appearing in them at all. Official company websites, another vendor reports, rarely appear in the lists of most-quoted sources; Wikipedia, Reddit and YouTube do.

Length does not matter. The correlation between word count and citation was measured at close to zero.

Freshness matters a little for assistants, which cited pages about a quarter newer than the ordinary results, and not for Google’s summaries, which cited slightly older ones.

And the one laboratory study of the question, from 2024, found that adding quotations, statistics and named sources to a page increased how much a model quoted it, by 30 to 40 per cent in a simulated engine, while stuffing it with keywords reduced it. The study used a simulated engine and an older model and measured share of the answer, not visits. It is the only controlled evidence there is, and what it rewards is evidence.

Google has published a plain list, and it is worth quoting because it contradicts most of the sales material. You do not need to create “machine readable files, AI text files, markup, or Markdown” to appear in Google’s AI features. There is “no requirement to break your content into tiny pieces”. You do not need “to write in a specific way just for generative AI search”. No special structured data exists for the AI features, and ordinary structured data must match what the page visibly says. Creating pages for each variation of a query to influence answers is against its spam policy. And “no third-party tool has access to our internal ranking or AI systems”, which is what every visibility score is a guess about.

The AI file, called llms.txt, is the clearest case. Of 137,000 sites with traffic that a vendor examined in May 2026, 28 per cent had published one. Of those files, 97 per cent were never requested by anyone. Of the requests that did arrive at the rest, about 1 per cent came from an AI retrieval crawler. Google says its search does not use the file. There is no measured evidence that any AI system does.

Schema markup has no proof behind it for AI citation, by the admission of the vendor most likely to find some. Author boxes and bylines have no study behind them at all.

Strip away what does not work and the list is short, and it is the list Google, the vendors and the laboratory agree on.

• Be readable without running code. Serve the page’s content as HTML. A static site does this by default; a site built on a JavaScript framework or a page builder needs checking, by fetching a page with the scripts turned off and seeing what remains.
• Allow the right crawlers. The search and user-fetch crawlers from OpenAI and Anthropic if you want to appear in their answers; Google’s ordinary crawler, which you already allow; the training crawlers on your own judgement.
• Answer the question on the page, with evidence. A specification, a figure with its source, a quoted standard. The pages that get quoted are the ones that say something checkable.
• Be mentioned. Directory listings, trade press, suppliers’ and customers’ pages, standards bodies: the mentions on other sites that the systems weigh, which is slow, ordinary work.
• Measure in Search Console, where impressions in Google’s AI features are now reported by page, and in your own analytics, where visits from assistants will show their referrer when they send one.

None of this is a new discipline, which is why we do not sell one. It is the work of a site that puts its facts where a machine and a buyer can both read them, which is what the Websites service builds and the AI projects page is careful to say AI is not.