AEO, GEO and AI search

AI crawler

An AI crawler is a bot run by an AI company that fetches web pages to train models or to answer user questions in real time, such as GPTBot, ClaudeBot or PerplexityBot.

AI companies run several kinds of bots. Some collect pages for training, some index pages for AI search, and some fetch a page only when a user asks a question about it. Each identifies itself with a user-agent name.

In more detail

You control access in robots.txt by naming each bot and allowing or disallowing paths. Blocking training bots does not always block search or user-triggered bots, which often have separate names. Many AI crawlers do not run JavaScript, so content that appears only after scripts load may be invisible to them.

Why it matters

If AI tools cannot read your pages, they cannot cite them. Brands that want AI visibility usually allow search and user bots, serve important content as plain HTML and check server logs to confirm the bots are visiting.

Example

A publisher finds it is missing from Perplexity answers. Its robots.txt blocks every bot except Googlebot, a rule added years ago. Allowing PerplexityBot and OAI-SearchBot, and serving articles as static HTML, gets it cited within weeks.

FAQ

AI crawler: quick answers

Should I block AI crawlers?

It depends on your goals. If you want to be cited in AI answers, allow the search and user-facing bots at least.

Do AI crawlers render JavaScript?

Many do not. Serve key content in the initial HTML so every crawler can read it.

Free audit

Find out where you stand in search and AI answers.

Get a free audit of your SEO, AI visibility and email setup. We send a short, specific report with the three changes that would move the needle most.