AI crawler
An AI crawler is a bot run by an AI company that fetches web pages to train models or to answer user questions in real time, such as GPTBot, ClaudeBot or PerplexityBot.
AI companies run several kinds of bots. Some collect pages for training, some index pages for AI search, and some fetch a page only when a user asks a question about it. Each identifies itself with a user-agent name.
In more detail
You control access in robots.txt by naming each bot and allowing or disallowing paths. Blocking training bots does not always block search or user-triggered bots, which often have separate names. Many AI crawlers do not run JavaScript, so content that appears only after scripts load may be invisible to them.
Why it matters
If AI tools cannot read your pages, they cannot cite them. Brands that want AI visibility usually allow search and user bots, serve important content as plain HTML and check server logs to confirm the bots are visiting.
A publisher finds it is missing from Perplexity answers. Its robots.txt blocks every bot except Googlebot, a rule added years ago. Allowing PerplexityBot and OAI-SearchBot, and serving articles as static HTML, gets it cited within weeks.