Technical SEO

robots.txt

robots.txt is a text file at the root of a website that tells crawlers which parts of the site they may or may not request.

The file lives at yourdomain.com/robots.txt. It lists rules for user agents, such as Googlebot or GPTBot, with Allow and Disallow lines for paths. It can also point to your XML sitemap.

In more detail

robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in results if other sites link to it, just without a description. To keep a page out of results, use a noindex tag and let crawlers see it. A single mistaken line, such as Disallow: /, can block an entire site, so changes should be tested carefully. Rules can also allow or block specific AI crawlers.

Why it matters

A well-written robots.txt keeps crawlers away from low-value areas like internal search results, while making sure important pages and resources stay accessible.

Example

After a redesign, a company's traffic falls to almost nothing. The developer had copied the staging site's robots.txt, which contained 'Disallow: /'. Removing that line restores crawling, and rankings return over two weeks.

FAQ

robots.txt: quick answers

Does robots.txt stop pages appearing in Google?

Not reliably. It stops crawling. Use a noindex tag to keep a page out of results.

Should I block AI bots in robots.txt?

Only if you do not want to appear in AI answers. Many brands allow AI search bots to stay visible.

Free audit

Find out where you stand in search and AI answers.

Get a free audit of your SEO, AI visibility and email setup. We send a short, specific report with the three changes that would move the needle most.