robots.txt
robots.txt is a text file at the root of a website that tells crawlers which parts of the site they may or may not request.
The file lives at yourdomain.com/robots.txt. It lists rules for user agents, such as Googlebot or GPTBot, with Allow and Disallow lines for paths. It can also point to your XML sitemap.
In more detail
robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in results if other sites link to it, just without a description. To keep a page out of results, use a noindex tag and let crawlers see it. A single mistaken line, such as Disallow: /, can block an entire site, so changes should be tested carefully. Rules can also allow or block specific AI crawlers.
Why it matters
A well-written robots.txt keeps crawlers away from low-value areas like internal search results, while making sure important pages and resources stay accessible.
After a redesign, a company's traffic falls to almost nothing. The developer had copied the staging site's robots.txt, which contained 'Disallow: /'. Removing that line restores crawling, and rankings return over two weeks.