robots.txt
robots.txt is a text file at the root of a site that tells crawlers which pages or sections they may or may not access.
robots.txt at a glance
robots.txt is a simple file placed at yoursite.com/robots.txt. It gives crawlers instructions about which parts of the site they can visit, and usually points to the sitemap. It is the first thing many bots read, so it shapes how search engines and AI crawlers explore your site.
Why robots.txt matters for your website
Used well, robots.txt keeps crawlers away from pages that do not belong in search, like thank-you or admin pages, and helps them focus on what matters. Used carelessly, it can accidentally block important pages or entire sections from being crawled. It controls crawling, not indexing, so sensitive pages still need other protection.
FAQ
Does robots.txt stop a page from being indexed?
Not reliably. It blocks crawling, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of results, use a noindex tag instead.
Should you block AI crawlers in robots.txt?
That is a strategic choice. If you want to appear in AI answers, allow crawlers like GPTBot and ClaudeBot; block them only if you deliberately want to opt out.
Want a website that does all this?
- ✓Custom design, never a template
- ✓Live in about 10 days
- ✓Hosting and support included