Robots.txt Checker

Fetch a site’s robots.txt, check that it does not block search engines, and see which crawlers are allowed or blocked.

Why it matters

One wrong line in robots.txt can hide a whole website from search. It is also where you decide whether AI crawlers such as GPTBot and PerplexityBot can read your content.

How to use the Robots.txt Checker

  1. Enter a URL Type the page or domain you want to check.
  2. Run the check We fetch it live, the way a crawler does.
  3. Fix what you find Each issue comes with a plain-English fix.

Frequently asked questions

Does robots.txt stop a page from being indexed?

Not reliably. It stops crawling, but a blocked URL can still appear in results if other sites link to it. Use a noindex tag to keep a page out of search.

Where must robots.txt live?

At the root of the domain, for example https://example.com/robots.txt. Each subdomain needs its own file.