Robots.txt Generator
Choose which folders to block and which AI crawlers to allow, and get a ready-to-upload robots.txt.
Why it matters
A clear robots.txt keeps crawlers focused on pages that matter and documents your choice on AI training versus AI search access.
How to use the Robots.txt Generator
- Fill in the form Enter your details or paste your text.
- See the result instantly Everything runs in your browser.
- Copy and use it Copy the output to your site or CMS.
Frequently asked questions
Should I block AI crawlers?
Blocking training crawlers (like GPTBot or CCBot) is a business choice. Blocking AI search crawlers (like OAI-SearchBot or PerplexityBot) means those tools cannot cite you.
Where do I upload it?
Save it as robots.txt in the top folder of your website so it loads at yourdomain.com/robots.txt.