Robots.txt Generator
Builds a robots.txt file that tells search engines and AI crawlers which parts of your site they may read.
This tool runs entirely in your browser. Nothing you enter is uploaded, transmitted or stored anywhere.
Common questions
Where does this file go?
Save it as robots.txt in the very top folder of your website, so it is reachable at yoursite.com/robots.txt.
Does blocking a page hide it?
No. robots.txt is a request, not a lock. Well-behaved crawlers respect it, but it does not stop anyone from opening the page directly. Never use it to protect private information.
Which AI crawlers get blocked?
GPTBot, ChatGPT-User, OAI-SearchBot, CCBot, Google-Extended, anthropic-ai, ClaudeBot, ClaudeBot-User, PerplexityBot, Bytespider, Diffbot, Applebot-Extended, Meta-ExternalAgent and Amazonbot.