robots.txt Generator
Create a robots.txt file to guide search engines, block AI training crawlers if you want, and test whether a page is allowed before you upload it.
About this tool
robots.txt sits at the root of your site (example.com/robots.txt) and tells crawlers which pages they may visit. Most sites should allow everything and only block pages with no value in search results, such as the basket, checkout and internal search results. Add your sitemap address so search engines find every page.
AI companies use separate crawlers for training models and for answering questions in AI search, so you can opt out of one without the other. “Block AI training only” blocks crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended, but keeps you visible in ChatGPT, Claude and Perplexity search results. Google-Extended and Applebot-Extended aren’t separate crawlers: they tell Google and Apple not to use pages their normal crawlers fetch for AI training, and blocking them doesn’t affect Google Search or Siri.
The tester applies the same rules Google uses: the most specific crawler group applies, the longest matching path wins, and Allow wins a tie. Remember that robots.txt is a request, not a lock. Well-behaved crawlers obey it, but it doesn’t protect private pages, and blocking a page doesn’t remove it from Google if other sites link to it; use a noindex tag for that. Your settings are remembered on this device.