robots.txt Generator
Short answer
Create a robots.txt file with blocked paths, sitemaps and a choice for each AI crawler.
Start each path with /, e.g. /admin/ or /cart/. Use * as a wildcard and $ to match the end of a URL.
Google ignores Crawl-delay; Bing reads it.
AI crawlers
Crawlers for AI search and answers decide whether your pages can be shown or cited there. Training crawlers collect pages for model training; blocking them does not remove your site from AI search.
AI crawler list last checked on 17/09/2026.
Your robots.txt
User-agent: * Allow: /
What the file allows on the home page
- GPTBotAllowed
- OAI-SearchBotAllowed
- ChatGPT-UserAllowed
- ClaudeBotAllowed
- Claude-SearchBotAllowed
- Claude-UserAllowed
- PerplexityBotAllowed
- Perplexity-UserAllowed
- Google-ExtendedAllowed
- Applebot-ExtendedAllowed
- meta-externalagentAllowed
- AmazonbotAllowed
- CCBotAllowed
Save the file as robots.txt at the root of your domain (for example https://example.com/robots.txt), then test the live file. Open the robots.txt checker
The file is created in your browser.
What It Does
Creates a robots.txt file with the paths to keep crawlers out of, sitemap lines, an optional Crawl-delay, and an allow or block choice for each known AI crawler, grouped into AI search, model training and other AI uses. It then reads the file back and shows what each AI crawler may fetch.
Why It Matters
robots.txt tells crawlers which URLs they may fetch. AI companies use separate crawlers for search, for fetching a page a user asks about, and for model training, so you can stay visible in AI search while opting out of training. Blocking a training crawler such as GPTBot does not remove a site from ChatGPT search. robots.txt is a request, not access control, and some fetchers that act for a user say they may not follow it.
How It Works
-
Choose whether to allow crawling or block the whole site
-
List the paths to keep crawlers out of and your sitemap URLs
-
Allow or block each AI crawler, or start from a preset
-
Copy or download the file, save it as /robots.txt and test the live file with the robots.txt checker
Sample input + output
crawling: allowed · blocked paths: /admin/, /cart/ AI preset: allow AI search, block the rest sitemap: https://example.com/sitemap.xml
User-agent: * Disallow: /admin/ Disallow: /cart/ User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: CCBot Disallow: / Sitemap: https://example.com/sitemap.xml
Who Uses This
-
Site owner
Stay in AI search results while opting out of model training.
-
SEO specialist
Keep crawlers out of cart, internal search and admin pages without blocking the rest of the site.
-
Web developer
Create a block-all file for a staging site.
Frequently Asked Questions
Where do I put robots.txt?
At the root of your domain, for example https://example.com/robots.txt. Each subdomain needs its own file.
Does blocking a page in robots.txt remove it from Google?
No. robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it; use noindex or password protection to keep a page out.
Which AI crawlers can I control?
The crawler tokens that OpenAI, Anthropic, Perplexity, Google, Apple, Meta and Amazon document, plus CCBot, the crawler of an open web archive. Operators add and retire tokens, so the list is checked regularly; the date is shown under it.
Why don't allowed AI crawlers get their own group?
A crawler with its own group ignores the rules for all crawlers (User-agent: *). Leaving allowed crawlers out keeps your blocked paths in force for them too.
Does Google read Crawl-delay?
No, Google ignores it. Bing reads it.