Skip to content

robots.txt Generator

Short answer

Create a robots.txt with blocked paths, sitemap lines and an allow or block choice for each AI crawler, then see what every AI bot may fetch from your site.

What it does not do: It writes the file but does not publish it, and robots.txt only asks crawlers: it does not remove pages from search results or stop crawlers that ignore it.

Free · No signup · Runs in your browser
Crawling

Start each path with /, e.g. /admin/ or /cart/. Use * as a wildcard and $ to match the end of a URL.

Google ignores Crawl-delay; Bing reads it.

AI crawlers

Crawlers for AI search and answers decide whether your pages can be shown or cited there. Training crawlers collect pages for model training; blocking them does not remove your site from AI search.

AI search and answers
  • OAI-SearchBotOpenAI: indexes pages so its AI search can show and cite them.
  • ChatGPT-UserOpenAI: opens a page when a user asks the assistant about it. Its operator says robots.txt may not apply to these requests.
  • Claude-SearchBotAnthropic: indexes pages so its AI search can show and cite them.
  • Claude-UserAnthropic: opens a page when a user asks the assistant about it.
  • PerplexityBotPerplexity: indexes pages so its AI search can show and cite them.
  • Perplexity-UserPerplexity: opens a page when a user asks the assistant about it. Its operator says robots.txt may not apply to these requests.
  • meta-webindexerMeta: indexes pages so its AI search can show and cite them.
  • meta-externalfetcherMeta: opens a page when a user asks the assistant about it. Its operator says robots.txt may not apply to these requests.
  • ExaSearchBotExa: indexes pages so its AI search can show and cite them.
  • ExaUserExa: opens a page when a user asks the assistant about it.
  • KeenableBotKeenable: indexes pages so its AI search can show and cite them.
  • Keenable-UserKeenable: opens a page when a user asks the assistant about it.
Model training
  • GPTBotOpenAI: collects pages to train AI models.
  • ClaudeBotAnthropic: collects pages to train AI models.
  • CCBotOpen web archive crawler; many AI models are trained on its public dataset.
Other AI uses
  • Google-ExtendedGoogle: not a separate crawler — a rule that decides whether pages it already crawls may be used for its AI models.
  • Applebot-ExtendedApple: not a separate crawler — a rule that decides whether pages it already crawls may be used for its AI models.
  • meta-externalagentMeta: crawls pages for its own products; the content may also be used to train AI models.
  • AmazonbotAmazon: crawls pages for its own products; the content may also be used to train AI models.

AI crawler list last checked on 17/09/2026.

Your robots.txt

User-agent: *
Allow: /

What the file allows on the home page

  • GPTBotAllowed
  • OAI-SearchBotAllowed
  • ChatGPT-UserAllowed
  • ClaudeBotAllowed
  • Claude-SearchBotAllowed
  • Claude-UserAllowed
  • PerplexityBotAllowed
  • Perplexity-UserAllowed
  • Google-ExtendedAllowed
  • Applebot-ExtendedAllowed
  • meta-externalagentAllowed
  • meta-webindexerAllowed
  • meta-externalfetcherAllowed
  • ExaSearchBotAllowed
  • ExaUserAllowed
  • KeenableBotAllowed
  • Keenable-UserAllowed
  • AmazonbotAllowed
  • CCBotAllowed

Save the file as robots.txt at the root of your domain (for example https://example.com/robots.txt), then test the live file. Open the robots.txt checker

The file is created in your browser.

What It Does

Creates a robots.txt file with the paths to keep crawlers out of, sitemap lines, an optional Crawl-delay, and an allow or block choice for each known AI crawler, grouped into AI search, model training and other AI uses. It then reads the file back and shows what each AI crawler may fetch.

Why It Matters

robots.txt tells crawlers which URLs they may fetch. AI companies use separate crawlers for search, for fetching a page a user asks about, and for model training, so you can stay visible in AI search while opting out of training. Blocking a training crawler such as GPTBot does not remove a site from ChatGPT search. robots.txt is a request, not access control, and some fetchers that act for a user say they may not follow it.

How It Works

  1. Choose whether to allow crawling or block the whole site

  2. List the paths to keep crawlers out of and your sitemap URLs

  3. Allow or block each AI crawler, or start from a preset

  4. Copy or download the file, save it as /robots.txt and test the live file with the robots.txt checker

Sample input + output

INPUT
crawling: allowed · blocked paths: /admin/, /cart/
AI preset: allow AI search only
sitemap: https://example.com/sitemap.xml
OUTPUT
User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

What the file allows on the home page:
  GPTBot Blocked · OAI-SearchBot Allowed · ChatGPT-User Allowed
  ClaudeBot Blocked · Claude-SearchBot Allowed · PerplexityBot Allowed
  Google-Extended Blocked · CCBot Blocked

How to read the result

The User-agent: * group
Rules for every crawler: here Disallow: /admin/ and Disallow: /cart/, the two paths entered in the sample. Everything not disallowed may be crawled. robots.txt is a request that well-behaved crawlers follow, not a lock.
The AI crawler group
Every AI crawler set to Block is listed in one group ending in Disallow: /. The sample used the preset that allows AI search only, so the training crawlers are blocked (GPTBot, ClaudeBot, CCBot and others) while the search and user agents that fetch pages to cite them stay allowed.
The Sitemap line
One line per sitemap, always with the full https:// address. It applies to every crawler, wherever it sits in the file.
What the file allows on the home page
The generated file is read back with the same RFC 9309 rules crawlers use and tested on the home page for each AI crawler. In the sample GPTBot is Blocked and OAI-SearchBot is Allowed: the same company runs separate crawlers for training and for search.
Google-Extended and Applebot-Extended
These are not crawlers but tokens that say whether content Google and Apple already crawl may be used for their AI models. Blocking them does not stop Googlebot or Applebot from crawling the site for search.

Who Uses This

  • Site owner

    Stay in AI search results while opting out of model training.

  • SEO specialist

    Keep crawlers out of cart, internal search and admin pages without blocking the rest of the site.

  • Web developer

    Create a block-all file for a staging site.

Common problems and how to fix them

  1. You want a page out of search results

    robots.txt does not remove pages: a blocked URL can still be listed if other sites link to it. Let the page be crawled and add a noindex robots meta tag instead.

    <meta name="robots" content="noindex">
  2. CSS or JavaScript is blocked

    Do not disallow folders with the stylesheets and scripts your pages need; search engines render pages and judge them on what they see. Block private areas such as admin, cart and account pages instead.

  3. Relying on Crawl-delay

    Google ignores Crawl-delay; Bing and some other crawlers follow it. If Googlebot crawls too fast, fix server speed or let the server answer 503 or 429 for a short time.

  4. The file is in the wrong place

    Crawlers read robots.txt only at the root of each host: https://example.com/robots.txt covers example.com, not www.example.com or shop.example.com, which need their own file.

  5. Blocking every AI crawler

    Blocking the AI search and user agents too keeps your pages out of the answers that link to sources. If your aim is to stay out of model training only, block the training crawlers and keep the search ones allowed.

Frequently Asked Questions

Where do I put robots.txt?
At the root of your domain, for example https://example.com/robots.txt. Each subdomain needs its own file.
Does blocking a page in robots.txt remove it from Google?
No. robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it; use noindex or password protection to keep a page out.
Which AI crawlers can I control?
The crawler tokens that OpenAI, Anthropic, Perplexity, Google, Apple, Meta and Amazon document, plus CCBot, the crawler of an open web archive. Operators add and retire tokens, so the list is checked regularly; the date is shown under it.
Why don't allowed AI crawlers get their own group?
A crawler with its own group ignores the rules for all crawlers (User-agent: *). Leaving allowed crawlers out keeps your blocked paths in force for them too.
Does Google read Crawl-delay?
No, Google ignores it. Bing reads it.

Sources

The documentation this tool and the explanations on this page are based on.