Robots.txt Checker & Validator
Short answer
Check your robots.txt: its rules, the sitemap line, whether a page may be crawled, and which AI crawlers may read it.
What it does not do: It reads the rules as written and cannot tell whether a crawler obeys them; some AI fetchers that act on a user's request may ignore robots.txt, and blocking crawling does not remove a page from search results.
What It Does
Fetches your `/robots.txt` and parses every directive — User-agent groups, Allow/Disallow rules, Sitemap and Crawl-delay. Checks whether the URL you entered is allowed, using the same longest-match precedence real crawlers use, and flags a blanket `Disallow: /`, a missing sitemap directive and an oversized file. It also lists the main AI crawlers and control tokens, shows whether each may fetch that URL and explains what each one is for: training AI models, AI search answers, or opening a page when a user asks.
Why It Matters
A single stray `Disallow: /` can stop search engines from crawling a whole site, and it is easy to ship by accident from a staging setup. AI crawlers add a second question: blocking a training crawler does not remove you from AI search answers, but blocking a search crawler can — so it pays to know which token does what.
How It Works
-
Enter the URL you want to test
-
We fetch `/robots.txt` and parse every directive
-
Check whether that URL is allowed for crawlers in general and for each AI crawler, using the precedence rules real crawlers use
-
Flag a full-site block, a blocked URL and a missing sitemap directive, and explain each AI crawler
Sample input + output
url: https://rankproof.eu/tools
robots.txt looks good and doesn't block this page. 100/100
HTTP status: 200 · File size: 5.3 KB · User-agent groups: 53 · Crawl delay: Not set
robots.txt found OK
No blanket disallow OK
This page is crawlable OK
Sitemap directive present OK
File size under 500KB OK
Issues found (10):
Disallow: / found for user-agent "Bytespider" — only that crawler is blocked from the whole site, which is often deliberate.
Disallow: / found for user-agent "Diffbot" — only that crawler is blocked from the whole site, which is often deliberate.
…
AI crawlers: 13 of 13 AI crawlers may read this page
GPTBot Allowed OpenAI: collects pages to train AI models.
OAI-SearchBot Allowed OpenAI: indexes pages so its AI search can show and cite them.
ChatGPT-User Allowed OpenAI: opens a page when a user asks the assistant about it. Its operator says robots.txt may not apply to these requests.
ClaudeBot Allowed Anthropic: collects pages to train AI models.
… How to read the result
- HTTP status (200) · File size (5.3 KB)
- Only a 200 counts as a robots.txt file. Any other answer is reported as no file (25 points), which for a 404 is right: crawlers then allow everything. A server error is different for Google: it stops crawling the site for up to 12 hours and then relies on the last copy it fetched, so a robots.txt that answers 5xx needs fixing quickly. Files over 500 KB cost 10 points, since crawlers may stop reading there.
- User-agent groups (53) · Crawl delay (Not set)
- How many crawler names the file gives rules to, and the Crawl-delay value if one is set. Google ignores Crawl-delay; some other crawlers honour it.
- This page is crawlable · No blanket disallow
- Both are read from the group for every crawler (User-agent: *), with the longest matching rule winning and Allow beating Disallow on a tie, as RFC 9309 sets out. A rule that blocks the tested address costs 50 points and one that blocks the whole site 55: these are the mistakes that take a site out of search.
- Issues found (10)
- Every crawler name blocked from the whole site is listed, but costs no points: blocking a named scraper is usually deliberate, as in the sample. A missing Sitemap line is listed too and costs 5 points.
- AI crawlers (13 of 13 may read this page)
- For each AI crawler and control token we apply its own group if the file names it, otherwise the group for every crawler, and test the address you entered. The line under each one says what it is for: training, AI search, fetching a page when a user asks, or a control token. "Its operator says robots.txt may not apply" marks fetchers acting on a user's request.
Who Uses This
-
Technical SEO
Before launch, check the production robots.txt so a staging "block everything" rule never goes live.
-
Site Owner
Decide which AI crawlers to allow: keep AI search crawlers open while choosing whether your pages may train AI models.
-
Content Manager
When a section disappears from Google Search Console, first check whether robots.txt blocks it.
Common problems and how to fix them
-
Disallow: / for every crawler
Remove it, or narrow it to the paths that really must stay out. It is often left over from a staging site; a robots.txt rule cannot protect a staging site anyway, so use a password there instead.
-
The page you want indexed is blocked
Find the rule that matches its address (a broad pattern such as /*? or a folder rule) and add a more specific Allow or narrow the Disallow. The longest matching rule wins, so a precise Allow overrides a short Disallow.
-
A named crawler lost rules you set for everyone
A crawler with its own group reads only that group, not the rules under User-agent: *. When you add a group for one crawler, repeat in it every Disallow it still needs.
# every crawler: keep admin and cart out User-agent: * Disallow: /admin/ Disallow: /cart # a named crawler reads ONLY its own group, so repeat what it needs User-agent: GPTBot Disallow: /admin/ Disallow: /cart Disallow: /members/ Sitemap: https://example.com/sitemap-index.xml -
No Sitemap line
Add a Sitemap line with the absolute address of your sitemap or sitemap index. Every crawler that reads robots.txt can find it there, not only the search engines where you submitted it.
# robots.txt Sitemap: https://example.com/sitemap-index.xml -
robots.txt used to hide a page from results
Blocking crawling does not keep a page out of search: a blocked address can still be listed if other pages link to it, and Google cannot see a noindex tag on a page it may not crawl. Allow crawling and add noindex instead.
<meta name="robots" content="noindex">
Frequently Asked Questions
- What is robots.txt?
- A plain-text file at your domain root that tells crawlers which URLs they may or may not request. It is standardized in RFC 9309 (2022).
- Can I block AI crawlers separately?
- Yes — each has its own user-agent token, such as `GPTBot` or `ClaudeBot`. Several AI companies run separate crawlers for model training and for search, so you can block training while staying visible in their AI search answers.
- What is Google-Extended?
- A control token, not a separate crawler. Google crawls with its usual crawlers either way; disallowing `Google-Extended` tells Google not to use your content to train Gemini models or to ground answers in Gemini apps. It does not affect Google Search or rankings.
- Do all AI crawlers follow robots.txt?
- The major AI companies say their crawlers do. Some fetchers that act on a user's request, such as ChatGPT-User and Perplexity-User, may not apply robots.txt rules, and the report marks them.
- What is the difference between robots.txt and noindex?
- robots.txt prevents crawling. `noindex` allows crawling but prevents indexing. If you block a page in robots.txt, Google cannot see its noindex tag — a common mistake.
- Should I include a sitemap directive?
- Yes — `Sitemap: https://example.com/sitemap.xml` helps every crawler find your sitemap, including Bing and other search engines.
- Does Google always respect robots.txt?
- For crawling, yes. But a blocked URL can still appear in search results if other pages link to it — without a snippet. Use noindex to keep a page out.
Sources
The documentation this tool and the explanations on this page are based on.
- RFC 9309: Robots Exclusion Protocol — RFC Editor
- How Google Interprets the robots.txt Specification — Google Crawling Infrastructure
- Robots.txt Introduction and Guide — Google Search Central
- Overview of OpenAI Crawlers — OpenAI