AI crawler blocking on EU websites
How popular national websites in each EU country treat AI search and training crawlers in robots.txt, and how many publish llms.txt.
Updated · Sample month 08/2026 · 4,645 of 5,400 websites measured in 27 countries · Press kit
Key findings
- 10% of the measured websites block at least one AI search or answer crawler in robots.txt.
- 17% block at least one AI training crawler.
- 82% block no AI crawler at all.
- 9% of the websites we could check publish a valid llms.txt file.
- The most blocked AI crawler is CCBot: 14% of the measured websites block it.
10%
Block an AI search or answer crawler
17%
Block an AI training crawler
82%
Block no AI crawler
9%
Publish a valid llms.txt
By country
The EU total comes first, followed by the countries from the highest share to the lowest.
| Country | Measured | AI search blocked | AI training blocked | No AI blocked | llms.txt |
|---|---|---|---|---|---|
| EU (27 countries) | 4,645 | 10% | 17% | 82% | 9% |
| Germany | 184 | 27% | 43% | 55% | 5% |
| Italy | 165 | 22% | 30% | 68% | 5% |
| France | 166 | 20% | 25% | 72% | 5% |
| Sweden | 189 | 17% | 24% | 75% | 6% |
| Finland | 188 | 16% | 22% | 78% | 8% |
| Netherlands | 176 | 16% | 22% | 76% | 10% |
| Ireland | 178 | 16% | 20% | 79% | 9% |
| Spain | 160 | 16% | 24% | 75% | 6% |
| Denmark | 178 | 13% | 19% | 80% | 17% |
| Belgium | 174 | 11% | 16% | 80% | 9% |
| Czechia | 186 | 11% | 20% | 78% | 11% |
| Austria | 175 | 10% | 19% | 79% | 5% |
| Romania | 171 | 8% | 13% | 84% | 18% |
| Portugal | 170 | 8% | 14% | 84% | 8% |
| Estonia | 167 | 7% | 10% | 88% | 10% |
| Luxembourg | 174 | 7% | 11% | 88% | 8% |
| Latvia | 165 | 7% | 8% | 90% | 7% |
| Poland | 181 | 7% | 24% | 71% | 11% |
| Cyprus | 163 | 6% | 9% | 91% | 9% |
| Greece | 169 | 5% | 11% | 88% | 8% |
| Slovakia | 174 | 4% | 10% | 88% | 9% |
| Malta | 154 | 4% | 8% | 92% | 18% |
| Bulgaria | 157 | 4% | 11% | 87% | 9% |
| Croatia | 170 | 4% | 9% | 88% | 10% |
| Hungary | 175 | 2% | 7% | 91% | 10% |
| Slovenia | 178 | 2% | 4% | 93% | 11% |
| Lithuania | 158 | 1% | 6% | 92% | 13% |
Blocked, per AI crawler
| Crawler | Use | Websites blocking it |
|---|---|---|
GPTBot · OpenAI | Model training | 14% |
OAI-SearchBot · OpenAI | AI search and answers | 7% |
ChatGPT-User · OpenAI | AI search and answers | 8% |
ClaudeBot · Anthropic | Model training | 12% |
Claude-SearchBot · Anthropic | AI search and answers | 5% |
Claude-User · Anthropic | AI search and answers | 5% |
PerplexityBot · Perplexity | AI search and answers | 8% |
Perplexity-User · Perplexity | AI search and answers | 5% |
Google-Extended · Google | Other AI uses | 11% |
Applebot-Extended · Apple | Other AI uses | 9% |
meta-externalagent · Meta | Other AI uses | 10% |
Amazonbot · Amazon | Other AI uses | 12% |
CCBot | Model training | 14% |
Includes websites that block every crawler by default (User-agent: *): 3% of those measured.
Methodology
Sample: for each of the 27 EU countries, the first 200 websites on the country’s own domain (such as .de or .gr) among the 10,000 websites that Chrome users in that country visited most in 08/2026. One website per registered domain, secure (https) websites only. Source: Chrome UX Report (Google, CC BY 4.0).
Our crawler identifies itself as RankProofBot and follows each website’s robots.txt. Once a month it reads robots.txt, /llms.txt and the homepage, one request at a time per website.
A website blocks a crawler when its robots.txt disallows the homepage for that crawler under the standard rules (RFC 9309), including rules written for all crawlers. 755 of the 5,400 websites are left out of the shares: robots.txt answered with a bot check, a rate limit or a server error, or the website could not be reached.
An llms.txt file counts as valid when it is a text file that starts with a Markdown heading, as the llms.txt proposal requires. 4,260 websites could be checked.
No website is named: only totals per country are stored and published.
Press kit: quote, cite and reuse
Ready-to-quote figures from the latest month of this study, each with its source attached.
Ready-to-quote figures (5)
-
“10% of the measured websites block at least one AI search or answer crawler in robots.txt.” — RankProof, AI crawler blocking on EU websites, 08/2026 (4,645 websites in 27 EU countries)
-
“17% block at least one AI training crawler.” — RankProof, AI crawler blocking on EU websites, 08/2026 (4,645 websites in 27 EU countries)
-
“82% block no AI crawler at all.” — RankProof, AI crawler blocking on EU websites, 08/2026 (4,645 websites in 27 EU countries)
-
“9% of the websites we could check publish a valid llms.txt file.” — RankProof, AI crawler blocking on EU websites, 08/2026 (4,645 websites in 27 EU countries)
-
“The most blocked AI crawler is CCBot: 14% of the measured websites block it.” — RankProof, AI crawler blocking on EU websites, 08/2026 (4,645 websites in 27 EU countries)
Download the data
Free to reuse with credit to RankProof and a link to this page. CC BY 4.0
How to cite this dataset
RankProof Research, "AI crawler blocking on EU websites", 2026. https://rankproof.eu/research/ai-crawler-access-eu
Credit line with link (HTML)
<a href="https://rankproof.eu/research/ai-crawler-access-eu">Source: RankProof, AI crawler blocking on EU websites (08/2026)</a> (<a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>) Free tools on this topic
Guides on this topic
Measured every month, after each new monthly list is published.