Skip to content
AI crawlers · monthly Live

AI crawler blocking on EU websites

How popular national websites in each EU country treat AI search and training crawlers in robots.txt, and how many publish llms.txt.

Updated · Sample month 10/2026 · 4,554 of 5,400 websites measured in 27 countries · Press kit

Key findings

  • 11% of the measured websites block at least one AI search or answer crawler in robots.txt.
  • 21% block at least one AI training crawler.
  • 77% block no AI crawler at all.
  • 10% of the websites we could check publish a valid llms.txt file.
  • The most blocked AI crawler is CCBot: 19% of the measured websites block it.

11%

Block an AI search or answer crawler

21%

Block an AI training crawler

77%

Block no AI crawler

10%

Publish a valid llms.txt

By country

The EU total comes first, followed by the countries from the highest share to the lowest.

Country Measured AI search blocked AI training blocked No AI blocked llms.txt
EU (27 countries) 4,554 11% 21% 77% 10%
Germany 176 26% 44% 55% 8%
Finland 170 22% 28% 71% 3%
Italy 161 22% 33% 65% 6%
Spain 162 19% 28% 72% 6%
Netherlands 173 18% 24% 74% 11%
France 162 17% 29% 69% 6%
Sweden 183 16% 22% 77% 6%
Denmark 169 15% 24% 76% 15%
Belgium 170 15% 22% 75% 10%
Ireland 167 13% 22% 76% 5%
Austria 163 12% 23% 74% 6%
Czechia 180 10% 23% 73% 9%
Poland 181 8% 27% 70% 12%
Estonia 171 8% 14% 85% 12%
Luxembourg 170 8% 12% 87% 9%
Portugal 164 7% 18% 80% 11%
Cyprus 158 7% 16% 83% 14%
Croatia 180 6% 12% 86% 12%
Bulgaria 170 6% 15% 84% 8%
Malta 143 6% 18% 80% 10%
Greece 164 5% 12% 86% 13%
Latvia 167 5% 19% 81% 10%
Slovakia 174 5% 17% 79% 13%
Lithuania 159 5% 21% 75% 14%
Romania 168 4% 17% 82% 14%
Hungary 179 3% 15% 83% 9%
Slovenia 170 1% 7% 90% 12%
Blocked, per AI crawler
Crawler Use Websites blocking it
GPTBot · OpenAI Model training 18%
OAI-SearchBot · OpenAI AI search and answers 7%
ChatGPT-User · OpenAI AI search and answers 9%
ClaudeBot · Anthropic Model training 16%
Claude-SearchBot · Anthropic AI search and answers 6%
Claude-User · Anthropic AI search and answers 6%
PerplexityBot · Perplexity AI search and answers 9%
Perplexity-User · Perplexity AI search and answers 7%
Google-Extended · Google Other AI uses 15%
Applebot-Extended · Apple Other AI uses 14%
meta-externalagent · Meta Other AI uses 15%
meta-webindexer · Meta AI search and answers —
meta-externalfetcher · Meta AI search and answers —
ExaSearchBot · Exa AI search and answers —
ExaUser · Exa AI search and answers —
KeenableBot · Keenable AI search and answers —
Keenable-User · Keenable AI search and answers —
Amazonbot · Amazon Other AI uses 17%
CCBot Model training 19%

Includes websites that block every crawler by default (User-agent: *): 4% of those measured.

Month by month
Month Measured AI search blocked AI training blocked llms.txt
10/2026 4,554 11% 21% 10%
09/2026 4,523 11% 18% 8%
08/2026 4,645 10% 17% 9%

Methodology

Sample: for each of the 27 EU countries, the first 200 websites on the country’s own domain (such as .de or .gr) among the 10,000 websites that Chrome users in that country visited most in 10/2026. One website per registered domain, secure (https) websites only. Source: Chrome UX Report (Google, CC BY 4.0).

Our crawler identifies itself as RankProofBot and follows each website’s robots.txt. Once a month it reads robots.txt, /llms.txt and the homepage, one request at a time per website.

A website blocks a crawler when its robots.txt disallows the homepage for that crawler under the standard rules (RFC 9309), including rules written for all crawlers. 846 of the 5,400 websites are left out of the shares: robots.txt answered with a bot check, a rate limit or a server error, or the website could not be reached.

An llms.txt file counts as valid when it is a text file that starts with a Markdown heading, as the llms.txt proposal requires. 4,065 websites could be checked.

No website is named: only totals per country are stored and published.

Press kit: quote, cite and reuse

Ready-to-quote figures from the latest month of this study, each with its source attached.

Ready-to-quote figures (5)

  • “11% of the measured websites block at least one AI search or answer crawler in robots.txt.” — RankProof, AI crawler blocking on EU websites, 10/2026 (4,554 websites in 27 EU countries)

  • “21% block at least one AI training crawler.” — RankProof, AI crawler blocking on EU websites, 10/2026 (4,554 websites in 27 EU countries)

  • “77% block no AI crawler at all.” — RankProof, AI crawler blocking on EU websites, 10/2026 (4,554 websites in 27 EU countries)

  • “10% of the websites we could check publish a valid llms.txt file.” — RankProof, AI crawler blocking on EU websites, 10/2026 (4,554 websites in 27 EU countries)

  • “The most blocked AI crawler is CCBot: 19% of the measured websites block it.” — RankProof, AI crawler blocking on EU websites, 10/2026 (4,554 websites in 27 EU countries)

Download the data

Free to reuse with credit to RankProof and a link to this page. CC BY 4.0

How to cite this dataset

RankProof Research, "AI crawler blocking on EU websites", 2026. https://rankproof.eu/research/ai-crawler-access-eu

Every country row and every figure above has its own address: click a country name and copy the link from your browser, for example https://rankproof.eu/research/ai-crawler-access-eu#country-de.

Credit line with link (HTML)

<a href="https://rankproof.eu/research/ai-crawler-access-eu">Source: RankProof, AI crawler blocking on EU websites (10/2026)</a> (<a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>)

Measured every month, after each new monthly list is published.