Here is a php script that blocks many AI bots from accessing my website.
$agent = $_SERVER['HTTP_USER_AGENT'];
$botarray = array("GPTBot", "ClaudeBot", ..., "Big Sur AI", "GrokBot");
if (in_array($agent, $botarray)) {
echo '<meta http-equiv="refresh" content="0;url=https://art-noah-s.eu/.htaccess">';
exit();
}…
if (in_array($agent, $botarray)) {[code to execute]} is a conditional code. It only executes whats between { and } if the condition is met. In this case it’s that $agent is listed somewhere in $botarray. In summary, if you’re trying to access my site with something listed in that long list of bots, the code activates and otherwise it doesn’t do anything.
echo ‘meta http-equiv=”refresh” content=”0;url=https://art-noah-s.eu/.htaccess”‘; (<> tags removed for functionality) is a piece of code that immediately redirects to my htaccess file.
…
My .htaccess starts with this code.
<Files .htaccess>
order allow,deny
deny from all
</Files>…
I like that you found a solution that works for you, but I wonder if this could have been optimised a little. Here’s what you’ve got so far:
- So long as the bot hits a URL that’s run through PHP, your code runs (limitation:probably doesn’t work on e.g. image URLs, stylesheets, static pages if you have any, etc.)
- If the user-agent matches a banlist, you send some HTML code to ask them to redirect (possible improvement: if you want to redirect, you can do that faster with a HTTP header than with HTML)
- The page you redirect them to is a
.htaccessfile, which applies to the whole directory, but contains a rule that only applies to itself and which blocks all access (possible improvement: if your server supports .htaccess file, you can probably do the whole thing with just that!)
I assume that you’re doing this rather than just a robots.txt file because you don’t trust those bots to respect robots.txt (although this does imply that
you do trust them enough not to spoof their user-agent to get around the blocks!).
Alternative 1: HTTP 403s in pure PHP
Let’s take some improvements one step at a time. First, in PHP: rather than doing a HTML-redirect to a blocked page, why don’t we just send the HTTP 403 right then and there:
if (in_array($agent, $botarray)) { header("HTTP/1.1 403 Forbidden"); exit(); }
This obviates the need for a .htaccess file at all and removes the need for a redirect. If you’re sticking to a PHP-only approach, that’s how I’d do it.
Alternative 2: Blocking at the webserver
An even stronger solution would be to drop the PHP code entirely and switch to blocking at the webserver! This saves your resources (webservers are great at filtering on headers and they’re less computationally-expensive than spinning up a FCGI thread/PHP interpreter) and it means that your blocks can apply not only to URLs served using PHP but also to static files!
Blocking badly-behaved bots in Caddy
My go-to webserver these days is Caddy, where I’d use some configuration like this in my Caddyfile:
@aibots header_regexp User-Agent "GPTBot|ClaudeBot|Claude-User|Claude-SearchBot|CCBot|Google-Extended|Applebot-Extended|Facebookbot|Meta-ExternalAgent|Meta-ExternalFetcher|diffbot|PerplexityBot|Perplexity‑User|Omgili|Omgilibot|webzio-extended|ImagesiftBot|Bytespider|TikTokSpider|Amazonbot|Youbot|SemrushBot-OCOB|Petalbot|VelenPublicWebCrawler|TurnitinBot|Timpibot|OAI-SearchBot|ICC-Crawler|AI2Bot|AI2Bot-Dolma|DataForSeoBot|AwarioBot|AwarioSmartBot|AwarioRssBot|Google-CloudVertexBot|PanguBot|Kangaroo Bot|Sentibot|img2dataset|Meltwater|Seekr|peer39_crawler|cohere-ai|cohere-training-data-crawler|DuckAssistBot|Scrapy|Cotoyogi|aiHitBot|Factset_spyderbot|FirecrawlAgent|bedrockbot|DeepSeekBot|GoogleAgent-Mariner|Gemini-Deep-Research|Google-NotebookLM|Google-Agent|GoogleAgent-URLContext|Google-Firebase|Googlebot|bingbot|MistralAI-User|SemrushBot|SemrushBot-FT|SemrushBot-ESI|AddSearchBot|bigsur.ai|Brightbot|Crawlspace|EchoboxBot|FriendlyCrawler|LinerBot|Panscient|Panscient.com|Poseidon Research Crawler|SBIntuitionsBot|TerraCotta|Thinkbot|Yak|YandexAdditional|YandexAdditionalBot|anthropic-ai|Slurp|Baiduspider|YandexBot|AhrefsBot|MJ12Bot|facebookexternalhit|TwitterBot|LinkedInBot|SlackBot|ChatGPT-User|Soguo web spider|Claude-Web|magpie-crawler|ChatGPT-User|OpenAI|Claude-Code|quillbot.com|MyCentralAIScraperBot|ChatGPT Operator|Grok|DigitalOceanGenAICrawler|LINER Bot|Big Sur AI|GrokBot" abort @aibots
Blocking badly-behaved bots in Apache
Based on your use of .htaccess files, though, it sounds like you’re using Apache. If you replaced your .htaccess file’s contents with this, you’d probably get
the same result (I’m assuming you’ve got mod_rewrite enabled already).
RewriteEngine On RewriteCond %{HTTP_USER_AGENT} ^.*(GPTBot|ClaudeBot|Claude-User|Claude-SearchBot|CCBot|Google-Extended|Applebot-Extended|Facebookbot|Meta-ExternalAgent|Meta-ExternalFetcher|diffbot|PerplexityBot|Perplexity‑User|Omgili|Omgilibot|webzio-extended|ImagesiftBot|Bytespider|TikTokSpider|Amazonbot|Youbot|SemrushBot-OCOB|Petalbot|VelenPublicWebCrawler|TurnitinBot|Timpibot|OAI-SearchBot|ICC-Crawler|AI2Bot|AI2Bot-Dolma|DataForSeoBot|AwarioBot|AwarioSmartBot|AwarioRssBot|Google-CloudVertexBot|PanguBot|Kangaroo Bot|Sentibot|img2dataset|Meltwater|Seekr|peer39_crawler|cohere-ai|cohere-training-data-crawler|DuckAssistBot|Scrapy|Cotoyogi|aiHitBot|Factset_spyderbot|FirecrawlAgent|bedrockbot|DeepSeekBot|GoogleAgent-Mariner|Gemini-Deep-Research|Google-NotebookLM|Google-Agent|GoogleAgent-URLContext|Google-Firebase|Googlebot|bingbot|MistralAI-User|SemrushBot|SemrushBot-FT|SemrushBot-ESI|AddSearchBot|bigsur.ai|Brightbot|Crawlspace|EchoboxBot|FriendlyCrawler|LinerBot|Panscient|Panscient.com|Poseidon Research Crawler|SBIntuitionsBot|TerraCotta|Thinkbot|Yak|YandexAdditional|YandexAdditionalBot|anthropic-ai|Slurp|Baiduspider|YandexBot|AhrefsBot|MJ12Bot|facebookexternalhit|TwitterBot|LinkedInBot|SlackBot|ChatGPT-User|Soguo web spider|Claude-Web|magpie-crawler|ChatGPT-User|OpenAI|Claude-Code|quillbot.com|MyCentralAIScraperBot|ChatGPT Operator|Grok|DigitalOceanGenAICrawler|LINER Bot|Big Sur AI|GrokBot).*$ [NC] RewriteRule .* - [R=403,L]
This solution blocks those bots’ access to your entire site (even e.g. images), and doesn’t hit PHP at all (so it’s fast and saves your processors for the traffic you care about).
I’ve not tested it (again: I don’t often use Apache any more: I migrated to Nginx for the fancy features and I’m now mostly-migrated to Caddy for the simple configuration), but I think I’m close to correct; some tweaking might be needed.
Testing your blocks
If you want to thoroughly test your blocks, you can trivially spoof your user-agent from curl, e.g.:
~ # Request the page as Firefox, get a HTTP 200: ✅ ~ curl -sI -H "User-Agent: Firefox" https://art-noah-s.eu/ | head -n1 HTTP/2 200 ~ # Request the page as GPTBot, get a HTTP 403: 🎉 ~ curl -sI -H "User-Agent: GPTBot" https://art-noah-s.eu/ | head -n1 HTTP/2 403
What you’ve already good seems good enough for your needs, Noah. But if you want to take it to the next level, hopefully the ideas above will help!