Re: Blunt Banhammer Blocks Basic Bots Ballistically

I cut bot traffic to my site by over 80% with a simple trap.

Last month, my hobby website OpenBenches.org transferred more than 2.5TB of data to visitors. As much as I'd like to think the site is enormously popular, it looks like it is just AI bots endlessly crawling the data and slurping up the images.

So I've built B⁶ to be an indiscriminate tool to eradicate these unwanted parasites.

It is very loosely inspired by Jeff Starr's Blackhole, but works even on sites with heavy caching.

Here's the high level description of how it works.

• Create a directory like /honeypot/ on your site
• Put some code in index.php within the directory
• Add the honeypot as an explicit "Disallow" in robots.txt
• Include a hidden link to to the honeypot on every page of your site
• If something accesses that page, write its IP address to your .htaccess
• The .htaccess bans the IP address from ever accessing your site again.

...

You've got the be really careful with this, and I don't think your implementation is careful enough! A friend of mine implemented something almost-identical to this on a forum they run, earlier this year, and I quickly found that it could be exploited as a denial-of-service attack. That, in turn, was based on my own experience of implementing something like this the other year on geohashing.site, which has long had a terrible problem with badly-behaved bots.

In the site you're defending, openbenches.org , the "poisoned" URL is at /quantum/index.php. It's easy for an attacker to find: under the assumption that you'd have tried to protect users of accessibility technologies from accidentally triggering it, I just ran document.querySelector('a[aria-hidden]') in my debug console and I saw it instantly.

Now to craft an attack: all I need to do is to trick a legitimate user into making a request to that URL, and they're permabanned! And that might be as simple as making a "trap" HTML page (or email!) that contains some code like this:

<img src="//openbenches.org/quantum/index.php">

Now, anybody who views that page gets their IP permanently blocked from accessing OpenBenches. Erk! That poisoned code could be embedded anywhere: in a forum post, in a GitHub issue (even against the actual repository!), in an email. And there's plenty of alternative ways to trigger it too: a plain hyperlink, a fetch() request (CORS won't save you if even the OPTIONS request hits the honeypot), the action of a <form>, etc.

I'm sure this honeypot is effective, but in my opinion it's also dangerous.

So what could we do instead? Well: here's my preferred approach:

1. When you hit the "honeypot" URL, don't instaban (yet). Instead, generate a signed token by hashing the current date and time, the suspected-attacker's IP address, and a secret salt.
2. Write this back to the page as an actual honeypot link, e.g. <a href="/honeypot/index.php?token=...">.
3. If a badly-behaved bot crawls this second link, you can detect it: just check the hash matches and is relatively-recent and if so: trigger the permaban.

If you're willing to accept the extra processing time, you can even skip the first step and embed the URLs it would generate onto every page, as hidden links. This is now safe from denial-of-service attacks because the attacker's hash won't match when they trick an unsuspecting user into requesting it, because their IP address isn't part of the signed token.

I made a demo of this at some point; I can probably dig up the code, tidy it and open-source it if you'd like.