Using ban_hammer.conf correctly?

becca

Bots, most prominently meta-externalagent (Meta's AI crawler) are hitting our public Pawtucket instance harder than usual lately, and I want to address this before it becomes a problem for our users. In the past we've had issues with server overload. I will add that I am not very experienced with server administration.

Facebook and Amazon to be the biggest culprits, though there are others. I am seeing a lot of this kind of thing in the httpd access log so maybe this is some kind of "bot trapped in the Browse page" situation:

57.141.0.16 - - [24/Jul/2025:14:11:06 -0400] "GET /Search/Objects/key/key/f0e2c0de69a4c520ea5f84d744055275/facet/subject/id/1123/view/list HTTP/1.1" 200 57999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"

52.54.15.103 - - [24/Jul/2025:14:56:24 -0400] "GET /Browse/Objects/key/d2b43bdc51bf1d1da81e7160c2da455f/facet/entity/id/3531/view/list HTTP/1.1" 200 57999 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36"

I thought that ban_hammer.conf might help me out here (https://docs.collectiveaccess.org/pawtucket/ban_hammer) but I am not seeing any progress.

What I tried:

I added specific IPs and UserAgents to the ban list.
I added some of the options included in the latest Github version to our ban_hammer.conf (https://github.com/collectiveaccess/pawtucket2/blob/master/app/conf/ban_hammer.conf), particularly hoping that the use_ip_ban_feed option would help.
I made these changes to both the version of ban_hammer.conf in our theme directory and the version in app/conf).
Afterwards, I ran both the clear-bans and the clear-whitelist commands in caUtils

In addition, I added rules to robots.txt to disallow access to Search and Browse:

Disallow: /index.php/Search/Objects/*
Disallow: /index.php/Browse/Objects/*
Disallow: /Search/Objects/*
Disallow: /Browse/Objects/*

These attempts do not seem to have made any difference. I still see the same UserAgents in my httpd logs. (Maybe I need to wait longer than 24 hours?)

Can anyone offer advice as to what I might be doing wrong, or other options I might try?

We are running Providence 2.0 and Pawtucket 1.8.

becca

becca Here is our current ban_hammer.conf in case it's helpful.

https://drive.google.com/file/d/1GdPX0C8rLdx3TF1E7n6DlKKZNNPtpfJv/view?usp=sharing

monica

becca the newer options in ban hammer wouldn’t be working for version 1.8
You will need to restart your web service/php service to make sure the configs are read. The one in your theme directory is the one it’ll read.

The ban hammer doesn’t stop the bots hitting your site, but it stops them getting the content. It shows a “you’re banned” type of message.

Changes to robots.txt will take longer to work, as bots don’t read this every time they visit, but maybe every week or so. And only honest bots will read and obey the instructions here.

If you want to stop these bots hitting, then you need to identify them and stop them before they get to the page. You do this via your web service configuration, but you will probably need an experienced web admin to set this up.