Bad Bot Blocker¶
The Nginx Ultimate Bad Bot Blocker is a set of configuration files for Nginx that block over 4000 bad referers, spam referrers, user-agents, bad bots, bad IP's, porn, gambling and clickjacking sites, lucrative seo companies, and wordpress theme detectors. It is regularly maintained and easily updateable.
https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
Installing Bad Bot Blocker¶
Obtain the globalblacklist.conf file and place into your /etc/nginx/conf.d folder.
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/conf.d/globalblacklist.conf -O /etc/nginx/conf.d/globalblacklist.conf
Create a config directory to store the include files:
sudo mkdir /etc/nginx/bots.d
Download and place the following files into that folder:
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/blockbots.conf -O /etc/nginx/bots.d/blockbots.conf
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/ddos.conf -O /etc/nginx/bots.d/ddos.conf
Download the whitelist configuration files:
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/whitelist-ips.conf -O /etc/nginx/bots.d/whitelist-ips.conf
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/whitelist-domains.conf -O /etc/nginx/bots.d/whitelist-domains.conf
Add your static IPs' to the whitelist-ips.conf file and your domains to the whitelist-domains.conf file.
Now download custom blacklist files for bad user-agents, bad referrers, and bad IP addresses or ranges:
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/blacklist-user-agents.conf -O /etc/nginx/bots.d/blacklist-user-agents.conf
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/custom-bad-referrers.conf -O /etc/nginx/bots.d/custom-bad-referrers.conf
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/blacklist-ips.conf -O /etc/nginx/bots.d/blacklist-ips.conf
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/bots.d/bad-referrer-words.conf -O /etc/nginx/bots.d/bad-referrer-words.conf
These files are for you to edit with your own additions and won't be overwritten when you update the lists. The instructions on how to use them are within the comments in the files. Just open in nano to read.
Now we get the botblocker-nginx-settings.conf file. This file contains Nginx settings for rate limiting:
sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/conf.d/botblocker-nginx-settings.conf -O /etc/nginx/conf.d/botblocker-nginx-settings.conf
Open up the Nginx config file:
sudo nano /etc/nginx/nginx.conf
and ensure it contains the following directive. Add it if not:
include /etc/nginx/conf.d/*;
And ensure you have the following 2 lines added to your virtual host files within a server block:
include /etc/nginx/bots.d/blockbots.conf;
include /etc/nginx/bots.d/ddos.conf;
Adding a dedicated log file¶
Now we'll create a seperate log in nginx specifically for badbots, which we can then use in fail2ban. First make the following changes in the blockbots.conf file:
sudo nano /etc/nginx/bots.d/blockbots.conf
# UNCOMMENT THE NEXT 4 LINES TO ACTIVATE THE SUPER WHITELIST
#if ($remote_addr ~ "(127.0.0.1)|(192.168.0.1)" ) {
#set $bad_bot '0'; #Uncommenting this line will disable bad_bots functionality for specified IP(s)
#set $validate_client '0'; #Uncommenting this line will disable validate_client ip blocking functionality for specified IP(s)
#}
set $bbcode '0';
# --------------
# BLOCK BAD BOTS
# --------------
# Section bot_1 Unused
#limit_conn bot1_connlimit 100;
#limit_req zone=bot1_reqlimitip burst=50;
limit_conn bot2_connlimit 10;
limit_req zone=bot2_reqlimitip burst=10;
if ($bad_bot = '3') {
set $logbb 1;
set $bbcode 'Type: Bad Bot';
return 444;
}
# ---------------------
# BLOCK BAD REFER WORDS
# ---------------------
if ($bad_words) {
set $logbb 1;
set $bbcode 'Type: Bad Word';
return 444;
}
# ------------------
# BLOCK BAD REFERERS
# ------------------
if ($bad_referer) {
set $logbb 1;
set $bbcode 'Type: Bad Referrer';
return 444;
}
# -----------------------------
# BLOCK IP ADDRESSES and RANGES
# -----------------------------
if ($validate_client) {
set $logbb 1;
set $bbcode 'Type: Bad IP Address';
return 444;
}
# Additional log for Bad Bots
access_log /var/log/nginx/access_badbots.log bb if=$logbb;
# If we have a global access log, we need to re-declare it here
access_log /var/log/nginx/access.log main;
Now create the custom log format in /etc/nginx/nginx.conf and also create a default value for the $logbb variable:
sudo nano /etc/nginx/nginx.conf
http {
...
log_format bb '$remote_addr - $remote_user [$time_local] $host "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" "$bbcode"';
...
map $host $logbb {
default 0;
}
}
Check the configuration with:
sudo nginx -t
And apply changes (if no errors) with:
sudo service nginx reload
Testing¶
Run the following commands one by one from a terminal on another linux machine against your own domain name. Substitute example.com in the examples below with your real domain name
curl -A "googlebot" http://example.com
Should respond with 200 OK
curl -A "80legs" http://example.com
curl -A "masscan" http://example.com
Should respond with: curl: (52) Empty reply from server
curl -I http://example.com -e http://100dollars-seo.com
curl -I http://example.com -e http://zx6.ru
Should respond with: curl: (52) Empty reply from server
Check the log:
sudo tail -n 10 /var/log/nginx/access_badbots.log
Updating¶
To update to the latest version automatically, create a script:
sudo nano /usr/local/sbin/update-badbotsblocker.sh
and enter the following:
wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/conf.d/globalblacklist.conf -O /etc/nginx/conf.d/globalblacklist.conf
service nginx reload
exit 0
Schedule it with a cron job:
sudo crontab -e
0 22 * * * /usr/local/sbin/update-badbotsblocker.sh > /dev/null 2>&1
Blocking bots with fail2ban¶
We can block persistent bad bots at firewall level using fail2ban in order to prevent excess stress on the Nginx server.
Create a filter:
sudo nano /etc/fail2ban/filter.d/nginx-badbots.conf
[Definition]
failregex = ^<HOST> .* 444 .*"Type: Bad Bot"$
ignoreregex =
Create a jail:
sudo nano /etc/fail2ban/jail.d/nginx-badbots.local
[nginx-badbots]
enabled = true
port = http,https
filter = nginx-badbots
logpath = /var/log/nginx/access_badbots.log
maxretry = 1
findtime = 36000
bantime = 86400
action = iptables-allports
Reload fail2ban:
sudo fail2ban-client reload