Why are my log files in the hundreds of megabytes? Have LLMS, bots, and other crawlers made your once fast server seem slow? Have you tried fail2ban, crowdsec, and others to fight them off? Look no further. Instead of having the need to monitor docker containers, different robots.txt files, and the plethora of documentation each "new" fighter needs, let's tailor it to your server and your traffic.

Prerequisites

General

*nix system
sudo user

Standard

bash 4+
awk
curl
flock
jq

Ban backend (at least one)

The script uses the first one it finds, in this order (see Ban backends):

CrowdSec (cscli). It also needs a bouncer to actually block anything; status warns if none is registered
nftables (nft)
iptables and ipset. ipset is required; if only iptables is installed, the script logs a warning and won't use it

Third party

nginx
standard nginx log files

Optional

Package	Install (Ubuntu)	Enables
`mmdb-bin` (`mmdblookup`)	`sudo apt install mmdb-bin`	MaxMind GeoIP2 ISP / GeoLite2 ASN lookups, see MaxMind GeoIP
`geoipupdate`	`sudo apt install geoipupdate`	Downloading and updating the MaxMind databases
`dnsutils` (`dig`)	`sudo apt install dnsutils`	Team Cymru DNS lookups (last-resort ISP provider)

Install

Guide

Clone the repo git clone https://git.dou.bet/iamdoubz/nginx-log-crowdsec-blocker.git
Enter repo directory cd nginx-log-crowdsec-blocker
Install the bash script sudo install -m 755 crawl-ban.sh /usr/local/bin/crawl-ban.sh
Install the cron task sudo install -m 644 crawl-ban.cron /etc/cron.d/crawl-ban. It runs every 30 minutes, plus once 60 seconds after boot to re-apply bans
Install the config file sudo install -m 600 -o root -g root crawl-ban.conf.example /etc/crawl-ban.conf
Perform a dry run sudo DRY_RUN=1 /usr/local/bin/crawl-ban.sh
(optional) Create a whitelist for your public IP address echo "YOUR.STATIC.IP.ADDRESS # home" | sudo tee /var/lib/crawl-ban/whitelist.txt
Edit /etc/crawl-ban.conf to match when and what to block i.e. THRESHOLD=1000 will start blocking individual IP after 1000 hits are recorded. Settings there override the script's defaults, so you can update the script without losing them
Check everything is detected sudo crawl-ban.sh status

What the script does

Counting: each run counts requests across all non-empty *access.log files combined. A bot that splits 1,500 + 900 across two vhosts still trips the 2,000 limit
Range bans: IPs are also grouped into subnets (/24 for IPv4, /64 for IPv6). A subnet with at least SUBNET_THRESHOLD requests spread over at least SUBNET_MIN_IPS different IPs is banned as a whole range. This catches crawlers that rotate through many addresses. IPs inside a banned range aren't banned separately
Skipped IPs: private ranges, CGNAT, loopback, link-local and IPv6 ULA are skipped, plus anything in whitelist.txt. Your public IP belongs there
No repeat decisions (two checks):
1. If the backend (CrowdSec, nftables or ipset) already has an active ban for the IP or range, the script skips it
2. Once that 8h ban expires, the old hits are still in the log until it rotates. The script remembers the count at the time of the ban and only re-bans after 2,000 new requests. If the count drops below that baseline, it assumes the log rotated and resets
Repeat offenders: each ban lasts longer than the last (BAN_DURATION + BAN_INCREMENT per previous ban). The PERMANENT_AFTER-th ban of an IP, or the RANGE_PERMANENT_AFTER-th ban of a range, is permanent. A target that stays quiet for FORGIVE_AFTER after its last ban ended starts over at 0
Self-healing: every run re-applies any active ban the backend is missing. This covers reboots, a wiped CrowdSec database, and switching backends
ISP lookups: each target's ISP is looked up once and cached. Providers are tried in order, and one that rate-limits or fails is skipped until its cooldown ends. See ISP lookup providers

Documentation

Flags

These are called after the script file sudo crawl-ban.sh <flag-name>

Name	Description
	Simply runs the script once. This is what cron uses
status	Shows the config file in use, backend, ban counts, ISP providers, health checks, and every setting's current value. Settings changed from the default are marked `*` with where they came from (config or env) and the default
report	Lists offenders found in the `STATE_FILE`
permanent	Lists any offenders that have been permanently banned
unban	Given an IP/CIDR, lift the ban, permanent flag, and keep history
pardon	Given an IP/CIDR, unban and reset the ban count to 0
ban-permanent	Given an IP/CIDR, promote straight to permanent ban status
lookup	Given an IP, ask every ISP provider about it and show each answer (or why it was skipped). Useful for testing providers
help	Show the built-in help (`--help` and `-h` work too)

Script variables

Set these in /etc/crawl-ban.conf (recommended) or as environment variables, e.g. on the cron line. Values in the config file win.

Name	Purpose	Default
THRESHOLD	Number of requests before ban takes effect per single IP	2000
SUBNET_THRESHOLD	Number of requests before ban takes effect per subnet	5000
SUBNET_MIN_IPS	Distinct IPs needed before a subnet range ban	5
SUBNET_V4	Subnet range for IPv4. Can be 8 (57.0.0.0/8), 16 (57.141.0.0/16), or 24 (57.141.4.0/24)	24
SUBNET_V6	Subnet range for IPv6. Can be 48 or 64	64
BAN_DURATION	How long the ban should last	8h
BAN_INCREMENT	Time added for each repeat ban: ban N lasts `BAN_DURATION` + (N-1) × `BAN_INCREMENT`	2h
MAX_BAN_DURATION	Set a cap on the length of `BAN_DURATION` for repeat offenders. 0 means no cap	0
PERMANENT_AFTER	After this many bans of an IP it becomes permanent. 0 = never	10
RANGE_PERMANENT_AFTER	After this many bans of a CIDR it becomes permanent. 0 = never	5
PERMANENT_DURATION	Length of a "permanent" CrowdSec decision (about 10 years). nftables/ipset permanent bans have no timeout	87600h
FORGIVE_AFTER	If an IP or CIDR stays quiet this long after its last ban ended, its ban count resets to 0. 0 = off	30d
BACKEND	Which backend to use: auto, crowdsec, nftables, iptables	auto
PERMANENT_ALSO_FIREWALL	If using CrowdSec, also add permanent bans to nftables (or ipset) as a backup	0
LOG_GLOB	Where to parse the nginx log files	`/var/log/nginx/*access.log`
CONFIG_FILE	Optional bash file of settings, loaded before the defaults. Refused if group/world-writable	`/etc/crawl-ban.conf`
STATE_DIR	Directory to store history, whitelist and lookup cooldown files	`/var/lib/crawl-ban`
STATE_FILE	File used for run and ban history	`$STATE_DIR/offenders.tsv`
PERM_FILE	Source of truth for permanent bans	`$STATE_DIR/permanent.txt`
BACKEND_FILE	Backend used on the last execution of the script	`$STATE_DIR/backend`
WHITELIST	If you want to exclude any IPs or CIDRs from being processed by this script. One per line. Comments `#` are okay to be in file after entry	`$STATE_DIR/whitelist.txt`
EVENT_LOG	File to store script logs	`/var/log/crawl-ban.log`
NGINX_DENY_FILE	Optional: write permanent bans as nginx "deny" lines i.e. `/etc/nginx/crawl-ban-deny.conf`
NGINX_RELOAD	What post run command to execute after deny file is (re)populated	nginx -t -q && systemctl reload nginx
LOOKUP_PROVIDERS	Ordered list of ISP lookup providers. A bash array, so set it in `CONFIG_FILE`. See ISP lookup providers	geoip, nordvpn, ipwhois, ip-api, ipapi.co, ipinfo, freeipapi, geolite, cymru
LOOKUP_URL	Deprecated. If set, tried before `LOOKUP_PROVIDERS` as `<LOOKUP_URL>/<ip>`
IPINFO_TOKEN	Free token from ipinfo.io; turns on the `ipinfo` provider
LOOKUP_TIMEOUT	Seconds allowed per HTTP lookup request	10
LOOKUP_COOLDOWN	How long to skip a rate-limited provider that didn't say how long to wait	1h
LOOKUP_ERROR_COOLDOWN	How long to skip a provider that is down, timing out or erroring	5m
ISP_REFILL_PER_RUN	Active bans with "Unknown ISP" to look up again each run	5
GEOIP_ISP_DB	Path to MaxMind `GeoIP2-ISP.mmdb`. Empty = search `GEOIP_DIRS`
GEOLITE_ASN_DB	Path to MaxMind `GeoLite2-ASN.mmdb`. Empty = search `GEOIP_DIRS`
GEOIP_DIRS	Directories searched for the MaxMind databases	`/var/lib/GeoIP /usr/share/GeoIP /usr/local/share/GeoIP`
GEOIP_MAX_AGE	Warn when a MaxMind database file is older than this	30d
DRY_RUN	Run the script using current configuration without banning	0
CSCLI	Crowdsec command/program name	cscli
NFT	nftables command/program name	nft
NFT_TABLE	nftables ban table name	crawlban
IPTABLES	iptables IPv4 command/program name	iptables
IP6TABLES	iptables IPv6 command/program name	ip6tables
IPSET	ipset command/program name	ipset
IPSET_PREFIX	ipset ban table prefix name	crawlban
MMDBLOOKUP	mmdblookup command/program name	mmdblookup
DIG	dig command/program name	dig

ISP lookup providers

The ISP name goes into the ban reason (Excessive crawling - <ISP>). Providers in LOOKUP_PROVIDERS are tried top to bottom until one answers.

Provider	Key needed	Free limit	Returns
`geoip`	Paid MaxMind subscription	None (local file)	ISP, then organization, then AS organization
`nordvpn`	No	Not documented	ISP
`ipwhois` (ipwho.is)	No	1,000/day	ISP, then organization
`ip-api` (ip-api.com)	No	45/minute, HTTP only, non-commercial use	ISP, then organization
`ipapi.co` (ipapi.co)	No	1,000/day, 30,000/month	Organization
`ipinfo` (IPinfo Lite)	Free token (`IPINFO_TOKEN`)	Unlimited	AS name
`freeipapi` (freeipapi.com)	No	60/minute	AS organization
`geolite`	Free MaxMind account	None (local file)	AS organization
`cymru` (Team Cymru)	No, needs `dig`	None stated (DNS)	AS name

AS names are the network owner's registered name (e.g. MICROSOFT-CORP-MSN-AS-BLOCK) rather than a friendly ISP name, which is why those providers come last. Providers that aren't set up (no database, no token, no dig) are skipped silently.

Rate limits and switching

A provider is put on cooldown, and the next one is used, when it:

answers HTTP 429: skipped for its Retry-After time, ip-api's X-Ttl, or LOOKUP_COOLDOWN if neither is given
answers with a rate-limit error in the body, even on HTTP 200 (ipapi.co's "reason":"RateLimited", ipwho.is's "Rate limit exceeded")
answers 401/403: skipped for LOOKUP_COOLDOWN
times out, can't be reached or answers 5xx: skipped for LOOKUP_ERROR_COOLDOWN
sends ip-api's X-Rl: 0 (window used up): the answer is still used, then ip-api is paused until the window resets. ip-api bans addresses that keep hitting 429s

Cooldowns are saved in $STATE_DIR/lookup-cooldowns.tsv, so they carry across runs, and each one is written to EVENT_LOG. sudo crawl-ban.sh status lists every provider with its state. If every provider fails, the target is banned with Unknown ISP, and later runs fill in the name (up to ISP_REFILL_PER_RUN per run).

Custom providers

Each LOOKUP_PROVIDERS entry is either a built-in name (geoip, geolite, cymru) or name|url|jq-filter for any JSON API. {ip} and {IPINFO_TOKEN} in the URL are filled in, and the jq filter picks the ISP field:

LOOKUP_PROVIDERS=(
    "geoip"
    "nordvpn|https://web-api.nordvpn.com/v1/ips/lookup/{ip}|.isp"
    "myapi|https://example.com/ip/{ip}|.network.isp // .network.org"
    "cymru"
)

Test the chain with sudo crawl-ban.sh lookup <ip>.

MaxMind GeoIP (optional)

MaxMind's databases are local files, so lookups are instant, have no rate limits and make no HTTP calls. The script finds them automatically in GEOIP_DIRS.

Database	Cost	Gives	Provider
GeoIP2 ISP	$49/month or $539/year (pricing)	ISP and organization names	`geoip`
GeoLite2 ASN	Free with an account	AS organization only	`geolite`

Setup

Buy the GeoIP2 ISP subscription (or create a free GeoLite account) and generate a license key in your MaxMind account
Install the tools sudo apt install geoipupdate mmdb-bin. If your release doesn't package them, both are on MaxMind's GitHub
In /etc/GeoIP.conf set AccountID, LicenseKey and EditionIDs GeoIP2-ISP GeoLite2-ASN (keep only the editions you have). MaxMind offers a pre-filled copy in your account
Download the databases sudo geoipupdate
Keep them updated, e.g. in /etc/cron.d/geoipupdate: 17 4 * * 1,4 root /usr/bin/geoipupdate. MaxMind's license requires current data, and the script warns once a day when a database is older than GEOIP_MAX_AGE
Check sudo crawl-ban.sh status shows geoip ready: ..., then try sudo crawl-ban.sh lookup 74.7.242.28

If DatabaseDirectory in /etc/GeoIP.conf points outside GEOIP_DIRS, set GEOIP_ISP_DB / GEOLITE_ASN_DB in /etc/crawl-ban.conf.

Ban backends

Backend	How bans are stored	Expiry
`crowdsec`	`cscli decisions add`; a CrowdSec bouncer enforces them	CrowdSec decision duration
`nftables`	Own table `inet crawlban`, sets `ban4`/`ban6`, dropped at raw/prerouting (priority -300)	Per-entry timeout; permanent bans have none
`iptables`	ipsets `crawlban4`/`crawlban6` with a `DROP` rule in the raw table's `PREROUTING` chain	Per-entry timeout; permanent bans have none

Dropping at raw/prerouting catches traffic before your other rules, including ports Docker publishes directly
nftables sets and ipsets don't survive a reboot; the @reboot cron line and the every-run re-apply put active bans back
When the backend changes (e.g. CrowdSec is uninstalled), active bans move to the new one with their remaining time. When CrowdSec comes back, the script removes its own nftables table and ipsets
With CrowdSec, status warns if no bouncer is registered, because then decisions are recorded but nothing enforces them

Future features

Use nftables and/or iptables+ipset if crowdsec not installed
Add more ISP providers and automatic switching if limit is reached
Add GeoIP ISP lookup (paid service)
Add method for repeat offenders to increase ban time and permanently ban once a threshold is reached
Convert *nix script to powershell for Windows users
Expand log types
- apache
- litespeed
- caddy
- node.js
- lighttpd
- IIS

Other

Here is some useful information and commands that you can run after getting everything setup.

Subject	Description	Example output
`grep crawl-ban /var/log/syslog`	Show all run cronjobs from the installed cron file	`Oct 2 10:39:01 computer CRON[2445869]: (root) CMD (/usr/local/bin/crawl-ban.sh >/dev/null 2>&1)`
`tail -f /var/log/crawl-ban.log`	Show script's own log file contents (only writes on bans/errors)	`2026-10-02 09:48:28 BANNED range(69 IPs) 57.141.4.0/24 (67596 reqs) for 8h: Excessive crawling - Facebook [ban #1]`
`sudo cscli decisions list`	Show currently blocked IPs and ranges by crowdsec	`1813803 cscli Ip:74.7.242.28 Excessive crawling - Microsoft Azure ban 1 7h9m42s 855`
`ls -lhS /var/log/nginx/*access.log \| awk '$5 != "0"' \| head -10`	Show top 10 log files that are not empty sorted by file size largest to smallest	`-rw-r----- 1 www-data www-data 35M Oct 2 09:32 /var/log/nginx/gitea_access.log`
`awk '{print $1}' /var/log/nginx/gitea_access.log \| sort \| uniq -c \| sort -rn \| head -5`	Show top 5 unique IPs in a certain log file	`2193 161.153.113.113`
`sudo crawl-ban.sh lookup 74.7.242.28`	Ask every ISP provider about an IP	`nordvpn Microsoft Azure`
`grep "ISP lookup" /var/log/crawl-ban.log`	Show provider cooldowns and lookup failures	`2026-10-06 13:44:01 ISP lookup: nordvpn rate limited (HTTP 429); skipping it for 3m 0s`
`sudo nft list table inet crawlban`	Show the nftables ban sets (nftables backend only)	`elements = { 57.141.4.0/24 timeout 8h expires 7h52m3s }`

README.md Unescape Escape

nginx-log-crowdsec-blocker

Why