If I didn’t enjoy the cat & mouse of it, I’d definitely walk away from running a site. Just out of curiosity, I created an hourly graph today showing malicious bot, legitimate bot and human ratio graphs for one of my sites. The malicious hits make up around 80% hourly and human hits are usually well under 10%.
This IP from Hong Kong hit my site over 5,000 times in a couple minutes. The first 12 page loads were legitimate URLs and got served. On their first exploit attempt, they were blocked and the remaining 4,98x page loads were no-effort 403s. This type of thing has made it possible for me to have a cheap public-facing site as I wouldn’t be able to afford an endless array of malicious IPs doing this day in and day out. Before I went this route, I would have to shut the site down during high impact periods of these visitors as they’d corrupt and crash the database.
As soon as an IP is determined to be malicious, subsequent pageloads result in a single query to match the IP and they get dumped to a 403 static error.
With the bots that hammer a site looking for the top 1,000 or so exploitable URLs or partake in speed scraping, this has worked wonders for my server load. I catch them in their 1st-5th page load and the rest just result in basically no effort on the part of the script.
If I didn’t enjoy the cat & mouse of it, I’d definitely walk away from running a site. Just out of curiosity, I created an hourly graph today showing malicious bot, legitimate bot and human ratio graphs for one of my sites. The malicious hits make up around 80% hourly and human hits are usually well under 10%.
What sorts of measure you have taken to prevent bots from exhausting your compute and bandwidth
Another example expanding on my previous comment.
This IP from Hong Kong hit my site over 5,000 times in a couple minutes. The first 12 page loads were legitimate URLs and got served. On their first exploit attempt, they were blocked and the remaining 4,98x page loads were no-effort 403s. This type of thing has made it possible for me to have a cheap public-facing site as I wouldn’t be able to afford an endless array of malicious IPs doing this day in and day out. Before I went this route, I would have to shut the site down during high impact periods of these visitors as they’d corrupt and crash the database.
As soon as an IP is determined to be malicious, subsequent pageloads result in a single query to match the IP and they get dumped to a 403 static error.
With the bots that hammer a site looking for the top 1,000 or so exploitable URLs or partake in speed scraping, this has worked wonders for my server load. I catch them in their 1st-5th page load and the rest just result in basically no effort on the part of the script.