When I said “let’s make the most of the higher site traffic limit,” I did not mean “Bingbot should start continuously body-slamming my comics with a new page request every few seconds.”
But that’s what happened. Here’s a graph of the past 30 days, where you can see, in late September, the CPU usage from the BICP site (for some reason, it has a special fixation on that one) tripled:
That’s not because a bunch of human readers decided to start bingeing the archive! It’s automated bot traffic, especially the Bing crawler.
On the advice of SiteGround customer support, I used a custom .htaccess rule to full-on block Bingbot (and a couple of other crawlers) about 24 hours ago. The dip at the end of the graph looks promising…but I checked this evening, it’s headed back up, and my access logs are still full of Bingbot hits.
I went back and belatedly added an extra block via robots.txt rule, along with individually blocking a bunch of Bing-specific IPs and IP ranges.
The bot hits are all getting logged as 403 errors now! But they haven’t stopped coming!
So, yeah. There is a real chance all my sites will go down again before the end of October — unless I pay for the next level of hosting service (an extra $50/month). Because a tiny number of bots are trying to spam their way through twice as many resources as all my real human readers put together.
In a decent world, I would just pay for it now, and then forward the bill to Microsoft.
I’ll keep an eye on the graph over the next few days. Best-case scenario, the small amount of creeping-up is manageable — but I’m leery about getting my hopes up for anything right now. (Hopefully my next update for you all will have a nicer ending than this one.)
Comment Header
11 Comments
I suppose it’s not the actual bot from Microsoft’s Bing, but another (broken) one masquerading as such, assuming that most websites want to be seen and thus won’t block search engine crawlers.
Unfortunately, it’s the real one. Searching for info about how to block Bingbot brings up a ton of posts from people with the same problem, and this one points out the ranges of verified-by-Microsoft Bing IP addresses, which also matches the IPs my spam was coming from.
Urgh. Can you block it at a Cloudflare level if your hosting has that option?
Frikkin’ crawlers. It’d be fine if they followed any of the rules, but noooo…
Cloudflare doesn’t show up in my admin panel. Not sure if the hosting company is using it, but I don’t seem to have access to mess with it.
You’d think they could internalize “no means no” some time during the first hour, let alone the first 24! If only.
Is investigating this the cause of no Sketchbook upload on Saturday, October 11?
Noticed that, huh. (Yes, yes it is.)
Yeah, it’s essentially unheard of for a day to go by without an upload. ❤️
… I know this is unhelpful, but if the bots consume JUST twice as many resources as real human users you are lucky.
There is still chance bingbot is just slow to react (it IS possible it needs more than 24 hours to update robots.txt, or to realize you are blocking it).
… wait. Shouldn’t returning 403 consume much less resources than rendering the page?
Yes, but 403 responses still consume some resources. Depending on the technology used, that some could be a fair amount.
I don’t know about the technology powering this website in particular, but a lot of websites written in either interpreted or compile at runtime languages like PHP and Perl will still consume most of the same CPU time, as loading the code to display the page takes more resources than the rest of the process combined. (To be clear, it’s my understanding both PHP and Perl are compile at runtime these days. I think they were both interpreted back in version 1, and I’m not sure when either made the switch.)
There may be tricks to get around this, depending on the language and platform in question, but these could be thwarted by ones hosting provider.
All of that said, returning 403s will almost certainly reduce the amount of bandwidth that gets used. I have seen some pretty heavy error pages, but most are pretty short and don’t include other content. I’d like to think that bots would not get super heavy error pages, but if bots count as click-throughs, then I would expect they might be especially heavy.
It’s true that the 403 errors consume far fewer resources.
The problem is, even if I got rid of the inauthentic bot traffic completely — if it burned through 90% of my monthly CPU quota before I figured out how, the traffic from legitimate human readers will probably use the last 10% before the month is over.
(Frankly, at this point I’m hoping some admin put a note on my account that says “disregard the quota this month, it was messed up by a wave of spambots which the user has dealt with.”)
Well that’s one reason why it’s not recommended to put PHP or Perl directly online but use reverse proxy server like nginx.
The other two are security and support of modern protocols like HTTP/2.0.
(Leif and Thorn IS using nginx. Of course, it’s question if Erin has access to its configuration or did her changes in Wordpress.)
Comment Footer