Funny Story: I Got DDOS Attacked By Google

twest

Kind of funny, kind of sad, kind of infuriating. I spent about 6 hours figuring this out, and it ended up being the dumbest thing.

So I host this one site, it's getting about 3k users/day, not much. This site has these plugins WP Tabs and WP Show Posts and WP Popular Posts and Shortcoder. Well, the genius designer used all these plugins to make custom designed Pages to replace the regular category archives... The combo of plugins makes such a mess, the Tabs for example adds a bunch of long strings in the url bar, for the various tabs. Anyhoo, these pages don't seem to be cacheable, too many queries from the various plugins and the query in the url breaking cache. That's kind of important for what's next.

I checked the search console while investigating the irregularly high load generating seemingly from this site. In the page index area, there's some 10 million urls "crawled but not indexed", it's those weird urls from the tabs plugins lol...

So I check the access log... 50,000 get requests from Google today, all on those weird urls. Wow.

I added a rate limit on Googlebot to disallow more than 1 crawl per 5 seconds on those pages with those garbage plugins, and got instant results:

Load dropped from 11 to 2... Oh, and bandwidth dropped from 200Mbps to 30Mbps... I have no idea what Googlebot was down/uploading to use so much bandwidth, maybe something to do with docker networking and the weird urls.

Amazing though, I never imagined getting ddosed by Google lol. Just goes to show too, one crappy designer off Fiverr can do enough damage to totally ruin your site AND hosting. Now I have to explain to this client how their bad decision is going to cost them more $ to get fixed, and the 10 million urls in their search console is probably going to be an issue as well lol

Btw, I wonder if there's a way for Enhance to more gracefully handle 404s or other such weird uncacheable content/urls? I had a site with two missing fonts that was generating massive load, and another site with missing favicon also generating massive load. It seems even cgroups can't stop 404s from hammering the cpu.

JohnB

twest WP Tabs and WP Show Posts and WP Popular Posts and Shortcoder. Well, the genius designer used all these plugins to make custom designed Pages to replace the regular category archives...

These idiots give Wordpress developers a bad name. They are not real programmers (yes im gatekeeping sue me) and are lost without a page builder like elementor and/or a bunch of plugins they can string together because editing or creating a new template is scary despite tons of documentation. If you say you're a Wordpress developer you have to preface it by saying you develop plugins and custom themes to avoid an assumption that you're the idiot above.

Pisses me off so much when people go on fiverr for ANYTHING programming/tech related. Do they expect quality? Fiverr should automatically mean garbage quality where your project will be rushed so they can move on to the next. Even video/image editing is the same way. It will be done as quick as possible because its the only way to make money at fiverr prices...

Anyway rant over. Good catch lol. Removing 10 million URLs is going to be an issue lol.

twest

JohnB not to mention the extreme security risk people take by hiring the lowest cost provider possible. I had 6 customers lose access to their Facebook accounts last year due to hiring the cheapest offshore help they could find, then they come to me "do you have a contact at Facebook that can help me get my account back?", ugh!

It's a constant uphill battle, but alas, that's the job lol

josedieguez

have seen this a few times, not in enhance yet, but it does happen.

JohnB

twest not to mention the extreme security risk people take by hiring the lowest cost provider possible. I had 6 customers lose access to their Facebook accounts last year due to hiring the cheapest offshore help they could find, then they come to me "do you have a contact at Facebook that can help me get my account back?", ugh!

I used to do some pentesting and had one client say "I lost my password to gmail don't you have like a backdoor or something" like... if I had that... I wouldn't be dealing with you lmao.

twest

Since rate limiting Googlebot last night, it has been throttled 90,000 times lol... How crazy 🤣

Rich

Well if you do try and capture a googlebot A.I. in a trap, expect mass compute/bandwidth in it's struggle to leave. 😁 I hear they don't make great pets. 🤔

Another tail to share. I've had clients come to me and they'd got their logo designed on the cheap. To find that the domain name they wanted to buy now suddenly was brought up by some other 3rd party. So they've had to buy another domain...

Worse is what them domains get turned into when you don't offer to $$$ to get it, or buy back if they didn't renew fast enough.

Terrible for their branding, one - or letter away from your brand as NSFW. It's entirely legit as well, it's heart breaking to tell them theres nothing that can be done. They legitimately have the right to host whatever on the domain they've sucked up.

twest

Rich yeah I couldn't afford to keep a Googlebot for long, they cost too much CPU lol

I get those domain woes all the time too, it boggles my mind that people don't just buy the domain first thing, they'll talk about it to other people for weeks before trying to buy. Then shocked that it was taken. Gee, I wonder if talking about it to other people was a bad idea 🤔 Everyone is freaking out about Google Domain's being sold to SquareSpace too, I suspect that is going to cause a huge wave of missed renewals as millions of people didn't pay attention to the change lol

SharedGrid

This is kind of totally unrelated, but your story reminded me of something a few years ago.

I was working for a software development company who developed software for public services, government, police, healthcare etc. I got a call at 3am that multiple services were down, I was the first and only person called and I just couldn't work out what was going on. Everything was accessible internally but services which needed external access to the Internet were very sparodic. Called up one of the network engineers who I'm close friends with and he couldn't see any issues with our circuits.

I can't even remember how I noticed it, but Google suddenly stopped us from querying any record from their public DNS service, and guess what? All services were set to use Google for external DNS querying. Great. Cue switching everything to Cloudflare as quickly as possible across 100s of services.

No idea why it happened but about 12 hours later the issue disappeared completely.

twest

SharedGrid blahaha, classic. Can't trust Google with anything, filthy buggers lol

charlesalmeida

Bot control needs to be efficient, otherwise it will fry CPU for free.