How I managed to DDoS myself using Amazon Route 53
It’s good to be able to have a laugh at yourself from time to time. It keeps you humble. I was left scratching my head earlier when this site and another I host started throwing back 504 status codes. This blog’s just for kicks, it’s not monetised and I’m happy using cheap hosting as there’s nothing mission critical. As such there’s no auto-scaling or high availability in place, so if it goes down, it’s down until I fix it.
Following some investigation (and roping in my friendly neighbourhood AWS sysops guru), we determined that there was excessive traffic coming from a number of AWS IP addresses, most of which were in the ap-southeast-1 (Singapore) region. The funny thing was, despite having caching, CDN, and Web Application Firewall in place, the traffic was still hammering the sites. The connections from Singapore were directly to the IP on port 443 rather than via DNS. So we took some steps to block the offending IP range using .htaccess and iptables, which seemed to eventually take effect. Finally everything came back up on this site and started working again.
So I turned my attention to the other site and thought I’d take the quick, scorched earth approach of attaching a different static IP and changing the DNS, because, well, I’m still learning linux and taking the approach we did with the first site seemed a bit much for a Friday night. So I switched the static IP, changed the DNS records and then went into Route 53 to change the IP address on the Health Check I’d set up the other day.
Route53 Health Checks let you poke at your endpoint from various locations around the globe.
Oh. Hang on. Asia Pacific (Singapore)? Hmmm. Those IPs look awfully familiar. A quick check on the Health Check settings showed:
If you choose an interval of 30 seconds, each of the Route53 health checkers in data centers around the world will send your endpoint a health check request every 30 seconds. On average, your endpoint will receive a health check request about every two seconds. - https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/health-checks-creating-values.html#health-checks-creating-values-request-interval
Yep. Choosing the 30 second interval means you’re actually getting a hit from each checker every 30 seconds. Imagine my surprise when I remove the health checks and everything goes back to normal in minutes.
So yeah. I basically DDoS’d myself using Route 53 Health Checks. Oops.