I began working with the Stack Overflow team at the beginning of May, and one of the first challenges I faced was to find a method to stop abusive users and bots from hammering the site. While we’re happy to make way for legitimate search engines and users of our API, we need to ensure that the majority of our cpu cycles are going towards helping everyone find answers to their questions. We needed a rate-limiting mechanism.
After discovering that our tried-and-true load balancer did not have a native facility for rate limiting, I turned my attention towards nginx. A quick scan of the documentation revealed that the limit_req_zone declaration should take care of things. After a few tests with the development team, it was determined that it might actually work. When any user surpassed the configured rate limit, a custom 503 error page would then be displayed.
Since HAProxy is already an important part of our stack, it didn’t make sense to rip it out. Instead, I opted to slide nginx in front of haproxy, ultimately allowing us to add rate limiting to Stack Overflow with minimal effort and less risk to everyone. If worse came to worse, nginx could easily be pulled back out of the stack without too much interruption. I chose to break the work up into two phases: nginx without load balancing, and nginx with load balancing.
I made the initial deployment on Sunday, May 16, and as you can see here, there were a few small glitches as our Monday morning traffic began to flow in. Thankfully, Geoff’s quick thinking helped me see that we had exceeded the maximum amount of connections that the Linux kernel was configured to track, and I was able to adjust things appropriately. After that, we had a few small bouts of slowness, which were overcome simply by increasing nginx’s worker process count and reloading the config. I left this in place for a week, making extra sure that the new layer wasn’t going to have any adverse effects.
Rate limiting was then officially enabled on Sunday, May 23. There was a bit of shakiness at first, as I did not enable the ‘nodelay’ feature. This caused nginx to actually try and slow down offenders instead of just handing them 503s. Once this was discovered, the feature was turned on and we were in business.
Over the past few days, I’ve been pouring over our http access logs, trying to determine if we have seen any amount of success. I’m happy to say that we have: where I normally would see dozens of bots hitting us somewhere between 16,000 and 40,000 times an hour, I saw… none. Victory for nginx!
More to follow as we continue to tweak nginx and employ other systems to help us keep the channels clean…