The very first post on this blog, Experiments in Rate Limiting, explained how our load balancer HAProxy did not support rate limiting. So in order to support rate limiting we placed Nginx in front of HAProxy. This has worked but it wasn’t ideal for a few reasons:
- Having a reverse proxy in front of another one is just a little bit too Rube Goldberg.
- We found that Nginx doesn’t allow rates to be calculated at rates greater than hits per second. So you can write rates as per minute but that is just another way of writing per second rates.
- We wanted the ability to track the amount of data transferred over period of time as well as hits.
So in order to continue the company’s goal which is to make the Internet a better place to get expert
answers rate limiting to your questions for your websites we contracted HAProxy’s author Willy Tarreau to add this feature to HAProxy. This solution made sense to us because:
- HAProxy has worked great for us and other companies like reddit.
- Willy is awesome and it so happens that Willy is a Server Fault user.
- Since HAProxy is free and open source our rate limiting is your rate limiting. So head on over and get the code if you like.
After working with Willy over the past month by putting on my testing/QA hat we ended up with a version we felt we could try in production. So on Saturday night it went live and survived the high traffic Monday.
How to Set it up: If you have multiple sites under HAProxy a good way we have found to configure it is to have a frontend/backend configuration. The frontend is what HAProxy listens on and it sends connections to the backend(s) which can be different sites and use different pools of web servers.
This is and a sample of the rate limiting features are best explained with a sample configuration so here is a working configuration that can be used to experiment with:
global log 127.0.0.1 local0 log 127.0.0.1 local1 notice stats socket /var/run/haproxy.stat mode 600 level operator maxconn 4096 user haproxy group haproxy daemon
defaults log global mode http option httplog option dontlognull retries 3 option redispatch maxconn 2000 contimeout 5000 clitimeout 50000 srvtimeout 50000
frontend http bind *:2550
stick-table type ip size 200k expire 10m store gpc0 # check the source before tracking counters, that will allow it to # expire the entry even if there is still activity. acl whitelist src 192.168.1.154 acl source_is_abuser src_get_gpc0(http) gt 0 use_backend ease-up-y0 if source_is_abuser tcp-request connection track-sc1 src if ! source_is_abuser acl is_test1 hdr_sub(host) -i test1.com acl is_test2 hdr_sub(host) -i test2.com use_backend test1 if is_test1 use_backend test2 if is_test2
backend test1 stick-table type ip size 200k expire 30s store conn_rate(100s),bytes_out_rate(60s) acl whitelist src 192.168.1.154
# values below are specific to the backend tcp-request content track-sc2 src acl conn_rate_abuse sc2_conn_rate gt 3 acl data_rate_abuse sc2_bytes_out_rate gt 20000000 # abuse is marked in the frontend so that it's shared between all sites acl mark_as_abuser sc1_inc_gpc0 gt 0 tcp-request content reject if conn_rate_abuse !whitelist mark_as_abuser tcp-request content reject if data_rate_abuse mark_as_abuser server local_apache localhost:80
backend test2 stick-table type ip size 200k expire 1m store conn_rate(100s),bytes_out_rate(60s) acl whitelist src 192.168.1.154
# values below are specific to the backend tcp-request content track-sc2 src acl conn_rate_abuse sc2_conn_rate gt 5 acl data_rate_abuse sc2_bytes_out_rate gt 20000000 # abuse is marked in the frontend so that it's shared between all sites acl mark_as_abuser sc1_inc_gpc0 gt 0 tcp-request content reject if conn_rate_abuse !whitelist mark_as_abuser tcp-request content reject if data_rate_abuse mark_as_abuser server local_apache localhost:80
backend ease-up-y0 mode http errorfile 503 /etc/haproxy/errors/503rate.http
So the backends both use the same single web server in our test environment which happens to run on the same machine: “server local_apache localhost:80″. test1.com and test2.com are directed to the appropriate backend with “acl is_test1 hdr_sub(host) -i test1.com” which is an access list that returns true if the host header in the HTTP request is test1.com. We then use “use_backend test1 if is_test1″ to redirect it to the backend. This is how the basic set up works in a very short summary because I want to focus on the actual rate limiting configuration.
If people violate the rate limiting in this example they are redirected to the backend ease-up-y0 which gives them 503 error page that can be customized. Different backends have different rates in our example but the blocking happens on the front end. Looking first at the backend for test1.com there are several pieces to this. First we track average connection rate and bytes rate per source ip with:
stick-table type ip size 200k expire 3m store conn_rate(100s),bytes_out_rate(60s)
This declares a table to store the source IP addresses that is up to 200,000 entries long. Each IP entry is about 50 bytes and the connection rate and bytes out rate are 12 bytes each which are stored with each source IP address. So at 74 Bytes an entry we are looking at a possible 14 MBytes of usage for this table. The expire argument is how long to keep an entry in the table (In this case it just needs to be twice the length of the longest rate argument for a smoothed average). The time arguments for connection rate and bytes out rate are how long to calculate the average over.
The next part enables the tracking and sets the limits:
tcp-request content track-sc2 src acl conn_rate_abuse sc2_conn_rate gt 5 acl data_rate_abuse sc2_bytes_out_rate gt 20000000
Finally we have the section that actually enforces these rules:
acl mark_as_abuser sc1_inc_gpc0 gt 0 tcp-request content reject if conn_rate_abuse !whitelist mark_as_abuser tcp-request content reject if data_rate_abuse !whitelist mark_as_abuser
This section is a little bit more cryptic but is not too complicated. The gpc0 is a generic counter and sc1_inc_gpc0 is incrementing the counter we use on the frontend — if it is evaluated. So in the tcp-request line if they have violated either the connection rate or the data rate and they are not whitelisted then the front-end gpc0 will be incremented. The mark_as_abuser does not get evaluated if conn_rate_abuse or data_rate_abuse because this uses short circuit evaluation. This incremented front end gpc is then handled by the frontend:
stick-table type ip size 200k expire 10m store gpc0 # check the source before tracking counters, that will allow it to # expire the entry even if there is still activity. acl whitelist src 192.168.1.154 acl source_is_abuser src_get_gpc0(http) gt 0 use_backend ease-up-y0 if source_is_abuser tcp-request connection track-sc1 src if ! source_is_abuser
In this stick table we store the general counter with the source ip address. If that has been incremented by a backend the user is redirected to the 503 page. Once this has happened they are not longer tracked during this period, as a result they are redirected to 503 until their entry expires which is set to 10 minutes in this example.
A good way to get a feel for this is to use the socket connection feature that HAProxy has. Here is an example session (Here the client happens to be on localhost so the client entry is 127.0.0.1):
sudo socat readline /var/run/haproxy.stat prompt > set timeout cli 1d > show table
table: http, type: 0, size:2048000, used:1
table: test1, type: 0, size:2048000, used:1
table: test2, type: 0, size:2048000, used:0
> show table http
table: http, type: 0, size:2048000, used:1
0x8370aac: key=127.0.0.1 use=0 exp=592347 gpc0=0
> show table test1
table: test1, type: 0, size:2048000, used:1
0x8370a68: key=127.0.0.1 use=0 exp=19020 conn_rate(100000)=2 bytes_out_rate(60000)=986
> help Unknown command. Please enter one of the following commands only : clear counters : clear max statistics counters (add 'all' for all counters) clear table : remove an entry from a table help : this message prompt : toggle interactive mode with prompt quit : disconnect show info : report information about the running process show stat : report counters for each proxy and server show errors : report last request and response errors for each proxy show sess [id] : report the list of current sessions or dump this session show table [id]: report table usage stats or dump this table's contents get weight : report a server's current weight set weight : change a server's weight set timeout : change a timeout setting disable server : set a server in maintenance mode enable server : re-enable a server that was previously in maintenance mode
The documentation for this new feature set is pretty thorough and will explain all of the above in more detail but I wanted to give a starting point to start experimenting with the rate limiting. I think this is an excellent new feature that introduces some web application firewall features into HAProxy that fit nicely. If you are using HAProxy I recommend giving this new feature a try and if you are not using HAProxy and have a use for a load balancer you should check it out.