The very first post on this blog, Experiments in Rate Limiting, explained how our load balancer HAProxy did not support rate limiting. So in order to support rate limiting we placed Nginx in front of HAProxy. This has worked but it wasn’t ideal for a few reasons:

  • Having a reverse proxy in front of another one is just a little bit too Rube Goldberg.

  • We found that Nginx doesn’t allow rates to be calculated at rates greater than hits per second. So you can write rates as per minute but that is just another way of writing per second rates.
  • We wanted the ability to track the amount of data transferred over period of time as well as hits.

So in order to continue the company’s goal which is to make the Internet a better place to get expert answers rate limiting to your questions for your websites we contracted HAProxy’s author Willy Tarreau to add this feature to HAProxy. This solution made sense to us because:

After working with Willy over the past month by putting on my testing/QA hat we ended up with a version we felt we could try in production. So on Saturday night it went live and survived the high traffic Monday.

How to Set it up: If you have multiple sites under HAProxy a good way we have found to configure it is to have a frontend/backend configuration. The frontend is what HAProxy listens on and it sends connections to the backend(s) which can be different sites and use different pools of web servers.

This is and a sample of the rate limiting features are best explained with a sample configuration so here is a working configuration that can be used to experiment with:

global
    log 127.0.0.1   local0
    log 127.0.0.1   local1 notice
    stats socket /var/run/haproxy.stat mode 600 level operator
    maxconn 4096
    user haproxy
    group haproxy
    daemon

defaults log global mode http option httplog option dontlognull retries 3 option redispatch maxconn 2000 contimeout 5000 clitimeout 50000 srvtimeout 50000

frontend http bind *:2550

stick-table type ip size 200k expire 10m store gpc0

# check the source before tracking counters, that will allow it to
# expire the entry even if there is still activity.
acl whitelist src 192.168.1.154
acl source_is_abuser src_get_gpc0(http) gt 0
use_backend ease-up-y0 if source_is_abuser
tcp-request connection track-sc1 src if ! source_is_abuser

acl is_test1 hdr_sub(host) -i test1.com
acl is_test2 hdr_sub(host) -i test2.com

use_backend test1  if is_test1
use_backend test2  if is_test2

backend test1 stick-table type ip size 200k expire 30s store conn_rate(100s),bytes_out_rate(60s) acl whitelist src 192.168.1.154

# values below are specific to the backend
tcp-request content  track-sc2 src
acl conn_rate_abuse  sc2_conn_rate gt 3
acl data_rate_abuse  sc2_bytes_out_rate  gt 20000000

# abuse is marked in the frontend so that it's shared between all sites
acl mark_as_abuser   sc1_inc_gpc0 gt 0
tcp-request content  reject if conn_rate_abuse !whitelist mark_as_abuser
tcp-request content  reject if data_rate_abuse mark_as_abuser

server local_apache localhost:80

backend test2 stick-table type ip size 200k expire 1m store conn_rate(100s),bytes_out_rate(60s) acl whitelist src 192.168.1.154

# values below are specific to the backend
tcp-request content  track-sc2 src
acl conn_rate_abuse  sc2_conn_rate gt 5
acl data_rate_abuse  sc2_bytes_out_rate gt 20000000

# abuse is marked in the frontend so that it's shared between all sites
acl mark_as_abuser  sc1_inc_gpc0 gt 0
tcp-request content reject if conn_rate_abuse !whitelist mark_as_abuser
tcp-request content reject if data_rate_abuse mark_as_abuser
server local_apache localhost:80

backend ease-up-y0 mode http errorfile 503 /etc/haproxy/errors/503rate.http

So the backends both use the same single web server in our test environment which happens to run on the same machine: “server local_apache localhost:80″. test1.com and test2.com are directed to the appropriate backend with “acl is_test1 hdr_sub(host) -i test1.com” which is an access list that returns true if the host header in the HTTP request is test1.com. We then use “use_backend test1 if is_test1″ to redirect it to the backend. This is how the basic set up works in a very short summary because I want to focus on the actual rate limiting configuration.

If people violate the rate limiting in this example they are redirected to the backend ease-up-y0 which gives them 503 error page that can be customized. Different backends have different rates in our example but the blocking happens on the front end. Looking first at the backend for test1.com there are several pieces to this. First we track average connection rate and bytes rate per source ip with:

stick-table type ip size 200k expire 3m store conn_rate(100s),bytes_out_rate(60s)

This declares a table to store the source IP addresses that is up to 200,000 entries long. Each IP entry is about 50 bytes and the connection rate and bytes out rate are 12 bytes each which are stored with each source IP address. So at 74 Bytes an entry we are looking at a possible 14 MBytes of usage for this table. The expire argument is how long to keep an entry in the table (In this case it just needs to be twice the length of the longest rate argument for a smoothed average). The time arguments for connection rate and bytes out rate are how long to calculate the average over.

The next part enables the tracking and sets the limits:

    tcp-request content  track-sc2 src
    acl conn_rate_abuse  sc2_conn_rate gt 5
    acl data_rate_abuse  sc2_bytes_out_rate gt 20000000

Finally we have the section that actually enforces these rules:

    acl mark_as_abuser  sc1_inc_gpc0 gt 0
    tcp-request content reject if conn_rate_abuse !whitelist mark_as_abuser
    tcp-request content reject if data_rate_abuse !whitelist mark_as_abuser

This section is a little bit more cryptic but is not too complicated. The gpc0 is a generic counter and sc1_inc_gpc0 is incrementing the counter we use on the frontend — if it is evaluated. So in the tcp-request line if they have violated either the connection rate or the data rate and they are not whitelisted then the front-end gpc0 will be incremented. The mark_as_abuser does not get evaluated if conn_rate_abuse or data_rate_abuse because this uses short circuit evaluation. This incremented front end gpc is then handled by the frontend:

    stick-table type ip size 200k expire 10m store gpc0
    # check the source before tracking counters, that will allow it to
    # expire the entry even if there is still activity.
    acl whitelist src 192.168.1.154
    acl source_is_abuser src_get_gpc0(http) gt 0
    use_backend ease-up-y0 if source_is_abuser
    tcp-request connection track-sc1 src if ! source_is_abuser

In this stick table we store the general counter with the source ip address. If that has been incremented by a backend the user is redirected to the 503 page. Once this has happened they are not longer tracked during this period, as a result they are redirected to 503 until their entry expires which is set to 10 minutes in this example.

A good way to get a feel for this is to use the socket connection feature that HAProxy has. Here is an example session (Here the client happens to be on localhost so the client entry is 127.0.0.1):

sudo socat readline /var/run/haproxy.stat
prompt
> set timeout cli 1d
> show table

table: http, type: 0, size:2048000, used:1

table: test1, type: 0, size:2048000, used:1

table: test2, type: 0, size:2048000, used:0

> show table http

table: http, type: 0, size:2048000, used:1

0x8370aac: key=127.0.0.1 use=0 exp=592347 gpc0=0

> show table test1

table: test1, type: 0, size:2048000, used:1

0x8370a68: key=127.0.0.1 use=0 exp=19020 conn_rate(100000)=2 bytes_out_rate(60000)=986

> help Unknown command. Please enter one of the following commands only : clear counters : clear max statistics counters (add 'all' for all counters) clear table : remove an entry from a table help : this message prompt : toggle interactive mode with prompt quit : disconnect show info : report information about the running process show stat : report counters for each proxy and server show errors : report last request and response errors for each proxy show sess [id] : report the list of current sessions or dump this session show table [id]: report table usage stats or dump this table's contents get weight : report a server's current weight set weight : change a server's weight set timeout : change a timeout setting disable server : set a server in maintenance mode enable server : re-enable a server that was previously in maintenance mode

The documentation for this new feature set is pretty thorough and will explain all of the above in more detail but I wanted to give a starting point to start experimenting with the rate limiting. I think this is an excellent new feature that introduces some web application firewall features into HAProxy that fit nicely. If you are using HAProxy I recommend giving this new feature a try and if you are not using HAProxy and have a use for a load balancer you should check it out.

It Must be True

Kyle Brandt

My teachers in grade school hammered at least one lesson into me:

“If it is in a book, it must be true”

Since I was such a good student I remember this little aphorism exactly as it was stated even to this day. This is great news for Server Fault as it was mentioned in the 4th Edition of the Unix and Linux System Administration Handbook as one of the most significant resources of UNIX and Linux information on the web. I already felt this was the case myself, but it is good to know that it must be true.

Thank you Zordache for pointing this out in the Server Fault chat room.

A question that comes up again and again in web development companies is:

“Should the developers have access to the production environment, and if they do, to what extent?”

My view on this is that as a whole they should have limited access to production. A little disclaimer before I attempt to justify this view is that this standpoint is in no way based on the perceived quality or attitude of the developers — so please don’t take it this way. First I want to cover a few common arguments of developers that dislike or hate this idea:

“We can’t get stuff done, the system administrators get in the way and take forever.” Well if this is actually the case, then they are right. If there are not enough administrators or the administrators are not good then they can become a bottleneck. However, developer access is not the solution because after this you still have crappy or not enough administrators. Sometimes there are other administrative specific concerns that might make things take longer, more on this later, but it shouldn’t take an unreasonable amount of time.

“We have always had access before.” Startup companies seem to rarely start out with administrators. For some reason system administrators are considered a luxury. Although this process might have worked before, as you grow there is probably more administration. Things get more complicated and this is probably why they went out and hired an administrator. So in this case, “this is what we have always done” isn’t really good enough argument.

“We need access to troubleshoot.” Maybe, maybe not. It is possible the administrators can just give you the information you need. If this particular area becomes a bottleneck, limited access might be in order.

Those are a few possible arguments against restricted access for developers, but lets move on to what I really want to talk about — why it is a good idea.

The Process Restricted Access Creates: If the developers can not access production one big implication is that they can’t install their own code. This means that the administrators must install the code. Two things then need to happen: 1) The developers and system administrators must communicate — with each other! The administrators learn how to install the software which I hope I don’t have to explain is probably a good thing. 2) The developers have to create an install or release process that is easy and effective. This is also a good idea. Being able to rebuild the environment is an essential part of disaster recovery. So what can’t happen with restricted access is that the installation of the code is some complex process that only lives in heads of a few developers. Also, the developers don’t have to spend time deploying and installing code when they could be writing new code. It might take them longer at first, but asymptotically this is will be faster (That is right, I used a fancy developer word).

Administrators will also probably learn a little bit more about what needs to be backed up through this process. If the administrator doesn’t know the application well they just have to trust that what the developer told them to back up is all that really needs to be backed up.

Developer’s Concerns are Often Not System Administrator’s Concerns: In general developers do not focus on security in the same areas that system administrators do. They usually have different areas of expertise when it comes to web site security. Topics such as cross site scripting and SQL injection are likely areas of security where developers have specific expertise and administrators do not. Account privileges, file permissions, web server configuration are often not what developers have experience in or are very interested in. These are all important areas in production environments are meant to the expertise of system administrators. Also while I am on the topic of security the less people with access the better (Principle of Least Privilege). This also helps when the call comes in at 2am because the system administrator doesn’t have to wonder if one of the 15 developers with access were on the system doing … something.

Change Control: I don’t think there is a decent developer out there that isn’t serious about change control. When it comes to their code. However, I haven’t seen to many developers that are serious about logging every single change they make to server as a whole (I have seen some configuration files under revision control however).

If this isn’t done it means that the production environment will not be able to be rebuilt properly. It also means that if there have been changes that might have caused a problem those changes might not be know to the person trying to solve the problem. The parallel to this would be if an administrator just went into the production code and changed some things without telling anyone or checking it in. Ya, the developers would freak.

The Person Who Owns it Should Have Control: One of Joel’s Spolsky’s beliefs when it comes to management is:

“Everybody owns some area. When they own it, they own it. If a manager, or anybody else, wants to provide input into how that area is managed, they have to convince the owner. The owner has final say.”

System administrators are generally considered to own the production environment. The administrators are the ones who keep track of uptime, the ones who get the phone calls at 2am, basically, they are the ones closest to the problem. When developers have direct access to production from what I have seen this control always gets undermined.

The System Administrators Responsibilities: In order for this to work, administrators have duties that must be fulfilled. 1) Invite the developers to request what they need from you and be pleasant about giving it to them. 2) Make sure the developers have a good development environment in which they have free rein. 3) Be reasonable and practical. Every company is different, for some companies maybe developers should just have no access because of the nature of the business (i.e. finance). However, if you are not a financial company, a work flow where developers have unprivileged access is likely the best solution. There might also be some developers that double as system administrators so every company has a different situation.

As I stated in the beginning my belief in the process doesn’t have to do with having great or not-so-great developers — many developers function great as system administrators (For example see this post on sending email without it being tagged as spam). Rather this is about a process that lets both people focus on their expertise as a company grows. Things may move a little bit slower. However, the trade in should be that you get a more reliable and secure production environment.