If you are not familiar with the concept, technical debt is essentially the idea that you build and program things quickly, skipping the niceties in order to ship, and then fix it later. By putting things off you build up debt that needs to be paid down later. One of the places this most commonly shows itself is in performance.

It works like this. Developers make features because the business and users want features. Performance is hard, and the benefits of good performance are not usually as obvious or concrete as the benefits of new features. Therefore, nobody really pays attention to performance or it is intentionally skipped until it gets so bad that people consciously notice it. Then the developers need to do a “feature freeze” and fix things until performance is at least “okay.” again. If you don’t mind the cliche, the feature freeze is the “Rinse.”, and then it all starts over again — “Repeat.” This is the cycle of technical debt.

At Stack Exchange I saw this happen, the developers had to stop working on features and fix performance because it got the point where we were getting timeouts. However, here is where things get interesting: After that, it never happened again.

“Impossible!” No, it is not impossible. In reality, of course there are still things that slip by, but the overall macro cycle of technical debt, when it comes to performance, is avoidable. And if you order my VHS series for 19.95, I will tell you how.

In all seriousness, even if there is no one recipe, from my viewpoint Stack Exchange escaped the cycle through culture, and making the right performance investments. The culture that lead to this consists of:

• Placing a value on performance: “Performance is a feature”
• Well integrated development and operations
• A sense of craftsmanship when it comes to performance

Good performance makes a system enjoyable to use, everyone has to believe this idea. When development and operations are well integrated the teams empower each other, and since performance takes both programming and systems knowledge this is needed. Lastly, if good performance is an aspect of good craftsmanship, it becomes a source of pride.

These cultural aspects at Stack Exchange and the performance investments made enforce each other. I don’t think we could have one without the other. But if there is a secret sauce, it feels to me like it is the performance investments we have made. These investments follow a development pattern that results in instant feedback when it comes to performance:

The 3 Step Process to Good Performance Investments:

Step 1: Collect your data in a queryable way

I can’t emphasize enough how important this initial step is. Your performance data such as logs and system data (i.e. CPU/Memory/Network etc) needs to be in a format that can easily be queryed, extracted, aggregated, and molded in a way that leads to discovery. We use SQL Server for our logs and system data. It doesn’t have to be SQL, but I think that rrd, the common storage format used by systems like Cacti, although good for displaying time series graphs does not fill this requirement due to the difficulty of extracting data.

Step 2: Discover the Important Metrics

Once you have the data in a queryable format, you can then explore that data and discover what the important metrics are. Once we started capturing our web logs in SQL we were able to add custom headers that tell us things like which route is being hit, and measure performance grouped by route. If your data isn’t queryable the discovery process has too much friction.

Step 3: Automate and Integrate the Important Data

Once you have found the important data by exploring it with various queries, those queries should be automated and integrated into your application. Then with every build (rapid integration or frequent building helps) you get instant feedback. At Stack Exchange we have a dashboard that includes graphs from log data, system data, profiling results, and exceptions. We can explore our web logs with a data explorer instance. Also, some of this such as our profiler results are part of every page load.

This process leads to an instantaneous and effortless return of performance information. This eliminates the friction around discovering how your performance is changing. With this information readily available and in your face, it enables a culture where keeping up with performance becomes an aspect of good craftsmanship.

These tools we have created are performance investments. Investments are the opposite of debt. Investments give returns where as debt has interest. When you make these sort of performance investments the cycle of debt is broken and you start collecting the returns. For the most part, people in this world are either collecting returns or paying debt — and collecting returns feels damn good.

Nearly every time we talk about our infrastructure, people ask us why we own and operate our servers rather than host Stack Overflow and the Stack Exchange network in the cloud. Usually when people ask us this, they seem to want to convince us that we should be in the cloud. The debate usually then centers around cost.

Cloud vs Self Hosting Cost?

The hypothetical cost of Stack Exchange being in the cloud has come up on meta. It turns out that the cost is difficult to actually figure out. Some of the things you need to take into account are:

• More or fewer Sysadmins required? (People say with the cloud you need fewer system administrators, never been convinced of this though)
• Licensing Costs
• Owned vs Rented Assets
• How many cloud “servers” or instances you would need vs real hardware
• Cost differences when you consider high availability

To really get this analysis correct you really have to invest a lot of time into the analysis, and even then it will only be an estimate. We have looked at cloud computing costs and we think it would actually be higher. When it comes down to it though the cost debate misses the point.

We Love Computers

and every aspect about them. We don’t just love programming and our web applications. We get excited learning about computer hardware, operating systems, history, computer games, and new innovations. Loving computers is an essential part of our company culture. Many of us have assembled our own workstations and our CTO even blogs about it in seven articles when he does. Most of us have grown up with computers as part of our identity. We all have a shared nostalgia of our first computers — if we haven’t taken our pilgrimage to the The Computer History Museum yet then we dream about it. We like to think about about the past, present, and future of computing. Owning and operating our own servers is part of how we get to live out our love of computers.

This culture means when we hire technical staff, we hire people who share this passion. I believe that this passion translates into a better product. Whenever someone does a cost analysis of cloud vs self hosting there is no row in the spreadsheet for “Work Productivity Increase due to Passion.” We are performance and control freaks and love to tweak everything including our hardware. If we outsourced our hosting to cloud computing, we would be outsourcing part of our passion. If you just want to use someone else’s computers, it means you don’t love computers — at least not every aspect to them. Sometimes cloud computing may be the best fit (for example if you have 20x the traffic around the holidays or tax season), but if you truly love computing, giving up control of computers to someone else will hurt.

We don’t just like computers, we love them. We have an emotional connection to them, and suggesting that we let someone else own, manage, and tweak them is like suggesting we get rid of what we love — just the thought of it offends.

Backups are just one of the many responsibilities of system administrators. IT Generalists have many areas to cover so they probably don’t take the time to make spreadsheets to measure the cost of data loss as they might in The Enterprise. However, investing time in trying to place a value on your backups can provide perspective on just what a terrific responsibility backups can be.

At Stack Exchange, I view our users and our user contributed content as the company’s most valuable asset. We have a lot of talent in the company, and our user contributed content isn’t even our direct source of revenue. However, if this data were totally lost (or a large portion of it) I have trouble envisioning how the company could bounce back from that. In addition to this, as a user myself I value this data as something we have created together that has intrinsic value for our professions.

Measuring Value

There are lots of ways to measure value. The obvious method is to use traditional business methods that put a dollar value on your company. When it comes to Stack Exchange some people somewhere put a big dollar value on the company which they call our valuation and in theory they don’t just make this up. If I accept that the loss of our user contributed content is the loss of the company, I could just say that our valuation is the value of our backups. The problem is that valuations tend to be pretty big numbers, and the abstraction there just doesn’t speak to me.

Also from a business perspective I can use the $18 million of VC funding we have taken and use that as a basis for value of our backups. That is a lot of money and I can’t help but start to feel the sense of importance of these backups. However, there is still a lot of abstraction there. The point of this exercise is to really feel the responsibility and not just be intellectually aware of it.

Another way to measure value is time. Our users and coworkers collectively have invested incredible amounts of time into our sites. I am user and know many of our users so I know that what we have created is important to us. I don’t have an accurate way to measure this, but I can do a back of the envelope calculation for Stack Overflow. To be conservative, looking only at the 1.4 million accepted answers on stackoverflow.com the total word count is about 100 million. According to Wikipedia people write about 19 words per minute, but I will assume people on SO are faster and can compose about 40 words per minute. That gives us 100,000,000 words / 40 words per minute / 60 minutes per hour / 24 hours a day / 365 days a year =~ 5 years of non-stop skilled work. Now I realized this calculation is perhaps, a bit, well, hair-brained, but it is reasonable for my purposes.

Another aspect to take into account is the profit generated by Stack Exchange. I don’t mean profit in the traditional sense, rather I look at what I call time profit. When a user answers someones question, they not only saved that person time but many other people who will eventually search for the same question and find that answer. This saves those people time. Because of this our sites like Stack Overflow are systems where the output is greater than the input. So in this sense of time profit, if our content was lost, future potential time profit would be lost.

We all have different ways of perceiving value. I value what our users and my coworkers have created, and when I attempt to measure just how much has been created, it becomes very apparent that safe guarding that creation though backups is an awesome responsibility.

One website defines production control as the following:

Activities involved in handling materials, parts, assemblies, and subassemblies, from their raw or initial stage to the finished product stage in an organized and efficient manner. It may also include activities such as planning, scheduling,routing, dispatching, storage, etc.

That definition is a fairly broad one, but can easily be applied to Information Technology when one is tasked with making sure a software or service that your consumers rely on remains responsive and available at all times.

Simply put, production control is establishing processes/checklists that help you and your fellow team members ensure that you do the right tasks, in the right order, every single time. What does this solve for? Firstly, production control assists in mitigating lost revenue by preventing unexpected downtimes. If you have a process where you catch most mistakes early, users will not be burdened with being your QA department. You also save the users from dealing with longer “we’re having trouble reverting something that we accidentally broke” downtimes.

Production control also encourages developers to be more cognizant of bugs/errors before the consumer finds them. If there’s an organized line-of-custody to a release, then it is easy to identify where a link in the chain fails. This is not to say that someone should be disciplined or chastised for mistakes! We’re all human, it happens. It does, however, help point towards where some extra assistance could be lent in the process to make sure things go smoothly.

As a corollary, production control does help break the “circle of blame” that can occur in organizations when a problem is found. If there are signoffs in the process, then glaringly obvious issues that aren’t detected can be tracked back to someone not properly executing the checklist. Again, this does not need to result in disciplinary action, it just further helps indicate where more manpower or alternative practices could help shore up the defenses.

An implicit benefit of production control is that it eliminates the “too many cooks in the kitchen” syndrome. If all of your developers have access to change production, you can run into situations where two people may not communicate that they’re fixing the same issue and unwittingly cause each other serious delays as they attempt to fix the problem. If you’ve ever heard “I swore I just changed that line of code!”, there’s a good chance that person did. Except, someone else changed it back when they uploaded the file with a different change.

For organizations that do not use production control methodology, there are some hurdles to adoption. If one looks at the situation from the top-down, obviously executive buy-in can be problematic. Small businesses generally have owners who are used to having complete and total control over everything in their organization, and in tech companies this generally means they want to be able to change whatever they want (even production), whenever they want (when they feel like something should be tweaked at 3am) and you’re expected to deal with that. This method of thinking is extremely disruptive to technologists that generally like things to run in smooth, planned ways. However, if you effectively present your argument, executives will generally buy-in on the idea. Worded properly, they will accept that the process, while seeming like unnecessary fluff that “only big companies need to do”, will actually make their product better and as a result bring in more revenue.

Are the benefits worth the pains? Not only do you need to convince the executives, but also the developers. This can be difficult if you have a development team full of “beautiful and unique snowflakes.” In my experience, however, developers have been more supportive of production control. I’ll admit that I had not considered the “circle of blame” breakage as a benefit until a developer mentioned it to me, once upon a time. His consideration was sound: “The less access I have to production, the less chance someone is going to say ‘You broke production while fiddling around on the server.’”

The exercise now is to define a sane production control methodology, and this varies wildly based on how important bugcatching is for your organization. The most basic form of production control is when you have developers promote their code to a staging server which mirrors production as closely as possible. Then, the developers can test their new changes at that server, prior to tossing notice over to the operations team to have them push the code to production. This method has a single handoff, and is best in lower-impact, low-opportunity-cost-of-downtime shops. The process can get more involved if you have people willing/able to do QA in a dedicated role. This can be helpful for developers, since developers often fall into the trap of “testing like the guy who developed it” rather than “testing like the user consuming it.” In this case, the developer would stage and hand off to QA, then QA would test and upon approval, toss it over to operations for the push. If you’re a fan of flowcharts, this type of procedure is a flowchart designer’s wet dream. There’s no limit to how bogged down this can get, but remember: If the process is cumbersome, people will not want to follow it.

Whether you choose to implement a process such as the one presented, or continue to move forward without it, remember that in all things, there are benefits and detriments, but by and large these processes exist because they work and eliminate unknowns from the equation. Dear readers, what are your thoughts on this topic?

I have always had the notion that some companies “Get It” when marketing to programmers, sysadmins, and hackers. A couple weeks ago I was in Silicon Valley at Tech Field Day listening to a lot of presentations with some fellow geeks. In some of these presentations they “got it”, in others they didn’t.

The fundamental difference is that the ones that get it tell me about their technology, and the ones that don’t tell me about their products. True geeks have one common trait, and that is that they like to learn how things work. We get excited about learning and news ideas. Exciting technologies solve problems in non-obvious ways. Since these are not obvious, we want to learn about them, and in as much detail as possible.

Products, on the other hand, are how these ideas are packaged. For the most part, we don’t care. If we do care, that comes later, only if it will solve a problem that we have will we start to care about implementation and packaging. The advantage of telling us about your technology is that we do care, even if we don’t have a use for it at the moment, because we are geeks – it is our nature. When done right, we will then associate your product with the technology and that will be enough.

This has some consequences that marketing should be aware of when targeting tech people. First, we generally don’t want to talk to you directly, at least, not for very long. This is not because you are not important or interesting, it is just that you probably can’t get us excited about your technology like your engineers can. You can enable your engineers to present your technology well, and that is what people who are good at geek marketing do. If you can’t get your actual engineers to present no mater how hard you try, your screwed. It’s not your fault, your company just sucks and you probably just need to move on. Lastly, everything targeted to tech people should be aimed at getting us excited about your technology, not your product.

With my recent experiences at tech field day there were some good examples of this done right. Pure Storage taught me about why I see SSDs fail and a new type of RAID they invented suited for SSDs. Arkeia educated me about various implementations of deduplication in backups. Data Direct Networks introduced me to the concept of object store filesystems. However, in that case I wanted to learn even more given that amount of time. How successful they were came down to how much time they all devoted to fulfilling my desire to learn more about technology vs. telling me about their products.

Our CEO Joel Spolsky wrote a book about hiring programmers that is titled Smart & Gets Things Done. The idea behind the title is that when it comes down to it, you really just want to hire someone who is both smart and will get things done:

“People who are smart, but don’t get things done … would rather mull over something academic about a problem than ship on time”.

This applies equally in the area of system administration. Our CTO Jeff Atwood recently tweeted a reference to this:

When I read that I felt like he was talking to the sysadmin team (I hope he wasn’t — but the fact that I felt like he might have been is enough). The reason for this isn’t that we are not getting things done; I think we work hard and get a lot of things done. Rather, I think we have tendency to pick the interesting tasks over the uninteresting or tedious tasks.

Assuming as a system administrator you actually like what you do, I think tasks end up looking like the following:

So there are some interesting tasks that would be good to do, but may not need to be done. Then you have most tasks, which if you like your job, both need to be done and are interesting. Lastly, you have the tasks that are not interesting but do need to be done.

With a tendency to give too much weight to the interesting tasks or pieces of a project, that category on the right will grow over time. I like to think of keeping that category under control as eating my vegetables. Ideally, one maintains a good balanced diet all the time. But in reality there are times when you need to get on the scale and come to grips with the fact that you might need to go on a healthy diet for a while.

Lastly, don’t worry, the irony of taking some time to write this post is not lost on me.

Most people in the system administration field I have talked to agree that the professionalization of system administration is happening faster. System administrators have always been paid, that isn’t what I mean by professionalization. What I do mean is that more leaders are emerging, participating in local groups, blogging, and participating on sites like Server Fault. The end results is that the standard of the profession is going up. That doesn’t mean that there hasn’t always been great people — but I believe the average expected expertise is increasing in the field.

One of the main ways I see this happening is in what people expect from answers on Server Fault. Lets take a hypothetical question:

“On my AFSRQ 2000, the system crashes every week or so. I managed to capture my logs before the crash: …. However, I am not sure what these lines mean?”

An acceptable answer to many system administrators would have been something like following as long as the AFSRQ 2000 stops crashing:

“Have you tried pressing the green button twice? Last time I did that it fixed my issues.”

The problem with this sort of answer is that it spreads ignorance. If there is no understanding of what the green button does, no wisdom can be gained from an answer like this. If the answer to how the fix works and why it works is missing, then this knowledge can not be applied to future related situations. Eventually, the AFSRQ 2000 will be upgraded to the AFSRQ 2008, and then button might not even exist, or, even worse, it might be a red button. Even if it solves this specific problem it doesn’t really help the profession advance. Also what happens is people attempt to press the green button to solve every problem — even when it wouldn’t help at all.

The better way to answer this is to try to understand why pressing the green button stops the system from crashing. Answer what causes the crash in the first place. The goal should be that answers should be expressed in the context of computing fundamentals and backed up by real data. By fundamentals I mean things like how operating systems work, system calls, memory management, network traces, hardware, etc. The reason is that when problems and answers are expressed in terms of fundamental building blocks patterns can emerge. Recognizing and understanding these patterns is what experience is system administration is really about. When answers are backed with real data (it helps when vendors empower you with tools to get the data), then this also gives context and proof, and other system administrators can verify the information. These are the sort of answers that raise the level of our field, pressing the green button twice does not.