Recently there was a question on the Programmers site asking “Why don’t all companies buy developers the best hardware?” This is actually a very interesting question and there was a good deal of discourse on the topic, as there always is when you get a group of professionals talking about the tools of their trade.

We here at Stack Exchange pride ourselves on not letting the technology get in the way of our devs. We want them to be able to do their job – which is writing code, and doing it well – with minimal hassle. Now, I have to take a brief moment to make a little bit of a note some of the things we do do not scale.

The philosophy that is maintained when getting a developer a new machine is a pretty simple one:

> If a dev is constantly struggling with their machine they aren’t getting work done, they aren’t happy, and they are producing bad code. All of these things are more expensive than a nice developer machine

As a sysadmin – and hence the guy in charge of getting the devs what they need you find out really quickly that they know what they want for the most part, and for those that you don’t ask if you give them a powerful machine with a lot of screen real estate they are generally very happy.

Since I mentioned earlier that there are some things we do here that aren’t all that scalable – in that they work for a company with 20 devs but not 100 i’ll split them up.

The Scalable

The base config that I get new devs in the NY office is:

• Dell Optiplex 980 class tower
• Max out the RAM
• Best i7 Processor that I can get in it
• SSD primary drive
• 7.2k large (500GB – 1TB) Secondary Drive
• 30″ Primary Monitor
• 20″ Secondary monitor (turned sideways)

Every dev picks their own keyboard – we may give them the crappy one that comes with the machine to get them up and running, but they can request any keyboard/mouse combo they want (and … i do mean any).

The not so scalable

• For our remote devs we get workstation replacement class laptops. Basically I go out and find the most powerful laptop I can get at the time they start, as well as a 30″ monitor to go with it. Not Scalable.
• At least one dev in the NY office has a fully customized hand built machine with 2 30″ monitors. VERY not scalable.
• Whatever they want – within reason

Basically, it boils down to get them what they need to get their job done. One of the biggest challenge to some people is that they do not have management’s buy-in to get the Devs what they need. I find that this is a sad state of affairs, but I have no real advice to offer – especially since I work somewhere that management has mandated great dev machines.

When you have an infrastructure problem, rebooting the machine(s) is something you should do as a last resort. The reason is that you likely will never learn what the problem was, and it is probably going to come up again. I generally deplore this sort of troubleshooting and wrote about that opinion in my previous “Push the Green Button Twice” post. That being said, this is what we resorted to this past Friday for our entire switching infrastructure. This brought us offline for several minutes.

It all started on a Rainy Evening this Past Wednesday…

On Wednesday evening of this past week we started to see network timeouts in our application logs. Digging into this further and checking more logs this seemed to be widespread. On our Linux routers which run carp on the LAN side we saw some flapping going on. On our load balancers, we saw messages about late heartbeat messages. We use failover Intel teaming on our web server NICs and saw errors about missing probes. The problem was wide spread enough that it seemed to be the switching infrastructure, however there were no significant errors in the switch logs. We did see some ASIC and interface drops, but the incrementing of these did not seem to always coincide with major network blips in our infrastructure.

We then tried to localize the problem. We took network captures, and lots of them. Some from SPAN ports covering all of our traffic. Some from examples between select servers from the viewpoint of both servers as well as the viewpoint of the switch ports they were attached to. In addition to this we did iperf tests and ping tests between all sorts of different points in our network. We did broadcast analysis, tcp analysis, latency analysis, and IO graphing. Several of us worked pretty much around the clock for three days trying to figure this out. Although from the outside we were pretty much up, users were seeing timeouts. We even brought Cisco support into the mix and went through 3 support techs.

After three days of this, we honestly didn’t know a whole lot more than we did when we started — we were losing packets. We thought a lot about what we changed when this all started to happen and couldn’t think of anything. About two weeks ago we changed our switch configuration to a stacked setup using flexstack. Although a major change, it was two weeks ago. When we start to go down this road we are just starting to guess. Unless you actually see evidence that points to something, you really could say it is just about anything. The switch stacking is more related to what is going on, but there have been more recent changes — like the fact that it was raining — perhaps it was the rain?

When the jokes about what might be causing the problem become just as frequent as reasonable theories, that is probably the time to just try turning it off and on again — and that is what we did. It seems to have fixed the problem, but the weekend is our low traffic point and it could just seem fine because of that. This could also be some sort time based bug or something that is only triggered under a certain conditions.

Our Best Current Theory

Although traffic on most of our interfaces is quite low, lower than 100Mbit/s on Gigabit ports, it occured to me that maybe we were saturating more small scale units of time. I posted a question about this on Server Fault. The basic idea is that 1GBit per second is also 1Mbit per millisecond, and we are spiking the one millisecond limt frequently. If that is a realistic limit, our captures confirm that we do hit a lot. Perhaps enough of these spikes punishes the switches enough to trigger an unknown IOS bug?

This is still just a guess, but it is at least a plausible theory. So the solution we are going implement is a network architecture change I had planned on if we ever approached the 1 GBit/s bottleneck. We are going to set up a dedicated VLAN between our web servers and database servers that uses dedicated NIC ports. This dedicated path also won’t traverse the router making sure there isn’t a gateway bottleneck. The database traffic from the web tier will have its own dedicated interfaces that don’t have to share the path with our redis caching traffic and http traffic. Lastly we will bond these with an active-active method that will give us more throughput.

We don’t know if this will help prevent this problem or not, but we all think it is a better architecture so either way it is an optimization worth doing.

A Lesson in Troubleshooting Complex Problems — Document As You Go

The biggest mistake we made in this process so far in my opinion was not documenting our troubleshooting while we are doing it. By the time we got to Friday, we had a lot of data points. There were enough that we had trouble keeping them all in our head. That made it hard to make sense of them and our thoughts would go in circles at times. Even worse, we questioned if what we remembered and if our tests were even accurate.

Going forward I think we should use a collaborative document system like Google Docs to document our troubleshooting and any ideas we have as we go. Each test we do should include:

• When the test was run in UTC time and who ran it
• Screenshot(s) of the test. This is very important so people can verify the results, and repeat the test.
• Attachments and/or links to where the file is of logs and things like capture. Captures should include screen shots of graphs and analysis as well.
• Whatever conclusions you think can draw at the time from the testing as it relates to the problem.

With this on day two we can look at what we have done so far and what the sum of it all what logically might mean. Also, when people are taking breaks or are away, when they come back they can get caught up on what is happening. In the long run it will save time and make the troubleshooting more effective. We can still use an open phone line to communicate, but this would record the most important tests and ideas.

I really hope we stay calm enough and have the discipline to do this text time we deal with a major problem.

With all my talk of doing things in a scalable way comes a requirement, and that is that we actually accomplish this stuff in practice at Stack Exchange. We have been making a lot of progress in this area. George has been refining and expanding our deployment process. He improved our Windows deployment to include most of the software we use and has made it so kickstart for CentOS/Linux installs are integrated into our deployment as well.

Scaling your ability to manage your environment in my mind means doing more with less. I find I really only have to ask myself one question to quickly gauge if we are doing it right or not.

Do I have to repeat steps to do this task on multiple servers?

I like this question because it is more specific than “Is it automated?” while still implying automation. It is akin to the DRY rule in programming: “Don’t repeat yourself.”

When it comes to our environment, here is where we are at. For this I will ignore some details — for instance, we log into servers to kick off a PXE install and then just let it go — I consider that a Yes to “No repetition required” since it is only one or two steps and we don’t really deploy more than a server at a time. We don’t want to automate to the point over engineering beyond the projected size of our environment.

¹We do have some software that can be deployed via GPO, and LiteTouch deploys a lot of stack on the web servers during deployment. But future software updates and software that doesn’t lend itself to GPO is manual.

† I am currently in the middle of rolling out puppet so it is partially deployed on some of our Linux servers

The big picture of all of this is deployment as phase 1 and maintenance as phase 2 for both Windows and Linux. Also, ideally these phases connect to each other seamlessly.

One of our main goals is to change all of the above “No” to “Yes” over the next few months and then refine each step. For the most point we have deployment taken care of for both Windows and Linux. As far as maintenance goes, I believe as I progress in rolling out Puppet that will solve the vast majority of our Linux needs. How we will manage our firmware is still an unknown. As far as Window maintenance goes we are going to start exploring System Center soon and hope that it can meet our needs.

What I really think all of this will buy us is consistency, recoverability, and most importantly — time. By having all of this centrally managed it will make our servers more consitent with each other — and make them effectively drones. By having these processes automated we will be able to recover fast and replace servers easily. Lastly, and most importantly it buys time. By making our management faster and more agile, George and I can focus on rapidly deploying improvements to our environment instead of just maintaining it. By having less friction to deploying changes to our infrastructure I believe more possibilities for improvement will start emerge.

Most people in the system administration field I have talked to agree that the professionalization of system administration is happening faster. System administrators have always been paid, that isn’t what I mean by professionalization. What I do mean is that more leaders are emerging, participating in local groups, blogging, and participating on sites like Server Fault. The end results is that the standard of the profession is going up. That doesn’t mean that there hasn’t always been great people — but I believe the average expected expertise is increasing in the field.

One of the main ways I see this happening is in what people expect from answers on Server Fault. Lets take a hypothetical question:

“On my AFSRQ 2000, the system crashes every week or so. I managed to capture my logs before the crash: …. However, I am not sure what these lines mean?”

An acceptable answer to many system administrators would have been something like following as long as the AFSRQ 2000 stops crashing:

“Have you tried pressing the green button twice? Last time I did that it fixed my issues.”

The problem with this sort of answer is that it spreads ignorance. If there is no understanding of what the green button does, no wisdom can be gained from an answer like this. If the answer to how the fix works and why it works is missing, then this knowledge can not be applied to future related situations. Eventually, the AFSRQ 2000 will be upgraded to the AFSRQ 2008, and then button might not even exist, or, even worse, it might be a red button. Even if it solves this specific problem it doesn’t really help the profession advance. Also what happens is people attempt to press the green button to solve every problem — even when it wouldn’t help at all.

The better way to answer this is to try to understand why pressing the green button stops the system from crashing. Answer what causes the crash in the first place. The goal should be that answers should be expressed in the context of computing fundamentals and backed up by real data. By fundamentals I mean things like how operating systems work, system calls, memory management, network traces, hardware, etc. The reason is that when problems and answers are expressed in terms of fundamental building blocks patterns can emerge. Recognizing and understanding these patterns is what experience is system administration is really about. When answers are backed with real data (it helps when vendors empower you with tools to get the data), then this also gives context and proof, and other system administrators can verify the information. These are the sort of answers that raise the level of our field, pressing the green button twice does not.

As a system administrator, participating on Server Fault has made me better at what I do and it has also been fun. Here’s why.

The All-Star IT Department

In many tech companies the IT department is pretty small. Even in the bigger ones you get more specialized admins, say a team of two network admins, a Unix guy with a beard and a Windows lady. That means when you have ideas that you want to bounce off of people or questions that you have, there is only a small group of people you can go to, and even then an even smaller chance that they will know what you are talking about.

Server Fault is a combination of a big IT department and a knowledge resource. Most of the visitor traffic is people finding answers to questions that they have that someone else already asked. However, there is also a community here. The top administrators on the site get to know each other and many also hang out in chat. Spending time with other people who are passionate about what you do improves the quality of your day to day life.

So in my mind Server Fault is an all-star IT department that I get to work with all year round.

The advantages of writing

One of the main things that people value in a system administrator is communication. It seems that a lot of system administrators are just not very good at it for some reason. Whatever that reason is doesn’t matter as much as the fact that writing will make you a better communicator. I would not promise that it will make you a great writer but it will make you a lot better. If you not convinced that this is the case have a look at What Stack Overflow Can Teach You.

When you write out a question it clarifies what you are thinking about. Before you even get an answer from someone else you might have an idea as you write as to what the answer is, or you might realize that you are going down the wrong path.

Staying on top of your game

In the same way that writing a question helps clarify your thoughts, the same is true of writing answers. Every time you write an answer you refresh your memory on the topic and make the thought more defined and fleshed out. In doing this not only does your knowledge on that particular topic get better but also your mental models about system administration improve. Much of system administration has themes that come up and up again with many different technologies. Mastering these themes means improving your mental models that you apply to learn new things. As a result you end up with more facility in that particular area but also at the general patterns that come up in system administration. I believe this is what many people mean when they talk about experience, so in a way you can get more experience in a shorter amount of time.

The site also helps you keep track of what the new technologies are in a field that is always changing. When you see lots of questions coming up on a topic over time you know that you should at least know what it is. You can then take a look at these and be more in touch with what is going on in your field.

Showing off and helping others

It is fun as a geek to show off your knowledge to others and helping other people with their problems satisfies a fundamental desire. Also Sever Fault is usually less rewarding to people who show off their knowledge in an unkind way (i.e. RTFM). Because of this you might develop a better attitude as a system administrator — another thing that the stereotypical system administrator is not always famed for.

It is worth pointing out that people will also help you with your problems. When you are stuck on something you can ask the question, go work on something else, and come back and you will likely have an answer.

A personal and public repository of system administration knowledge

When you answer a question on Server Fault, you not only help the person asking the question, but you also help lots of other fellow system administrators who find that answer through a Google search which saves them time and frustration. That can also be true for yourself, answering questions or having your own questions on Server Fault is like keeping a notebook. I have actually used Google to look up something I did not know and found the answer on Server Fault that I myself wrote but had forgotten.

You might get a better job

By participating on Server Fault you will likely grow as an administrator in your technical knowledge and facility as well as your communication skills. With a body of answers you have written on Server Fault you can demonstrate to any future employers that you are interested in what you do beyond it just being your day job. Next time you walk into an interview you might get asked a question you answered before on Server Fault and will have a lot of confidence in answering that question well. If you are asked about something you don’t know you can have an intelligent conversation with the interviewer about it as he explains it to you since your mental models about system administration will have improved. This will demonstrate that you learn things quickly.

Server Fault is also a way to have a public presence on the web that is a little bit easier to do than a blog. If you do this over time employers will start to seek you out and you might not even have to go looking for your next job.

There are lots of different reasons people spend time in our community. If you are already one of them I hope that this might remind you why you do it or maybe gives you more reasons as to the value of being here. I personally greatly appreciate learning from all of you and have enjoyed getting to know the members of this community. If you haven’t spent much time with us I hope you consider giving our community a try or even just stopping by once in a while.

We are going to take a break from our regularly scheduled program to ask a very important question!

I’ve looked a few times, and there are precious few “SysAdmin” conferences. Most of them are very *nix centric. I’m also very interested in Networking, Windows, emerging technologies and all of that fun stuff.

What I have found happens in reality is there are vendor or developer focused conferences with a SysAdmin type track that you can follow. Which when I think about it for a while is probably a good thing. In my mind it allows me to sneak off the beaten path and go check out what the devs are doing. I feel it’s important to not only stay abreast of sysadmin developments, but also keep up with technology in general.

I’ve been to some really good smaller conferences run by vendors that seemed like they would be just one big sales pitch (ok, maybe they were) but they turned out to have some really great breakout sessions run by some really smart people. On the other side of the equation I’ve been to some large conferences that were just awful, where I didn’t pick up anything that wasn’t already general knowledge and everyone just sat there looking bored out of their minds.

So I wanted to run a little informal poll of you, the Great All Knowing Internet.  What conferences to you look forward to going to year after year?  What do you like about them? Also in a more general sense, what are the pros and cons of going to conferences at all? I want to harness this internet power for nothing but good I swear!

Right now Kyle and I have a short list of conferences we would really like to attend:

• O’Reilly Velocity 2011

• NJ-LOPSA PICC

• Tech Ed

• LinuxCon

Also, in addition to conferences you get excited to go to every year, are there any you are especially excited about this year? Let us know over on this Server Fault question: http://meta.serverfault.com/questions/1089/conferences-to-go-to-2011-edition

Please let us know in the comments what conferences you attend every year – maybe we’ll see you there!

Any programmer worth their salt has used and praised revision control systems. The weird thing to me though is the fact that it seems as if the majority of sysadmins havn’t embraced revision control.  If you look at what you do as a Sysadmin, it really isn’t that much different from what programmers do. I’ll bet you spend more time scripting solutions so that you don’t have to do the same task over and over again.  I’ve worked both with and without VCS’s, and I can tell you that working without them is a horrible way to work. If you don’t have some sort of VCS to put all of your scripts into you quickly end up in the untenable situation where you have random scripts laying all over the place. You’ll have some scripts that only exist on the server they were written for, some scripts sitting out on the file share, while others are just sitting on one of your admin’s computers. Or, very possibly,  just sitting on your desktop!

I’ll be the first to admit, Sysadmins generally don’t need all the fancy features of some of the modern VCS’s. But, we do need something to centrally manage our scripts and config files – these really our life blood as Sysadmins.

When looking for a solution to implement, there are a plethora of  options out there to choose from:

• CVS
• Subversion
• Git
• Mercurial
• TFS

All of these different systems have their ups and downs, a deep dive into the details of each is well beyond the scope of this post. I’ll just leave you with this bit of advice that has always been good to me – “Look at all the options and choose what is best for you.” Personally I’ve been very happy with just plain old CVS and that has served me very well. However, like any of the systems you need to get used to its little quirks. Once you choose one and get it running the way you want, the world gets much much brighter.

Take the following scenarios as an example and learn from them – I know all of you out there have run into a similar situation in the past. These very types of situations are drawn from my own experiences as a sysadmin, and have molded , and even changed my personal view on Version Control systems.

You write a little utility script for a seriously one-off function that your boss swears up and down you’ll never have to do again.

Point 1. Yes you will need to do this again because your result will be expected at every board meeting from now to infinity.

Point 2. You are going to think of a way to make it even more awesome and will completely refactor the script to make it more awesome.

Now, lets tackle the first situation erm first.

You’ve just tossed this script together on your local machine (sans comments, or even a unified coding style of course)  you run it and get the results your boss has been clamoring for all day (he even got out the pointy stick ). Your boss is happy because he has the info he needed to make his boss happy, you are happy because he put the pointy stick away before causing permanent damage. You move on to working on more important things like figuring out why the HA cluster keeps failing over at noon on Fridays. Three months later your boss comes around 1 hour before the next board meeting asking for “that thing you did last time with the thing.” I mean you should be able to pull it right up since you did the hard work for him 3 months ago right?  Ahh you say, I still have that script! this should only take 5 mins. Of course what you forgot is that you re-imaged that laptop you had the only copy of the script on, of course you backed your laptop up like a good sysadmin before re-imaging – it’s on one of those …. usb .. drives… over … there.

Now, you poor soul if you had checked your script into your “random stuff” VCS just in case you could have just pulled the most recent version down, run the script and forgotten about it for three more months.

Next, we have the much more probable situation where you write the same script as above, and decide “I can make this more awesome!” So you start hacking away. tweaking this and that moving this, removing that. Then suddenly you realize you have completely forgotten what you just changed that makes the script drop that image of your dog into the output of everything instead of the data you need. You start pushing that code, tweaking that bit there, move this around delete that. And now … and now the script does ….. nothing. You have no way back. You need to start over.

Yet another situation where a VCS system would have saved your skin. You could just simply have reverted to the last working version and moved on from there.

Beyond those – obviously made up – situations, a few more of the added benefits of a VCS system are:

• Centralize backups of your scripts, configs, anything you put into the VCS
• Check and see who checked in that change that broke the production environment
• Persist your scripts easily across machines and employees
• Feel smug in the fact you do this when you hear of someone losing “the script”

What this is really all about is preserving your hard work. You work hard to preserve the work of others, why not work hard to preserve your own work?

Just a quick note, we have moved blog.serverfault off of the Tumblr platform and onto a WordPress based self-hosted solution. And, no this has nothing to do with the outage that recently happend. We had been wanting to do this for a while, the outage just made us re-prioritize things.

I know we have been a little lacking in the posts recently, but this will change soon. We have been very busy working hard to make the Stack Exchange network better, stronger, and faster. I promise we will have more insightful posts coming soon!