It is pretty hard to get a bunch of system administrators together for any period of time and not have a conversation about command line for managing things. In my experience, command line always wins the debate for any medium to large installation. Microsoft caught on to this and has introduced powershell. From my own experience and people I have talked to, powershell adoption has been slow. Like powershell or dislike it, it is an interesting take on command line with its object oriented versus text-based approach. It is also quite powerful, so why the slow adoption?
In my mind it isn’t a problem with powershell itself, but rather a result of human nature, culture, and the Windows ecosystem. The problem lies in workflow, and I think the best way to illustrate this is to contrast Windows versus Linux administration. When managing Linux systems, scripting something flows naturally from the experience of trying things out in the first place. For example, lets say someone with some Linux experience (but not an expert) is building a piece of software and installing it on Linux for the very first time. The steps usually involve something like:
- Run the traditional, configure, make, make install from the command line. See what libraries or binaries are missing that are required to build, from the errors. Install them (often via yum or apt), keep doing this until it works.
- Possibly add some users to the system, adjust permissions etc with commands like useradd, chown, chmod.
- Edit some text files
- Set it to start at boot with commands like chkconfig or update-rc.d
As you do this more and more, you learn the patterns of what fails, and get the urge to automate it. Although things like puppet are the end goal these days, the next normal step is to write shell scripts to do this. This is the lynchpin of the whole experience, a shell script is just a little flow control, error handling, and a few variables on top of what you were already doing. With the Linux experience, scripting flows naturally as a next step in your experience. It is a gradual learning experience, and your previous experience of doing things manually taught you much of what you need to know to script it — what goes wrong, what to check, how to do everything you need to do. Even the first time tutorials on the web are almost always command line. From the second you start learning how to administrate a Linux box, you are learning how to script.
Unless you are extremely disciplined or have been taught Windows administration recently by a “benevolent” dictator, your first steps in Windows administration are via the GUI and Wizards. Don’t get me wrong, for many things this gets the job done faster and with far more ease. The generally accepted problem with the GUI though is that it doesn’t scale. When you want to scale in management you often turn to scripts (GPOs are very useful, but you are generally out of luck if what you want isn’t predefined.) With Windows, scripting these tasks is generally a complete departure from your previous experience. Scripting doesn’t flow naturally from previous experience, and scripting something is scary because of this. This means the windows ecosystem, due to human nature, is far less likely to produce administrators with scripting experience. Therefore, powershell adoption is logically slower. This also means administrators are less likely to be testing powershell and pressuring each other and Microsoft to improve it.
The cure to this workflow to me seems to be Windows Core since changing human nature isn’t easy. With Windows core, even though you can use the Management Console, you really have to start doing many things via the command line and powershell. With use of these tools, just as with the shell in Linux, scripting will flow naturally. The Catch-22 however though is that since adoption is slow, everything you need might not be available via command line options. At my company we haven’t used Windows core, but I’m hoping with Server 2012 it is time for Core, does anyone with more experience know if it is time for this shift?
Sorry this blog has been a bit quiet lately, we’ve been very busy making some big changes behind the scenes. So what are we up to? Let’s start with just the SQL infrastructure moves, here’s a list of servers in play as they started out:
- NY-DB01 – SQL2008 R2 Hosts all sites except Stack Overflow and the Sites DB
- NY-DB02 – SQL2008 R2 Daily backups restored from NY-DB01, and the Dev DBs
- NY-DB03 – SQL2008 R2 StackOverflow’s DB
- NY-DB04 – SQL 2008 R2 StackOverflow 5 minute behind hot spare in restore mode
- OR-DB01 – SQL2008 R2 Chat’s DBs
- OR-DB02 – SQL2008 R2 SEDE, Internal SEDE and Chat Dev DBs
- New Dell R710 w/ 2x OS Drives + 6x Data 300GB Intel 320 SSDs in RAID10 and 96GB RAM
- 3xNew Dell R720 w/ 2x OS Drives + 12x Data 200GB Intel 710 SSDs in RAID10 and 384GB RAM
First we set up the first SQL2012 Cluster with those new R720 machines. The new R720s are identical; they became NY-SQL01, NY-SQL02 and OR-SQL01:
- Primary: Sites DB, Stack Overflow
- Backups: Sites DB & StackOverflow Full and Transaction Logs -> NY
- Replica: Sites DB, Stack Overflow, Chat DBs
- Backups: Chat DBs Full -> NY
- Primary: Chat DBs
- Replica: Sites DB, Stack Overflow
- Backups: Sites SB, StackOverflow & Chat DBs Full -> OR
For this we have 2 new availability groups, StackOverflow_AG and Chat_AG. The primary server for StackOverflow_AG is NY-SQL01 replicated to a secondary in the same data center (NY-SQL02) and across the country to Oregon (OR-SQL01). The Chat_AG is only on 2 servers: the OR-SQL01 primary (chat is hosted in Oregon) and the replica NY-SQL02. The reason chat is only on 2 servers is because SQL2012 availability groups do not have the ability to distinguish between sites and replicate that way…so it would send the same transaction stream across the country twice to replicate to the NY servers, rather than echoing the transactions through one to the other…this is an unnecessary use of bandwidth we feel.
The StackOverflow and Sites DB portion of the first cluster was completed on the 2012-08-11 maintenance window; chat will be completed on 2012-08-18 (part of chat has moved, we want to give it a week to observe any problems). Now what happens in the following week? We need to shuffle some hardware around.
With the StackOverflow DB moved off of the NY-DB03 and NY-DB04 pair, they’re ready to be re-tasked. Currently these servers are identical Dell R710s with 288GB of RAM, 2x OS Drives in a RAID 1 and 6x 200GB Intel 710 SSDs in RAID10. These boxes get re-tasked to be NY-SQL03 and NY-SQL04. Joining them in this second SQl2012 cluster is OR-SQL02, that new Dell R710 above. Here’s a breakdown:
- Primary: All Stack Exchange 2.0 Sites other than SO
- Backups: SE 2.0s Full + [diff of trans] -> NY
- Primary: SE 2.0 & SO Dev DBs
- Replica: All Stack Exchange 2.0 Sites other than SO
- Primary: Chat Dev DBs
- Replica: All Stack Exchange 2.0 Sites other than SO
- SE 2.0s Full + [diff of trans] -> OR
Now we’ve freed up the NY-DB01 and NY-DB02 boxes, they’ll also be nuked, get some new drives and be re-tasked for some other purposes (for example, one goes to Oregon to be the HAProxy traffic log out there).
For the miscellaneous bits, OR-DB01 will be freed up after the move to OR-SQL01 and OR-SQL02 of the chat DBs. We’ll then take OR-DB01 and install 2012 re-tasking it to host the data.stackexchange.com databases. It has double the memory of the current server and should provide a nice boost to performance there.
Why? What does all this moving get us? Well it turns out SQL 2012 Always on Availability Groups give us quite a bit. Here are the big ones for our architecture:
- Near real-time replicas of every production database, ready to go
- No more copying backups across to the offsite datacenter for redundancy
- We can read from the replicas, eliminating the need for an entire server and allows us to spread the read load out (e.g. API can point at a replica)
- A backup DR location is now doable
First, we can have very near real-time hot spares for all production databases (previously, we had up to 8 hours data loss between differentials). Second, we don’t need to do these wasteful copies of databases across the country purely for backup purposes…we have a nearly-in-sync replica across the country we can do speedy local backups from. That’s a huge cross-country VPN bandwidth savings as an added bonus. Third, we can spread the read load out across multiple servers (and we can add another 2 more to either of these availability groups if needed). Performance-wise, we don’t even have a need for any read load spreading, but it’s very nice to have as an option. Now for the last one: a DR site.
PEAK Internet in Oregon is where Stack Overflow began on a single server, and we’ve been very happy with the service provided ever since. Chat’s been all alone out there for over a year now, it deserves some company. In another blog post coming up I’ll detail how we’re setting up as a read-only disaster recovery location out there, as well as our intention to actively use that while we move datacenters in New York.
P.S. Make sure to RAID your PCIe SSD drives, we’ll put up a post with that story a bit later…
Observe, Orient, Decide, Act. I love simple but elegant models, and The OODA loop developed for combat operations by John Boyd is just that. Designed for situations like fighter jet combat, it fits high stress situations that require quick responses. Although comically less extreme, it is a very useful model for handling system administration incidents because it highlights what goes right and what goes wrong when a sysadmin or devops team deals with the unexpected.
OODA in Practice
As an example let’s say you have reports that your website is slow and sometimes timing out. Step one is to gather facts and Observe. For sysadmins this means looking through your logs, the reports themselves, and/or your monitoring system.
Once you have collected data it needs to be digested so you can Orient yourself to the situation. Orienting is the act of analyzing and interpreting the data. For example, logs contain many fields, but to turn that data into information the logs need to be queried to find anomalies or patterns. We create graphs or generate summary statistics, whatever we need to do to understand the situation. This often is naturally done alongside observation. In order to truly fix problems we try to come up with a hypothesis based on the data and our experience to find the real cause.
Eventually somebody has to Decide to do something, even it is just deciding to jump back to observation to get more information. For example, if there are indications that the database is slow, then you might decide to go back and collect more information about the performance of the database server and restart the loop.
The last stage is to actually Act and make some changes that will either fix the problem, test a hypothesis, or allow you to observe more information that can be analyzed. If you think certain queries are making the database server slow eventually someone has to decide to fix them and take action.
This is a loop that will almost always have many iterations. With this model a good sysadmin team can iterate the loop rapidly, smoothly, and intelligently. Also over time a good team develops tools to make the loop go faster and gets better at working together to tighten the loop.
This framework brings light to problematic patterns that come up in system administration. Each stage of the loop has common problems and often the loop isn’t navigated in a logical way.
When it comes to observation, the most common problem seems to be a lack of data or a willful skipping of this phase. Often there just isn’t anywhere near enough logging and monitoring to diagnose problems in a scientific way. There can also be a lack of discipline to take the time to actually collect the data needed to pinpoint issues in a smart way. If there is too much friction around getting the data or collecting it in the first place it can lead to skipping this phase. All of this leads to one of my pet peeves that comes up in system administration — guessing.
Guessing also shows up in the orientation phase. If the observation phase has been skipped or done poorly then you can’t really orient, all you can do is grope around hoping to get lucky. Sometimes guessing can make sense when it is based on experience — but that is really using heuristics and not guessing. A lack of good analytical skills and/or experience can also lead to guessing. If the data is there but nobody knows how to interpret it well then all you can do is guess. Also if the observation and orientation phases are too slow then the pressure builds and in panic people will just start trying random things.
If there are problems with deciding and acting then there tends to be organizational or personality problems. If it isn’t clear who should be making decisions, or if there is a lot of fighting around what decision to take then the team needs to sit down and have some frank conversations to hash out their problems. Everyone should be willing to move forward with choices and trust each other or the loop can get bogged down in this phase. Failure to act during a crisis can be frustrating so the team needs to have the skill and confidence to act with expertise.
OODA Done Right
Contrast all those problems with your ideal sysadmin team facing an urgent incident. Each stage is highly automated and is constantly improving. In a great team when major problems come up instantly everyone starts collecting and sharing data. The monitoring systems have all the information they need and they have already built tools to quickly analyze it. The alerts themselves have already automated much of the observation phase because they describe the components of the problem. With a good team this sort of monitoring likely exists if the there is continuous improvement around monitoring and they learn and implement what is needed based on past experience.
With good monitoring and analysis tools a smart team quickly comes up with several good possibilities based on their experience and what they are seeing after orienting themselves. They can then quickly decide to pick a theory and implement it because they know they can try other ideas quickly and they trust each other. They also will accept feedback (new information) at any stage and adjust smoothly.
Why it Matters
If there are problems at any stage of the model, then all of the other stages will suffer when it comes to facing incidents. The same model can be applied to longer term projects or strategy as well. It gives us a framework to analyze how we have performed and where we can focus on improving to prepare for the next unknown incident. Facing incidents with skill can make a failure feel like a success and the OODA loop can help you make sure that happens every time.
I have to apologize to the serverfault community for a few things:
First, we’ve been really, really busy around the offices of Stack Exchange, and we’ve just not had a good amount of time to write blog posts. Luckily, now that we’ve added a new Systems Administrator, Mr. Bart Silverstrim (http://serverfault.com/users/13647/bart-silverstrim) we might have some time for posts more often.
Secondly, we’d like to sincerely apologize for taking Bart from the ServerFault community. I’m sure with all the coffee he’ll be fetching, there will be little time for him to increase his already sizable 25k reputation (and give me time to catch up with him.)
All joking aside, we are very pleased that Bart is joining our team; he’s not only a very smart admin (which his serverfault profile will prove), he’s also a great guy to be around.
So, WELCOME SERVERFAULT VALUED ASSOCIATE #0000004! May you not crash the blogs with large images.
In order to make upgrades to our network infrastructure, we will be doing maintenance that will involve intermittent downtime tomorrow. The maintenance will start at 2PM UTC on April 14th, 2012 and will continue for several hours.
During this window we will be:
- Replacing our routers and firewalls
- Making some L3 Network design changes
The long term design change will involve changing our WAN and load balancing infrastruture from Active/Passive failover to Active/Active. When this is completed in the following weeks there will be a detailed post explaining our new design!
I have yet to meet a systems administrator who got into the field purely because they enjoyed stressful, sleepless nights for often-lower-than-average salaries. Being a sysadmin really requires a love of computers. For a lot of us, this came about in our earlier years as we acquired secondhand equipment and disassembled it to find out how it worked. How amazing and wondrous this world could be! So many cool toys! “I’m going to love working with this kind of stuff for the rest of my life,” you might have said.
One thing that we had no incentive or guidance to learn as young, fledgling hackers was a process for being methodical and thorough in the execution of our projects. I, for one, have several projects open on my workbench in the basement that are in varying states of completion. I can get away with this because they’re personal endeavors and the only person I’m accountable to is me. The problem comes when we translate this type of behavior into the world of business.
Your Office is Not Your Basement
The business world exists really for one purpose: turning a profit. Companies are built around processes and manpower which are (in a perfect world) optimized to produce the most profit for the least amount of cost. Systems administration is not immune to this; we are expected to do things right the first time and do them in such a way that we’re not deferring productivity costs down the road so that someone else will need to “clean up our messes.” All too often, however, we can fall into the habit of treating our job as the hobby we’ve loved since we were younger, and down this path lies the specter of carelessness or incomplete projects.
Lets face it: we love working on fun computer projects. However, I’m sure none of us gets a lot of joy out of pre-planning or cleaning up after we’re done. They’re what I like to call the “toilet plunging” of being a data janitor. Nobody wants to do it, but it has to be done. I have met a few sysadmins in the past who will work on a project as long as it is fun, but then leave the rest of the work to others or leave the project in a semi-functional state declaring that its status was “good enough for now” or “we’ll fix it later.” Doing this sort of thing is extremely harmful to productivity, though oftentimes you won’t realize this until months later where you’re forced to redo the old project before you can move forward with another.
Being Thorough is a Learned Habit
We don’t come out of the womb with an innate ability to understand all of the implications of our choices. This is an ability that one has to learn over time; arguably this is the reason why adolescence is such a trying time in most of our lives. Some people never fully pick up the habit of being thorough in everything they do, and for the most part one can cruise through life without needing to have this skill. Sysadmins, however, do need to have this skill-set and it needs to be taught as early as possible in their education so that the ability to plan and execute is deeply engrained in their own mythos of Systems Administration.
Everyone will eventually learn this lesson the hard way. At some point, some project you’ve taken shortcuts on will inevitably fail in a spectacularly unexpected way and you’ll realize the only person to blame was the original architect, yourself. Did you plan how you were going to execute the project? Did you consider any kind of pitfalls? Did you do any research, if necessary, beforehand? When executing the project, were you sure to do it in a way that would be easy to maintain in the future? Did you clean up after yourself or did you leave the worksite a virtual (or physical) hazard for all who come after you? These are all things we need to be aware of when we’re doing our jobs.
It Might be a Reason, But Not An Excuse
It’s important to understand the pitfalls of complacency, especially when talking about Systems Administration as a proper job. There are many ways that we try to excuse not being thorough, but ultimately they’re justifications for a choice that we feel is wrong but are too lazy to correct. A big one I hear is “this is only going to be temporary.” It won’t be, trust me. You might have a strong suspicion that the temporary fix will only be necessary for another month or two, but those projects you count on can easily slip in the timeline. Before you know it, that patch cable you cross-wired across three racks in a diagonal from top to bottom is in the way of a whole lot of equipment that might need to be maintained that now cannot be because production traffic is going over your “temporary fix.”
Another one I hear many times is “our downtime window was closing and we needed to cut corners to make the deadline.” Every time I hear this I want to strangle someone! First of all, nine times out of ten, the Sysadmins are responsible for setting the downtime window in the first place, so why didn’t they put in a lot of fluff time in case things went south? Also, believe it or not, the majority of users could care less how long the environment is down as long as they’ve been notified first that there was maintenance planned. Most user anguish comes from them being taken unawares that the system is down when they haven’t had proper time to come up with alternative things to do. If your executive has been procrastinating about doing a particularly time-consuming project, having the environment down when she’s spent the last hour giving herself a mental pep-talk about doing the project will assuredly cause angry calls to the helpdesk. If she’s aware that there’s changes afoot, though, she’ll be more likely to understand if the work took longer than initially anticipated. This isn’t representative of everyone, of course, but it’s been my experience with everyone I’ve had to support.
On Feeling Overwhelmed and Slog Overflow
There is another casualty of this problem and it doesn’t often hit your consciousness until long after you’ve gone down the path of convenience, forgoing proper planning. There can come a point where an environment is held together by so many rubber bands and duct tape that it becomes a chore to maintain. Once this happens, you’re in “Slog Overflow.” This condition, which I’ve just now given a name, is something that I’ve experienced in the past and I’ve seen others exhibit the same symptoms. It is the point where procrastination begins to win out over doing a job you once loved. At some point, you wake up and realize you don’t want to go to work anymore. You know that the day will be filled with countless firefights and you will have to tell consumers that their issues are ones you’d like to fix, but to do so you’d have to rip off the bandage on another system which nobody wants to risk. To fix the environment at this point becomes a huge sink of productivity and brain-capital and will vary in cost only based on how much downtime you want to incur to fix everything the “Right Way.” You don’t ever want to experience Slog Overflow, as it can really cause you to dislike a job that you’ve spent years enjoying. If you feel like your world is moving in this direction, TAKE ACTION NOW!
Here are some thoughts on how to fix these problems or prevent them from happening in the future:
- Always plan ahead – Put together (at the least) a rough outline of the work you’re going to do. Be sure that it includes pre- and post-task steps so that the whole process is considered. If you have more than one person on your team, ask them to go over your outline and see if there’s anything you might have forgotten.
- Embrace feedback and criticisms – We’re all in this together! While some sysadmins may not have the best people skills, they usually have good ideas and aren’t afraid to share them. Take what is useful, try not to be offended. Nobody’s perfect!
- Emphasize quality and demand it from everyone – We need to fight the urge to become complacent and make sure we do things right the first time. That sometimes means you should stop yourself mid-task and make sure that what you’re doing is the best choice in that situation. Demand a higher level of awesome from the people you work with, too. It only takes one hole for the boat to sink.
- Teach young Systems Administrators to be thorough – Lead by example. Junior sysadmins absolutely pick up behaviors from senior team members. If your seniors are procrastinators and always do the easy fix eschewing the right-but-harder choice, you’re setting a ridiculously horrible example for the young guys who need proper mentoring.
What are your experiences with these problems? Feel free to comment below; I love feedback.
Like any good family, when we saw that our older sibling had something we wanted what did we do? Well, we whined and complained until we got it!
Stack Overflow recently finished their 2011 survey, and as soon as the results where announced, Server Fault being the proper younger sibling that it is wanted one too.
So I call all of you Server Fault users to come and take our survey, you wanted it and now you’ve got it! (oh and you really should get your sysadmin friends to take it too – the more data the better!)
Just like the Stack Overflow survey we will be releasing the data right when the survey ends!
This topic was suggested by one of our users, Bart Silverstrim. He was curious about how companies could maintain an environment where maintenance downtimes would not actually affect customer experience. Think of large sites like Google, Facebook, CNN, pretty much any site in the quantcast or alexa top 100. These sites maintain extremely good uptime numbers, but how do they do it?
It’s no secret that sites like this employ a lot of servers to handle content delivery. There are not only load balanced server clusters but also CDNs and caching proxies that help mitigate some of the load on the environment. Eventually, though, all of these machines are going to need maintenance at some point. How does one do this and not affect perceived uptime from the users? This question is actually rather complex because it depends on multiple ecosystems to properly execute.
Monolithic Deployment vs. Continuous Deployment
For many years, the method of software development followed a pretty static and time consuming process:
- Define scope for this release
- Program features for the release
- Debug, if errors, back to #2
- QA (generally someone testing who ISNT the developer), if errors, back to #2
Now, I know a lot of shops that would laugh at this list and claim that they do only #2 and #3, then shove it up to the servers. This can very well be the case, but I’d hazard a guess that those companies have never gotten close to scraping the alexa top 100, nor would I believe with a straight face if they told me they had uptimes in excess of 99%. There is another way, though: Continuous Deployment.
Simply put, Continuous Deployment is when instead of having large, multi-bug/multi-feature releases, you instead implement many, many small changes continuously throughout the product lifetime. At first glance this method might sound dangerous to many sysadmins who like to plan a task to an atomic point algorithmically, but it’s actually extremely safe if you follow some standard methodologies, namely having an environment where you can test changes (for Stack Exchange, this is meta.stackoverflow.com) and use a build proctor like CruiseControl or TeamCity to facilitate the build process.
One of the main benefits of Continous Deployment is the fact that small changes are easy to deploy and usually just as easy to revert should problems arise. It becomes significantly harder to revert a deployment if there is a large corpus of changes contained in the update.
Code it like you’ll need to change it
One thing that new programmers often fall prey to is the laziness of not properly architecting their application. I am sure that every programmer remembers their earlier code shenanigans when they’ve needed to update something they wrote earlier in their career only to find that it was fraught with all sorts of freshman mistakes. To put it simply: you need to write your code to be portable, easily understood, and utilizing the best practices available for that language. If the language is OO, this means obeying standards like MVC, creating class interfaces and making the app as dynamic as possible. The reason should be clear: code that’s easier to interface with is easier to change and update, and it generally means that bugs introduced in most modules should not have global impact to the site.
Load Balancing is Good
Load Balancing is paramount to a seamless customer experience. It allows you to do some pretty cool tricks, but only so long as your web application is coded to properly handle a load balanced environment. The main thing one needs to think about when coding an app in a load balanced environment is that there’s no guarantee that request ‘n+1′ is going to land on the same server as request ‘n’, so you need to handle sessions in a centralized/db manner so that the cookies in the browser link you up to the proper session regardless of what server you hit. This does NOT mean, however, that you should disable persistance or affinity in your load balancer! There are benefits to keeping a session on the same server with regards to file caching and the like; we just want to make sure the app is ready should you want to take one of the servers down for maintenance.
Once properly load balanced, you gain several levels of win. First off, your app will be much more performant if there are more servers available to handle the load. Secondly, if you have multiple servers available, you can bring one of them offline and your users should never know the difference. A side benefit of Load Balancing is the peace of mind you’ll get knowing that if a box eats itself alive at 3am, your site will survive without you needing to fire up the laptop or, heaven forbid, head into the office at an extremely early hour.
How Stack Exchange does it
It may become clear after reading this that what I’m talking about isn’t necessarily how Stack Exchange works. People who visit us often know that we do have site downtimes for various reasons. For those curious, the below section is how we handle our deployment and development process.
There is one big place where the Stack Exchange Core Q&A Services are vulnerable, and this is at the database. Currently we employ SQL 2008 R2 database servers (currently SP1 as of this writing), with the primary server constantly replicating to the backup server via transaction log shipping. Those familiar with transaction log shipping will know that employing this method is really only usable in an active/”hot passive” mode. One of the downsides of using SQL Server is the lack of solid high availability given our transaction load. Simply put, both the asynchronous and synchronous methods of active/active can’t keep up with the sheer volume of transactions we throw at it. We’re hoping that when Denali comes out this spring, the HA features will improve to the point where we can be fully active/active.
The reason we incur downtimes for upgrades these days has to do partially with ease of execution and cleanliness; because we use transaction log shipping, if we wanted to go active on the backup node we’d have to break the replication and convert the backup server to the master server, then re-setup replication. Would we do this for a couple windows patches? Absolutely not. It’s a nontrivial amount of work that can incur human error and is unnecessary. We reserve this process for when we do have a major database emergency. We’re hoping that SQL 2012’s enhanced failover capabilities will permit us to enter read-only mode for brief maintenance windows, but this will need serious testing first, once Denali (2012) is released.
Development wise, this is the continuous deployment procedure that Stack Exchange uses:
- A developer will get the latest HEAD from the mercurial repository.
- That developer makes a change, then commits that change back to the Mercurial repository.
- TeamCity queries Mercurial (every 60 seconds) to check for new changes. If a change is found, TeamCity builds it immediately in the development testing environment.
- Once the developers have given the change a test in development, the developer then deploys the change to meta.stackoverflow.com.
- If the dev is having a good day, meta.so’s users won’t report any bugs and after a period of time the dev will push the code to everything except stackoverflow.
- If the change stands up to rigorous testing on the other approximately 80 sites in our network, the change will be pushed to stackoverflow, aka “the fire hose.”
One might be curious as to why we wait till the last minute to push code changes to stackoverflow. The reason behind this has to do with the fact that stackoverflow gets several orders of magnitude more traffic than the rest of Stack Exchange combined. A case study in this procedure: when we deployed ProtoBuf v2, the change worked great everywhere else, but as soon as the change was applied to stackoverflow.com, a “cold start” bug seriously degraded users’ experience.
An important thing to note as well is that a great deal of the sites’ code changes are toggleable by configuration, so if a problem is found it can be reversed and mitigated much, much faster than it would take a developer to debug and fix the problem. Employing this method where possible in your applications will be helpful for many reasons.
Uptime! Fsck Yeah!
For most of our readership, the problems described above might not apply to you. You may not have quite as many hits as Stack Exchange, or you might have less need for high availability and uptime. This isn’t to say that the above doesn’t apply; architecting for uptime should be in every developer’s best interests and best practices.
As always, your questions and comments are most welcome, feel free to post below.
If you are not familiar with the concept, technical debt is essentially the idea that you build and program things quickly, skipping the niceties in order to ship, and then fix it later. By putting things off you build up debt that needs to be paid down later. One of the places this most commonly shows itself is in performance.
It works like this. Developers make features because the business and users want features. Performance is hard, and the benefits of good performance are not usually as obvious or concrete as the benefits of new features. Therefore, nobody really pays attention to performance or it is intentionally skipped until it gets so bad that people consciously notice it. Then the developers need to do a “feature freeze” and fix things until performance is at least “okay.” again. If you don’t mind the cliche, the feature freeze is the “Rinse.”, and then it all starts over again — “Repeat.” This is the cycle of technical debt.
At Stack Exchange I saw this happen, the developers had to stop working on features and fix performance because it got the point where we were getting timeouts. However, here is where things get interesting: After that, it never happened again.
“Impossible!” No, it is not impossible. In reality, of course there are still things that slip by, but the overall macro cycle of technical debt, when it comes to performance, is avoidable. And if you order my VHS series for 19.95, I will tell you how.
In all seriousness, even if there is no one recipe, from my viewpoint Stack Exchange escaped the cycle through culture, and making the right performance investments. The culture that lead to this consists of:
- Placing a value on performance: “Performance is a feature”
- Well integrated development and operations
- A sense of craftsmanship when it comes to performance
Good performance makes a system enjoyable to use, everyone has to believe this idea. When development and operations are well integrated the teams empower each other, and since performance takes both programming and systems knowledge this is needed. Lastly, if good performance is an aspect of good craftsmanship, it becomes a source of pride.
These cultural aspects at Stack Exchange and the performance investments made enforce each other. I don’t think we could have one without the other. But if there is a secret sauce, it feels to me like it is the performance investments we have made. These investments follow a development pattern that results in instant feedback when it comes to performance:
The 3 Step Process to Good Performance Investments:
Step 1: Collect your data in a queryable way
I can’t emphasize enough how important this initial step is. Your performance data such as logs and system data (i.e. CPU/Memory/Network etc) needs to be in a format that can easily be queryed, extracted, aggregated, and molded in a way that leads to discovery. We use SQL Server for our logs and system data. It doesn’t have to be SQL, but I think that rrd, the common storage format used by systems like Cacti, although good for displaying time series graphs does not fill this requirement due to the difficulty of extracting data.
Step 2: Discover the Important Metrics
Once you have the data in a queryable format, you can then explore that data and discover what the important metrics are. Once we started capturing our web logs in SQL we were able to add custom headers that tell us things like which route is being hit, and measure performance grouped by route. If your data isn’t queryable the discovery process has too much friction.
Step 3: Automate and Integrate the Important Data
Once you have found the important data by exploring it with various queries, those queries should be automated and integrated into your application. Then with every build (rapid integration or frequent building helps) you get instant feedback. At Stack Exchange we have a dashboard that includes graphs from log data, system data, profiling results, and exceptions. We can explore our web logs with a data explorer instance. Also, some of this such as our profiler results are part of every page load.
This process leads to an instantaneous and effortless return of performance information. This eliminates the friction around discovering how your performance is changing. With this information readily available and in your face, it enables a culture where keeping up with performance becomes an aspect of good craftsmanship.
These tools we have created are performance investments. Investments are the opposite of debt. Investments give returns where as debt has interest. When you make these sort of performance investments the cycle of debt is broken and you start collecting the returns. For the most part, people in this world are either collecting returns or paying debt — and collecting returns feels damn good.
At some point in your career as a Systems Administrator or other Person-Of-Responsibility-in-IT, you will find yourself stuck in the unfortunate position of cleaning up a mess that was totally preventable, if you had known the signs beforehand that the problem was imminent. This fact is 100% assurable, as I have yet to meet a seasoned SysAdmin who didn’t have a war story similar to “man, if I was only monitoring disk space on server X…”
Monitoring is an extremely important tool in your arsenal of preventative measures. Monitoring is important for a great deal of different reasons:
- Monitoring allows you to send alerts if certain conditions are met,
- It allows you to visualize trends in data,
- Provides a method of assurance to the customer that their consumed services are guarded,
- Allows you to do internal benchmarking for when you need to come up with budget/spend numbers.
There are many different products in the monitoring sphere. Some are extremely expensive and meant only for enterprise use and there are many that are open source and therefore free to use. My personal favorite is Nagios, though it does have some shortcomings that I will touch on later. Most monitoring systems follow the same basic recipe: You configure hosts, which in turn have services or metrics you want to monitor. The monitoring system will “optionally” alert you if you configure it to do so. Most monitoring systems have a method of keeping historical data and graphing it. This is not only a great way for you to look at pretty graphs; the management staff will get excited seeing information visualized in a way they’re used to seeing.
There are several methods of monitoring. The most basic and least useful method alone is a simple ping test. Products that provide this feature give you a simple up/down alert if there’s an outage, but honestly, the users breaking down your door will be a more effective alert. Most monitoring systems will give you the ability to not only run ping tests, but also have checks that incorporate SNMP (Simple Network Monitoring Protocol) statistics. This is better than a simple ping test, but in my opinion still short of the complete picture I’d like to see. SNMP does have its benefits, though: since it’s been around practically since the beginning of time, lots of equipment supports it out-of-the-box. It’s the primary method one uses to gather statistics about your routers and switch interfaces, including drops/discards and packet saturation rates.
Going beyond simple ping and snmp monitoring, many monitoring applications allow you to have custom checks that give you metrics for items SNMP misses.
For instance, the nagios plugin exchange provides a plethora of check-metrics that other users have written with varied themes from temperature probe
monitoring to advanced Microsoft SQL statistics checking. In particular, one Nagios addon that I cannot live without is the “nagios-wsc” project, which you
install on a windows IIS server and it in turn provides the ability for Nagios to check windows statistics over WMI. Being able to do this vastly
improves the metrics you can extract from windows servers.
At the time of this writing, I’m not sure if a similar interface for PowerShell exists or is
in the works, but that would be a “must-have” addon, as Microsoft has said that they’re moving away from WMI in favor of PowerShell, at least as far as Exchange
is concerned. (As commenter Jim Butts points out, I don’t have citation for this and so I’m going to strike it from the post, though I swear I remember reading at one point that Microsoft intended to replace WMI with PowerShell. This might have only been related to the Exchange family of products, though, so don’t take it as gospel. Also worth noting, as another commenter explained, WMI and PowerShell are two different technologies meant to do two different things. WMI is an instrumentation interface, whereas PowerShell is a scripting language. It just so happens that you can get some information with PowerShell that you cannot easily get through the WMI interface.)
One of the major pieces of any monitoring environment is the ability to alert administrators of an impending problem. Many admins default to e-mail for this, but not many people realize that most mobile phones are fantastic SMS modems. Find a prepaid model that lets you send SMS’s from a serial/usb connection via AT commands, and now you have not only an out-of-band notification method, but you’ve saved yourself a bunch of money on specialist hardware. I’ve also heard of some people using Asterisk to do voice-dial alerts; when the alert hits the system, it Text-to-Speech’s the alert and then plays the audio over a telephone call to the remote party. Pretty slick and high tech, but in my opinion that’s a rather big system to rely on for monitoring. Generally, simple methods of alerting, with less moving parts, makes for a more stable and trustworthy alerting platform.
A helpful part of many monitoring systems is being able to group hosts and services into logical containers. This ability lets you not only monitor a whole logical service from one view, but also allows you to quick-add new servers to a group and immediately have that server’s checks already applied to it by virtue of being a member of the host or service group. If your monitoring system supports grouping and you are not using it “you are doing it wrong.”
Do you need a monitoring environment? Yes. There is no other answer to this question. If you have even a single server in your environment, monitoring it will provide a treasure trove of information about the system. The only question is, how much do I have to monitor? This depends a lot on your customer SLAs and the expectactions of uptime. As the uptime target grows and the margin for error shrinks, you will need to squeeze more and more information out of your environment to maximize the “magic twilight” between a server showing symptoms of impending troubles and “THE SYSTEM IS DOWN.”
Having a lot of stuff monitored also helps with correlation and causation. For instance, you might have a website error showing up on one of your servers, and you start diagnosing that error. Thirty minutes later, you come to find the problem was that the SQL server is bogged down and replying to queries too slowly. If you were just monitoring the web server, you just lost thirty minutes. If you were monitoring both the SQL and the web server, you would have a greater chance of knowing that the problem lay with SQL, not with the web server. All of this data can lead to a condition I call “data addiction,” and it’s a condition that I will attest is pervasive at Stack Exchange. Many of our developers rely heavily on our monitoring data to give them metrics into how the sites are operating.
One thing that needs to be considered when you setup monitoring is the “Who Watches The Watcher” paradox. Simply put, if you become reliant on your monitoring system, you want to trust that the monitoring system is active. There are a few ways to solve this. First off, if your organization has multiple sites, setup a monitoring server at each site and have the monitoring servers monitor each other as well as their other systems. If you have only a single site, then you should probably consider getting a simpler monitoring system to monitor your monitoring system. You’ll never be able to have 100% faith that your monitoring system is foolproof, it’s tough to rely on software that was written by human hands to be 100% failure free, all the time. Regularly auditing the monitoring environment is the best way to keep your faith in the system.
In closing, I’d like to reiterate that even if you feel you don’t need a monitoring system, I’m pretty sure you would still benefit from one. Start small if this is your first time; if you run into issues, sites like ServerFault are a great resource to get good answers. Over time, I think you’ll grow to enjoy having the peace of mind that comes from knowing what your network is doing without having to spend additional time manually collecting statistics on your own.