Sorry this blog has been a bit quiet lately, we’ve been very busy making some big changes behind the scenes. So what are we up to? Let’s start with just the SQL infrastructure moves, here’s a list of servers in play as they started out:
- NY-DB01 – SQL2008 R2 Hosts all sites except Stack Overflow and the Sites DB
- NY-DB02 – SQL2008 R2 Daily backups restored from NY-DB01, and the Dev DBs
- NY-DB03 – SQL2008 R2 StackOverflow’s DB
- NY-DB04 – SQL 2008 R2 StackOverflow 5 minute behind hot spare in restore mode
- OR-DB01 – SQL2008 R2 Chat’s DBs
- OR-DB02 – SQL2008 R2 SEDE, Internal SEDE and Chat Dev DBs
- New Dell R710 w/ 2x OS Drives + 6x Data 300GB Intel 320 SSDs in RAID10 and 96GB RAM
- 3xNew Dell R720 w/ 2x OS Drives + 12x Data 200GB Intel 710 SSDs in RAID10 and 384GB RAM
First we set up the first SQL2012 Cluster with those new R720 machines. The new R720s are identical; they became NY-SQL01, NY-SQL02 and OR-SQL01:
- Primary: Sites DB, Stack Overflow
- Backups: Sites DB & StackOverflow Full and Transaction Logs -> NY
- Replica: Sites DB, Stack Overflow, Chat DBs
- Backups: Chat DBs Full -> NY
- Primary: Chat DBs
- Replica: Sites DB, Stack Overflow
- Backups: Sites SB, StackOverflow & Chat DBs Full -> OR
For this we have 2 new availability groups, StackOverflow_AG and Chat_AG. The primary server for StackOverflow_AG is NY-SQL01 replicated to a secondary in the same data center (NY-SQL02) and across the country to Oregon (OR-SQL01). The Chat_AG is only on 2 servers: the OR-SQL01 primary (chat is hosted in Oregon) and the replica NY-SQL02. The reason chat is only on 2 servers is because SQL2012 availability groups do not have the ability to distinguish between sites and replicate that way…so it would send the same transaction stream across the country twice to replicate to the NY servers, rather than echoing the transactions through one to the other…this is an unnecessary use of bandwidth we feel.
The StackOverflow and Sites DB portion of the first cluster was completed on the 2012-08-11 maintenance window; chat will be completed on 2012-08-18 (part of chat has moved, we want to give it a week to observe any problems). Now what happens in the following week? We need to shuffle some hardware around.
With the StackOverflow DB moved off of the NY-DB03 and NY-DB04 pair, they’re ready to be re-tasked. Currently these servers are identical Dell R710s with 288GB of RAM, 2x OS Drives in a RAID 1 and 6x 200GB Intel 710 SSDs in RAID10. These boxes get re-tasked to be NY-SQL03 and NY-SQL04. Joining them in this second SQl2012 cluster is OR-SQL02, that new Dell R710 above. Here’s a breakdown:
- Primary: All Stack Exchange 2.0 Sites other than SO
- Backups: SE 2.0s Full + [diff of trans] -> NY
- Primary: SE 2.0 & SO Dev DBs
- Replica: All Stack Exchange 2.0 Sites other than SO
- Primary: Chat Dev DBs
- Replica: All Stack Exchange 2.0 Sites other than SO
- SE 2.0s Full + [diff of trans] -> OR
Now we’ve freed up the NY-DB01 and NY-DB02 boxes, they’ll also be nuked, get some new drives and be re-tasked for some other purposes (for example, one goes to Oregon to be the HAProxy traffic log out there).
For the miscellaneous bits, OR-DB01 will be freed up after the move to OR-SQL01 and OR-SQL02 of the chat DBs. We’ll then take OR-DB01 and install 2012 re-tasking it to host the data.stackexchange.com databases. It has double the memory of the current server and should provide a nice boost to performance there.
Why? What does all this moving get us? Well it turns out SQL 2012 Always on Availability Groups give us quite a bit. Here are the big ones for our architecture:
- Near real-time replicas of every production database, ready to go
- No more copying backups across to the offsite datacenter for redundancy
- We can read from the replicas, eliminating the need for an entire server and allows us to spread the read load out (e.g. API can point at a replica)
- A backup DR location is now doable
First, we can have very near real-time hot spares for all production databases (previously, we had up to 8 hours data loss between differentials). Second, we don’t need to do these wasteful copies of databases across the country purely for backup purposes…we have a nearly-in-sync replica across the country we can do speedy local backups from. That’s a huge cross-country VPN bandwidth savings as an added bonus. Third, we can spread the read load out across multiple servers (and we can add another 2 more to either of these availability groups if needed). Performance-wise, we don’t even have a need for any read load spreading, but it’s very nice to have as an option. Now for the last one: a DR site.
PEAK Internet in Oregon is where Stack Overflow began on a single server, and we’ve been very happy with the service provided ever since. Chat’s been all alone out there for over a year now, it deserves some company. In another blog post coming up I’ll detail how we’re setting up as a read-only disaster recovery location out there, as well as our intention to actively use that while we move datacenters in New York.
P.S. Make sure to RAID your PCIe SSD drives, we’ll put up a post with that story a bit later…
Observe, Orient, Decide, Act. I love simple but elegant models, and The OODA loop developed for combat operations by John Boyd is just that. Designed for situations like fighter jet combat, it fits high stress situations that require quick responses. Although comically less extreme, it is a very useful model for handling system administration incidents because it highlights what goes right and what goes wrong when a sysadmin or devops team deals with the unexpected.
OODA in Practice
As an example let’s say you have reports that your website is slow and sometimes timing out. Step one is to gather facts and Observe. For sysadmins this means looking through your logs, the reports themselves, and/or your monitoring system.
Once you have collected data it needs to be digested so you can Orient yourself to the situation. Orienting is the act of analyzing and interpreting the data. For example, logs contain many fields, but to turn that data into information the logs need to be queried to find anomalies or patterns. We create graphs or generate summary statistics, whatever we need to do to understand the situation. This often is naturally done alongside observation. In order to truly fix problems we try to come up with a hypothesis based on the data and our experience to find the real cause.
Eventually somebody has to Decide to do something, even it is just deciding to jump back to observation to get more information. For example, if there are indications that the database is slow, then you might decide to go back and collect more information about the performance of the database server and restart the loop.
The last stage is to actually Act and make some changes that will either fix the problem, test a hypothesis, or allow you to observe more information that can be analyzed. If you think certain queries are making the database server slow eventually someone has to decide to fix them and take action.
This is a loop that will almost always have many iterations. With this model a good sysadmin team can iterate the loop rapidly, smoothly, and intelligently. Also over time a good team develops tools to make the loop go faster and gets better at working together to tighten the loop.
This framework brings light to problematic patterns that come up in system administration. Each stage of the loop has common problems and often the loop isn’t navigated in a logical way.
When it comes to observation, the most common problem seems to be a lack of data or a willful skipping of this phase. Often there just isn’t anywhere near enough logging and monitoring to diagnose problems in a scientific way. There can also be a lack of discipline to take the time to actually collect the data needed to pinpoint issues in a smart way. If there is too much friction around getting the data or collecting it in the first place it can lead to skipping this phase. All of this leads to one of my pet peeves that comes up in system administration — guessing.
Guessing also shows up in the orientation phase. If the observation phase has been skipped or done poorly then you can’t really orient, all you can do is grope around hoping to get lucky. Sometimes guessing can make sense when it is based on experience — but that is really using heuristics and not guessing. A lack of good analytical skills and/or experience can also lead to guessing. If the data is there but nobody knows how to interpret it well then all you can do is guess. Also if the observation and orientation phases are too slow then the pressure builds and in panic people will just start trying random things.
If there are problems with deciding and acting then there tends to be organizational or personality problems. If it isn’t clear who should be making decisions, or if there is a lot of fighting around what decision to take then the team needs to sit down and have some frank conversations to hash out their problems. Everyone should be willing to move forward with choices and trust each other or the loop can get bogged down in this phase. Failure to act during a crisis can be frustrating so the team needs to have the skill and confidence to act with expertise.
OODA Done Right
Contrast all those problems with your ideal sysadmin team facing an urgent incident. Each stage is highly automated and is constantly improving. In a great team when major problems come up instantly everyone starts collecting and sharing data. The monitoring systems have all the information they need and they have already built tools to quickly analyze it. The alerts themselves have already automated much of the observation phase because they describe the components of the problem. With a good team this sort of monitoring likely exists if the there is continuous improvement around monitoring and they learn and implement what is needed based on past experience.
With good monitoring and analysis tools a smart team quickly comes up with several good possibilities based on their experience and what they are seeing after orienting themselves. They can then quickly decide to pick a theory and implement it because they know they can try other ideas quickly and they trust each other. They also will accept feedback (new information) at any stage and adjust smoothly.
Why it Matters
If there are problems at any stage of the model, then all of the other stages will suffer when it comes to facing incidents. The same model can be applied to longer term projects or strategy as well. It gives us a framework to analyze how we have performed and where we can focus on improving to prepare for the next unknown incident. Facing incidents with skill can make a failure feel like a success and the OODA loop can help you make sure that happens every time.
I have to apologize to the serverfault community for a few things:
First, we’ve been really, really busy around the offices of Stack Exchange, and we’ve just not had a good amount of time to write blog posts. Luckily, now that we’ve added a new Systems Administrator, Mr. Bart Silverstrim (http://serverfault.com/users/13647/bart-silverstrim) we might have some time for posts more often.
Secondly, we’d like to sincerely apologize for taking Bart from the ServerFault community. I’m sure with all the coffee he’ll be fetching, there will be little time for him to increase his already sizable 25k reputation (and give me time to catch up with him.)
All joking aside, we are very pleased that Bart is joining our team; he’s not only a very smart admin (which his serverfault profile will prove), he’s also a great guy to be around.
So, WELCOME SERVERFAULT VALUED ASSOCIATE #0000004! May you not crash the blogs with large images.
In order to make upgrades to our network infrastructure, we will be doing maintenance that will involve intermittent downtime tomorrow. The maintenance will start at 2PM UTC on April 14th, 2012 and will continue for several hours.
During this window we will be:
- Replacing our routers and firewalls
- Making some L3 Network design changes
The long term design change will involve changing our WAN and load balancing infrastruture from Active/Passive failover to Active/Active. When this is completed in the following weeks there will be a detailed post explaining our new design!
I have yet to meet a systems administrator who got into the field purely because they enjoyed stressful, sleepless nights for often-lower-than-average salaries. Being a sysadmin really requires a love of computers. For a lot of us, this came about in our earlier years as we acquired secondhand equipment and disassembled it to find out how it worked. How amazing and wondrous this world could be! So many cool toys! “I’m going to love working with this kind of stuff for the rest of my life,” you might have said.
One thing that we had no incentive or guidance to learn as young, fledgling hackers was a process for being methodical and thorough in the execution of our projects. I, for one, have several projects open on my workbench in the basement that are in varying states of completion. I can get away with this because they’re personal endeavors and the only person I’m accountable to is me. The problem comes when we translate this type of behavior into the world of business.
Your Office is Not Your Basement
The business world exists really for one purpose: turning a profit. Companies are built around processes and manpower which are (in a perfect world) optimized to produce the most profit for the least amount of cost. Systems administration is not immune to this; we are expected to do things right the first time and do them in such a way that we’re not deferring productivity costs down the road so that someone else will need to “clean up our messes.” All too often, however, we can fall into the habit of treating our job as the hobby we’ve loved since we were younger, and down this path lies the specter of carelessness or incomplete projects.
Lets face it: we love working on fun computer projects. However, I’m sure none of us gets a lot of joy out of pre-planning or cleaning up after we’re done. They’re what I like to call the “toilet plunging” of being a data janitor. Nobody wants to do it, but it has to be done. I have met a few sysadmins in the past who will work on a project as long as it is fun, but then leave the rest of the work to others or leave the project in a semi-functional state declaring that its status was “good enough for now” or “we’ll fix it later.” Doing this sort of thing is extremely harmful to productivity, though oftentimes you won’t realize this until months later where you’re forced to redo the old project before you can move forward with another.
Being Thorough is a Learned Habit
We don’t come out of the womb with an innate ability to understand all of the implications of our choices. This is an ability that one has to learn over time; arguably this is the reason why adolescence is such a trying time in most of our lives. Some people never fully pick up the habit of being thorough in everything they do, and for the most part one can cruise through life without needing to have this skill. Sysadmins, however, do need to have this skill-set and it needs to be taught as early as possible in their education so that the ability to plan and execute is deeply engrained in their own mythos of Systems Administration.
Everyone will eventually learn this lesson the hard way. At some point, some project you’ve taken shortcuts on will inevitably fail in a spectacularly unexpected way and you’ll realize the only person to blame was the original architect, yourself. Did you plan how you were going to execute the project? Did you consider any kind of pitfalls? Did you do any research, if necessary, beforehand? When executing the project, were you sure to do it in a way that would be easy to maintain in the future? Did you clean up after yourself or did you leave the worksite a virtual (or physical) hazard for all who come after you? These are all things we need to be aware of when we’re doing our jobs.
It Might be a Reason, But Not An Excuse
It’s important to understand the pitfalls of complacency, especially when talking about Systems Administration as a proper job. There are many ways that we try to excuse not being thorough, but ultimately they’re justifications for a choice that we feel is wrong but are too lazy to correct. A big one I hear is “this is only going to be temporary.” It won’t be, trust me. You might have a strong suspicion that the temporary fix will only be necessary for another month or two, but those projects you count on can easily slip in the timeline. Before you know it, that patch cable you cross-wired across three racks in a diagonal from top to bottom is in the way of a whole lot of equipment that might need to be maintained that now cannot be because production traffic is going over your “temporary fix.”
Another one I hear many times is “our downtime window was closing and we needed to cut corners to make the deadline.” Every time I hear this I want to strangle someone! First of all, nine times out of ten, the Sysadmins are responsible for setting the downtime window in the first place, so why didn’t they put in a lot of fluff time in case things went south? Also, believe it or not, the majority of users could care less how long the environment is down as long as they’ve been notified first that there was maintenance planned. Most user anguish comes from them being taken unawares that the system is down when they haven’t had proper time to come up with alternative things to do. If your executive has been procrastinating about doing a particularly time-consuming project, having the environment down when she’s spent the last hour giving herself a mental pep-talk about doing the project will assuredly cause angry calls to the helpdesk. If she’s aware that there’s changes afoot, though, she’ll be more likely to understand if the work took longer than initially anticipated. This isn’t representative of everyone, of course, but it’s been my experience with everyone I’ve had to support.
On Feeling Overwhelmed and Slog Overflow
There is another casualty of this problem and it doesn’t often hit your consciousness until long after you’ve gone down the path of convenience, forgoing proper planning. There can come a point where an environment is held together by so many rubber bands and duct tape that it becomes a chore to maintain. Once this happens, you’re in “Slog Overflow.” This condition, which I’ve just now given a name, is something that I’ve experienced in the past and I’ve seen others exhibit the same symptoms. It is the point where procrastination begins to win out over doing a job you once loved. At some point, you wake up and realize you don’t want to go to work anymore. You know that the day will be filled with countless firefights and you will have to tell consumers that their issues are ones you’d like to fix, but to do so you’d have to rip off the bandage on another system which nobody wants to risk. To fix the environment at this point becomes a huge sink of productivity and brain-capital and will vary in cost only based on how much downtime you want to incur to fix everything the “Right Way.” You don’t ever want to experience Slog Overflow, as it can really cause you to dislike a job that you’ve spent years enjoying. If you feel like your world is moving in this direction, TAKE ACTION NOW!
Here are some thoughts on how to fix these problems or prevent them from happening in the future:
- Always plan ahead – Put together (at the least) a rough outline of the work you’re going to do. Be sure that it includes pre- and post-task steps so that the whole process is considered. If you have more than one person on your team, ask them to go over your outline and see if there’s anything you might have forgotten.
- Embrace feedback and criticisms – We’re all in this together! While some sysadmins may not have the best people skills, they usually have good ideas and aren’t afraid to share them. Take what is useful, try not to be offended. Nobody’s perfect!
- Emphasize quality and demand it from everyone – We need to fight the urge to become complacent and make sure we do things right the first time. That sometimes means you should stop yourself mid-task and make sure that what you’re doing is the best choice in that situation. Demand a higher level of awesome from the people you work with, too. It only takes one hole for the boat to sink.
- Teach young Systems Administrators to be thorough – Lead by example. Junior sysadmins absolutely pick up behaviors from senior team members. If your seniors are procrastinators and always do the easy fix eschewing the right-but-harder choice, you’re setting a ridiculously horrible example for the young guys who need proper mentoring.
What are your experiences with these problems? Feel free to comment below; I love feedback.
Like any good family, when we saw that our older sibling had something we wanted what did we do? Well, we whined and complained until we got it!
Stack Overflow recently finished their 2011 survey, and as soon as the results where announced, Server Fault being the proper younger sibling that it is wanted one too.
So I call all of you Server Fault users to come and take our survey, you wanted it and now you’ve got it! (oh and you really should get your sysadmin friends to take it too – the more data the better!)
Just like the Stack Overflow survey we will be releasing the data right when the survey ends!
This topic was suggested by one of our users, Bart Silverstrim. He was curious about how companies could maintain an environment where maintenance downtimes would not actually affect customer experience. Think of large sites like Google, Facebook, CNN, pretty much any site in the quantcast or alexa top 100. These sites maintain extremely good uptime numbers, but how do they do it?
It’s no secret that sites like this employ a lot of servers to handle content delivery. There are not only load balanced server clusters but also CDNs and caching proxies that help mitigate some of the load on the environment. Eventually, though, all of these machines are going to need maintenance at some point. How does one do this and not affect perceived uptime from the users? This question is actually rather complex because it depends on multiple ecosystems to properly execute.
Monolithic Deployment vs. Continuous Deployment
For many years, the method of software development followed a pretty static and time consuming process:
- Define scope for this release
- Program features for the release
- Debug, if errors, back to #2
- QA (generally someone testing who ISNT the developer), if errors, back to #2
Now, I know a lot of shops that would laugh at this list and claim that they do only #2 and #3, then shove it up to the servers. This can very well be the case, but I’d hazard a guess that those companies have never gotten close to scraping the alexa top 100, nor would I believe with a straight face if they told me they had uptimes in excess of 99%. There is another way, though: Continuous Deployment.
Simply put, Continuous Deployment is when instead of having large, multi-bug/multi-feature releases, you instead implement many, many small changes continuously throughout the product lifetime. At first glance this method might sound dangerous to many sysadmins who like to plan a task to an atomic point algorithmically, but it’s actually extremely safe if you follow some standard methodologies, namely having an environment where you can test changes (for Stack Exchange, this is meta.stackoverflow.com) and use a build proctor like CruiseControl or TeamCity to facilitate the build process.
One of the main benefits of Continous Deployment is the fact that small changes are easy to deploy and usually just as easy to revert should problems arise. It becomes significantly harder to revert a deployment if there is a large corpus of changes contained in the update.
Code it like you’ll need to change it
One thing that new programmers often fall prey to is the laziness of not properly architecting their application. I am sure that every programmer remembers their earlier code shenanigans when they’ve needed to update something they wrote earlier in their career only to find that it was fraught with all sorts of freshman mistakes. To put it simply: you need to write your code to be portable, easily understood, and utilizing the best practices available for that language. If the language is OO, this means obeying standards like MVC, creating class interfaces and making the app as dynamic as possible. The reason should be clear: code that’s easier to interface with is easier to change and update, and it generally means that bugs introduced in most modules should not have global impact to the site.
Load Balancing is Good
Load Balancing is paramount to a seamless customer experience. It allows you to do some pretty cool tricks, but only so long as your web application is coded to properly handle a load balanced environment. The main thing one needs to think about when coding an app in a load balanced environment is that there’s no guarantee that request ‘n+1′ is going to land on the same server as request ‘n’, so you need to handle sessions in a centralized/db manner so that the cookies in the browser link you up to the proper session regardless of what server you hit. This does NOT mean, however, that you should disable persistance or affinity in your load balancer! There are benefits to keeping a session on the same server with regards to file caching and the like; we just want to make sure the app is ready should you want to take one of the servers down for maintenance.
Once properly load balanced, you gain several levels of win. First off, your app will be much more performant if there are more servers available to handle the load. Secondly, if you have multiple servers available, you can bring one of them offline and your users should never know the difference. A side benefit of Load Balancing is the peace of mind you’ll get knowing that if a box eats itself alive at 3am, your site will survive without you needing to fire up the laptop or, heaven forbid, head into the office at an extremely early hour.
How Stack Exchange does it
It may become clear after reading this that what I’m talking about isn’t necessarily how Stack Exchange works. People who visit us often know that we do have site downtimes for various reasons. For those curious, the below section is how we handle our deployment and development process.
There is one big place where the Stack Exchange Core Q&A Services are vulnerable, and this is at the database. Currently we employ SQL 2008 R2 database servers (currently SP1 as of this writing), with the primary server constantly replicating to the backup server via transaction log shipping. Those familiar with transaction log shipping will know that employing this method is really only usable in an active/”hot passive” mode. One of the downsides of using SQL Server is the lack of solid high availability given our transaction load. Simply put, both the asynchronous and synchronous methods of active/active can’t keep up with the sheer volume of transactions we throw at it. We’re hoping that when Denali comes out this spring, the HA features will improve to the point where we can be fully active/active.
The reason we incur downtimes for upgrades these days has to do partially with ease of execution and cleanliness; because we use transaction log shipping, if we wanted to go active on the backup node we’d have to break the replication and convert the backup server to the master server, then re-setup replication. Would we do this for a couple windows patches? Absolutely not. It’s a nontrivial amount of work that can incur human error and is unnecessary. We reserve this process for when we do have a major database emergency. We’re hoping that SQL 2012′s enhanced failover capabilities will permit us to enter read-only mode for brief maintenance windows, but this will need serious testing first, once Denali (2012) is released.
Development wise, this is the continuous deployment procedure that Stack Exchange uses:
- A developer will get the latest HEAD from the mercurial repository.
- That developer makes a change, then commits that change back to the Mercurial repository.
- TeamCity queries Mercurial (every 60 seconds) to check for new changes. If a change is found, TeamCity builds it immediately in the development testing environment.
- Once the developers have given the change a test in development, the developer then deploys the change to meta.stackoverflow.com.
- If the dev is having a good day, meta.so’s users won’t report any bugs and after a period of time the dev will push the code to everything except stackoverflow.
- If the change stands up to rigorous testing on the other approximately 80 sites in our network, the change will be pushed to stackoverflow, aka “the fire hose.”
One might be curious as to why we wait till the last minute to push code changes to stackoverflow. The reason behind this has to do with the fact that stackoverflow gets several orders of magnitude more traffic than the rest of Stack Exchange combined. A case study in this procedure: when we deployed ProtoBuf v2, the change worked great everywhere else, but as soon as the change was applied to stackoverflow.com, a “cold start” bug seriously degraded users’ experience.
An important thing to note as well is that a great deal of the sites’ code changes are toggleable by configuration, so if a problem is found it can be reversed and mitigated much, much faster than it would take a developer to debug and fix the problem. Employing this method where possible in your applications will be helpful for many reasons.
Uptime! Fsck Yeah!
For most of our readership, the problems described above might not apply to you. You may not have quite as many hits as Stack Exchange, or you might have less need for high availability and uptime. This isn’t to say that the above doesn’t apply; architecting for uptime should be in every developer’s best interests and best practices.
As always, your questions and comments are most welcome, feel free to post below.
If you are not familiar with the concept, technical debt is essentially the idea that you build and program things quickly, skipping the niceties in order to ship, and then fix it later. By putting things off you build up debt that needs to be paid down later. One of the places this most commonly shows itself is in performance.
It works like this. Developers make features because the business and users want features. Performance is hard, and the benefits of good performance are not usually as obvious or concrete as the benefits of new features. Therefore, nobody really pays attention to performance or it is intentionally skipped until it gets so bad that people consciously notice it. Then the developers need to do a “feature freeze” and fix things until performance is at least “okay.” again. If you don’t mind the cliche, the feature freeze is the “Rinse.”, and then it all starts over again — “Repeat.” This is the cycle of technical debt.
At Stack Exchange I saw this happen, the developers had to stop working on features and fix performance because it got the point where we were getting timeouts. However, here is where things get interesting: After that, it never happened again.
“Impossible!” No, it is not impossible. In reality, of course there are still things that slip by, but the overall macro cycle of technical debt, when it comes to performance, is avoidable. And if you order my VHS series for 19.95, I will tell you how.
In all seriousness, even if there is no one recipe, from my viewpoint Stack Exchange escaped the cycle through culture, and making the right performance investments. The culture that lead to this consists of:
- Placing a value on performance: “Performance is a feature”
- Well integrated development and operations
- A sense of craftsmanship when it comes to performance
Good performance makes a system enjoyable to use, everyone has to believe this idea. When development and operations are well integrated the teams empower each other, and since performance takes both programming and systems knowledge this is needed. Lastly, if good performance is an aspect of good craftsmanship, it becomes a source of pride.
These cultural aspects at Stack Exchange and the performance investments made enforce each other. I don’t think we could have one without the other. But if there is a secret sauce, it feels to me like it is the performance investments we have made. These investments follow a development pattern that results in instant feedback when it comes to performance:
The 3 Step Process to Good Performance Investments:
Step 1: Collect your data in a queryable way
I can’t emphasize enough how important this initial step is. Your performance data such as logs and system data (i.e. CPU/Memory/Network etc) needs to be in a format that can easily be queryed, extracted, aggregated, and molded in a way that leads to discovery. We use SQL Server for our logs and system data. It doesn’t have to be SQL, but I think that rrd, the common storage format used by systems like Cacti, although good for displaying time series graphs does not fill this requirement due to the difficulty of extracting data.
Step 2: Discover the Important Metrics
Once you have the data in a queryable format, you can then explore that data and discover what the important metrics are. Once we started capturing our web logs in SQL we were able to add custom headers that tell us things like which route is being hit, and measure performance grouped by route. If your data isn’t queryable the discovery process has too much friction.
Step 3: Automate and Integrate the Important Data
Once you have found the important data by exploring it with various queries, those queries should be automated and integrated into your application. Then with every build (rapid integration or frequent building helps) you get instant feedback. At Stack Exchange we have a dashboard that includes graphs from log data, system data, profiling results, and exceptions. We can explore our web logs with a data explorer instance. Also, some of this such as our profiler results are part of every page load.
This process leads to an instantaneous and effortless return of performance information. This eliminates the friction around discovering how your performance is changing. With this information readily available and in your face, it enables a culture where keeping up with performance becomes an aspect of good craftsmanship.
These tools we have created are performance investments. Investments are the opposite of debt. Investments give returns where as debt has interest. When you make these sort of performance investments the cycle of debt is broken and you start collecting the returns. For the most part, people in this world are either collecting returns or paying debt — and collecting returns feels damn good.
At some point in your career as a Systems Administrator or other Person-Of-Responsibility-in-IT, you will find yourself stuck in the unfortunate position of cleaning up a mess that was totally preventable, if you had known the signs beforehand that the problem was imminent. This fact is 100% assurable, as I have yet to meet a seasoned SysAdmin who didn’t have a war story similar to “man, if I was only monitoring disk space on server X…”
Monitoring is an extremely important tool in your arsenal of preventative measures. Monitoring is important for a great deal of different reasons:
- Monitoring allows you to send alerts if certain conditions are met,
- It allows you to visualize trends in data,
- Provides a method of assurance to the customer that their consumed services are guarded,
- Allows you to do internal benchmarking for when you need to come up with budget/spend numbers.
There are many different products in the monitoring sphere. Some are extremely expensive and meant only for enterprise use and there are many that are open source and therefore free to use. My personal favorite is Nagios, though it does have some shortcomings that I will touch on later. Most monitoring systems follow the same basic recipe: You configure hosts, which in turn have services or metrics you want to monitor. The monitoring system will “optionally” alert you if you configure it to do so. Most monitoring systems have a method of keeping historical data and graphing it. This is not only a great way for you to look at pretty graphs; the management staff will get excited seeing information visualized in a way they’re used to seeing.
There are several methods of monitoring. The most basic and least useful method alone is a simple ping test. Products that provide this feature give you a simple up/down alert if there’s an outage, but honestly, the users breaking down your door will be a more effective alert. Most monitoring systems will give you the ability to not only run ping tests, but also have checks that incorporate SNMP (Simple Network Monitoring Protocol) statistics. This is better than a simple ping test, but in my opinion still short of the complete picture I’d like to see. SNMP does have its benefits, though: since it’s been around practically since the beginning of time, lots of equipment supports it out-of-the-box. It’s the primary method one uses to gather statistics about your routers and switch interfaces, including drops/discards and packet saturation rates.
Going beyond simple ping and snmp monitoring, many monitoring applications allow you to have custom checks that give you metrics for items SNMP misses.
For instance, the nagios plugin exchange provides a plethora of check-metrics that other users have written with varied themes from temperature probe
monitoring to advanced Microsoft SQL statistics checking. In particular, one Nagios addon that I cannot live without is the “nagios-wsc” project, which you
install on a windows IIS server and it in turn provides the ability for Nagios to check windows statistics over WMI. Being able to do this vastly
improves the metrics you can extract from windows servers.
At the time of this writing, I’m not sure if a similar interface for PowerShell exists or is
in the works, but that would be a “must-have” addon, as Microsoft has said that they’re moving away from WMI in favor of PowerShell, at least as far as Exchange
is concerned. (As commenter Jim Butts points out, I don’t have citation for this and so I’m going to strike it from the post, though I swear I remember reading at one point that Microsoft intended to replace WMI with PowerShell. This might have only been related to the Exchange family of products, though, so don’t take it as gospel. Also worth noting, as another commenter explained, WMI and PowerShell are two different technologies meant to do two different things. WMI is an instrumentation interface, whereas PowerShell is a scripting language. It just so happens that you can get some information with PowerShell that you cannot easily get through the WMI interface.)
One of the major pieces of any monitoring environment is the ability to alert administrators of an impending problem. Many admins default to e-mail for this, but not many people realize that most mobile phones are fantastic SMS modems. Find a prepaid model that lets you send SMS’s from a serial/usb connection via AT commands, and now you have not only an out-of-band notification method, but you’ve saved yourself a bunch of money on specialist hardware. I’ve also heard of some people using Asterisk to do voice-dial alerts; when the alert hits the system, it Text-to-Speech’s the alert and then plays the audio over a telephone call to the remote party. Pretty slick and high tech, but in my opinion that’s a rather big system to rely on for monitoring. Generally, simple methods of alerting, with less moving parts, makes for a more stable and trustworthy alerting platform.
A helpful part of many monitoring systems is being able to group hosts and services into logical containers. This ability lets you not only monitor a whole logical service from one view, but also allows you to quick-add new servers to a group and immediately have that server’s checks already applied to it by virtue of being a member of the host or service group. If your monitoring system supports grouping and you are not using it “you are doing it wrong.”
Do you need a monitoring environment? Yes. There is no other answer to this question. If you have even a single server in your environment, monitoring it will provide a treasure trove of information about the system. The only question is, how much do I have to monitor? This depends a lot on your customer SLAs and the expectactions of uptime. As the uptime target grows and the margin for error shrinks, you will need to squeeze more and more information out of your environment to maximize the “magic twilight” between a server showing symptoms of impending troubles and “THE SYSTEM IS DOWN.”
Having a lot of stuff monitored also helps with correlation and causation. For instance, you might have a website error showing up on one of your servers, and you start diagnosing that error. Thirty minutes later, you come to find the problem was that the SQL server is bogged down and replying to queries too slowly. If you were just monitoring the web server, you just lost thirty minutes. If you were monitoring both the SQL and the web server, you would have a greater chance of knowing that the problem lay with SQL, not with the web server. All of this data can lead to a condition I call “data addiction,” and it’s a condition that I will attest is pervasive at Stack Exchange. Many of our developers rely heavily on our monitoring data to give them metrics into how the sites are operating.
One thing that needs to be considered when you setup monitoring is the “Who Watches The Watcher” paradox. Simply put, if you become reliant on your monitoring system, you want to trust that the monitoring system is active. There are a few ways to solve this. First off, if your organization has multiple sites, setup a monitoring server at each site and have the monitoring servers monitor each other as well as their other systems. If you have only a single site, then you should probably consider getting a simpler monitoring system to monitor your monitoring system. You’ll never be able to have 100% faith that your monitoring system is foolproof, it’s tough to rely on software that was written by human hands to be 100% failure free, all the time. Regularly auditing the monitoring environment is the best way to keep your faith in the system.
In closing, I’d like to reiterate that even if you feel you don’t need a monitoring system, I’m pretty sure you would still benefit from one. Start small if this is your first time; if you run into issues, sites like ServerFault are a great resource to get good answers. Over time, I think you’ll grow to enjoy having the peace of mind that comes from knowing what your network is doing without having to spend additional time manually collecting statistics on your own.
Welcome back to my series on WiFi. In Part 1 of the series, I began with some basics of RF and explained some differences about antennas. It should be apparent at this point that there is a science behind this activity, and I’ll take this moment to warn you thoroughly before we move on: These posts are a good way for you to become familiar with WiFi and should provide you with some solid knowledge to help improve your WiFi coverage. However, this brief education is not a replacement for having an actual RF engineer do a site survey of your environment! If you have a “must work right the first time” environment, and you’re reading this because you’re the decision-maker and don’t have the slightest hint about what all this is about, Get An Expert. They do this all day long. It’s money well spent.
If you do use these techniques below, Your Mileage May Vary. It’s also important to note that if you go to all this work, setup your access points, then your neighbor goes and installs his AP right next to yours on the same channel, then you’re going to be stuck re-doing these activities all over again. WiFi isn’t a static situation; as people get more WiFi-connected devices, the playing field changes, and it will change on you, I guarantee it.
Understanding RF Interference and What It Does to WiFi
You hear people joke about microwave ovens interfering with WiFi equipment pretty often. Most people laugh it off as an urban legend. It’s not. Below, I have included some RF spectrograms for your entertainment. If you haven’t seen images like this before, they are a visualization of signal frequency and intensity over time. Past-to-present is a top-to-bottom relationship, and the colors are a heatmap (with red being a strong signal.) As you look at both types of graphs, the channels start from 1 at the left hand side of the graph, and go up through 12 in the right hand side. NOTE: Quick Shout-Out to the guys at metageek.net for creating the awesome Wi-Spy and accompanying Chanalyzer Pro software. We paid full price for the DBx bundle (Comes with the Wi-Spy DBx and Chanalyzer Pro) and I definitely feel like it was money well spent. Check them out if you want to do these types of visualizations yourself.
These images show what the wireless signal looks like in my suburban home. Not a lot of interference in this visualization, you can see my home Cisco Aironet 1240 AP humming along happily as visualized by the wavy lines in the waterfall spectrogram, above. In the lower graph, we see signal strength (Amplitude) measured by frequency.
Let’s shake things up, and show what happens when you fire up a microwave oven:
Look out, here comes that microwave burrito exploding all over your RF Spectrum! For about 30 seconds, I nuked a mug of water and this was the result. You can see through the swamp of RF that the access point does its best to compensate for the signal interference, but that’s a pretty strong blast of RFI.
Do you have a baby monitor at home? Is it on 2.4ghz? Ready to see what it’s doing to your wireless signal?
These two charts were captures I took from my friend’s house (incidentally, the gentleman who I mentioned in the previous post – he has a penchant for wifi problems.) I was over his house and ran some traces to get a visualization of his wireless conditions in preparation for installing a new wireless router. I asked him if he noticed the WiFi being slower at night and he’d mentioned that it did indeed seem to be more problematic at night. Welcome to the wonderful world of random equipment in your home causing issues with your wifi. In the above trace, you can see the telltale wavy lines of the access point, trying to power its way through the interference. The graph below has just the slighest hint of bell curve, which is where his AP was situated in the RF Spectrum. I believe in this case his AP was on channel 3. Needless to say, we popped his new wifi router on channel 11, which is quiet in these graphs.
One final graph to show. If you scroll back up to the initial image I showed of my suburban home, this will give you an idea of what your general household’s 2.4gHz spectrum might look like. Now, compare that image to this:
This, my friends, is what the 2.4gHz spectrum looks from the Stack Exchange offices. We’re located down by Wall Street, on the 26th Floor of One Exchange Plaza. Our scenic vantage point does come with a cost! These spectrograms show just how much RF interference we are subject to at this location. Astute readers may notice the timescale difference on the graphic, but I assure you that the 30 second view is just as nasty.
What can we take away from these charts? One could safely summarize this entire section as “Location and the gadgets in your home both play a significant role in how your WiFi might perform.”
Mapping Your Wireless Landscape
It’s worth noting that even though the above charts were taken using a very expensive measuring tool, your laptop’s WiFi card is a potent ally in your quest to improve your coverage. For the next section, I am assuming that you firstly are running Windows and have downloaded and installed both Vistumbler as well as Microsoft SQL express. I am aware that a strong number of our readership are Linux based, and there is also a big Mac contingent. I’ll unabashedly say that the steps I’m following and the software choices I made were purely for my convenience, but I hope that I explain the process in easy enough terms so that the tinkerers out there can take the wheat from the chaff, so to speak.
Step 1 – Take some measurements!
Vistumbler is a wardriving utility that, when attached to a gps, can help you map where there are wireless access points in range of your device. We’re going to borrow it for a more sedentary purpose. Fire up Vistumbler, set your laptop in the areas where you want to consume your WiFi signal, and then start the scan/capture process. Leave the laptop there for at least 30-60 minutes, as we want a whole lot of datapoints to work with. It will keep track of every time it hears of an access point and record the relative strength of the signal. Once the time is up, you can either run the detailed export to CSV now, or “Exit (Save DB)” and come back to export the file later. NOTE: If you’re in a tight urban region like we are at Stack Exchange, leaving Vistumbler up for 30+ minutes will result in a tremendous amount of data! It’s wise to have a very powerful PC to handle the vistumbler export process, or do seperate scans and aggregate the data together in a later step.
Step 2 – Massage the data!
Sadly, Vistumbler’s export to CSV does leave a bit to be desired with its field quoting. We’re going to open the csv in Excel, since it seems to be especially forgiving. Once we’ve opened it in Excel, we’ll do the following steps:
- Make a new column for Location. Populate the column with a location name. You’ll want this when you’re querying SQL later on.
- Save the file as an Excel spreadsheet.
- Fire up SQL Server Management Studio.
- Create a new database if you don’t already have a good scratch database, then right-click the db and select tasks->import data.
- Using “Microsoft Excel” as a datasource, submit your new excel file as the source data. HINT: If you’re getting an error about unable to find a particular OLE provider, and you’re on 2010 like me, with 64bit windows, you will likely need this link to load said provider.
- Select the destination database; elementary stuff here. From this point on, “Just Keep Hitting Next,” except for the prompt where you specify the destination table name. I strongly advise you to change that name to something easier to type rather than the default date/time string. Finally, Finish to start the import job. This may take a while, so don’t be afraid if it seems like it’s taking too long.
- Repeat these steps for each file of data. Be sure to specify the same table name for each import, it will append to the database.
Step 3 – Analyze!
With this complete, congratulations! You now have data in a sql server that you can use to leverage the power of SQL to get some statistics from. I’ll admit I’m a SQL neophyte — I can do some joins and “GROUP BY”s but I’m sure others could tease a lot more information out of it than I have. Here are some basic queries for your dataporn pleasure:
Get a sorted list of strongest access points across all locations:
select LOCATION, SSID, AVG(SIGNAL) AS AVGSIGNAL FROM [dbo].[wifilog] GROUP BY LOCATION,SSID ORDER BY AVGSIGNAL DESC
In the above query, we see that in our sysadmin office, as well as somewhere in our office (for shame, I forget where I took the trace!) the strongest signals are SO-GUEST (our current guest wireless AP) and ROVIO (for our cute little mobile webcam.) We’ve also got a couple shadow AP’s (as specified by NULL) followed up by some other AP’s that I’m not sure who or where they are. Suffice to say, our current AP is pretty strong in these two locations.
Find the average signal strength of all APs at a particular location:
select SSID, AVG(SIGNAL) AS AVGSIGNAL FROM [dbo].[wifilog] WHERE LOCATION='sysadmin' GROUP BY SSID ORDER BY AVGSIGNAL DESC
Similar to the first query, you can drill down by a particular location and see the top AP’s seen at that location. This is useful, but what we’re really looking for is the least used channels at a certain location.
Get a channel utilization chart
select LOCATION,CHANNEL,SSID,AVG(SIGNAL) AS AVGSIGNAL FROM [dbo].[wifilog] GROUP BY LOCATION,CHANNEL,SSID ORDER BY AVGSIGNAL DESC
The above gives us some pretty useful information. We can see here that channels 8 and 11 have several entries, and only one device in range of our scans is on channel 6. Using some critical thinking, it’d indicate that channel 3 might be a good choice should we want to add a new AP to this environment. Lets massage the query a bit to see if that’s confirmed by our other data:
select LOCATION,CHANNEL,SSID,AVG(SIGNAL) AS AVGSIGNAL FROM [dbo].[wifilog] WHERE CHANNEL in (1,2,3,4,5,6) GROUP BY LOCATION,CHANNEL,SSID ORDER BY AVGSIGNAL DESC
At first blush, one might be enthusiastic about channel 3 given the fact it’s not used as the main carrier frequency in any of our entries. Be careful, though, for WiFi channels have some pretty strong overlap:
As this graphic shows, each wifi “Channel” is merely just a 5MHz swatch of the 2.4ghz ISM band. WiFi signals have a 22Mhz bandwidth, so realistically there’s only 3 channels one can use in an environment without any fear of interference or overlap. Because of this, one needs to take into account not only the channel but also the signal strength of potentially interfering access points.
In our case at Stack Exchange, there are just so many APs utilizing so many channels that we’ve ultimately decided to go with a Cisco controller-based access point layout, which will dynamically change channels based on signal conditions in realtime. For those of us at home, this is way too expensive of an option for most. Sadly, we’ll just have to take these data queries and give it our best shot.
I hope this blog series helps you a bit with your next WiFi installation. In summary:
- Antenna choice matters when you’re trying to cut through interference or travel long distances.
- Most residential building materials will not diffuse wireless signals to an appreciable amount unless you’re talking about very far distances, (i.e. trying to use your laptop on the third floor at the far side of your house when the AP is in the basement, for instance.)
- Be aware of electronics in your home that might share the 2.4GHz radio spectrum; they can seriously affect your wireless transfer rate and signal strength.
- Apps like Vistumbler can catalog used channels in your environment and you can then use this data to find a quiet spot in the spectrum.
As always, I welcome your comments and criticisms, below. Also, feel free to share any specific SQL queries you used that might help glean even more information from the datasets you’ve gathered!