Shane Madden at the Denver StackExchange officeThe ServerFault Systems Administration team continues its growth with the addition of sysadmin Shane Madden.  Shane lives in Denver and will work out of the Denver office on days he wants lunch.

You may know Shane already from his contributions in the open source world.  He’s an avid Puppet programmer and Python coder.  Shane’s very active on Server Fault… he has over 69k reputation points which makes him the 7th highest rep user there!

Shane’s hobbies include skiing, board games, video games and hockey.  He has many pets including a mantis shrimp.

Please join me in welcoming Shane to the team!

Site reliability engineers, in the most general sense, are charged with a clear mission: efficiently keep the sites reliable. Reliability can be broken down into two main facets: availability and performance. This is about where it stops being straightforward and everything becomes nuanced. This is because you have to start defining what availability and performance means for your systems (which is generally driven by the mission of your organization and how your systems fit into that). Even more complexity comes into play when you consider all the activities an SRE team engages in to achieve these things. For example: configuration management, capacity planning, restores, fault tolerance, and security to name some of them.

How you define availability and performance in your organization is a topic worthy of its own set of posts; and the details of all the activities an SRE team participates could fill a library. An SRE team needs to start somewhere and have a strategy to tackle all of this. There is no one answer, but achieving a high level of observability needs to be a key strategic component for any SRE team.

Observability is the Foundation

Observability is the degree and facility in which your team can gain insight into the behavior of your systems. It is worth noting that the scope of your systems is likely quite broad; it includes the obvious things like your applications and hosts, but also includes things like processes, workflows, and team dynamics. Having insight in your systems means:

  • Questions operators have about their systems can be quantifiably answered with minimal effort
  • Operators have rich mental models of how their systems function

When you have to decide something you can either guess or use “the science.” Without a set of systems for observability in place you will end up guessing (not the educated kind) or be terribly inefficient. A good understanding of how systems work is what allows operators to be effective and avoid disastrous mistakes: observability can drive that.

Decision Making and Incident Preparedness

Observability is key to the strategy for an SRE team because it informs and impacts nearly every other activity that team engages in. I’ve written about the OODA loop before which stands for Observe, Orient, Decide, Act (You can think of Orient as “Analyze.”) It is a military strategy that suggests you can be successful when you can rapidly and successfully iterate through this loop quickly. It is also a tool that is useful for thinking about site reliability operations as well.

OODA is carried out at both the macro and micro levels (planning and incidents) by SRE teams. As an example, we can imagine what making system design decisions as a team is like without good observability (and since we have likely all been there, you can probably just remember.) The observation phase will be based on people’s memory and is frequently skipped. Orienting or analyzing that information as a group will have conflicts because people don’t agree on what the facts are. This can result in arguments about the person’s recollection of the facts instead of the issue at hand. Decisions end up being prolonged and half hearted because of the uncertainty of their basis. Lastly, action will be hindered because a strong consensus hasn’t been reached because people don’t trust the baseless decision. Even worse, people question if this is even the system they should be working on at all.

Many have also probably been through outages when observability is lacking. Lots of time is lost trying to figure out what is even going on. Orienting is difficult because operators lack the internal model of the system that observability provides over time. As a result of these things decisions and actions are chaotic. Or more simply put, it’s amateur hour.

In contrast, the picture is entirely different with a solid foundation in observability because everything becomes data informed. This is different from “data driven” because you can trust people’s intuition. Due to good observability they have developed keen instincts about systems over time. When it comes to system design decisions you are in a much better position because chances are you are designing the right thing in the first place. Team members will bring their observations to the discussion. If there are questions about the facts, instead of arguing then you can just look them up. Decisions will be made with more confidence and faster because they are based on evidence. Lastly, action will have more consensus behind it, even if people didn’t agree they at least know the choice was based on something.

You never know what the next incident will be, but if you have good observability then your operators will have a deeper understanding of the system and will be far more prepared for the unknown.

Other Benefits

Observability positions a team to do more capacity planning by enabling them to see constrained resources and forecast growth. This can help reduce the vicious cycle of fire fighting that many SRE teams are locked into.

Since observability leads to insight, team members are learning more about their systems which generally is a common source of fulfillment for engineering types.

Convinced? 5 Steps to Achieving Good Observability:

In order to achieve good observability an SRE team (often in conduction with the rest of the organization) needs to do the following steps.

  1. Instrument your systems by publishing metrics and events
  2. Gather those metrics and events in a queryable data store(s)
  3. Make that data readily accessible
  4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior
  5. Establish the resources to drill down into abnormal or out of bounds behavior

Each of these steps largely depends on the previous step to be successful.

1. Instrument your Systems

Brainstorm what key and useful metrics exist for your system. Make those metrics easily accessible (i.e. standard APIs like json via REST or by providing a destination to push to) and document what they are and what the implications of those metrics are. This largely falls on the developers of systems, and DevOps culture can go a long way encourage application developers to empower the operations side of things by doing this. At the highest level you can break metrics and events into two categories:

  1. Objective Oriented: These metrics reflect the mission of your organization. For example they include client facing measurements like response time, availability, error codes, items sold, number of users, number of active users and rate of content created.
  2. Diagnostic Oriented: These measure aspects of the system that allow you to achieve your objects. These include system measures such as OS, network, hardware, middleware, cluster, and application metrics. These also include response time and availability metrics but they measure components and parts of the pipeline that contribute to your objectives.

Good Metrics also tend to have these properties:

  • High Resolution: “High” is qualitative, but a higher frequency of data collection means you will have more insight into the shape of your data (i.e. is it bursty)
  • Lossless: This means that there isn’t missing information from your metric. This can often be achieved by publishing counters instead of rates and letting the display side of things calculate a rate from that information. Also not pre-aggregating things into averages can be useful (or if you are going to do that also aggregate the data into multiple percentiles)
  • Specific: More specific metrics can often be more useful to understanding a system and drilling down into a problem. For example, with something like CPU utilization it is better to report something like %user, %system CPU time breakdowns and let something later in the pipeline aggregate them.

It is also worth making a point to instrument your own internal “meta” systems such as bug tracking and documentation.

2. Gather those metrics in a queryable data store(s)

This is a key intermediate step to making this data accessible. Data generally needs to be stored over time in order to give it context (although the time of each datapoint isn’t always important for things like histograms when it is processed later). Having this step enables things like:

  • Building dashboards
  • Enabling capacity planning
  • Allowing operators to explore the data and learn
  • Allowing people to invent cool stuff you didn’t anticipate

As a rule of thumb, less data stores are better because it makes it easier to work with the data (although specialized databases for things like time series might be worth the tradeoff because of features and scalability.) For time series data in particular, a couple of useful qualities are:

  • Scalability: This enables one to collect a lot of metrics, at high resolution, and high retention
  • Aggregation: This encourages a shift from host/process oriented views to cluster and service oriented views

3. Make that data Readily Accessible

If there is a lot of friction to view the data then people won’t have time or energy to do it. This is why it is important to have good dashboards and APIs to allow easy access for your operators. Good dashboards tend to have the following attributes:

  • A fast responsive UI to allow for operators to drill down and explore easily
  • Enables operators to create their own dashboards and graphs
  • Highlight problems

4. Highlight metrics that are, or are trending towards abnormal or out of bounds behavior

Ideally a team ends up collecting a lot of data. This means humans can’t process it all and therefore your systems need to ask for operator attention. Essentially this is alerting. However it is important to understand that alerting doesn’t always mean “emailing”. It can also mean things like publishing something to a dashboard or logging it.

Traditionally alerting has been done on current values, but anomaly detection and forecasting are becoming a reality thanks to some work done at Etsy.

Alert noise / desensitization is a plague in our field, my belief is that future systems will allow for more carefully crafted and adjustable rules to reduce the noise. Keeping this under control is also largely about discipline and remembering that every alert requires action.

5. Establish the resources to drill down into abnormal or out of bounds behavior

The above steps are a gateway to observability. This is because the nature of collecting metrics is resource constrained. You can only collect so much information without noticeably impacting what you are trying to observe. Eventually you are going to need to drill down into problems or explore further why metrics are behaving in a certain way. There are three common activities for this:

  1. Log analysis: Digging into your system logs for information. System logs can also be a powerful source of metrics (especially things like web logs) if you parse them and feed the results into your monitoring systems
  2. Profiling: This the activity of sampling programs to figure out what they are doing – generally at a much higher resolution than collecting metrics (computer time (sub 1ms) instead of human time)
  3. Tracing: Collecting every single thing a system is doing (i.e. strace or DTrace)

Although my path to observability puts an emphasis on collecting metrics and events, this step is also crucial to observability.

Use the science, Luke

If observability is one of the key components of the strategy for your team, then it sets the tone and foundation for everything else. It can create a culture of constant learning as it provides a medium for learning about your systems and proves a source of information for productive analytical arguments. Whatever your strategy is, you need to consider what role observability plays in your team. And remember: Use The Science.

A lot of tools available in IT/Sysadmin/Ops/DevOps are disappointing:

  • They don’t fit your environment. They lack features or our designed for a different sort of environment (i.e cloud vs hardware, Linux vs Windows, distributed vs centralized etc)
  • You can’t interact with them programmatically
  • They cost too much
  • They are not customizable enough, or require too much customization to get off the ground
  • Feel kludgy, unreliable, outdated, or like the programmers were stoned
  • Don’t fit with your company’s culture (i.e. Enterprise vs Agile)

In short a lot of stuff is too expensive, isn’t a good fit, or is simply bad software. This ends up leaving an ops team with two options. They can whine about it, or create their own tools. So at Stack Exchange we build our own DevOps tools.

Status

Nick Craver’s baby, which we just call “Status” is at first glance a monitoring dashboard, but is essentially a collection of tools that filled various needs:

  • An Overview of CPU, Memory, and Network utilization for all our servers as well as a detailed view. Done with responsive and interactive D3 graphs as well as sparklines it helps compensate for Solar Wind’s terrible interface. statussqlscreen2-png
  • SQL Server monitoring. SQL’s built in Clustering views are deeply flawed. If a node loses connectivity, it stops updating remote nodes status, so it could show everything as connected and fine, even if there is no connectivity. We also get to see the most expensive queries, active queries utilizing whoisactive, current connections, and which DBs are on which server
  • HAProxy Monitoring and Administration: With multiple instances of HAProxy we needed a single view instead of HAProxy’s built-in display. Also, this gave us a nice web interface to take servers out of rotation statusdashboardscreen-png
  • Redis: A nice presentation of Redis Info across all instances and all servers. Also a display that shows what is slaved to what in at a quick glance
  • Elastic Search: Health overview of or clusters (as well as index and shard data)
  • A dashboard of all the exceptions generated by our applications

Status is C# / .NET app. It polls data from various sources – sometimes the system directly and other times it gets it from Orion. There is a lot more to status that makes it awesome. The real accomplishment is that status enables us to see the general health of our main infrastructure at a glance.

Web Logging

If you business is creating and running websites, your web logs are gold. We use the logs generated by our load balancer, HAProxy, as our canonical web logs. In their raw text format, web logs are often not that useful (this is particularly true with over 100 million records a day). However we parse and structure our web logs in a few different ways:

realog.ds.stackexchange.com

  • We have C# service that Jarrod Dixon wrote that inserts them into SQL so we can query them. In order to query them we use an instance of Data Explorer, SQL management studio, and also have certain lookups directly from our sites
  • Displaying realtime graphs of various log information with Realog, a system I created with Go, Redis, and NVD3.js so we could view activity live without having to write queries

One of the interesting things we do with our weblogs is to add extra information by adding headers inside the app and striping them from the response at HAProxy. For example, we capture how many Redis and SQL queries were involved in that request and how long they took.

Patch Dashboard

OS updates can be a bit tedious, even more so in a mixed Windows and Linux environment. PartialPatchDashboard Steven Murawski and George Beech created a dashboard that allows us:

  • View the outstanding patches and patch count for both Linux and Windows
  • Trigger updates on either Linux or Windows
  • Schedule time frames for automatic Linux updates

What’s Next

If you want to learn more about these tools and DevOps at Stack Exchange, come see George, Nick, and Steven present “Building for Operations” at Velocity.

Keeping all this stuff to ourselves feels a bit greedy. However, for something open sourced to be very useful it usually needs to be made a bit more generic which takes time. We also want to build a lot more. Our inventory system Racktables lacks an API so we need a new one or a way to extend it. We want to build our own monitoring system (likely on top of OpenTSDB). In order to create more, and open source it we need help. So we are looking a full time developer with ops experience to join our SRE team. So if you are awesome, want to build awesome ops stuff and open source it, come join us!

Keeping Your Cool

Steve Murawski

I’m not talking about data center cooling here…

I was recently listening to The Ship Show podcast titled “Keep calm and PROD on”.  In this podcast, the hosts were discussing whether or not all devs and operations personnel should have production access.

The conversation really hit home with me when it changed from having access to production to how people handle dealing with outages/incidents.  The hosts asked for some feedback on the topic on Twitter, but I have just a few more than 140 characters of thoughts on the topic.

The Scenarios

The hosts outlined a couple of scenarios for discussion.  They didn’t use these exact terms, but it helps me to group them under certain archetypes (it must be all those role playing years catching up with me..).  In no particular order (going from memory and making up the classifications here), we have the “cowboy” response, the “neophyte” response, and the “deer-in-the-headlights” response.  I’ll provide a quick synopsis of these responses.

The “Cowboy” Response

This incident response archetype is very identifiable in many environments.  Picture it, the year is 2010, it’s 2 AM and your cell phone is blowing up with monitoring alerts.  The website is down and everyone is waking up.  You log in to your VPN and start gathering information.  Everything goes dark, and then everything comes back up, seemingly magically fixed.  It comes to light that one of the developers, on his own volition, decided to take drastic actions to restore service.

This response highlights the lack of communication and command and control that is typical of frantic incident response in many IT departments I’ve observed over my years in IT.

The “Neophyte” Response

This incident response archetype is your typical “newbie”.  The neophyte does not have to be new to the field, just new to high pressure outage scenarios.  Taking the scenario above, the neophyte may be taking his first turn at the on-call rotation or this may be the first time a particular problem has cropped up.  The neophyte might not  be comfortable with getting a more senior engineer out of bed, or getting some developers involved.  Another pitfall with the neophyte is that they might not be willing to take charge as others begin to respond to the incident, meaning that the incident response can be confused with no coordinated direction.

The “Deer-In-The-Headlights” Response

The last one I remember the podcast covering was the “deer-in-the-headlights” response, where regardless of a person’s experience level, they just don’t respond well in a crisis.  Many of the downsides of the neophyte are felt here too.. The primary responder may not reach out for help or may not be able to control other responders.  This leads to a fractured response, where people may be working at cross purposes.

What Should We Do?

What I’ve Learned In Previous Careers

Training To Be A Cop

You may not know this about me, but before I got into IT, I was training to be a cop.  I went to school and did all the fun training on how to drive fast, arrest people, and shoot guns.  A big part of that training also included how to respond to medical situations (trauma or illness) or hazardous materials situations.  (Despite what you may think.. watching re-runs of Cops is not adequate training – except as what not to do!)

In that training, we drilled how to respond to emergencies as individuals and as part of a team.  Each course drilled scenarios, but our firearms course drilled scenarios most heavily and over time included skill sets learned in other classes.  Nothing wakes you up to a training scenario like walking into a situation with a vague description of a problem and, as you start to gather information and stabilize the scene, you get drilled in the head by several rounds of simunitions.  Even though you know that you are using training rounds, it shakes you to your core to realize you could be dead in seconds.

Part of the training regimen includes working past the failures and mistakes.  In that first scenario, even though every recruit gets ambushed and shot in the head, our instructors make us follow through with the techniques we’ve drilled in the classroom and on the range.  We are coached to get to cover, return fire, and call for backup.  We can’t dwell on our mistakes (there will be time for that later.. everyone is videotaped for review with the whole class afterwards), we have to follow our training and deal with the problem and stabilize the scene.

Over time, we progressed through a number of other scenarios, some by ourselves, and some with other recruits.  In all cases, we were responsible for communicating status back to our dispatch center, requesting resources as needed, and dealing with any subjects in person that the scenario called for.  We were tested with a variety of actions, all potentially threatening, but each requiring evaluation for how we could respond and always under the watchful eyes of our instructors and the unblinking eye of the video camera.  This training reinforced our more static, isolated drills, allowing us to respond to dynamic situations with a combination of intentional action and reflexive reaction based on our other drills.  Training helps minimize the “deer-in-the-headlights” and the “neophyte” responses and allows people who would experience those responses to fall back on training.

Working In A Public Safety Agency

I also worked as a dispatcher, clerk, and auxiliary officer before and alongside my role as IT guy for a local police department.  Over that time, I observed, interacted with, and sometimes responded with our emergency responders as they dealt with life and death situations.  I observed the growth in how the police and fire agencies learned to respond to incidents together, using the National Incident Management System (NIMS), the emphasis on which grew after 9/11/2001.

Under the NIMS model, the first responding public safety officer is the incident commander.  As the situation develops, the incident commander role can change, based on who’s best suited to deal with the incident.  For example, in the case of a fire, the police officer first on scene will be the incident commander, until a fire department official is set up and ready to take over command.  (In my experience, this is after police officers have saved all the people in the building. ;) ).  Let’s look at another scenario: officers respond to a report of a burglary in progress.  The first officer responding is the incident commander.   As the scene develops and a perimeter is established, the officer in command is responsible for requesting the resources needed and beginning to stabilize the situation.  Next a sergeant, lieutenant, or captain arrives on the scene and takes over the coordination of the perimeter, allowing the officer to focus on his area of responsibility.

NIMS also defines several other key roles for the command staff in an incident, a Public Information Officer (PIO), a Safety Officer, and an Liaison Officer.  The PIO’s responsibility is to keep the stakeholders and public in general informed as to what is going on.  Part of the role is determining what information is helpful to share and what should not be disclosed.  The Safety Officer monitors conditions and ensures the safety of all incident personnel.  The Liaison Officer is responsible for dealing with all the coordinating agencies.  Defining these roles can help deal with the “cowboy” response.  If incident response is structured, the right resources can be directed to a problem and a sustainable fix is more likely an outcome, versus “just get x involved, he fixed it last time.”

How We Should Respond

These two experiences provide some basic thoughts into how we can approach incident response.

Drilling

Just because we are sysadmins, site reliability engineers, devops engineers, etc. doesn’t immediately grant one the intrinsic knowledge and skill necessary to deal with an outage, especially if you are a specialist and the outage deals with technology you are less familiar with.

For operations personnel (developers, dba’s – yes them too – and sysadmins), this is critical.

For someone with deep intuition about their environment, the answers are easier than for newer or more narrowly focused personnel.  Guess what?  The knowledgeable guy isn’t always around when things go ill.

Focused drills, around dealing with one sort of problem or technology, as well as combined drills with multiple components are vital.  Not only do you need to be familiar with the systems you are responsible for, but also everything they interact with, internally and externally.  Quick – what do you do when your CDN stops serving content?  Have you drilled that scenario?  If you haven’t where do you even begin?

This is something we are going to be focusing more on here at Stack Exchange and I’m super excited about that.  We’ve decided that this is a priority for our organization and we’ll be dedicating time to this.

Roles

In the police department where I worked there was a definite command structure, but individual officers had a great deal of latitude to respond to most situations.  The latitude enjoyed by officers in that agency is similar to the latitude I have as a sysadmin on my team.  Certain situations we can just deal with and not need to involve others.  If I need backup (additional resources), I can request those, but if a situation escalates, it’s time to bring in more support.  We don’t have a strict command and control environment;  as we grow our technical staff I think that’ll be more defined.

When it comes to incident response though, the cops and my fellow sysadmins have a bit of a different experience.  Since the officers drill a variety of scenarios and have those drills and training reinforced by continual engagement with an unpredictable public, their escalations from situation to incident are much more fluid, as are their transitions of the incident commander role.  We currently don’t have anything defined like that, though I hope as we start to drill these scenarios more, I’ll be able to lobby effectively for the establishment of at least two of the NIMS roles, the Incident Commander and the Public Information Officer.

In the podcast, one of the themes discussed included how detrimental demands for status updates and presence on conference bridges could be.  By assigning (and training) someone (and at least one alternate) to fill the PIO role, the remainder of the technical staff is freed up to deal with the issue.

In our case, I can envision that as having our PIO designate one chat room  or google hangout as our internal status update location and during the incident he/she’d make regular updates to those areas.  In addition, the PIO would be responsible for updating our status Twitter account and status blog to keep the public informed as needed.

As for the Incident Commander role, we’d need to train all of our on-call personnel, as well  the rest of the technical staff, so everyone is on the same page as to who is in charge and directs the resolution of an incident.  While we don’t have this defined yet, we had a short outage a few weeks back that illustrates how this can work.

  • About 9 AM UTC, while I was dreaming of servers with 4TB of RAM and many multi-core processors, my phone began to blow up with alerts from Pingdom.  I wasn’t the on-call person, but I always monitor for severe external alert failures.
  • I rolled out of bed and ran stumbled down to my office and got online to start investigating.
  • First order of business, check our chat site.. oops! It’s down!  Normally our chat servers are in the opposite data center from where our Q&A sites run, but we are preparing for maintenance in our secondary data center, so it’s running in New York with the rest of our infrastructure.
  • Next up, VPN..  which connects.. that means internet to our data center in NY is still good.
  • I jump in a Google Hangout (fortunately unfortunately for my co-workers,  I look as good as I do right out of bed as I do after getting ready in the morning) that we have set up for our site reliability team.
  • Chat comes back online.  I dropped a note in chat that I’m in the SRE hangout and troubleshooting.
  • I’m soon joined by Geoff Dalgas, one of our Core Q&A developers.
  • He and I discuss the situation and began validating the different bit of our infrastructure.
  • We determined that we were seeing an issue with keepalived on our load balancers.
  • Just then, Tom Limoncelli, another of our SRE team joined us to help with the issue.
  • We also had several more developers pop in and see if they could offer any help.
  • We determined a course of action to remediate the problem and began to implement it.
  • Soon, Stack Overflow (and the rest of the network) was back online and the twitters began to calm themselves.

In this situation, I was first on-scene and acted as incident commander.  If the situation had continued to develop into something more complex involving the load balancers, I might have had to defer to one of our other engineers or Geoff.

Open Line

So, what do you do in your organization?  Do you drill and train for failures?  Do you test your backups?  Do you prepare your operations personnel for how to respond in an incident?

I’ve been getting some great response to my previous post, and I wanted to make a few things very clear.

Clarifications

What I’m not doing -

  1. I’m not advocating or dismissing any particular configuration management tool.
  2. I’m not discounting the tough work done by companies and community projects that have created abstractions on managing disparate systems.

What I am attempting to do -

  1. Highlight the challenges of cross-platform management and application management.
  2. Show one of the efforts in providing a standards based management abstraction.
  3. Offer my thoughts on why I see value in that direction and what challenges I see.

Managing the Operating System vs. Managing Applications

There is definitely some confusion around using CIM to manage the OS vs managing applications.

  1. CIM Classes can be used to model applications as well as OS resources.
  2. Most of the “applications” that are packaged as roles and features in Windows Server expose a CIM management API.
  3. WMI is an implementation of the CIM standard and starting with Server 2012 and the Windows Management Framework V3, CIM is exposed via WSMAN.

Hyper-V, File Shares, Clustering, IIS, and others all offer CIM based management models.  Other applications can expose a CIM management model as well.  As long as the host CIM server (WMI on Windows and OMI or OpenPegasus or ???? on Linux based Operating Systems) is operational, applications can also offer their configuration and status via that channel by creating a provider.  To do this on Windows, there is some documentation to get you started:

To do this with OMI, you can find some documentation and source at https://collaboration.opengroup.org/omi/documents.php.

The Current State – Reprise

But, but, tool {fill in your favorite tooling here} already does THAT!!!

There are a number of tools that valiantly strive to provide cross-platform management.  I mentioned several of them in my last post, but there are a number of others.

Yep, it does.  Until…

things change.  The challenge these tools have is that they have had to implement their abstractions against very different implementations.  The problem there is that these things are not stagnant.  The management APIs can change over time and since there is not a standard description of the API or underlying configuration.

If CIM were the standard API exposing the configuration, the underlying implementation details can change, but configuration management and monitoring tooling don’t have to care about that.  The tool vendors and community projects can focus on other value adds for their particular tooling, rather than being forced to continually update the basics.

The Next Steps

We are still in the early stages of the push for CIM and WSMAN.  We’ll have to see how adoption picks up.  The continuing work around OMI holds promise, but it needs a deployment or integration story for various Linux distros and more public providers for managing components of the Linux OS and attendant applications.

There have been some interesting announcements at TechEd in relation to Windows Server 2012 R2 (watch this video and pay attention around 49 minutes in).  I’m going to talk more about this feature and it’s implications and my plans with it in the very near future.

 

​Configuration management today is mess if you work in a heterogeneous platform.

There is tooling that takes a stab at it, and is getting better (from the *nix world – Puppet, CFEngine, and Chef and from the Windows world – System Center Configuration Manager, Group Policy, among other third party application deployment platforms).  These tools are all well and good, but they fall down when reaching across the OS divide.  Puppet, Chef, and CFEngine (there are others as well, but these are some of the more popular) all have some cross platform support, but it feels unnatural (especially in module or recipe development).

Why is this a mess?

Windows is traditionally described as having an API oriented management model, whereas *nix has a document based management model.

Well, that’s a load of crappy, crap, crap.  What does that actually mean?

It means that the two operating systems offer two different management models.  The two different models have different abstractions and idioms for operating system constructs.  Let’s look at a concrete example, setting a static IP on a network interface (just the rough strokes.. I’m not going to spend too much time on the minutia). As I stated before, Linux uses a document oriented management model, so to configure my network interface, I’ll edit a document or two.

The Linux (Centos) example:

  1. Find the correct interface file under /etc/sysconfig/network-scripts
  2. Open it in your text editor of choice
  3. Edit it to contain your desired settings for the network interface and save the file
  4. If you need to add/modify DNS servers, find /etc/resolv.conf
  5. When done, you can bring your interface online with a command line call to
    ifup eth0

You’ll have something that looks like this for your network configuration file:

DEVICE=eth0
IPADDR=192.168.10.80
NETMASK=255.255.255.0
NETWORK=192.168.10.0
BROADCAST=192.168.10.255
GATEWAY=192.168.10.1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
And something like this for your resolv.conf:
domain serverfault.com
search serverfault.com
nameserver 192.168.10.60
nameserver 192.168.10.61
That wasn’t so bad, and as an added benefit, they are just text files, so I could check them in to a revision control system (Versioning FTW!). Now, let’s look at what we’d need to do on the Windows side.  Since this is a blog for a community of “professional” systems administrators, we are going to dispense with any GUI example for doing this.

The Windows Server (2008 R2) example:

  1. Use WMI to retrieve the network adapter interface index.
  2. Use WMI to retrieve the network adapter configuration by the index.
  3. Set the desired IP address, gateway, and DNS servers and suffix against the WMI object.

You can use the following PowerShell commands to make those changes:

$NetworkAdapter = Get-WMIObject Win32_NetworkAdapter -filter "NetConnectionID = 'Local Area Connection'"
$NetworkAdapterConfiguration = Get-WMIObject Win32_NetworkAdapterConfiguration -filter "InterfaceIndex = $($NetworkAdapter.InterfaceIndex)"
$NetworkAdapterConfiguration.EnableStatic('192.168.10.80','255.255.255.0')
$NetworkAdapterConfiguration.SetGateways('192.168.10.1','1')
$NetworkAdapterConfiguration.SetDNSServerSearchOrder('192.168.10.60','192.168.10.61')
$NetworkAdapterConfiguration.SetDNSSuffixSearchOrder('192.168.10.60')
and

The Windows Server (2012) example:

  1. Set the desired IP address and gateway based on the interface name.
  2. Set the DNS servers and with a few more PowerShell commands.

You can use the following PowerShell commands to make those changes:

$IPAddressParameters = @{
            IPAddress = '192.168.10.80'
            InterfaceAlias = 'Local Area Connection'
            AddressFamily = 'IPv4'
            PrefixLength = 24
            DefaultGateway = '192.168.10.1'
}
Set-NetIPAddress @IPAddressParameters
Set-DNSClientServerAddress -InterfaceAlias 'Local Area Connection' -ServerAddresses '192.168.10.60','192.168.10.61'
Both of these examples are interactive commands, but I could easily save them in a file and place that under version control (and I should).

So what?

The examples don’t look all too different, but they do illustrate the difference between similar operations.  In both  examples, I end up with an artifact, but one is for a one time application of the setting (the Windows side) and the other is the setting storage location (the Centos example).

On the Centos box, we had to edit a file where the configuration was read from.  On the Windows servers, we updated settings via a WMI API (in both cases.. on Server 2012 there are more built in cmdlets, but many of them are thin wrappers over the WMI APIs) and not the actual end storage location.

This is what

Any configuration management tool that works in a cross platform capacity needs to understand these distinctions and check based on OS type which implementation to use when configuring a system.  This means for most configuration types, you’d have a big “IF” block where *nix based OS’s follow this line of processing and Windows based machines follow  the other line of processing.  This can become a maintenance nightmare as OS versions change the API on the Windows side or modify location and or layout of the configuration files on the *nix side.

And it’s even worse…

Now, what happens when you have a model that doesn’t translate across both worlds?

For example, how do manage file permissions?

Posix style permissions (used on most *nix variants) assigns permissions are nowhere as discreet as NTFS file permissions.  In addition on Windows, the file system auditing is also configured via the permissions configuration. In the reverse, on *nix files can be set as executable, where that is handled by file type mappings based on file extensions in Windows. This fragmentation leads to more complex implementations on the side of configuration management software developers or missing feature coverage.  In either case, this is a loss for the sysadmin who maintains cross platform environments.

But what if….

there was a common method of interacting with operating systems, regardless of what was running underneath? What if this method used a common transport (open standard) and communications were defined by an open standard? This is the direction Microsoft is taking with CIM and WS-Management.

CIM (Common Information Model) is a DMTF (Distributed Management Task Force) standard for describing management information for systems, networks, applications, and services.

WS-Management is another DMTF standard for management communication, focused on CIM traffic.

Microsoft has contributed to an open source project hosted by the Open Group called OMI.  OMI is a CIM server that communicates over WS-Management and is implemented to run on *nix based operating systems.

I’m personally interested in where this will go, given Microsoft’s market power (Cisco and Arista are working on incorporating OMI into their network switches).  The idea of a shared management model is very appealing to me, as I work in a cross platform environment.  I’m responsible for our Windows infrastructure, but I have to be able to work with our *nix infrastructure as well.  If I could use one model for interacting with both, that’s a huge win for me and my team.

This wouldn’t eliminate any domain specific knowledge on either OS side, as you’d still need to know what buttons to push and knobs to tweak to get things going and do some deep troubleshooting.  It does, however, make the idea standardizing how various OS components can be accessed, making basic configuration, monitoring, and troubleshooting much easier.

I’m interested because…

this pushes the implementation down to the OS provider (or the CIM provider provider) and gives vendors one target to hit for configuration standards.  In the Microsoft case, they can say “Follow this standard and any Windows system can manage you with minimal effort.”  If other OS’s support CIM and WS-MAN as well, it becomes easy to offer management interfaces there as well.

Obviously this would be huge change to the existing way of doing things for OS and application developers, not to mention systems administrators that are invested in their existing ways of doing things.

I don’t see another good alternative though, as the numbers and variety of systems continue to scale up and “cloud” becomes more of a factor in our environments, yet the number of admins is staying stagnant or being reduced.  Simplifying the management and monitoring surface makes sense in today’s and likely tomorrow’s data center landscape.

It doesn’t solve every problem and vendors can still implement vendor specific extensions (and we know how well that’s worked with SNMP).

 

NOTE – Be sure to check out some clarifications and expanded discussion in my followup post.

The ServerFault Systems Administration team continues its growth with the addition of sysadmin icon Thomas Limoncelli.  You may know Tom from his books, Time Management for System Administrators and The Practice of System and Network Administration, or from his many conference appearances at events like LOPSA-East, the Cascadia IT Conference, and USENIX LISA.  Tom’s also been a ServerFault user, though references to his books outnumber the number of direct answers he’s supplied.

When we saw that Tom was just finishing up his time at Google as we were posting the next ServerFault opening, it was kismet.  Tom is the quintessential systems administrator and a great fit for our team.

I’ve been lucky enough to know Tom for the past couple of years and I am very excited to be able to work with and learn from him (and hopefully teach him a trick or two on our Windows stuff).  His The Practice of System and Network Administration was one of the first resources I had when I got started working as a systems administrator and really got my career headed in the right direction (well, if you consider getting to work with a small crack team of sysadmins and developers on one of the more dynamic environments out there “the right direction”).

Tom is joining Bart and George in working out of our (newly constructed) New York headquarters. Tom hails from New Jersey and frequently attends local LOPSA meetings.

Join me in welcoming Tom to our team!

Hey everyone, I’m one of the newer ServerFault sysadmins and I’ve got the fun job of explaining why we have been moving to Windows Server 2012 just almost as fast as WDS can deploy images.

 

What’s Wrong With Windows Server 2008 R2?

Where do I begin?

2008 R2 was a great operating system in its day.  Automation was becoming a core part of the platform.  PowerShell was shipped in the box, enabled by default, and supported several roles and features.  Many of the kinks were worked out after the Vista/2008 release cycle and people were starting to acclimate to the User Account Control (or figured out how to turn it off).

Server 2008 R2 held up well for a long time, but was starting to show it’s age with our SQL Server cluster.  There are a bunch of hotfixes required to make clustering have a chance of working with SQL Server 2012 AlwaysOn in our environment.  Brent Ozar, in part based on experiences helping us, now recommends not building AlwaysOn Availability Groups on Server 2008 R2.

Additionally, the management story for Windows Server up to this point has been very focused on single machines.  Server Manager in 2008 R2 targets one machine at a time.  Most of the MMCs deal with one machine at a time.  PowerShell remoting was not on be default.  It is definitely work to build a multiple machine management story.

 

So, how does Server 2012 help?

First of all, Windows Server 2012 helps address our major pain point of how SQL Server 2012 Availability Groups interact with Windows Clustering.  Clustering received a lot of attention in Windows Server 2012.  The addition of dynamic quorum helps maintain the cluster state in the case of site-to-site VPN interruptions.  Clustering itself has just become more resilient and offers a wider feature set.

The second (and I think most important feature) that Windows Server 2012 offers is SMB3.  SMB3 fixes many of the performance considerations with access to files stored centrally on a file share.  SMB Multichannel allows SMB clients and servers to leverage multiple network connections, makes SMB transfers resilient to TCP disconnects (by having multiple streams per transfer, even with one NIC), and provides the foundation for scale out file servers, providing a way for multiple servers to share the same connection concurrently, isolating storage transactions for single server failures and allowing connections to scale across multiple servers.  SMB3 provides a number of opportunities to radically change the structure of your application data access and I talked with Richard Campbell about this at length on RunAs Radio.

The third huge win is how Windows Server 2012 supports three levels of GUI-ness.  You have the traditional full UI to support the traditional administrative experience, and since this is the same codebase as Windows 8, you get the Metro/Modern UI start screen as a “bonus”.  The next level of GUI-ness offered by Server 2012 is the minimal shell experience, or what I like to call “Server Core with Training Wheels”.  The minimal shell removes Explorer, Internet Explorer and most GUI apps, but leaves behind Server Manager and the MMCs.  This provides a bit of a security blanket for those admins who aren’t quite ready to give up managing a server locally, but want to break their reliance on the UI and grow up to be SYSTEMS administrators who can manage multiple systems.  By removing explorer.exe and iexplore.exe, you’ve eliminated a huge source of patches, particularly the ones that require a reboot.  Additionally, since there is no Explorer, the RDP experience is a command prompt and Server Manager – discouraging random RDP sessions.  The final level of GUI-ness is full on Server Core.  This is the DEFAULT install option for Windows Server 2012.  Server Core lacks even the minimal GUI tooling provided by the minimal shell configuration.  Server Core, as of Server 2012, is no longer a life choice.  In Server 2008 and Server 2008 R2, when you selected Server Core as the install option, you were stuck in Server Core.  With Server 2012, the GUI bits are features and can be added and removed through the standard feature management tooling.  I talked a bit about this with Richard Campbell on RunAs Radio earlier this year.

Next up in the “pro” column for Windows Server 2012 is PowerShell V3 and the increased cmdlet coverage in the operating system.  PowerShell v3 brings a number of great enhancements to the party like improved performance, resilient remoting sessions, commands for working with the web and REST endpoints, workflow, ISE enhancements and more.  Windows Server 2012 also brought about 2400 commands across the in-box roles and features (up from under 500 in 2008 R2).  This explosion in coverage was due to improvements in the WMI APIs and a new API in PowerShell for being able to take an XML mapping file and generate PowerShell commands from a WMI API.  This got us tons of additional coverage, including the network stack. Since servers really aren’t intended to be managed individually (despite what some sysadmins, product teams, and third parties software vendors believe), the increased PowerShell coverage makes that a much more reasonable task.  If you are interested in further talk about PowerShell V3, you guessed it, there was a RunAs Radio talking about that as well.

The final consideration we had for choosing Server 2012 was in-box NIC teaming.  Server 2012 offers OS level NIC teaming with a variety of teaming options.  Given the problems we’ve faced with third-party NIC teaming (http://blog.serverfault.com/2011/03/04/broadcom-die-mutha/ and some other fun stories), I was gung-ho to get switched over to the native teaming features (which I’ve monitored in production environments for over a year at that point).  Having the network load-balancing and failover (LBFO) teaming included in the OS is a major plus.  Don’t worry, I won’t point you to yet another RunAs Radio show this time.

 

Where are we now?

I’m currently working with our team to continue the roll-out of Server 2012.  Part of that process is updating our installation and configuration management plan.  Nick Craver, one of our developers who likes to help with the sysadmin work and has been featured on this blog, and I have spent a good amount of time tweaking our deployment scripts (server and application deployment) to facilitate the roll out of new servers.  We’ve got our SQL Server and web tier installs fully scripted, to the point where we can go from bare metal to a fully functional web server in about 3 hours (mainly OS install time and running Windows Updates).

We are still working out the rest of our deployment story, looking for integration points with our Puppet configuration management, making the configuration idempotent, and creating artifacts that can be stored in source control to version our environment.  But each week sees more Server 2012 (and PowerShell use…  Pete’s started using a bunch of stuff I wrote and now I have bugs to fix… grrr.. ) in our environment.

 

Where do we want to be?

I would like to see our Windows deployments that are in a production capacity (internally or externally facing) under full configuration management control, with changes pushed via a configuration management infrastructure and configurations enforced regularly.

I would like to have most of our servers running Server Core (doesn’t make much sense to do that for our management stations, but for most servers it works).

Most of all, I would like our server deployment to help serve as a template for a different way of Windows administration, one with rigid standards but quick to respond to changing needs.  I’d like to see our infrastructure as agile as our code base, where changing our OS configuration is as easy as pushing a build is for our development teams.  I’d like to be able to validate infrastructure changes quickly, closing the feedback loop and allowing worry free infrastructure changes.

You all have a front row seat to our environmental changes and I plan to blog extensively about my configuration management efforts in the near future, giving you an opportunity to learn from my mistakes.

 

Moving about 35 servers, something like 2000 pounds of computer hardware 50 blocks, doesn’t seem like that big of a thing. However, in our geek microcosm moving to a new colocation facility was a year long adventure with lessons learned, arguments, designs and redesigns, a hurricane, and many weekends of preparation that somehow resulted in a setup that we are all very proud of. We always want to share our experiences, so this post has some of the lessons we learned, and in the next post have all the technical detail of our new facility.

Lesson 1: Be Prophets

Sometimes you run your systems, and sometimes they run you. If you are not always looking forward, they eventually will ruin you and your job can seem like a post-apocalyptic nightmare. What your bottlenecks will be in capacity planing isn’t always obvious. Our network design was going to break down or get ugly unless we went from cabinets to a cage, and farther down the road our provider had limited power for our growth.

eye-terminator-salvation2

We did make the time to look to the future. We realized our capacity limits months before we started to feel the pressure, but because the process took a year there still was the feeling of the systems running us. They are cruel task masters – anyone that has seen Terminator or the Matrix know this. When they rule you, you end up in a reactive position and you start to lose control. The move was a reminder that this can’t be allowed to happen, we need to stay vigilant and look to the future.

Lesson 2: Beware the Purveyors of Colocation Space

They are tricksters and they want your gold. Some of them are masters of space and power and manipulate them to create the illusion of a good deal. Although a bit of an oversimplification, as a customer you generally care about a metric like “dollars per year to host a server”. However, colocation facilities generally don’t bill that way. They bill on space, power, and internet. So they do things like:

  • Offer cheaper square footage, but require more square footage per rack (Although more space can mean more room in a cage to move around in)
  • Only provide lower power options (i.e. 120/20 instead of something like 208/30) so you need more racks for the same amount of servers (Which will cause a bigger price difference over time with growth)
  • Force you to grow in increments of multiple racks so you are paying for space (And maybe even power) you don’t need as you grow
  • Put in some initial fees and initial higher prices just so they can remove or lower them to make you think you are getting a deal

If you model the costs based on your needs, accounting for growth, and look at the total cumulative costs you can see through their illusions.

Lesson 3: Holistic Design and Reality

We are control freaks. We decided to own everything inside our cage including the racks and PDUs. Short of building our own facility, this gave us a blank slate for the genesis of our perfect facility:

  • There are constraints of what is actually available to buy, and how much you can actually know what you are getting
  • Team members have different visions

Each choice you make along the way affects all the other choices. For example, Vertical PDUs mean you need a place on the rack to put them, certain types of cable management and cable arms might also take up that space. We discovered this as we went through a few different passes of various equipment that we had to return because it all didn’t work together.

Our biggest error with this was not making one person ultimately responsible for the physical design. Choices need to be made and not everyone’s ideas can be reconciled with each other and the constraints of reality. For a holistic design, eventually someone has to reconcile reality with what everyone wants or you end up with a bunch of individually well thought out pieces that don’t fit together (as well as a bit of frustration.)

Lesson 4: If it isn’t Right, Tear it Down and Do it Again

“One of my most productive days was throwing away 1000 lines of code.”

–Ken Thompson

When something isn’t right and you decide to move forward regardless then you may have to live with it for a long time. Even worse, it can create ripples forcing you to make further bad choices in other areas and cause broken window syndrome. The discpline to take a step backwards is a quality that is easy to respect and hard to have.

drill_sergeant

When the wiring wasn’t great, even though it took a lot of time: “AGAIN!” When we purchased the wrong stuff, we returned it and started “AGAIN!” In both these instances we did this more than once. It is often a hard call to make, but it will pay in the long run if you can summon your inner drill sergeant. Eventually in order to maintain momentum you have to move forward, but a rule that can help: If you are moving forward out of laziness then it is the wrong call.

Lesson 5: Disasters Really do Happen

“Superstorm Sandy” hit the week before we were supposed to move doing significant damage to the current facility. It did manage to stay up the whole time, but only because of a bucket brigade of people carrying diesel fuel. Oddly for us it was more like “Serendipitous Sandy”:

  • Gave us time to rectify some mistakes
  • Gave us confidence in our secondary datacenter since we failed over to it before the storm
  • Gave us time during our move since we were comfortable running out of our secondary facility

Mostly it was a good taste of reality, disasters happen and you better be prepared.

Lesson 6: It is Better When the Hosting Provider Owns the Building

If your building isn’t owned by the datacenter then there are more likely to be conflicts of interest and complications. We don’t know if our new provider is going to renew their lease, so we might be moving again. Also, during Sandy there were some issues with building access since the building owner wasn’t too worried about the needs of it’s customers’ customers. Don’t overlook this factor.

Lesson 7: Pace Yourself

The actual move itself is exciting but also exhausting. We set Alex Miller on the task of finding great physical movers and he delivered with Morgen Industries. They were fast, on time, personable, and flexible. They handled de-racking, moving, and re-racking the servers. Having that part handled by movers made it so we had more energy for configuring, cabling, and fixing problems that came after that. It played a major part in ending up with a great result.

When moving in we had shifts. You can only fit so many people in the space so this seemed like the most efficient method. However, the second shift spent it’s whole shift learning what the first shift figured out. So it is better to leave yourself the time to make sure there is overlap. A move is often a chance to get things right so make sure you pace yourself in a way that will allow your team to get it right.

ThingOfBeauty

Lesson 8: Value Craftsmanship

Moving was in many ways a chance to start over and fix issues that arose from rapid organic growth over the past few years. The whole team is proud of our result. It might make us weirdoes, but when cabling looks this neat it is a sexy and sleek piece of art. I am proud of the team and we are all proud of the work we did. That has real value, even if it isn’t easily measured. In the next post I’ll get into all the detail of the beautiful result of our chaotic journey.

StackExchange welcomes our newest system administrator, Steven Murawski!

Steven Murawski Steven followed a bit of an eclectic path to systems administration, having started out operating a garden center, floral design and landscape business. After eight years he decided to move into law enforcement; he went to school for Police Science and worked at a local police department as a clerk, dispatcher and auxiliary officer. While he enjoyed shooting various firearms and driving at high rates of speed, he also began working with the police department’s IT systems, and slowly fell in love with systems administration. He became a member of LOPSA, Microsoft MVP, and continues to lead two local user groups (the Greater Milwaukee IT Pro User Community and Greater Milwaukee Script Club). Steven combined his love of systems administration with public speaking by presenting technology talks at various technology conferences and user groups.

Today Steven distracts himself from technology by playing with his eight month old son, puttering around with home-improvement projects and hurling pieces of lead through paper targets, although he is seriously considering once again picking up the mantle of host for a systems administration and Powershell podcast!