A lot of tools available in IT/Sysadmin/Ops/DevOps are disappointing:

  • They don’t fit your environment. They lack features or our designed for a different sort of environment (i.e cloud vs hardware, Linux vs Windows, distributed vs centralized etc)
  • You can’t interact with them programmatically
  • They cost too much
  • They are not customizable enough, or require too much customization to get off the ground
  • Feel kludgy, unreliable, outdated, or like the programmers were stoned
  • Don’t fit with your company’s culture (i.e. Enterprise vs Agile)

In short a lot of stuff is too expensive, isn’t a good fit, or is simply bad software. This ends up leaving an ops team with two options. They can whine about it, or create their own tools. So at Stack Exchange we build our own DevOps tools.

Status

Nick Craver’s baby, which we just call “Status” is at first glance a monitoring dashboard, but is essentially a collection of tools that filled various needs:

  • An Overview of CPU, Memory, and Network utilization for all our servers as well as a detailed view. Done with responsive and interactive D3 graphs as well as sparklines it helps compensate for Solar Wind’s terrible interface. statussqlscreen2-png
  • SQL Server monitoring. SQL’s built in Clustering views are deeply flawed. If a node loses connectivity, it stops updating remote nodes status, so it could show everything as connected and fine, even if there is no connectivity. We also get to see the most expensive queries, active queries utilizing whoisactive, current connections, and which DBs are on which server
  • HAProxy Monitoring and Administration: With multiple instances of HAProxy we needed a single view instead of HAProxy’s built-in display. Also, this gave us a nice web interface to take servers out of rotation statusdashboardscreen-png
  • Redis: A nice presentation of Redis Info across all instances and all servers. Also a display that shows what is slaved to what in at a quick glance
  • Elastic Search: Health overview of or clusters (as well as index and shard data)
  • A dashboard of all the exceptions generated by our applications

Status is C# / .NET app. It polls data from various sources – sometimes the system directly and other times it gets it from Orion. There is a lot more to status that makes it awesome. The real accomplishment is that status enables us to see the general health of our main infrastructure at a glance.

Web Logging

If you business is creating and running websites, your web logs are gold. We use the logs generated by our load balancer, HAProxy, as our canonical web logs. In their raw text format, web logs are often not that useful (this is particularly true with over 100 million records a day). However we parse and structure our web logs in a few different ways:

realog.ds.stackexchange.com

  • We have C# service that Jarrod Dixon wrote that inserts them into SQL so we can query them. In order to query them we use an instance of Data Explorer, SQL management studio, and also have certain lookups directly from our sites
  • Displaying realtime graphs of various log information with Realog, a system I created with Go, Redis, and NVD3.js so we could view activity live without having to write queries

One of the interesting things we do with our weblogs is to add extra information by adding headers inside the app and striping them from the response at HAProxy. For example, we capture how many Redis and SQL queries were involved in that request and how long they took.

Patch Dashboard

OS updates can be a bit tedious, even more so in a mixed Windows and Linux environment. PartialPatchDashboard Steven Murawski and George Beech created a dashboard that allows us:

  • View the outstanding patches and patch count for both Linux and Windows
  • Trigger updates on either Linux or Windows
  • Schedule time frames for automatic Linux updates

What’s Next

If you want to learn more about these tools and DevOps at Stack Exchange, come see George, Nick, and Steven present “Building for Operations” at Velocity.

Keeping all this stuff to ourselves feels a bit greedy. However, for something open sourced to be very useful it usually needs to be made a bit more generic which takes time. We also want to build a lot more. Our inventory system Racktables lacks an API so we need a new one or a way to extend it. We want to build our own monitoring system (likely on top of OpenTSDB). In order to create more, and open source it we need help. So we are looking a full time developer with ops experience to join our SRE team. So if you are awesome, want to build awesome ops stuff and open source it, come join us!

Keeping Your Cool

Steve Murawski

I’m not talking about data center cooling here…

I was recently listening to The Ship Show podcast titled “Keep calm and PROD on”.  In this podcast, the hosts were discussing whether or not all devs and operations personnel should have production access.

The conversation really hit home with me when it changed from having access to production to how people handle dealing with outages/incidents.  The hosts asked for some feedback on the topic on Twitter, but I have just a few more than 140 characters of thoughts on the topic.

The Scenarios

The hosts outlined a couple of scenarios for discussion.  They didn’t use these exact terms, but it helps me to group them under certain archetypes (it must be all those role playing years catching up with me..).  In no particular order (going from memory and making up the classifications here), we have the “cowboy” response, the “neophyte” response, and the “deer-in-the-headlights” response.  I’ll provide a quick synopsis of these responses.

The “Cowboy” Response

This incident response archetype is very identifiable in many environments.  Picture it, the year is 2010, it’s 2 AM and your cell phone is blowing up with monitoring alerts.  The website is down and everyone is waking up.  You log in to your VPN and start gathering information.  Everything goes dark, and then everything comes back up, seemingly magically fixed.  It comes to light that one of the developers, on his own volition, decided to take drastic actions to restore service.

This response highlights the lack of communication and command and control that is typical of frantic incident response in many IT departments I’ve observed over my years in IT.

The “Neophyte” Response

This incident response archetype is your typical “newbie”.  The neophyte does not have to be new to the field, just new to high pressure outage scenarios.  Taking the scenario above, the neophyte may be taking his first turn at the on-call rotation or this may be the first time a particular problem has cropped up.  The neophyte might not  be comfortable with getting a more senior engineer out of bed, or getting some developers involved.  Another pitfall with the neophyte is that they might not be willing to take charge as others begin to respond to the incident, meaning that the incident response can be confused with no coordinated direction.

The “Deer-In-The-Headlights” Response

The last one I remember the podcast covering was the “deer-in-the-headlights” response, where regardless of a person’s experience level, they just don’t respond well in a crisis.  Many of the downsides of the neophyte are felt here too.. The primary responder may not reach out for help or may not be able to control other responders.  This leads to a fractured response, where people may be working at cross purposes.

What Should We Do?

What I’ve Learned In Previous Careers

Training To Be A Cop

You may not know this about me, but before I got into IT, I was training to be a cop.  I went to school and did all the fun training on how to drive fast, arrest people, and shoot guns.  A big part of that training also included how to respond to medical situations (trauma or illness) or hazardous materials situations.  (Despite what you may think.. watching re-runs of Cops is not adequate training – except as what not to do!)

In that training, we drilled how to respond to emergencies as individuals and as part of a team.  Each course drilled scenarios, but our firearms course drilled scenarios most heavily and over time included skill sets learned in other classes.  Nothing wakes you up to a training scenario like walking into a situation with a vague description of a problem and, as you start to gather information and stabilize the scene, you get drilled in the head by several rounds of simunitions.  Even though you know that you are using training rounds, it shakes you to your core to realize you could be dead in seconds.

Part of the training regimen includes working past the failures and mistakes.  In that first scenario, even though every recruit gets ambushed and shot in the head, our instructors make us follow through with the techniques we’ve drilled in the classroom and on the range.  We are coached to get to cover, return fire, and call for backup.  We can’t dwell on our mistakes (there will be time for that later.. everyone is videotaped for review with the whole class afterwards), we have to follow our training and deal with the problem and stabilize the scene.

Over time, we progressed through a number of other scenarios, some by ourselves, and some with other recruits.  In all cases, we were responsible for communicating status back to our dispatch center, requesting resources as needed, and dealing with any subjects in person that the scenario called for.  We were tested with a variety of actions, all potentially threatening, but each requiring evaluation for how we could respond and always under the watchful eyes of our instructors and the unblinking eye of the video camera.  This training reinforced our more static, isolated drills, allowing us to respond to dynamic situations with a combination of intentional action and reflexive reaction based on our other drills.  Training helps minimize the “deer-in-the-headlights” and the “neophyte” responses and allows people who would experience those responses to fall back on training.

Working In A Public Safety Agency

I also worked as a dispatcher, clerk, and auxiliary officer before and alongside my role as IT guy for a local police department.  Over that time, I observed, interacted with, and sometimes responded with our emergency responders as they dealt with life and death situations.  I observed the growth in how the police and fire agencies learned to respond to incidents together, using the National Incident Management System (NIMS), the emphasis on which grew after 9/11/2001.

Under the NIMS model, the first responding public safety officer is the incident commander.  As the situation develops, the incident commander role can change, based on who’s best suited to deal with the incident.  For example, in the case of a fire, the police officer first on scene will be the incident commander, until a fire department official is set up and ready to take over command.  (In my experience, this is after police officers have saved all the people in the building. 😉 ).  Let’s look at another scenario: officers respond to a report of a burglary in progress.  The first officer responding is the incident commander.   As the scene develops and a perimeter is established, the officer in command is responsible for requesting the resources needed and beginning to stabilize the situation.  Next a sergeant, lieutenant, or captain arrives on the scene and takes over the coordination of the perimeter, allowing the officer to focus on his area of responsibility.

NIMS also defines several other key roles for the command staff in an incident, a Public Information Officer (PIO), a Safety Officer, and an Liaison Officer.  The PIO’s responsibility is to keep the stakeholders and public in general informed as to what is going on.  Part of the role is determining what information is helpful to share and what should not be disclosed.  The Safety Officer monitors conditions and ensures the safety of all incident personnel.  The Liaison Officer is responsible for dealing with all the coordinating agencies.  Defining these roles can help deal with the “cowboy” response.  If incident response is structured, the right resources can be directed to a problem and a sustainable fix is more likely an outcome, versus “just get x involved, he fixed it last time.”

How We Should Respond

These two experiences provide some basic thoughts into how we can approach incident response.

Drilling

Just because we are sysadmins, site reliability engineers, devops engineers, etc. doesn’t immediately grant one the intrinsic knowledge and skill necessary to deal with an outage, especially if you are a specialist and the outage deals with technology you are less familiar with.

For operations personnel (developers, dba’s – yes them too – and sysadmins), this is critical.

For someone with deep intuition about their environment, the answers are easier than for newer or more narrowly focused personnel.  Guess what?  The knowledgeable guy isn’t always around when things go ill.

Focused drills, around dealing with one sort of problem or technology, as well as combined drills with multiple components are vital.  Not only do you need to be familiar with the systems you are responsible for, but also everything they interact with, internally and externally.  Quick – what do you do when your CDN stops serving content?  Have you drilled that scenario?  If you haven’t where do you even begin?

This is something we are going to be focusing more on here at Stack Exchange and I’m super excited about that.  We’ve decided that this is a priority for our organization and we’ll be dedicating time to this.

Roles

In the police department where I worked there was a definite command structure, but individual officers had a great deal of latitude to respond to most situations.  The latitude enjoyed by officers in that agency is similar to the latitude I have as a sysadmin on my team.  Certain situations we can just deal with and not need to involve others.  If I need backup (additional resources), I can request those, but if a situation escalates, it’s time to bring in more support.  We don’t have a strict command and control environment;  as we grow our technical staff I think that’ll be more defined.

When it comes to incident response though, the cops and my fellow sysadmins have a bit of a different experience.  Since the officers drill a variety of scenarios and have those drills and training reinforced by continual engagement with an unpredictable public, their escalations from situation to incident are much more fluid, as are their transitions of the incident commander role.  We currently don’t have anything defined like that, though I hope as we start to drill these scenarios more, I’ll be able to lobby effectively for the establishment of at least two of the NIMS roles, the Incident Commander and the Public Information Officer.

In the podcast, one of the themes discussed included how detrimental demands for status updates and presence on conference bridges could be.  By assigning (and training) someone (and at least one alternate) to fill the PIO role, the remainder of the technical staff is freed up to deal with the issue.

In our case, I can envision that as having our PIO designate one chat room  or google hangout as our internal status update location and during the incident he/she’d make regular updates to those areas.  In addition, the PIO would be responsible for updating our status Twitter account and status blog to keep the public informed as needed.

As for the Incident Commander role, we’d need to train all of our on-call personnel, as well  the rest of the technical staff, so everyone is on the same page as to who is in charge and directs the resolution of an incident.  While we don’t have this defined yet, we had a short outage a few weeks back that illustrates how this can work.

  • About 9 AM UTC, while I was dreaming of servers with 4TB of RAM and many multi-core processors, my phone began to blow up with alerts from Pingdom.  I wasn’t the on-call person, but I always monitor for severe external alert failures.
  • I rolled out of bed and ran stumbled down to my office and got online to start investigating.
  • First order of business, check our chat site.. oops! It’s down!  Normally our chat servers are in the opposite data center from where our Q&A sites run, but we are preparing for maintenance in our secondary data center, so it’s running in New York with the rest of our infrastructure.
  • Next up, VPN..  which connects.. that means internet to our data center in NY is still good.
  • I jump in a Google Hangout (fortunately unfortunately for my co-workers,  I look as good as I do right out of bed as I do after getting ready in the morning) that we have set up for our site reliability team.
  • Chat comes back online.  I dropped a note in chat that I’m in the SRE hangout and troubleshooting.
  • I’m soon joined by Geoff Dalgas, one of our Core Q&A developers.
  • He and I discuss the situation and began validating the different bit of our infrastructure.
  • We determined that we were seeing an issue with keepalived on our load balancers.
  • Just then, Tom Limoncelli, another of our SRE team joined us to help with the issue.
  • We also had several more developers pop in and see if they could offer any help.
  • We determined a course of action to remediate the problem and began to implement it.
  • Soon, Stack Overflow (and the rest of the network) was back online and the twitters began to calm themselves.

In this situation, I was first on-scene and acted as incident commander.  If the situation had continued to develop into something more complex involving the load balancers, I might have had to defer to one of our other engineers or Geoff.

Open Line

So, what do you do in your organization?  Do you drill and train for failures?  Do you test your backups?  Do you prepare your operations personnel for how to respond in an incident?

I’ve been getting some great response to my previous post, and I wanted to make a few things very clear.

Clarifications

What I’m not doing –

  1. I’m not advocating or dismissing any particular configuration management tool.
  2. I’m not discounting the tough work done by companies and community projects that have created abstractions on managing disparate systems.

What I am attempting to do –

  1. Highlight the challenges of cross-platform management and application management.
  2. Show one of the efforts in providing a standards based management abstraction.
  3. Offer my thoughts on why I see value in that direction and what challenges I see.

Managing the Operating System vs. Managing Applications

There is definitely some confusion around using CIM to manage the OS vs managing applications.

  1. CIM Classes can be used to model applications as well as OS resources.
  2. Most of the “applications” that are packaged as roles and features in Windows Server expose a CIM management API.
  3. WMI is an implementation of the CIM standard and starting with Server 2012 and the Windows Management Framework V3, CIM is exposed via WSMAN.

Hyper-V, File Shares, Clustering, IIS, and others all offer CIM based management models.  Other applications can expose a CIM management model as well.  As long as the host CIM server (WMI on Windows and OMI or OpenPegasus or ???? on Linux based Operating Systems) is operational, applications can also offer their configuration and status via that channel by creating a provider.  To do this on Windows, there is some documentation to get you started:

To do this with OMI, you can find some documentation and source at https://collaboration.opengroup.org/omi/documents.php.

The Current State – Reprise

But, but, tool {fill in your favorite tooling here} already does THAT!!!

There are a number of tools that valiantly strive to provide cross-platform management.  I mentioned several of them in my last post, but there are a number of others.

Yep, it does.  Until…

things change.  The challenge these tools have is that they have had to implement their abstractions against very different implementations.  The problem there is that these things are not stagnant.  The management APIs can change over time and since there is not a standard description of the API or underlying configuration.

If CIM were the standard API exposing the configuration, the underlying implementation details can change, but configuration management and monitoring tooling don’t have to care about that.  The tool vendors and community projects can focus on other value adds for their particular tooling, rather than being forced to continually update the basics.

The Next Steps

We are still in the early stages of the push for CIM and WSMAN.  We’ll have to see how adoption picks up.  The continuing work around OMI holds promise, but it needs a deployment or integration story for various Linux distros and more public providers for managing components of the Linux OS and attendant applications.

There have been some interesting announcements at TechEd in relation to Windows Server 2012 R2 (watch this video and pay attention around 49 minutes in).  I’m going to talk more about this feature and it’s implications and my plans with it in the very near future.

 

​Configuration management today is mess if you work in a heterogeneous platform.

There is tooling that takes a stab at it, and is getting better (from the *nix world – Puppet, CFEngine, and Chef and from the Windows world – System Center Configuration Manager, Group Policy, among other third party application deployment platforms).  These tools are all well and good, but they fall down when reaching across the OS divide.  Puppet, Chef, and CFEngine (there are others as well, but these are some of the more popular) all have some cross platform support, but it feels unnatural (especially in module or recipe development).

Why is this a mess?

Windows is traditionally described as having an API oriented management model, whereas *nix has a document based management model.

Well, that’s a load of crappy, crap, crap.  What does that actually mean?

It means that the two operating systems offer two different management models.  The two different models have different abstractions and idioms for operating system constructs.  Let’s look at a concrete example, setting a static IP on a network interface (just the rough strokes.. I’m not going to spend too much time on the minutia). As I stated before, Linux uses a document oriented management model, so to configure my network interface, I’ll edit a document or two.

The Linux (Centos) example:

  1. Find the correct interface file under /etc/sysconfig/network-scripts
  2. Open it in your text editor of choice
  3. Edit it to contain your desired settings for the network interface and save the file
  4. If you need to add/modify DNS servers, find /etc/resolv.conf
  5. When done, you can bring your interface online with a command line call to
    ifup eth0

You’ll have something that looks like this for your network configuration file:

DEVICE=eth0
IPADDR=192.168.10.80
NETMASK=255.255.255.0
NETWORK=192.168.10.0
BROADCAST=192.168.10.255
GATEWAY=192.168.10.1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
And something like this for your resolv.conf:
domain serverfault.com
search serverfault.com
nameserver 192.168.10.60
nameserver 192.168.10.61
That wasn’t so bad, and as an added benefit, they are just text files, so I could check them in to a revision control system (Versioning FTW!). Now, let’s look at what we’d need to do on the Windows side.  Since this is a blog for a community of “professional” systems administrators, we are going to dispense with any GUI example for doing this.

The Windows Server (2008 R2) example:

  1. Use WMI to retrieve the network adapter interface index.
  2. Use WMI to retrieve the network adapter configuration by the index.
  3. Set the desired IP address, gateway, and DNS servers and suffix against the WMI object.

You can use the following PowerShell commands to make those changes:

$NetworkAdapter = Get-WMIObject Win32_NetworkAdapter -filter "NetConnectionID = 'Local Area Connection'"
$NetworkAdapterConfiguration = Get-WMIObject Win32_NetworkAdapterConfiguration -filter "InterfaceIndex = $($NetworkAdapter.InterfaceIndex)"
$NetworkAdapterConfiguration.EnableStatic('192.168.10.80','255.255.255.0')
$NetworkAdapterConfiguration.SetGateways('192.168.10.1','1')
$NetworkAdapterConfiguration.SetDNSServerSearchOrder('192.168.10.60','192.168.10.61')
$NetworkAdapterConfiguration.SetDNSSuffixSearchOrder('192.168.10.60')
and

The Windows Server (2012) example:

  1. Set the desired IP address and gateway based on the interface name.
  2. Set the DNS servers and with a few more PowerShell commands.

You can use the following PowerShell commands to make those changes:

$IPAddressParameters = @{
            IPAddress = '192.168.10.80'
            InterfaceAlias = 'Local Area Connection'
            AddressFamily = 'IPv4'
            PrefixLength = 24
            DefaultGateway = '192.168.10.1'
}
Set-NetIPAddress @IPAddressParameters
Set-DNSClientServerAddress -InterfaceAlias 'Local Area Connection' -ServerAddresses '192.168.10.60','192.168.10.61'
Both of these examples are interactive commands, but I could easily save them in a file and place that under version control (and I should).

So what?

The examples don’t look all too different, but they do illustrate the difference between similar operations.  In both  examples, I end up with an artifact, but one is for a one time application of the setting (the Windows side) and the other is the setting storage location (the Centos example).

On the Centos box, we had to edit a file where the configuration was read from.  On the Windows servers, we updated settings via a WMI API (in both cases.. on Server 2012 there are more built in cmdlets, but many of them are thin wrappers over the WMI APIs) and not the actual end storage location.

This is what

Any configuration management tool that works in a cross platform capacity needs to understand these distinctions and check based on OS type which implementation to use when configuring a system.  This means for most configuration types, you’d have a big “IF” block where *nix based OS’s follow this line of processing and Windows based machines follow  the other line of processing.  This can become a maintenance nightmare as OS versions change the API on the Windows side or modify location and or layout of the configuration files on the *nix side.

And it’s even worse…

Now, what happens when you have a model that doesn’t translate across both worlds?

For example, how do manage file permissions?

Posix style permissions (used on most *nix variants) assigns permissions are nowhere as discreet as NTFS file permissions.  In addition on Windows, the file system auditing is also configured via the permissions configuration. In the reverse, on *nix files can be set as executable, where that is handled by file type mappings based on file extensions in Windows. This fragmentation leads to more complex implementations on the side of configuration management software developers or missing feature coverage.  In either case, this is a loss for the sysadmin who maintains cross platform environments.

But what if….

there was a common method of interacting with operating systems, regardless of what was running underneath? What if this method used a common transport (open standard) and communications were defined by an open standard? This is the direction Microsoft is taking with CIM and WS-Management.

CIM (Common Information Model) is a DMTF (Distributed Management Task Force) standard for describing management information for systems, networks, applications, and services.

WS-Management is another DMTF standard for management communication, focused on CIM traffic.

Microsoft has contributed to an open source project hosted by the Open Group called OMI.  OMI is a CIM server that communicates over WS-Management and is implemented to run on *nix based operating systems.

I’m personally interested in where this will go, given Microsoft’s market power (Cisco and Arista are working on incorporating OMI into their network switches).  The idea of a shared management model is very appealing to me, as I work in a cross platform environment.  I’m responsible for our Windows infrastructure, but I have to be able to work with our *nix infrastructure as well.  If I could use one model for interacting with both, that’s a huge win for me and my team.

This wouldn’t eliminate any domain specific knowledge on either OS side, as you’d still need to know what buttons to push and knobs to tweak to get things going and do some deep troubleshooting.  It does, however, make the idea standardizing how various OS components can be accessed, making basic configuration, monitoring, and troubleshooting much easier.

I’m interested because…

this pushes the implementation down to the OS provider (or the CIM provider provider) and gives vendors one target to hit for configuration standards.  In the Microsoft case, they can say “Follow this standard and any Windows system can manage you with minimal effort.”  If other OS’s support CIM and WS-MAN as well, it becomes easy to offer management interfaces there as well.

Obviously this would be huge change to the existing way of doing things for OS and application developers, not to mention systems administrators that are invested in their existing ways of doing things.

I don’t see another good alternative though, as the numbers and variety of systems continue to scale up and “cloud” becomes more of a factor in our environments, yet the number of admins is staying stagnant or being reduced.  Simplifying the management and monitoring surface makes sense in today’s and likely tomorrow’s data center landscape.

It doesn’t solve every problem and vendors can still implement vendor specific extensions (and we know how well that’s worked with SNMP).

 

NOTE – Be sure to check out some clarifications and expanded discussion in my followup post.

The ServerFault Systems Administration team continues its growth with the addition of sysadmin icon Thomas Limoncelli.  You may know Tom from his books, Time Management for System Administrators and The Practice of System and Network Administration, or from his many conference appearances at events like LOPSA-East, the Cascadia IT Conference, and USENIX LISA.  Tom’s also been a ServerFault user, though references to his books outnumber the number of direct answers he’s supplied.

When we saw that Tom was just finishing up his time at Google as we were posting the next ServerFault opening, it was kismet.  Tom is the quintessential systems administrator and a great fit for our team.

I’ve been lucky enough to know Tom for the past couple of years and I am very excited to be able to work with and learn from him (and hopefully teach him a trick or two on our Windows stuff).  His The Practice of System and Network Administration was one of the first resources I had when I got started working as a systems administrator and really got my career headed in the right direction (well, if you consider getting to work with a small crack team of sysadmins and developers on one of the more dynamic environments out there “the right direction”).

Tom is joining Bart and George in working out of our (newly constructed) New York headquarters. Tom hails from New Jersey and frequently attends local LOPSA meetings.

Join me in welcoming Tom to our team!

Hey everyone, I’m one of the newer ServerFault sysadmins and I’ve got the fun job of explaining why we have been moving to Windows Server 2012 just almost as fast as WDS can deploy images.

 

What’s Wrong With Windows Server 2008 R2?

Where do I begin?

2008 R2 was a great operating system in its day.  Automation was becoming a core part of the platform.  PowerShell was shipped in the box, enabled by default, and supported several roles and features.  Many of the kinks were worked out after the Vista/2008 release cycle and people were starting to acclimate to the User Account Control (or figured out how to turn it off).

Server 2008 R2 held up well for a long time, but was starting to show it’s age with our SQL Server cluster.  There are a bunch of hotfixes required to make clustering have a chance of working with SQL Server 2012 AlwaysOn in our environment.  Brent Ozar, in part based on experiences helping us, now recommends not building AlwaysOn Availability Groups on Server 2008 R2.

Additionally, the management story for Windows Server up to this point has been very focused on single machines.  Server Manager in 2008 R2 targets one machine at a time.  Most of the MMCs deal with one machine at a time.  PowerShell remoting was not on be default.  It is definitely work to build a multiple machine management story.

 

So, how does Server 2012 help?

First of all, Windows Server 2012 helps address our major pain point of how SQL Server 2012 Availability Groups interact with Windows Clustering.  Clustering received a lot of attention in Windows Server 2012.  The addition of dynamic quorum helps maintain the cluster state in the case of site-to-site VPN interruptions.  Clustering itself has just become more resilient and offers a wider feature set.

The second (and I think most important feature) that Windows Server 2012 offers is SMB3.  SMB3 fixes many of the performance considerations with access to files stored centrally on a file share.  SMB Multichannel allows SMB clients and servers to leverage multiple network connections, makes SMB transfers resilient to TCP disconnects (by having multiple streams per transfer, even with one NIC), and provides the foundation for scale out file servers, providing a way for multiple servers to share the same connection concurrently, isolating storage transactions for single server failures and allowing connections to scale across multiple servers.  SMB3 provides a number of opportunities to radically change the structure of your application data access and I talked with Richard Campbell about this at length on RunAs Radio.

The third huge win is how Windows Server 2012 supports three levels of GUI-ness.  You have the traditional full UI to support the traditional administrative experience, and since this is the same codebase as Windows 8, you get the Metro/Modern UI start screen as a “bonus”.  The next level of GUI-ness offered by Server 2012 is the minimal shell experience, or what I like to call “Server Core with Training Wheels”.  The minimal shell removes Explorer, Internet Explorer and most GUI apps, but leaves behind Server Manager and the MMCs.  This provides a bit of a security blanket for those admins who aren’t quite ready to give up managing a server locally, but want to break their reliance on the UI and grow up to be SYSTEMS administrators who can manage multiple systems.  By removing explorer.exe and iexplore.exe, you’ve eliminated a huge source of patches, particularly the ones that require a reboot.  Additionally, since there is no Explorer, the RDP experience is a command prompt and Server Manager – discouraging random RDP sessions.  The final level of GUI-ness is full on Server Core.  This is the DEFAULT install option for Windows Server 2012.  Server Core lacks even the minimal GUI tooling provided by the minimal shell configuration.  Server Core, as of Server 2012, is no longer a life choice.  In Server 2008 and Server 2008 R2, when you selected Server Core as the install option, you were stuck in Server Core.  With Server 2012, the GUI bits are features and can be added and removed through the standard feature management tooling.  I talked a bit about this with Richard Campbell on RunAs Radio earlier this year.

Next up in the “pro” column for Windows Server 2012 is PowerShell V3 and the increased cmdlet coverage in the operating system.  PowerShell v3 brings a number of great enhancements to the party like improved performance, resilient remoting sessions, commands for working with the web and REST endpoints, workflow, ISE enhancements and more.  Windows Server 2012 also brought about 2400 commands across the in-box roles and features (up from under 500 in 2008 R2).  This explosion in coverage was due to improvements in the WMI APIs and a new API in PowerShell for being able to take an XML mapping file and generate PowerShell commands from a WMI API.  This got us tons of additional coverage, including the network stack. Since servers really aren’t intended to be managed individually (despite what some sysadmins, product teams, and third parties software vendors believe), the increased PowerShell coverage makes that a much more reasonable task.  If you are interested in further talk about PowerShell V3, you guessed it, there was a RunAs Radio talking about that as well.

The final consideration we had for choosing Server 2012 was in-box NIC teaming.  Server 2012 offers OS level NIC teaming with a variety of teaming options.  Given the problems we’ve faced with third-party NIC teaming (http://blog.serverfault.com/2011/03/04/broadcom-die-mutha/ and some other fun stories), I was gung-ho to get switched over to the native teaming features (which I’ve monitored in production environments for over a year at that point).  Having the network load-balancing and failover (LBFO) teaming included in the OS is a major plus.  Don’t worry, I won’t point you to yet another RunAs Radio show this time.

 

Where are we now?

I’m currently working with our team to continue the roll-out of Server 2012.  Part of that process is updating our installation and configuration management plan.  Nick Craver, one of our developers who likes to help with the sysadmin work and has been featured on this blog, and I have spent a good amount of time tweaking our deployment scripts (server and application deployment) to facilitate the roll out of new servers.  We’ve got our SQL Server and web tier installs fully scripted, to the point where we can go from bare metal to a fully functional web server in about 3 hours (mainly OS install time and running Windows Updates).

We are still working out the rest of our deployment story, looking for integration points with our Puppet configuration management, making the configuration idempotent, and creating artifacts that can be stored in source control to version our environment.  But each week sees more Server 2012 (and PowerShell use…  Pete’s started using a bunch of stuff I wrote and now I have bugs to fix… grrr.. ) in our environment.

 

Where do we want to be?

I would like to see our Windows deployments that are in a production capacity (internally or externally facing) under full configuration management control, with changes pushed via a configuration management infrastructure and configurations enforced regularly.

I would like to have most of our servers running Server Core (doesn’t make much sense to do that for our management stations, but for most servers it works).

Most of all, I would like our server deployment to help serve as a template for a different way of Windows administration, one with rigid standards but quick to respond to changing needs.  I’d like to see our infrastructure as agile as our code base, where changing our OS configuration is as easy as pushing a build is for our development teams.  I’d like to be able to validate infrastructure changes quickly, closing the feedback loop and allowing worry free infrastructure changes.

You all have a front row seat to our environmental changes and I plan to blog extensively about my configuration management efforts in the near future, giving you an opportunity to learn from my mistakes.

 

Moving about 35 servers, something like 2000 pounds of computer hardware 50 blocks, doesn’t seem like that big of a thing. However, in our geek microcosm moving to a new colocation facility was a year long adventure with lessons learned, arguments, designs and redesigns, a hurricane, and many weekends of preparation that somehow resulted in a setup that we are all very proud of. We always want to share our experiences, so this post has some of the lessons we learned, and in the next post have all the technical detail of our new facility.

Lesson 1: Be Prophets

Sometimes you run your systems, and sometimes they run you. If you are not always looking forward, they eventually will ruin you and your job can seem like a post-apocalyptic nightmare. What your bottlenecks will be in capacity planing isn’t always obvious. Our network design was going to break down or get ugly unless we went from cabinets to a cage, and farther down the road our provider had limited power for our growth.

eye-terminator-salvation2

We did make the time to look to the future. We realized our capacity limits months before we started to feel the pressure, but because the process took a year there still was the feeling of the systems running us. They are cruel task masters – anyone that has seen Terminator or the Matrix know this. When they rule you, you end up in a reactive position and you start to lose control. The move was a reminder that this can’t be allowed to happen, we need to stay vigilant and look to the future.

Lesson 2: Beware the Purveyors of Colocation Space

They are tricksters and they want your gold. Some of them are masters of space and power and manipulate them to create the illusion of a good deal. Although a bit of an oversimplification, as a customer you generally care about a metric like “dollars per year to host a server”. However, colocation facilities generally don’t bill that way. They bill on space, power, and internet. So they do things like:

  • Offer cheaper square footage, but require more square footage per rack (Although more space can mean more room in a cage to move around in)
  • Only provide lower power options (i.e. 120/20 instead of something like 208/30) so you need more racks for the same amount of servers (Which will cause a bigger price difference over time with growth)
  • Force you to grow in increments of multiple racks so you are paying for space (And maybe even power) you don’t need as you grow
  • Put in some initial fees and initial higher prices just so they can remove or lower them to make you think you are getting a deal

If you model the costs based on your needs, accounting for growth, and look at the total cumulative costs you can see through their illusions.

Lesson 3: Holistic Design and Reality

We are control freaks. We decided to own everything inside our cage including the racks and PDUs. Short of building our own facility, this gave us a blank slate for the genesis of our perfect facility:

  • There are constraints of what is actually available to buy, and how much you can actually know what you are getting
  • Team members have different visions

Each choice you make along the way affects all the other choices. For example, Vertical PDUs mean you need a place on the rack to put them, certain types of cable management and cable arms might also take up that space. We discovered this as we went through a few different passes of various equipment that we had to return because it all didn’t work together.

Our biggest error with this was not making one person ultimately responsible for the physical design. Choices need to be made and not everyone’s ideas can be reconciled with each other and the constraints of reality. For a holistic design, eventually someone has to reconcile reality with what everyone wants or you end up with a bunch of individually well thought out pieces that don’t fit together (as well as a bit of frustration.)

Lesson 4: If it isn’t Right, Tear it Down and Do it Again

“One of my most productive days was throwing away 1000 lines of code.”

–Ken Thompson

When something isn’t right and you decide to move forward regardless then you may have to live with it for a long time. Even worse, it can create ripples forcing you to make further bad choices in other areas and cause broken window syndrome. The discpline to take a step backwards is a quality that is easy to respect and hard to have.

drill_sergeant

When the wiring wasn’t great, even though it took a lot of time: “AGAIN!” When we purchased the wrong stuff, we returned it and started “AGAIN!” In both these instances we did this more than once. It is often a hard call to make, but it will pay in the long run if you can summon your inner drill sergeant. Eventually in order to maintain momentum you have to move forward, but a rule that can help: If you are moving forward out of laziness then it is the wrong call.

Lesson 5: Disasters Really do Happen

“Superstorm Sandy” hit the week before we were supposed to move doing significant damage to the current facility. It did manage to stay up the whole time, but only because of a bucket brigade of people carrying diesel fuel. Oddly for us it was more like “Serendipitous Sandy”:

  • Gave us time to rectify some mistakes
  • Gave us confidence in our secondary datacenter since we failed over to it before the storm
  • Gave us time during our move since we were comfortable running out of our secondary facility

Mostly it was a good taste of reality, disasters happen and you better be prepared.

Lesson 6: It is Better When the Hosting Provider Owns the Building

If your building isn’t owned by the datacenter then there are more likely to be conflicts of interest and complications. We don’t know if our new provider is going to renew their lease, so we might be moving again. Also, during Sandy there were some issues with building access since the building owner wasn’t too worried about the needs of it’s customers’ customers. Don’t overlook this factor.

Lesson 7: Pace Yourself

The actual move itself is exciting but also exhausting. We set Alex Miller on the task of finding great physical movers and he delivered with Morgen Industries. They were fast, on time, personable, and flexible. They handled de-racking, moving, and re-racking the servers. Having that part handled by movers made it so we had more energy for configuring, cabling, and fixing problems that came after that. It played a major part in ending up with a great result.

When moving in we had shifts. You can only fit so many people in the space so this seemed like the most efficient method. However, the second shift spent it’s whole shift learning what the first shift figured out. So it is better to leave yourself the time to make sure there is overlap. A move is often a chance to get things right so make sure you pace yourself in a way that will allow your team to get it right.

ThingOfBeauty

Lesson 8: Value Craftsmanship

Moving was in many ways a chance to start over and fix issues that arose from rapid organic growth over the past few years. The whole team is proud of our result. It might make us weirdoes, but when cabling looks this neat it is a sexy and sleek piece of art. I am proud of the team and we are all proud of the work we did. That has real value, even if it isn’t easily measured. In the next post I’ll get into all the detail of the beautiful result of our chaotic journey.

StackExchange welcomes our newest system administrator, Steven Murawski!

Steven Murawski Steven followed a bit of an eclectic path to systems administration, having started out operating a garden center, floral design and landscape business. After eight years he decided to move into law enforcement; he went to school for Police Science and worked at a local police department as a clerk, dispatcher and auxiliary officer. While he enjoyed shooting various firearms and driving at high rates of speed, he also began working with the police department’s IT systems, and slowly fell in love with systems administration. He became a member of LOPSA, Microsoft MVP, and continues to lead two local user groups (the Greater Milwaukee IT Pro User Community and Greater Milwaukee Script Club). Steven combined his love of systems administration with public speaking by presenting technology talks at various technology conferences and user groups.

Today Steven distracts himself from technology by playing with his eight month old son, puttering around with home-improvement projects and hurling pieces of lead through paper targets, although he is seriously considering once again picking up the mantle of host for a systems administration and Powershell podcast!

About a  year ago, I wrote a post about the basics of getting a job in Information Technology.  Unfortunately, that post (linked here) did not cover much relevant material when it comes to putting your resume in and getting an interview.  This blog post has been in the making for a long time, however the major impetus came when a serverfault regular tossed me a message over the weekend asking for suggestions on how to answer “The Salary Question.” I address this below, but first I will cover what I think is the single most important thing to remember in IT when it comes to getting hired.

“It’s not what you know, it’s who you know!”

I have heard many people bemoan the fact that it is near-impossible to get an interview for tech jobs in this market.  Their stories are generally the same, listed below:

  • See there is a position at Awesome Company, LLC.
  • See that the resume is asking for every technology buzzword imaginable
  • Asks for 10 years experience with Windows 8
  • Decide to send in your resume for it, even though you only know 3 or 4 of the 15 buzzwords they’ve specified.
  • Become depressed when you don’t get a callback.

This is not an unusual turn of events.  The reason behind this oftentimes is that HR is putting together a job description and they’re (optionally!) asking the team that’s hiring for “some technologies they should look for.”  Oftentimes that hiring team says “Well, if they know ABC, XYZ, TLA, ETLA, or Java, it’d be a plus.”   This turns into “applicant must have X years experience in ABC, XYZ, TLA, ETLA and Java.”  HR just screwed not only your chances of getting hired, but the hiring team’s chance at getting a potentially good candidate.

The thing that is tough for most people to understand is that all of the above is what I like to call an “HR Trap.”  It’s there to trap your resume and never get it to the hiring team for consideration.  So, as problem solving IT thinkers, we need to think around this problem.

Do you have any friends that work at Awesome Company, LLC?  Are they friends with the IT guys, per chance?  Ask your friends if they’d be willing to hand your resume to the team that’s doing the hiring along with a recommendation.  (NOTE: Don’t sound needy!  It’s a good way to sour friendships.)  Some people advocate saying “Hi James, I noticed your company is hiring for a position that I’d be great at, and I’d really love to get an interview.  Would you put my resume on Tim The IT Director’s desk for me and possibly put in a good word or two on my behalf?”  This can really work.

Network, Network, Network!

There’s another thing we can do to increase our chances of getting hired in the IT community: go meet people!  This is called “Networking” to the sales world and is critical in getting their job done.  I do know that a lot of us consider ourselves antisocial, but realistically, if all of us “antisocial” types get together, we generally find things to talk about.  Don’t be afraid to approach people, please!  This process only works when you are the one taking initiative.  Here are some options for networking:

  • Conferences (LISA, LOPSA-East (nee PICC), Cascadia, Velocity, many others) are great places to network with other people and rub shoulders with the giants of our industry.  For instance, Vint Cerf will be delivering the keynote at LISA this year.  If you ever wanted to meet him, if you were at LISA you could just go up and say hi.
  • Local user groups (Like LOPSA presentations, Linux User Groups, Windows User Groups, Tech Meetups) help a great deal as well, and for most of us without a big pocketbook, they’re much cheaper to attend than a conference.  You still get an opportunity to meet a gaggle of new people, and that’s helpful when you’re looking for a job.  I know that LOPSA especially, at least the chapter that I belong to, asks if anyone has job openings they’d like to talk about.  If someone’s asking for something you can do, go talk to them!  You’ve just instantly gotten ahead of the game by meeting the possible stakeholder(s) before HR has had the chance to accidentally delete your e-mailed resume.

Show the world how awesome you are.

An oft overlooked part of the hiring process is the inevitable google search that your potential employer is going to do.  What kind of stuff are they going to find?  Are your Facebook photos from your buddy’s beer pong tournament open to the public?  This is what could be called a “negative public artifact.”  Some tech companies would find it funny, but stuffier business types would consider that an immediate turn off.  On the other hand, if you have a blog full of insightful comments on the industry and diary entries of your (mis)adventures in technology, you show that you’re not simply a piece of paper with a lot of certifications, but someone who actually produces good things.  This is where being a member on a Stack Exchange site is very helpful.  If you mention you’re a serverfault user on your resume and provide the link to your profile, your potential employer has one place to look for all of your questions and answers to judge how well you know your stuff.

Fun With Resumes

I know a lot of us are holding onto the same resume format we had when we started, and are just adding fields as necessary; it’s not always a smart idea.  The job market and what people are looking for on resumes has changed since some of us started looking for jobs.  The emphasis is shifting from “show your experience” to “show how you’re awesome;” if you want your resume to get a second glance, it has to clearly and concisely illustrate why you’re a slam dunk.

1972-1977  Sanford and Son, Salvage

  • Drove a salvage truck
  • Excellent problem solver

If your resume has sections like the above, you are boring the heck out of the person reading your resume.  Sure, it shows what you literally did in that job, but it doesn’t convey what you’re proud of.  Did you make the salvage equivilent of the Kessel run in less than 12 parsecs?  How is the potential employer going to know about this if you didn’t say so?  Lets try this again:

Sanford and Son, Salvage (1972-1977)

  • Frequently noted as having the best one-liners in the company
  • Kept a 1952 Ford Pickup in running order much longer than previously considered humanly possible
  • Voted the fastest salvager on the east side after completing the Kessel Run in under 12 parsecs! (City record!)

This is not unusual advice.  I’m sure many of us learned it in school when they talked about writing resumes.  At the very least, we’ve spiced up an otherwise dull list of abilities/accomplishments.  Why stop there, we can still do better:

Sanford and Son, Salvage (1972-1977)

Accomplishments

  • Ran the Los Angeles Kessel Run in under 12 blocks, beating the standing record of 16 blocks. (1972)

Accolades

  • In Trash We Trust Award – 1973
  • Salvage Monthly’s “Top of the Heap” – 1975

Skills Utilized

  • Advanced engine mechanics and collision repair
  • Navigation of South-Central Los Angeles by automobile and commercial trucks
  • Lifting heavy loads

If we look at the above, two things are important to note.  First, it has easy to read sections.  Second, it has superfluous space taken up by sections.  If you were to put the whole body of your experience down in this format, you’d have a 4-5 page resume.  Some places might like this, but the common knowledge is 2-3 pages, max.  Without some massaging, your long-winded resume will be too troublesome to read.  Why?  Oftentimes we’re handed a stack of resumes and after a while, it all gets so repetitive that you confuse which applicant had which experience.  The key to resolving this is massaging your work experience to be relevant to the job that you’re applying for.

Tailor that resume, chief

I know that it’s hard work to tailor resumes for every position, but at the very least you should have a few tailored resumes in the areas that you are most proficient.  For example, I have three separate resumes that I give out if I’m applying for a developer, IT, or management position.  I’ve done all three jobs, so I have three tailored sets of job experience that subtly (or perhaps not-so-subtly) emphasize those roles in the full body of my experience.  Sometimes I leave a job out that isn’t relevant to the position, but I’ve found some interviewers are taken aback when you mention you left out some job experience.  Why would  you care if I slung lattes at a coffee bar while in high school, I ask?

Cover letters are mandatory

You might have heard people say that cover letters are dying out in this age of digital resumes.  They’re lying so they can take your jobs.  Every single resume you sent out needs a cover letter, and it needs to be customized to the company you want to work for.  I know that this is a tall order for some, but if you’re a prudent applicant you’ve already studied the company you’re applying for and know what they do or provide.  Why not explain to the person who’s going to hire you why they should waste their day interviewing you?  The cover letter is your “First Impression” on paper.  It helps to show how eloquently you convey yourself on paper, and it’s  pretty important in tehnology jobs because a lot of our jobs involve some level of technical writing.  Ignore this advice at your peril.

Please provide your starting salary…

Yeah, if you want to get priced out of the market.  The answer to that question should always be “we can negotiate that after the interview when you’re sure I’ll be a great fit!”  If they press, try to dodge as much as possible.  The salary question allows them to automatically throw out resumes.  If you come in low, they think you’re not in touch with the job requirements or the job market.  If you come in high, they’ll think you’re either presumptuous or literally are too experienced and therefore too expensive to hire.

Your resume will give them an accurate idea of how much they should be paying you just by the virtue of how much experience you’re showing.  If you’d like more details on this, check out this year’s Robert Half Salary Guide.  This gives an accurate starting point to figure out how much money you should be asking for or how much they should be paying you.  However, don’t tell them this number (or your imaginary What-Am-I-Worth-A-Year number) until after they’ve decided they want to hire you.

Closing thoughts

I don’t want to end this on a somber note, but please keep in mind that if you’re trying to get interviews at companies where you have no inroads or networking contacts, please be ready to get a callback from 1 out of every 20 or more resumes you send.  It is a cut-throat job market and there are hundreds if not thousands of people applying for the same positions you are, but they’re better at bullshitting on resumes than you are.  It’s tough to get a job without having friends in the industry already.  Don’t get discouraged!  Do what I advise above, especially on the socializing front, and you’ll be much better off.

Appendix 1: What a well-formed Systems Administrator job reference might look like

I was asked to provide an actual SysAdmin-focused job description to help some people along.  Below is a completely fictitious entry that I’ve generated off the top of my head.

Senior Systems Administrator, ImaginedCompany, LLC (2008-Present)

Relevant Technologies

  • Advanced Active Directory provisioning and maintenance for a 100,000 user company
  • Deployment automation via Windows Deployment Services in Windows and KickStart in Linux
  • vSphere setup, troubleshooting and administration with multiple Datacenters and Hosts
  • Deep Telephony – PRI, TDM, FXO/FXS setup and maintenance
  • Configuration Automation via Puppet and Systems Center
  • Revision control with Mercurial, Git and Subversion

Accomplishments

  • Independently developed the Windows Deployment Services installation for ImaginedCompany, decreasing provisioning time from hours to minutes.
  • Converted legacy servers from bare-metal hardware to vmware, saving the company thousands of dollars a year in electricity
  • Quickly reimplemented a PBX solution when the legacy telephone system hardware failed, enabling the company to continue to make money until a permanent solution was in place.
  • Led-by-example with new project to use source and configuration management by way of Mercurial and Puppet/Systems Center

Accolades

  • 2008 – LOPSA Most Active Mentor
  • 2009 – Golden Baseball Bat recipient for best client-facing customer service.
  • 2011 – Employee of the Year

An update on Sandy

Nick Craver

We’re still up and running in Oregon, and have powered down all services in our New York datacenter at 75 broad.

While we’re not actually running anything out of the datacenter, here’s what’s happening:

After our last update the Peer1 datacenter continues to stay online any way they can.  When Sandy hit, their basement fuel store was contaminated and the pumps necessary to get it up 17 stories to the generator went offline.  Since then employees from several companies hosted there have shown up and continue fueling the generator via bucket brigade.

Our sister company Fog Creek has many hands down there right now, you can follow their status updates here.  Squarespace is in the exact same boat as well, you can see their updates here.

The concern has been fuel supply to carry upstairs, as I’m writing this a truck has pulled up and the barrels are filling.  Several of those companies with people on the ground have been fetching fuel from gas stations for the past few hours.

While this is happening the basement is being pumped out so that the facility can be repaired and grid power restored.  We don’t have an ETA at this time.

We’ll try and post an update in a few hours.