A lot of tools available in IT/Sysadmin/Ops/DevOps are disappointing:

  • They don’t fit your environment. They lack features or our designed for a different sort of environment (i.e cloud vs hardware, Linux vs Windows, distributed vs centralized etc)
  • You can’t interact with them programmatically
  • They cost too much
  • They are not customizable enough, or require too much customization to get off the ground
  • Feel kludgy, unreliable, outdated, or like the programmers were stoned
  • Don’t fit with your company’s culture (i.e. Enterprise vs Agile)

In short a lot of stuff is too expensive, isn’t a good fit, or is simply bad software. This ends up leaving an ops team with two options. They can whine about it, or create their own tools. So at Stack Exchange we build our own DevOps tools.

Status

Nick Craver’s baby, which we just call “Status” is at first glance a monitoring dashboard, but is essentially a collection of tools that filled various needs:

  • An Overview of CPU, Memory, and Network utilization for all our servers as well as a detailed view. Done with responsive and interactive D3 graphs as well as sparklines it helps compensate for Solar Wind’s terrible interface. statussqlscreen2-png
  • SQL Server monitoring. SQL’s built in Clustering views are deeply flawed. If a node loses connectivity, it stops updating remote nodes status, so it could show everything as connected and fine, even if there is no connectivity. We also get to see the most expensive queries, active queries utilizing whoisactive, current connections, and which DBs are on which server
  • HAProxy Monitoring and Administration: With multiple instances of HAProxy we needed a single view instead of HAProxy’s built-in display. Also, this gave us a nice web interface to take servers out of rotation statusdashboardscreen-png
  • Redis: A nice presentation of Redis Info across all instances and all servers. Also a display that shows what is slaved to what in at a quick glance
  • Elastic Search: Health overview of or clusters (as well as index and shard data)
  • A dashboard of all the exceptions generated by our applications

Status is C# / .NET app. It polls data from various sources – sometimes the system directly and other times it gets it from Orion. There is a lot more to status that makes it awesome. The real accomplishment is that status enables us to see the general health of our main infrastructure at a glance.

Web Logging

If you business is creating and running websites, your web logs are gold. We use the logs generated by our load balancer, HAProxy, as our canonical web logs. In their raw text format, web logs are often not that useful (this is particularly true with over 100 million records a day). However we parse and structure our web logs in a few different ways:

realog.ds.stackexchange.com

  • We have C# service that Jarrod Dixon wrote that inserts them into SQL so we can query them. In order to query them we use an instance of Data Explorer, SQL management studio, and also have certain lookups directly from our sites
  • Displaying realtime graphs of various log information with Realog, a system I created with Go, Redis, and NVD3.js so we could view activity live without having to write queries

One of the interesting things we do with our weblogs is to add extra information by adding headers inside the app and striping them from the response at HAProxy. For example, we capture how many Redis and SQL queries were involved in that request and how long they took.

Patch Dashboard

OS updates can be a bit tedious, even more so in a mixed Windows and Linux environment. PartialPatchDashboard Steven Murawski and George Beech created a dashboard that allows us:

  • View the outstanding patches and patch count for both Linux and Windows
  • Trigger updates on either Linux or Windows
  • Schedule time frames for automatic Linux updates

What’s Next

If you want to learn more about these tools and DevOps at Stack Exchange, come see George, Nick, and Steven present “Building for Operations” at Velocity.

Keeping all this stuff to ourselves feels a bit greedy. However, for something open sourced to be very useful it usually needs to be made a bit more generic which takes time. We also want to build a lot more. Our inventory system Racktables lacks an API so we need a new one or a way to extend it. We want to build our own monitoring system (likely on top of OpenTSDB). In order to create more, and open source it we need help. So we are looking a full time developer with ops experience to join our SRE team. So if you are awesome, want to build awesome ops stuff and open source it, come join us!

«
»
  • ChrisF
    • Kyle Brandt

      That was stupid of me, should be okay now. Thanks a bunch for pointing that out Chris!

  • Mike Robinson

    I really like the Status overview screen – it’s clean, well aligned, and easy to scan. Definitely a good example of how to design a dashboard. The SQL screen is still a bit lacking though. I see lots of numbers and was wondering what the max values were, and how they relate to each other. Is 105/138 conns/sess good or bad?

    • cdhunt

      That’s always the question and a dashboard like Status helps you figure that out for your application/environment.

    • Status will allow setting thresholds both globally and per instance for warnings, so you control your environment since what’s “right” really depends on your environment. If a server goes red we’ll tell you why it’s red in a very simple way via balloon right on the dashboard, with a link to drill down to the actual problem if we have a screen that provides more detail on that issue. For instance, we’d link to the connection list that shows connections on that instance and what queries they’re currently running for your example.

  • danlucraft

    Is Realog open source?

    • Kyle Brandt

      Not yet, but it is on my wishlist todo. Getting this stuff open sourced is one of the reasons we want a full time dev (see the link at the end of the post). To make realog opensource I would need to: – Get some proper Config files going (i.e. the redis server isn’t in the code) – I think I have a memory leak I need to figure out – Write up some deployment instructions – Clean up the more embarrassing sections in the code 😉

  • Oskar Gewalli

    I’ve tried doing logging easier for web apps using elastic and log4net: https://github.com/wallymathieu/ElasticElmah

  • Pingback: Homegrown DevOps Tools at Stack Exchange | Rocketboom()

  • Jeff Moser

    This sounds great!

    It’d be nice to have an open source offering that works well with Windows machines and combines the types of things New Relic does for performance monitoring (especially its profiling support), Splunk does for logging, and Pager Duty for alerting.

    One relatively simple feature you could borrow from Etsy is to integrate it with deploy info so that deploys appear as colored vertical lines on all the graphs so you can visually tell if a deploy messed something up.

    Another nice term feature would be to have something like Etsy’s Kale (Skyline & Oculus) for anomaly detection.

    • Kyle Brandt

      “Status” actually does plot builds on the graphs as points (which show the build # etc on hover).

      The anomaly detection would be a nice addition at some point. I replicated a simple version of their methodology (StdDev’s from avg) in SQL queries against Orion. I’m also interested in things like making autocorrelation tables etc (Done a little bit of that by sucking all of Realog’s Redis data into Pandas). There are often a bunch of “obvious” correlations, but when you see things that are correlated that maybe should be (i.e. the number of hits on a certain HTTP route and the number of 5xx status codes) it can highlight an issue.

      • Jeff Moser

        It’d be nice to make this anomaly detector as its own DLL/project. It could even work on something like IObservable of TimeSeriesValue of T making it play nice with the Reactive Extensions monad ecosystem.

        I’d be interested in helping out with statistics/machine learning parts of this piece if it’s open sourced and the rest of the Status tool works in our environment.

    • We actually pull builds from TeamCity and put them on our graphs for the same reason. I’d like the integration to be better since we narrow it to builds that actually affected THAT server, but haven’t found a cleaner way that allows the narrowing down we need as well.

      Since we deploy many, many times a day across many projects, showing them all isn’t an option really. I’m wide open to thoughts on this, as we’d want it to work with any number of deployment systems with data available as well.

      • Jeff Moser

        Perhaps each “metric” (plottable thingy) can be tagged with multiple tags.

        You could tag metrics by project (web, api, etc) as well as having a hierarchical “instance” tag of that metric (i.e. per server in a farm).

        Then your deploys could also be tagged by project and optionally by instances they’ve landed on (and when!).

        The deploy lines would appear if the deploy tags matched the metric tags (i.e. web == web).

        They could also be roll-up savvy so that if you have a metric that isn’t instance specific (i.e. it’s roll-up of all instances), you’d see all deploy lines of that project (or coalesce to a single bar). However, if the metric is instance-specific, you’d only see it if all tags, including the hierarchical tags matched (i.e. web == web && instance==”NY-WEB01″)

  • Kit

    This is wonderful stuff and I can’t wait to see more of it! I work as a DBA in an operations team and put pride in building my own tools to monitor SQL Server outages (monitoring of services like SQL Agent, SQL Service) and also SQL job monitoring from database servers that require immediate attention. Such tools, customized to the department’s needs and requirements, greatly improves our productivity.

  • Pingback: Interne Tools – billig und geschmacklos, richtig? | Code-Inside Blog()

  • Justizin

    Did anyone else stop reading at “SQL Server” ? 😉

    • siculars

      The StackExchange folks are a well known Microsoft shop. Windows, MS SQL server, C#, etc. Might not be for you, as it is not for me, but don’t hate.

    • carlivar

      I’m a long-time UNIX bigot but I’ll take SQL Server over Oracle any day.

      • AndrewBurton

        HERE, HERE!

  • Damian Harvey

    What is the time x-axis: “33:10”? does it roll left to right and that is 33 hours 10 minutes from now?

    • Kyle Brandt

      The x-axis is Minute:Second of Wall clock time. Realog is designed to be entirely ephemeral and currently only has a 30 minute retention. Because all of our weblogs are sent to SQL, Realog is just a supplement.

      The retention period is mostly easily changed (There are a few variables to adjust in the project, one of the things that would be cleaned up before open sourcing).

      Why 30 Minutes?

      • Realog “buckets” everything into 10 second data points (this can also be adjusted fairly easily, but not clean yet). With multiple JS graphs updating automatically every few seconds, 30*6=180 datapoints is a fair amount for the browser to process. When I do a refresh, I do a union of the updated data so I am not fetching all 180 points, but the entire graph redraws. Getting around this would need more advanced D3.js skills than I have (either replacing/abandoning NVD3.js or modifying it). Also longer periods mean a spike is going to distort the graph. I do have “hidden” query parameters that limit the amount of time displayed so this could be expanded upon.

      • One of the features “Top IPs” Tracks bandwidth and bytes per IP for the past 1 minute, 5 minute, 10 minutes, and 30 minutes. Calculating this and updating it frequently is done using ZSETs in Redis. This is a pretty expensive operation inside redis even though I have done a couple of optimizations: Culling the lists to keep the sets smaller and using the already calculated smaller ZSETS (For example, 10 minutes is done by using the 5min zsets, and the 6,7,8,9, and 10 minute ZSET). Since this all runs in a VM right now, extending this would be pretty taxing. If I really wanted to do this, a second Redis instance for the ZSET processing would allow this not to interfere with other processes (Redis is single threaded). I could have different retention settings for the TOP IPs feature and everything else. So a possibility, but out of scope based on my available time at the moment.

      • More retention equals more memory. I’d like to push Realog to its limits some day, but since we have SQL we don’t really need it, so low priority for me.

      • Lastly, and I know this really pretty lame 😉 but keeping it under an hour allowed me to ignore trying to worry about timezones (D3.js’ parse function currently doesn’t support timezones (%Z).

  • DougN

    Nice article Kyle — especially like the charts. I have a specific question. I work at a tiny competitor of SolarWinds, so when you said their interface is terrible, that really got my attention. Would you mind telling me what you don’t like about it so we can be sure and not make the same mistake (or fix it if we already have)?

    • I can’t speak for Kyle, but for me the Orion UI is terribly slow and non-responsive. It really doesn’t follow many common usage patterns.. like the node search doesn’t actually search if you type a term and hit return, it just refreshes the page. The web UI doesn’t expose all the management functionality either.. you need about 16 other separate GUI apps on the server to configure a number of things.. including alerting.

      • DougN

        Thanks for the input Steven. Sounds like we’re good then 🙂

  • aater

    Stay on top of evolving cloud storage pricing models As a faculty at Austin University and CEO Flux7 Labs, I am holding some discussions this Tuesday specifically on cost effective cloud architecture. Inviting Managers and Cloud professionals. http://ohours.org/aatersuleman Looking forward to your presence.

  • Zhoudong

    Does this scale up properly when connected to a JQuery server?

    • Thisis Myname

      what?

  • kyro

    Any reason why you don’t put this up on github/somewhere and let people see what it looks like now? I know you say needs to be more generic, but couldn’t that be done over time while the community can submit patches/improvements ?

  • Jason Ashby

    I really dig the Patch Dashboard and could use that in my environment. Are there any existing open source solutions similar to this? I’m hoping you guys get this on github. What language is it built on?

  • We’ve been wanting something like this at Parse.ly for a long time. We’ve been making due with dashboards built using Graphite, and then the usual soup of tools like Munin, NewRelic, Pingdom, PagerDuty, Sentry/Raven, etc. But one dashboard to rule them all would be awesome! It seems like this would be easier to do these days thanks to Chef / Cloud servers, but we still haven’t found a good open source project to host it all. OpenTSDB does, indeed, look very promising as a generic place to put time series monitoring data.

  • Nicky Helmkamp
  • tukuktuk

    kfykkutkkluk dfsdfsdf dfsdfsdf

  • Slawa

    Is the Status available (going to be available) as an open-source project? Is it again a .NET project?

  • Bob

    So did any of this ever get opened up or has it fallen into the vanity trap?

  • Meerkat Monitor

    Meerkat-Monitor http://meerkat-monitor.org/ Free tool for monitoring network services. It has notifications and allows integration with CI tools and scripts.Check it out!

  • ankso

    There is any clue if sometime you will add also a documentation about the opserver suite? Thanks.

  • ankso

    Hello, can you inform how to create the authentication for every database i have? because i have databases in different domains so the Service Account doesn’t work. Can i set them up somewhere in the configuration files for every sql another login credentials and if yes, where do i have to setup up them and how? Thank you. Awesome stuff.

  • mohammadpouya
  • heshmat
  • sofia
  • aroosha
  • shirinazar

    I think it is useful for each user..کرکره برقی

  • amir