RRD graphs are used by a lot of monitoring tools such as Cacti, Munin, and n2rrd (the Nagios plugin we are currently using). Editing RRD graph templates yourself isn’t terribly hard and is worth while since they are part of many of the graphing solutions available. Also by learning how to write the graph templates you can create your own graphs combining different data with complete freedom.

Bigger is Better:

At Stack Overflow we are spoiled with 30 inch monitors with a 2560×1600 resolution. Looking at a Cacti instance, which I think is a default setup, has graphs of 603×280. If everyone has the screen real estate don’t waste it and go ahead and make these graphs a lot bigger so you can see more information. Consider doubling or even tripling the size.

When making them bigger think also about the ratio of width to height. If you make the graph a tall one but not as wide it will exaggerate value differences better if that is what you are looking for.

Consider hard limits for vertical boundaries:

If you have a spiky graph, but care more about seeing detailed changes in the normal values the automatic graph scaling can get out of preportion for the information you are trying to get. What happens is the graph zooms out so the spike data is included fully in the graph. You can put hard limits on the y-axis values and allow the spikes to get cut off so you get more detail on the information you actually want. You can do this with the rigid option:

-u 300
-l "-300"
--rigid

Show values as negatives:

This could go either way as it confuses some people, but I find it helpful. You can “CDEF” a value to make it negative and make the graph less cluttered.

So for example:

CDEF:NegWrites=Writes,-1,*

I find that this works well for graphs with two values that generally go together and will never have actual negative values such as read/writes, network input/output, and GET/POST HTTP methods. However if you find this inverse visualization confusing then it is best to just differentiate in other ways.

Use trends:

If a graph is too spiky to see the patterns consider using the trend function. Over larger periods of time RRD graphs naturally eliminate the spikes as data is consolidated, however trends are very helpful for seeing patterns in shorter time frames. The trend creates a sliding window average of a specified time frame. So for example to get the 12 hour trend in the above graph I used the following:

"CDEF:ReadTrend=Reads,43200,TREND"

You can read more about the details of this function and other functions like it in the graph_rpn documentation.

Include helpful calculations:

One of the nice things with RRD graphs is that you can do calculations and display them at the bottom. A common one is to include the the 95th percentile on Internet bandwidth graphs since this is a common method of billing.

For my read/write graphs I set up a write percentage and a read percentage:

RRD does use RPN (Reverse Polish Notation) but writing rules such as these is not too bad once you play around with the notation a little.

"CDEF:IOPS=Reads,Writes,+"
"CDEF:PerReads=Reads,IOPS,/,100,"
"CDEF:PerWrites=Writes,IOPS,/,100,"

They are actually useful:

Before I made these changes the default graph I had for both read/writes per second and the amount of time they took was:

Sure it looks like a graph — but it is pretty useless.

I picked these graphs as examples not only because they show my points about the graph tips but also because they show the results of our recent 32 GB database RAM upgrade. As expected right after the upgrade there was a drop in both the amount of reads on our database drive and the amount of time reads take. This is because more of the data being read is hitting the memory cache. Also, since less reads are being done there is less load on the array and the reads are faster. This didn’t have any significant impact on writes since all writes need to be flushed to disk (or at least the RAID/Disk cache which is the disk as far as the OS is concerned). In theory I would think less reads would alleviate some of the write pressure as well on the array — but this doesn’t seem to be the case to any significant extent (I would have to make the graph adjusted against something like the amount of traffic or DB queries to see a change if there is one). Had I stuck with that useless graph then this change would not have been obvious without some data analysis.

«
»
  • jscott

    “CDEF” a value to make it negative…

    Completely unexpected, but incredibly delightful. I will be implementing this.

  • Pingback: Tweets that mention Some Tips for Better RRD Graphs - blog.serverfault.com -- Topsy.com

  • http://users.elis.ugent.be/~wheirman/wim/ Wim

    I like to use a logarithmic Y axis for things like access times (-o or --logarithmic switch, usually together with -r, -l and -u for rigid bounds). This allows me to see both the .5 -> .2 second optimization I made, as the worst-case spikes, in the same graph.

  • http://anthony.derobert.net/ Anthony

    Rather than (or in addition to) number of reads/writes per second, the wait time would probably be useful. Same with the percent of the time there is a request outstanding.

    On Linux you can get these from iostat -x, /sys/block/DEV/stat, or /proc/diskstats, not sure where you get it on Windows.

  • Pingback: What to do about Storage? - blog.serverfault.com

  • http://www.webthingsconsidered.com ryanw

    Keep up the great posts. I’m curious, are you using a nagios plugin to capture sql server metrics? Which one do you use?

  • Farhad Arshad

    My rrd_graph function is showing boxes(garbage value boxes) instead of titles in graph

    <?php

    $sourcerrd = &quot;/home/rrd/dd.u/2189_udosti_new_sub.rrd:ds:AVERAGE&quot;;

    $filename = &quot;graphs/2189_udosti_new_sub.png&quot;;

    $title = &quot;Ufone Dosti&quot;;

    $color =&quot;#FF0000&quot;;

    $opts = array(

        &#039;--start=-6400&#039;,
    
        &#039;--end=-30&#039;,
    
        &#039;--title=&#039;.$title,
    
        &#039;--alt-autoscale-max&#039;,          
    
        &#039;--base=1000&#039;,
    
        &#039;--height=400&#039;,
    
        &#039;--width=600&#039;,
    
        &#039;--lower-limit=0&#039;,
    
        &#039;--units-exponent=0&#039;,
    
        &#039;--vertical-label=&#039;,
    
        &#039;DEF:a=&#039;.$sourcerrd,
    
        &#039;LINE2:a&#039;.$color.&#039;:Live Charging&#039;
    

    );

    $result = rrd_graph($filename,$opts);

    if(!$result){

     echo rrd_error();
    

    }else{

    echo &#039; ';
    
    echo '';
    

    }

    ?>