If you are really observant you may have noticed in my RRD graph post that our write times on our database server are not so hot:

Also looking at this from SQLs perspective confirms this:

Our sequential log writing however is really fast (<10ms), but there is a write performance problem when it comes to our database files that we need to fix.

Too Many Variables

Figuring out what direction to go in with our storage is proving to be quite challenging. The main reason is that there are just so many variables:

• Cost and The Total Cost of Ownership (TCO)
• What will satisfy our needs and prepare us for growth?
• What gives the best performance for our workload and how is that workload going to change?
• Reliability and fail over options
• What direction is storage heading in as far as technology goes
• What fits our scaling pattern

Right now we are thinking about 3 main options:

• A PCI card storage solution like a FusionIO drive
• A SAN
• SSDs in the servers themselves

However, before looking at these options there are some fundamentals and issues to consider when it comes to storage.

The Problem with Figuring Out what is Needed for an IO Workload:

When it comes to analyzing IO workload there are some basic questions that must be answered:

How Much?

There are two main measurements used to answer the question of “how much?” with storage. There is how much data is moved within a certain amount of time (MB/s) and/or how many logical operations there are (IOPS or input/output operations per second). When looking at the workload the amount of operations in the queue is also considered for data that has a steady rate.

How Much of What?

The operations are either going to be reads or writes. The reads or writes will also be either sequential or random. So you end up with some blend of the following four possibilities:

• Sequential Reads
• Sequential Writes
• Random Reads
• Random Writes

What is the Shape Over Time?

Sometimes the shape of IO will be a steady stream of data, but often disk operations will get batched to make the IO more efficient. This means that the shape of the IO will usually be spiky on a micro level (say over several seconds). There will also be a shape on the macro level because most services have peak usage times and there are also scheduled IO intensive operations such as backups.

How Fast?

With whatever workload, operations need to be fulfilled within a certain amount of time. If the disk IO system is busy, then operations have to wait in a queue to be fulfilled.

My main point with all of this is that it is not advisable to just take an average of the amount of IOPS and Megabytes per second over a day and go buy a storage solution. This does not account for how fast these IOPS are satisfied or the shape of data over time. Even if these are taken into account, and taken into account correctly, the workload still should be tested on the actual equipment. The best an analysis can do is give a hypothesis. This leaves two possible courses of action:

1. Take an educated guess and buy something
2. Set up a demo unit or go to a demo location and load test the application

The main problem with option number 2 is that this is a large time investment, and that you have to have the capability to load test your application in a way that accurately reflects real world usage.

My Personal Gripe with Storage Vendors

My gripe is that most large or fast storage solutions can not be purchased at Micro Center. More seriously, even when talking to a sales person on the phone they won’t say what all the options are and what they really cost. Sometimes there is an option to price them out on the website, but the real cost is what the sales person will knock it down to. The sales people want the data I was just talking about (often a limited subset which will only give them a rough ballpark). I have always hated this sales method, I want to see what I get for certain costs — not tell them how much I can spend and have them tell me after that.

The other big thing is that vendors are not public with their actual performance numbers. The SPC1 benchmarks are the best effort I have seen to provide useful information, but the amount of devices in that repository is limited. To a degree this is understandable given all the different workloads as I mentioned, but some basic numbers on various workloads under a Raid 10 configuration would be nice. In other words “Give me some data, please”.

Our Particular Situation and Scaling Model

The sheer size of the stackoverflow.com database compared to our other sites at the moment is a major factor for us. Looking at the above image the amount of IOPS on the stackoverflow.com database is 30 times the amount of IOPS for the superuser.com database. Because of this treating stackoverflow.com’s database as a separate entity for the other databases does make sense.

We also don’t need that much capacity. Going off of Nick Craver’s growth analysis the SO Database will grow from 85 GB to 256 GB over the next 36 months (note: this is just a projection).

I have mentioned this before but our model has been to strike a balance between scaling up and out. We are not particularly attracted to building giant monster systems, nor do we want a bunch of cheap little boxes. We want a balanced amount of medium powered servers. In my mind this fits well with a Microsoft stack.

The Current Options

So taking the above into account here is my current thinking on what the following options might mean for us. We don’t have any demo units in hand yet but our plan is to evaluate FuisionIO as the PCI card option, Equallogic as the SAN option, and Dell approved SSDs put into our current Dell r710 database servers.

Option 1: FusionIO

Pros:

• FusionIO is going to be the fastest option out there. To quote Brent Ozar in his review of FusionIO: “The only way to outperform a Fusion-IO drive is to invest six figures in a SAN and hire a really sharp SAN admin.”
• Simplicity. There is a lot that goes into configuring a SAN correctly, with FusionIO we would copy our database file to the FusionIO drive and be done.

Cons:

• Limited single system availability. There doesn’t seem to be a simple RAID equivalent. For each single server there will likely be only one of these cards in each server. Two can be put in a server and set up to use software RAID but I wonder if that might just end up lowering the availability. In theory since these are solid state devices and not mechanical I would expect to have better reliability than hard drives, but the technology is still fairly new.
• Limited multi-system availability. Any sort of SQL clustering options are out the window and what you have left is log shipping and synchronous or asynchronous mirroring.

I think the FusionIO option fits our scaling model well. We currently have two DB servers — a primary and a fail over. We are planning on expanding to 4 servers so Stack Overflow (and maybe the rest of the original trilogy) can have its own primary and secondary server. There are different options, but one 640GB FusionIO would cover growth for the trilogy and provide the fastest speed compared to a SAN or SSDs. We could then have asynchronous mirroring to the secondary server and in a failover situation Brent Ozar estimated 90 seconds of data loss. Downtime might be around 30 minutes until we get the secondary server up and going manually. We generally favor speed over the highest possible uptime. It is not that we are glib about the uptime of our service, but we don’t have the uptime requirements of a financial institution. For our sites with higher uptime requirements such as Careers we can use the storage in the servers and possibly synchronous mirroring. I also imagine either this or SSDs in the servers themselves will be the cheapest solution — the 640GB drives were quoted at about 10k each for us and the 320GB at about 6.5k from a vendor.

Option 2: A 10GE EqualLogic San with some SSDs

Pros:

• Flexibility in growth and tiered storage. With a SAN we can add shelves as we grow and can tier our storage effectively. So for example we could have an SSD array for the trilogy and a SAS array for our smaller sites. As our sites grow we could move them accordingly. We could also use storage for logs or tempdb.
• Flexibility in availability options. Unlike a fusion drive or SSDs in the server clustering options are now open to us.
• 10GE might be useful for other things if we start to hit network bottlenecks. This is the main reason why 10GE appeals to us more than fiber.

Cons:

• The SAN as a logical unit is a single point of failure unless you buy two. I know these have lots of built-in redundancy but nonetheless our current thinking is that we would want two if we went the SAN route.
• Cost. These things are not cheap. The EqualLogic PS6010S with 8 SSDs is priced at 46k on their site. The redundant 10GE switches if we go with Dell would be about 20k. So without even factoring in other various total cost of ownership factors if we want two SANs we are talking well over 100k. That would be the same cost as getting at least 8 more of our current database servers.

The flexibility and growth options that a SAN offers are appealing. The cost could drastically change if we decided we could live with one SAN, look at different vendors, or give up on the option of having SSDs in the SAN. The performance won’t be as high as the FusionIO would be but for our workload that extra performance might not really matter.

Option 3: SSDs in the Dell Servers

This option would be pretty similar to the FusionIO option except that there will be a trade off of an increase in single server availability options but a decrease in speed. With SSDs we can use a traditional RAID configuration in the servers. These drives on Dell’s site are 4.6k for a 2.5 inch SAS 3 Gbps drive, so for a mirror of two drives it would be $9.2k and would give us 150GB capacity. This wouldn’t leave us much room for growth so we would probably want 300GB capacity. In RAID 10 that will cost about 18.4k. At the moment I don’t have any data on how this would perform but with the current cost of the Dell approved SSDs for our servers this option isn’t too appealing to me yet.

Conclusion

Right now we are still in the preliminary stages. Initially I am fond of the FusionIO option where as George is more in favor of a SAN. One of the main reasons I favor FusionIO at the moment is that it would satisfy our growth in the short term, interestingly enough this is one of the reasons George is less fond of it. George’s main reason for a SAN is that we get greater flexibility with the features a SAN offers such as the ability to fully cluster, use snapshotting, full LUN replication, and dynamic expansion (on some models). By going the SAN route earlier and not later we don’t put off solving the problem until about a year from now when it might be harder to change our infrastructure. If we were to get a SAN now we will learn how to use the advantages given to us by the SAN’s flexibility. I generally agree with this philosophy but not in this particular case. The reason I favor putting off a SAN is that I feel in a year or so the SAN options that include SSDs might be a lot cheaper and more attractive. Also the FusionIO option fits well with our current scaling model. Although the growth of our Stack Exchange sites looks very promising to me, I feel it is too early to predict how they will grow. This is not so much in terms of visitors but more in terms of the IO workload growth. Our developers could make changes that greatly effect our IO workload. So we might have a much better understanding of what we need 6 months to a year from now than we do currently.

What we really need is to get more data on the performance of these various options and get our hands on some demo units. I feel like all 3 of these paths are valid options. Also there are valid options we haven’t looked as closely at yet (for example SANs that don’t support SSDs). At this point it is clear that choosing a storage route to take is no small task.

As a system administrator, participating on Server Fault has made me better at what I do and it has also been fun. Here’s why.

The All-Star IT Department

In many tech companies the IT department is pretty small. Even in the bigger ones you get more specialized admins, say a team of two network admins, a Unix guy with a beard and a Windows lady. That means when you have ideas that you want to bounce off of people or questions that you have, there is only a small group of people you can go to, and even then an even smaller chance that they will know what you are talking about.

Server Fault is a combination of a big IT department and a knowledge resource. Most of the visitor traffic is people finding answers to questions that they have that someone else already asked. However, there is also a community here. The top administrators on the site get to know each other and many also hang out in chat. Spending time with other people who are passionate about what you do improves the quality of your day to day life.

So in my mind Server Fault is an all-star IT department that I get to work with all year round.

The advantages of writing

One of the main things that people value in a system administrator is communication. It seems that a lot of system administrators are just not very good at it for some reason. Whatever that reason is doesn’t matter as much as the fact that writing will make you a better communicator. I would not promise that it will make you a great writer but it will make you a lot better. If you not convinced that this is the case have a look at What Stack Overflow Can Teach You.

When you write out a question it clarifies what you are thinking about. Before you even get an answer from someone else you might have an idea as you write as to what the answer is, or you might realize that you are going down the wrong path.

Staying on top of your game

In the same way that writing a question helps clarify your thoughts, the same is true of writing answers. Every time you write an answer you refresh your memory on the topic and make the thought more defined and fleshed out. In doing this not only does your knowledge on that particular topic get better but also your mental models about system administration improve. Much of system administration has themes that come up and up again with many different technologies. Mastering these themes means improving your mental models that you apply to learn new things. As a result you end up with more facility in that particular area but also at the general patterns that come up in system administration. I believe this is what many people mean when they talk about experience, so in a way you can get more experience in a shorter amount of time.

The site also helps you keep track of what the new technologies are in a field that is always changing. When you see lots of questions coming up on a topic over time you know that you should at least know what it is. You can then take a look at these and be more in touch with what is going on in your field.

Showing off and helping others

It is fun as a geek to show off your knowledge to others and helping other people with their problems satisfies a fundamental desire. Also Sever Fault is usually less rewarding to people who show off their knowledge in an unkind way (i.e. RTFM). Because of this you might develop a better attitude as a system administrator — another thing that the stereotypical system administrator is not always famed for.

It is worth pointing out that people will also help you with your problems. When you are stuck on something you can ask the question, go work on something else, and come back and you will likely have an answer.

A personal and public repository of system administration knowledge

When you answer a question on Server Fault, you not only help the person asking the question, but you also help lots of other fellow system administrators who find that answer through a Google search which saves them time and frustration. That can also be true for yourself, answering questions or having your own questions on Server Fault is like keeping a notebook. I have actually used Google to look up something I did not know and found the answer on Server Fault that I myself wrote but had forgotten.

You might get a better job

By participating on Server Fault you will likely grow as an administrator in your technical knowledge and facility as well as your communication skills. With a body of answers you have written on Server Fault you can demonstrate to any future employers that you are interested in what you do beyond it just being your day job. Next time you walk into an interview you might get asked a question you answered before on Server Fault and will have a lot of confidence in answering that question well. If you are asked about something you don’t know you can have an intelligent conversation with the interviewer about it as he explains it to you since your mental models about system administration will have improved. This will demonstrate that you learn things quickly.

Server Fault is also a way to have a public presence on the web that is a little bit easier to do than a blog. If you do this over time employers will start to seek you out and you might not even have to go looking for your next job.

There are lots of different reasons people spend time in our community. If you are already one of them I hope that this might remind you why you do it or maybe gives you more reasons as to the value of being here. I personally greatly appreciate learning from all of you and have enjoyed getting to know the members of this community. If you haven’t spent much time with us I hope you consider giving our community a try or even just stopping by once in a while.

RRD graphs are used by a lot of monitoring tools such as Cacti, Munin, and n2rrd (the Nagios plugin we are currently using). Editing RRD graph templates yourself isn’t terribly hard and is worth while since they are part of many of the graphing solutions available. Also by learning how to write the graph templates you can create your own graphs combining different data with complete freedom.

Bigger is Better:

At Stack Overflow we are spoiled with 30 inch monitors with a 2560×1600 resolution. Looking at a Cacti instance, which I think is a default setup, has graphs of 603×280. If everyone has the screen real estate don’t waste it and go ahead and make these graphs a lot bigger so you can see more information. Consider doubling or even tripling the size.

When making them bigger think also about the ratio of width to height. If you make the graph a tall one but not as wide it will exaggerate value differences better if that is what you are looking for.

Consider hard limits for vertical boundaries:

If you have a spiky graph, but care more about seeing detailed changes in the normal values the automatic graph scaling can get out of preportion for the information you are trying to get. What happens is the graph zooms out so the spike data is included fully in the graph. You can put hard limits on the y-axis values and allow the spikes to get cut off so you get more detail on the information you actually want. You can do this with the rigid option:

-u 300
-l "-300"
--rigid

Show values as negatives:

This could go either way as it confuses some people, but I find it helpful. You can “CDEF” a value to make it negative and make the graph less cluttered.

So for example:

CDEF:NegWrites=Writes,-1,*

I find that this works well for graphs with two values that generally go together and will never have actual negative values such as read/writes, network input/output, and GET/POST HTTP methods. However if you find this inverse visualization confusing then it is best to just differentiate in other ways.

Use trends:

If a graph is too spiky to see the patterns consider using the trend function. Over larger periods of time RRD graphs naturally eliminate the spikes as data is consolidated, however trends are very helpful for seeing patterns in shorter time frames. The trend creates a sliding window average of a specified time frame. So for example to get the 12 hour trend in the above graph I used the following:

"CDEF:ReadTrend=Reads,43200,TREND"

You can read more about the details of this function and other functions like it in the graph_rpn documentation.

Include helpful calculations:

One of the nice things with RRD graphs is that you can do calculations and display them at the bottom. A common one is to include the the 95th percentile on Internet bandwidth graphs since this is a common method of billing.

For my read/write graphs I set up a write percentage and a read percentage:

RRD does use RPN (Reverse Polish Notation) but writing rules such as these is not too bad once you play around with the notation a little.

"CDEF:IOPS=Reads,Writes,+"
"CDEF:PerReads=Reads,IOPS,/,100,"
"CDEF:PerWrites=Writes,IOPS,/,100,"

They are actually useful:

Before I made these changes the default graph I had for both read/writes per second and the amount of time they took was:

Sure it looks like a graph — but it is pretty useless.

I picked these graphs as examples not only because they show my points about the graph tips but also because they show the results of our recent 32 GB database RAM upgrade. As expected right after the upgrade there was a drop in both the amount of reads on our database drive and the amount of time reads take. This is because more of the data being read is hitting the memory cache. Also, since less reads are being done there is less load on the array and the reads are faster. This didn’t have any significant impact on writes since all writes need to be flushed to disk (or at least the RAID/Disk cache which is the disk as far as the OS is concerned). In theory I would think less reads would alleviate some of the write pressure as well on the array — but this doesn’t seem to be the case to any significant extent (I would have to make the graph adjusted against something like the amount of traffic or DB queries to see a change if there is one). Had I stuck with that useless graph then this change would not have been obvious without some data analysis.

When it comes to system administration there are a two main questions that come up in relation to RAM:

• Which and how much RAM do I buy?
• How do I properly install it?

In order to be able to figure this out there are some fundamentals regarding RAM which a sysadmin has to know for servers. To be practical lets figure out what a particular stick of RAM is about (Which happens to be in our database servers):

> Registered ECC Dual Rank 8GB DDR3-1333 (PC3-10600)

Frequency and Bandwidth:

Wikipedia tells us that “frequency is the number of occurrences of a repeating event per a unit of time.” In the case of memory the “repeating event” we are concerned with is bits of data being sent from the cpu to RAM and the “unit of time” is a second. With memory this is done so many times we measure it in millions of times per second (aka megahertz or MHz). The bits are chunks of 64 bits which are often referred to as “words”. So the transfer frequency of our example RAM is 1333 MHz (the clock frequency is different, more on this later).

So how “fast” is this? In computing, speed is often measured in how much data you move in a certain amount of time (There is also latency). A common measurement is the amount of megabytes in a second. There are 8 bits per byte. We said before that this RAM can push 64 bits 1,333 million times per second. So 64 * 1,333,000,000 / 8 gives us 10,664,000,000 bytes a second, or roughly 10,600 MegaBytes per second. So we now have our bandwidth, 10,600 MB/s:

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

DDR3:

DDR stands for Double Data Rate. In the case of DDR memory the data transfer event actually manages to carry two chunks of data. With DDR2, you get 4, and with DDR3 you get 8. This does not include the rate of operations, but rather the amount of data chunks carried in each operation. What we end up having is different frequencies going on — clock ticks from the memory controller (which correspond to operations), bus and cpu frequencies, and finally an effective transfer frequency. These frequencies are all related to each other since this memory is synchronous. At the heart of all of this is the system clock, but memory is labeled by the transfer frequency because higher numbers sell better and it is probably the number most people would be interested in anyways. So the 10,600 MB/s throughput is based off of the transfer frequency and therefore takes DDR3 into account.

When it comes to “PC3” I am not entirely sure what that stands for, but DDR2 = PC2 and DDR3 = PC3. I am hoping a reader knows what it is for sure and can back it with some evidence.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Ranks and Capacity:

A rank is a independently accessible 64-bit area of memory. Today you buy memory in single, double, and quad ranks. To really get into ranks you will need to understand the basic signals used in ram which you can find in the “What every programmer should know about memory” in the referecences and then there is some detail on ranks in Addressing Increased Capacity Demand Using Commodity Memories. As a system administrator the main thing to understand is that there may be limitations on your motherboard such as the ability have mixed ranks and where certain ranks can go (On the R710, Quad-Ranks must go in the first DIMM slot of each channel).

The capacity is “8GB” — simply how much memory there is on the chip.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Registered and ECC

Registered RAM is also referred to as buffered RAM. It has a buffer between the memory controller and the memory chips. The end result is that you can have a higher capacity of RAM in the system but there is a little more latency. Memory you buy for servers will generally be registered memory. ECC stands for Error Correcting Code and includes parity on the memory to help ensure that the memory operation is valid.

> Registered ECC Dual Rank 8GB DDR3-1333 PC3-10600

Architecture, Channels, and the i7:

In the frequency and bandwidth section I mentioned that 64 bit chunks are being sent at a time. These chunks are being sent over a channel. When you have multiple channels the basic idea is that you are increasing your width — so dual channel is chunks of 128 bits. The current Intel i7 architecture supports 3 channels per processor (as in a chip you would hold in your hand). So in a dual processor system you have up to six channels. So with our simplified model we are talking rougly 30 GB/s of memory throughput per processor (10,600 * 3).

With recent architectures the memory was connected to the processor via the Front Side Bus (FSB). However with the i7 the memory controller is on the processors themselves and the memory is directly connected. There is also QPI (Quick Path Interconnect) which connects the PCI express bus to the processor. In multiple processor systems it also connects the processors together, so if Processor B wants Processor’s A memory it goes over the QPI.

There is a lot more to learn about memory. See the references section at the end of this post. Reading and writing about memory is difficult because there are a lot of differences between the general theory and what is actually happening so I and others skip over details. Reading the references or at least parts of them is probably needed to have a good picture of memory functions. You might also want to read about signal timings and interleaving, but I think the above covers the basics for a generalist system administrator.

Trade Offs — Channels, Capacity, and Speed:

So why does all this matter to a system administrator? The fundamental answer is that there are tradeoffs. Besides cost you have memory speed, error handling, and total capacity tradeoffs in the configuration you choose.

In our current database server, Dell R710, we are using the Intel 5600 series architecture. We have 2 processors which each have 3 channels of memory directly connected to each processor for a total of 6 channels. There are three general RAM population choices (Reference: Dell Poweredge 11g Whitepaper):

1. Optimize for speed (throughput). This configuration does not give the most RAM in the system in terms of capacity, but does give the fastest throughput possible for the memory. For the Dell R710, 4GB Registered DIMMs achieve this with a maximum amount of memory of 12 GB per CPU. This will let you run your memory at 1333 MHz.
2. A balance of speed and capacity. For this you would want dual-rank 1066 GB registered DIMMs. This allows for a capacity of 48GB per CPU but the memory can be clocked at 1033 MHz.
3. Optimize for capacity. With this option you would use the same chips as the balanced configuration but they get down clocked to 800 MHz for a total of 144 GB capacity.

According to Intel you get the best speed out of a balanced three channel configuration at this capacity. We just upgraded to 96 GB from 64 GB and have gone from a 2 channel configuration to a 3 channel configuration. This upgrade requires that the memory gets down clocked from DDR3-1333 to DDR3-1033. Although the clock speed has decreased for our memory, the number of channels or width has increased along with our capacity. Doing the math this in theory yields more memory throughput:

Two Channel:
(128 * 1,333,000,000) / 8 = 21,328 MB/s
Three Channel:
(192 * 1,033,000,000) / 8 = 24,792 MB/s

So our 3 channel configuration in our server is as follows:

So we match exactly option number 2 currently. Part of the reason you buy from a vendor like Dell is that they should be helping you get the ideal configuration, but knowing your RAM at a basic level will help make sure you are getting what you need.

Update:

George had informed me that he saw a memory speed 1333 MHz at boot. After looking at the output of CPU-Z and seeing that his was confirmed there as well I became confused as to how we could have 1333 MHz at a 96GB capacity:

After speaking with Dell I found that there was an information update regarding system memory for the R710. For 2R DIMMs they clock at 1333 not 1066 when you have 2 DIMMs per channel. So the proper calculation for our configuration is

(192 * 1,333,000,000) / 8 = 31,992 MB/s

This means that there was no trade off in clock speed when upgrading our capacity and to three channels.

References:

Kingston Technology memory ranking technical brief
What every programmer should know about memory — Highly Recommend, for those that want a lot of detail (PDF Version).
Memory technology evolution: an overview of system memory technologies, 8th ed.

I am going to be doing a bit of DB maintenance tonight at 9PM Eastern. The sites will be down for about an hour while I do a couple of upgrades to our primary database hardware:

1.  Add 32 GB of RAM to the server to bring it up to 96GB of RAM
2. Run Firmware updates for the RAID card
3. Update the RAID Card Driver firmware

The Trilogy and SE sites will be down for about an hour while I do this work. If the outage is going to last longer than that I will update this post.

Also, sorry for not posting this earlier — I’ll try to get these notifications up sooner than an hour before work is scheduled to start in the future.

We are going to take a break from our regularly scheduled program to ask a very important question!

I’ve looked a few times, and there are precious few “SysAdmin” conferences. Most of them are very *nix centric. I’m also very interested in Networking, Windows, emerging technologies and all of that fun stuff.

What I have found happens in reality is there are vendor or developer focused conferences with a SysAdmin type track that you can follow. Which when I think about it for a while is probably a good thing. In my mind it allows me to sneak off the beaten path and go check out what the devs are doing. I feel it’s important to not only stay abreast of sysadmin developments, but also keep up with technology in general.

I’ve been to some really good smaller conferences run by vendors that seemed like they would be just one big sales pitch (ok, maybe they were) but they turned out to have some really great breakout sessions run by some really smart people. On the other side of the equation I’ve been to some large conferences that were just awful, where I didn’t pick up anything that wasn’t already general knowledge and everyone just sat there looking bored out of their minds.

So I wanted to run a little informal poll of you, the Great All Knowing Internet.  What conferences to you look forward to going to year after year?  What do you like about them? Also in a more general sense, what are the pros and cons of going to conferences at all? I want to harness this internet power for nothing but good I swear!

Right now Kyle and I have a short list of conferences we would really like to attend:

• O’Reilly Velocity 2011

• NJ-LOPSA PICC

• Tech Ed

• LinuxCon

Also, in addition to conferences you get excited to go to every year, are there any you are especially excited about this year? Let us know over on this Server Fault question: http://meta.serverfault.com/questions/1089/conferences-to-go-to-2011-edition

Please let us know in the comments what conferences you attend every year – maybe we’ll see you there!

Our main backup system for files is Bacula running on Ubuntu server. I was able to get it up and going quickly but it has taken a while to get it running fair smoothly and doing what I want. My main impression of Bacula is that it favors flexibility over ease of use. So I wanted to share some of my discoveries in this process that might help others setting up and working with Bacula for the first time.

Our Current Setup and keeping it simple

Hardware:

• Tandberg 8 Tape StorageLoader that includes one LTO-4 drive
• Dell R610 with 209 GB of storage

With these resources George and I had a couple discussions about how the backups would work. We had a long conversation about how were were going to implement Grandfather-Father-Son (GFS) with Bacula:

Grandfather-father-son backup refers to the most common rotation scheme for rotating backup media. Originally designed for tape backup, it works well for any hierarchical backup strategy. The basic method is to define three sets of backups, such as daily, weekly and monthly. The daily, or son, backups are rotated on a daily basis with one graduating to father status each week. The weekly or father backups are rotated on a weekly basis with one graduating to grandfather status each month. Often one or more of the graduated backups is removed from the site for safekeeping and disaster recovery purposes.

Once we started to get pretty deep into how this would work with the recycling of tapes we realized that we just don’t need to be doing this. Instead we are going to have full backups and daily differential backups on the disk (possibly overflowing into a tape pool) and then just copy the most recent full backups to a tape once a week and take that tape off site once a week. This is right, for us, for the time being because:

• To keep a year of offsite tapes with a one year retention isn’t that expensive
• There is less room for error or complications with reusing the tapes

The lesson for us was to KISS (Keep it Simple Stupid) and to not over engineer our backups.

Don’t Make File Storage Volumes too Big

File storage is represented in Bacula as if it were tape backup solution. Your filesystem is basically the loader and each file on the filesystem that holds backed up data is like a tape. You can make these volumes any size you want really, if there isn’t room for a backup it goes onto the next one. However, do not make them too big. When you start out making your backup selections (what you want to backup) you might grab things that are big and that you don’t want. The problem is that if these are in the middle of a volume from what I can tell there is no easy way to just erase that job or file in the middle of the volume, so you will have to wait for the volume to become available for reuse (recycling) and it will sit there taking up space. If they are small you can manually recycle the volumes with the job that grabbed something large that you didn’t want to back up.

Know the Retention Periods

Bacula has three retention Periods:

• File
• Job
• Volume

When the file retention period has passed the records of files will be purged from the catalog and you will only be able to restore whole jobs. When the job retention has passed you will only be able to restore backups by scanning the whole tape. When volume retention passed and you are recycling volumes the backup might be overwritten.

Copy Jobs

The copy job is how I replicate the full backup to the offsite tape. This took me a little while to figure out as the documentation focuses more on the migration jobs. The main two points for me were that there has to be a bunch of Null configuration items so the job passes the configuration check and that you specify the destination pool of the copy job in the pool configuration of the source pool.

Here is a configuration for copy jobs that will copy the full backups from the past 2 days to my tape pool:

Job {
        Name = "OffsiteCopyFull"
        Type = Copy
        Pool = WeeklyFile
        Schedule = "OffsiteFull"
        Client = None
        Fileset = None
        Selection Type = SQLQuery
        Selection Pattern = "select * from Job where (EndTime BETWEEN NOW() - INTERVAL 2 DAY AND NOW()) AND Level = 'F' AND (JobStatus = 'e' OR JobStatus = 'T') AND PoolID = 8;"
        Maximum Concurrent Jobs = 1
        Allow Duplicate Jobs = Yes
        Messages = Standard
        Storage = File
        Write Bootstrap = "/var/lib/bacula/offsite-copy-%c.bsr"
}


Fake fileset for copy jobs

Fileset {
  Name = None
  Include {
    Options {
      signature = MD5
    }
  }
}

Fake client for copy jobs

Client {
  Name = None
  Address = localhost
  Password = "NoNe"
  Catalog = MyCatalog
}

Pool {
        Name = WeeklyFile
        Pool Type = Backup
        Recycle = yes                       # Bacula can automatically recycle Volumes
        AutoPrune = yes                     # Prune expired volumes
        Volume Retention = 14 days         # one year
        Maximum Volume Bytes = 10G          # Limit Volume size to something reasonable
        Maximum Volumes = 15               # Limit number of Volumes in Pool
        Next Pool = WeeklyOffsiteTapes          #This is for the copy job
}

Configuration Layout

Instead of one massive file you can have includes in the main bacula director configuration:

@/etc/bacula/backupjobs.conf
@/etc/bacula/copyjobs.conf
@/etc/bacula/schedules.conf
...

I find this keeps things a little bit neater in general.

Use JobDefs

Jobdefs allow you to define a template (directives that will be defaults to all jobs that inherit that job) for jobs. This makes for a cleaner configuration in the same way using inheritance in Nagios does.

Use Different Pools for Full and Differential Backups

I think using different file pools for full and differential (or incremental) backups makes sense for at least two reasons:

• If you need space you might want to blow away some of the older full backups when you have more recent ones. So for the same reason as using small volumes having different pools can make volumes easier to recycle.
• It makes sizing a little bit clearer. All you need is the du command to see how much your full vs differential backups are taking.

To specify pools for different types of backups is simple, just put something like the following in your JobDefs or Job configuration section:

 Pool = WeeklyFile
 Full Backup Pool = WeeklyFile
 Differential Backup Pool = DailyFile

Take Advantage of the SQL Backend

Since the catalog is built on SQL you can create your own queries. I chose to use MySQL and can interact with catalog using the console. Also you can run predefined queries by adding them to the /etc/bacula/scripts/query.sql file and then using the query command from bconsole.

As an example I added the following to my query.sql to the average size of full backups per client:

:Average Size of Full Backups per Client
SELECT Client.Name, AVG(Job.JobBytes) as Avg, STDDEV(Job.JobBytes) as StandardDev
 FROM Job INNER JOIN Client ON Job.ClientId=Client.ClientId
 WHERE Level = 'F'
 GROUP BY Client.Name
 ORDER BY Avg DESC;

You could also add something like

AND (EndTime BETWEEN NOW() - INTERVAL 1 WEEK AND NOW())

to the WHERE clause if you want to limit it to say the past week.

Final Thoughts:

As always with backups:

• Run test restores. Time is always limited but you need to run at least a few to sleep well at night.
• Check the backup logs regularly.