Backups are just one of the many responsibilities of system administrators. IT Generalists have many areas to cover so they probably don’t take the time to make spreadsheets to measure the cost of data loss as they might in The Enterprise. However, investing time in trying to place a value on your backups can provide perspective on just what a terrific responsibility backups can be.

At Stack Exchange, I view our users and our user contributed content as the company’s most valuable asset. We have a lot of talent in the company, and our user contributed content isn’t even our direct source of revenue. However, if this data were totally lost (or a large portion of it) I have trouble envisioning how the company could bounce back from that. In addition to this, as a user myself I value this data as something we have created together that has intrinsic value for our professions.

Measuring Value

There are lots of ways to measure value. The obvious method is to use traditional business methods that put a dollar value on your company. When it comes to Stack Exchange some people somewhere put a big dollar value on the company which they call our valuation and in theory they don’t just make this up. If I accept that the loss of our user contributed content is the loss of the company, I could just say that our valuation is the value of our backups. The problem is that valuations tend to be pretty big numbers, and the abstraction there just doesn’t speak to me.

Also from a business perspective I can use the $18 million of VC funding we have taken and use that as a basis for value of our backups. That is a lot of money and I can’t help but start to feel the sense of importance of these backups. However, there is still a lot of abstraction there. The point of this exercise is to really feel the responsibility and not just be intellectually aware of it.

Another way to measure value is time. Our users and coworkers collectively have invested incredible amounts of time into our sites. I am user and know many of our users so I know that what we have created is important to us. I don’t have an accurate way to measure this, but I can do a back of the envelope calculation for Stack Overflow. To be conservative, looking only at the 1.4 million accepted answers on stackoverflow.com the total word count is about 100 million. According to Wikipedia people write about 19 words per minute, but I will assume people on SO are faster and can compose about 40 words per minute. That gives us 100,000,000 words / 40 words per minute / 60 minutes per hour / 24 hours a day / 365 days a year =~ 5 years of non-stop skilled work. Now I realized this calculation is perhaps, a bit, well, hair-brained, but it is reasonable for my purposes.

Another aspect to take into account is the profit generated by Stack Exchange. I don’t mean profit in the traditional sense, rather I look at what I call time profit. When a user answers someones question, they not only saved that person time but many other people who will eventually search for the same question and find that answer. This saves those people time. Because of this our sites like Stack Overflow are systems where the output is greater than the input. So in this sense of time profit, if our content was lost, future potential time profit would be lost.

We all have different ways of perceiving value. I value what our users and my coworkers have created, and when I attempt to measure just how much has been created, it becomes very apparent that safe guarding that creation though backups is an awesome responsibility.

Our main backup system for files is Bacula running on Ubuntu server. I was able to get it up and going quickly but it has taken a while to get it running fair smoothly and doing what I want. My main impression of Bacula is that it favors flexibility over ease of use. So I wanted to share some of my discoveries in this process that might help others setting up and working with Bacula for the first time.

Our Current Setup and keeping it simple

Hardware:

• Tandberg 8 Tape StorageLoader that includes one LTO-4 drive
• Dell R610 with 209 GB of storage

With these resources George and I had a couple discussions about how the backups would work. We had a long conversation about how were were going to implement Grandfather-Father-Son (GFS) with Bacula:

Grandfather-father-son backup refers to the most common rotation scheme for rotating backup media. Originally designed for tape backup, it works well for any hierarchical backup strategy. The basic method is to define three sets of backups, such as daily, weekly and monthly. The daily, or son, backups are rotated on a daily basis with one graduating to father status each week. The weekly or father backups are rotated on a weekly basis with one graduating to grandfather status each month. Often one or more of the graduated backups is removed from the site for safekeeping and disaster recovery purposes.

Once we started to get pretty deep into how this would work with the recycling of tapes we realized that we just don’t need to be doing this. Instead we are going to have full backups and daily differential backups on the disk (possibly overflowing into a tape pool) and then just copy the most recent full backups to a tape once a week and take that tape off site once a week. This is right, for us, for the time being because:

• To keep a year of offsite tapes with a one year retention isn’t that expensive
• There is less room for error or complications with reusing the tapes

The lesson for us was to KISS (Keep it Simple Stupid) and to not over engineer our backups.

Don’t Make File Storage Volumes too Big

File storage is represented in Bacula as if it were tape backup solution. Your filesystem is basically the loader and each file on the filesystem that holds backed up data is like a tape. You can make these volumes any size you want really, if there isn’t room for a backup it goes onto the next one. However, do not make them too big. When you start out making your backup selections (what you want to backup) you might grab things that are big and that you don’t want. The problem is that if these are in the middle of a volume from what I can tell there is no easy way to just erase that job or file in the middle of the volume, so you will have to wait for the volume to become available for reuse (recycling) and it will sit there taking up space. If they are small you can manually recycle the volumes with the job that grabbed something large that you didn’t want to back up.

Know the Retention Periods

Bacula has three retention Periods:

• File
• Job
• Volume

When the file retention period has passed the records of files will be purged from the catalog and you will only be able to restore whole jobs. When the job retention has passed you will only be able to restore backups by scanning the whole tape. When volume retention passed and you are recycling volumes the backup might be overwritten.

Copy Jobs

The copy job is how I replicate the full backup to the offsite tape. This took me a little while to figure out as the documentation focuses more on the migration jobs. The main two points for me were that there has to be a bunch of Null configuration items so the job passes the configuration check and that you specify the destination pool of the copy job in the pool configuration of the source pool.

Here is a configuration for copy jobs that will copy the full backups from the past 2 days to my tape pool:

Job {
        Name = "OffsiteCopyFull"
        Type = Copy
        Pool = WeeklyFile
        Schedule = "OffsiteFull"
        Client = None
        Fileset = None
        Selection Type = SQLQuery
        Selection Pattern = "select * from Job where (EndTime BETWEEN NOW() - INTERVAL 2 DAY AND NOW()) AND Level = 'F' AND (JobStatus = 'e' OR JobStatus = 'T') AND PoolID = 8;"
        Maximum Concurrent Jobs = 1
        Allow Duplicate Jobs = Yes
        Messages = Standard
        Storage = File
        Write Bootstrap = "/var/lib/bacula/offsite-copy-%c.bsr"
}


Fake fileset for copy jobs

Fileset {
  Name = None
  Include {
    Options {
      signature = MD5
    }
  }
}

Fake client for copy jobs

Client {
  Name = None
  Address = localhost
  Password = "NoNe"
  Catalog = MyCatalog
}

Pool {
        Name = WeeklyFile
        Pool Type = Backup
        Recycle = yes                       # Bacula can automatically recycle Volumes
        AutoPrune = yes                     # Prune expired volumes
        Volume Retention = 14 days         # one year
        Maximum Volume Bytes = 10G          # Limit Volume size to something reasonable
        Maximum Volumes = 15               # Limit number of Volumes in Pool
        Next Pool = WeeklyOffsiteTapes          #This is for the copy job
}

Configuration Layout

Instead of one massive file you can have includes in the main bacula director configuration:

@/etc/bacula/backupjobs.conf
@/etc/bacula/copyjobs.conf
@/etc/bacula/schedules.conf
...

I find this keeps things a little bit neater in general.

Use JobDefs

Jobdefs allow you to define a template (directives that will be defaults to all jobs that inherit that job) for jobs. This makes for a cleaner configuration in the same way using inheritance in Nagios does.

Use Different Pools for Full and Differential Backups

I think using different file pools for full and differential (or incremental) backups makes sense for at least two reasons:

• If you need space you might want to blow away some of the older full backups when you have more recent ones. So for the same reason as using small volumes having different pools can make volumes easier to recycle.
• It makes sizing a little bit clearer. All you need is the du command to see how much your full vs differential backups are taking.

To specify pools for different types of backups is simple, just put something like the following in your JobDefs or Job configuration section:

 Pool = WeeklyFile
 Full Backup Pool = WeeklyFile
 Differential Backup Pool = DailyFile

Take Advantage of the SQL Backend

Since the catalog is built on SQL you can create your own queries. I chose to use MySQL and can interact with catalog using the console. Also you can run predefined queries by adding them to the /etc/bacula/scripts/query.sql file and then using the query command from bconsole.

As an example I added the following to my query.sql to the average size of full backups per client:

:Average Size of Full Backups per Client
SELECT Client.Name, AVG(Job.JobBytes) as Avg, STDDEV(Job.JobBytes) as StandardDev
 FROM Job INNER JOIN Client ON Job.ClientId=Client.ClientId
 WHERE Level = 'F'
 GROUP BY Client.Name
 ORDER BY Avg DESC;

You could also add something like

AND (EndTime BETWEEN NOW() - INTERVAL 1 WEEK AND NOW())

to the WHERE clause if you want to limit it to say the past week.

Final Thoughts:

As always with backups:

• Run test restores. Time is always limited but you need to run at least a few to sleep well at night.
• Check the backup logs regularly.