Some Notes on Setting Up Backups with Bacula
Kyle Brandt
Our main backup system for files is Bacula running on Ubuntu server. I was able to get it up and going quickly but it has taken a while to get it running fair smoothly and doing what I want. My main impression of Bacula is that it favors flexibility over ease of use. So I wanted to share some of my discoveries in this process that might help others setting up and working with Bacula for the first time.
Our Current Setup and keeping it simple
Hardware:
- Tandberg 8 Tape StorageLoader that includes one LTO-4 drive
- Dell R610 with 209 GB of storage
With these resources George and I had a couple discussions about how the backups would work. We had a long conversation about how were were going to implement Grandfather-Father-Son (GFS) with Bacula:
Grandfather-father-son backup refers to the most common rotation scheme for rotating backup media. Originally designed for tape backup, it works well for any hierarchical backup strategy. The basic method is to define three sets of backups, such as daily, weekly and monthly. The daily, or son, backups are rotated on a daily basis with one graduating to father status each week. The weekly or father backups are rotated on a weekly basis with one graduating to grandfather status each month. Often one or more of the graduated backups is removed from the site for safekeeping and disaster recovery purposes.
Once we started to get pretty deep into how this would work with the recycling of tapes we realized that we just don’t need to be doing this. Instead we are going to have full backups and daily differential backups on the disk (possibly overflowing into a tape pool) and then just copy the most recent full backups to a tape once a week and take that tape off site once a week. This is right, for us, for the time being because:
- To keep a year of offsite tapes with a one year retention isn’t that expensive
- There is less room for error or complications with reusing the tapes
The lesson for us was to KISS (Keep it Simple Stupid) and to not over engineer our backups.
Don’t Make File Storage Volumes too Big
File storage is represented in Bacula as if it were tape backup solution. Your filesystem is basically the loader and each file on the filesystem that holds backed up data is like a tape. You can make these volumes any size you want really, if there isn’t room for a backup it goes onto the next one. However, do not make them too big. When you start out making your backup selections (what you want to backup) you might grab things that are big and that you don’t want. The problem is that if these are in the middle of a volume from what I can tell there is no easy way to just erase that job or file in the middle of the volume, so you will have to wait for the volume to become available for reuse (recycling) and it will sit there taking up space. If they are small you can manually recycle the volumes with the job that grabbed something large that you didn’t want to back up.
Know the Retention Periods
Bacula has three retention Periods:
- File
- Job
- Volume
When the file retention period has passed the records of files will be purged from the catalog and you will only be able to restore whole jobs. When the job retention has passed you will only be able to restore backups by scanning the whole tape. When volume retention passed and you are recycling volumes the backup might be overwritten.
Copy Jobs
The copy job is how I replicate the full backup to the offsite tape. This took me a little while to figure out as the documentation focuses more on the migration jobs. The main two points for me were that there has to be a bunch of Null configuration items so the job passes the configuration check and that you specify the destination pool of the copy job in the pool configuration of the source pool.
Here is a configuration for copy jobs that will copy the full backups from the past 2 days to my tape pool:
Job { Name = "OffsiteCopyFull" Type = Copy Pool = WeeklyFile Schedule = "OffsiteFull" Client = None Fileset = None Selection Type = SQLQuery Selection Pattern = "select * from Job where (EndTime BETWEEN NOW() - INTERVAL 2 DAY AND NOW()) AND Level = 'F' AND (JobStatus = 'e' OR JobStatus = 'T') AND PoolID = 8;" Maximum Concurrent Jobs = 1 Allow Duplicate Jobs = Yes Messages = Standard Storage = File Write Bootstrap = "/var/lib/bacula/offsite-copy-%c.bsr" }
Fake fileset for copy jobs
Fileset { Name = None Include { Options { signature = MD5 } } }
Fake client for copy jobs
Client { Name = None Address = localhost Password = "NoNe" Catalog = MyCatalog }
Pool { Name = WeeklyFile Pool Type = Backup Recycle = yes # Bacula can automatically recycle Volumes AutoPrune = yes # Prune expired volumes Volume Retention = 14 days # one year Maximum Volume Bytes = 10G # Limit Volume size to something reasonable Maximum Volumes = 15 # Limit number of Volumes in Pool Next Pool = WeeklyOffsiteTapes #This is for the copy job }
Configuration Layout
Instead of one massive file you can have includes in the main bacula director configuration:
@/etc/bacula/backupjobs.conf @/etc/bacula/copyjobs.conf @/etc/bacula/schedules.conf ...I find this keeps things a little bit neater in general.
Use JobDefs
Jobdefs allow you to define a template (directives that will be defaults to all jobs that inherit that job) for jobs. This makes for a cleaner configuration in the same way using inheritance in Nagios does.
Use Different Pools for Full and Differential Backups
I think using different file pools for full and differential (or incremental) backups makes sense for at least two reasons:
- If you need space you might want to blow away some of the older full backups when you have more recent ones. So for the same reason as using small volumes having different pools can make volumes easier to recycle.
- It makes sizing a little bit clearer. All you need is the du command to see how much your full vs differential backups are taking.
To specify pools for different types of backups is simple, just put something like the following in your JobDefs or Job configuration section:
Pool = WeeklyFile Full Backup Pool = WeeklyFile Differential Backup Pool = DailyFile
Take Advantage of the SQL Backend
Since the catalog is built on SQL you can create your own queries. I chose to use MySQL and can interact with catalog using the console. Also you can run predefined queries by adding them to the /etc/bacula/scripts/query.sql file and then using the query command from bconsole.
As an example I added the following to my query.sql to the average size of full backups per client:
:Average Size of Full Backups per Client SELECT Client.Name, AVG(Job.JobBytes) as Avg, STDDEV(Job.JobBytes) as StandardDev FROM Job INNER JOIN Client ON Job.ClientId=Client.ClientId WHERE Level = 'F' GROUP BY Client.Name ORDER BY Avg DESC;You could also add something like
AND (EndTime BETWEEN NOW() - INTERVAL 1 WEEK AND NOW())to the WHERE clause if you want to limit it to say the past week.
Final Thoughts:
As always with backups:
- Run test restores. Time is always limited but you need to run at least a few to sleep well at night.
- Check the backup logs regularly.
Pingback: Tweets that mention Some Notes on Setting Up Backups with Bacula - blog.serverfault.com -- Topsy.com()
Pingback: Exectweets » spolsky at 01/10/11 05:09:09()
Pingback: Sysadmin Sunday #14 « Boxed Ice Blog()