What’s your backup backup?

Ouch.

I was logging in to the MovableType web interface to delete some comment spam, but the login page failed to be displayed – instead I got an error telling me that Postgres didn’t know anything about my MT database.

So, I logged in to the server and asked for a list of the databases available, and sure enough, Postgres had become absent-minded and forgotten them all. More specifically, the recent Debian upgrade of the Postgres package had not smoothly migrated the data. I recall doing the upgrade (it was only a couple of days ago), and everything seemed fine, but I trusted what the upgrade told me and didn’t check any of the databases or tables. Ooops. Time to get worried – all the databases on that machine can be rebuilt from source files (they are test installations of sites in development), except for my blog. The last dump I have of MovableType is from the middle of last year.

This is where it gets really entertaining. I was zapping comment spam to kill time whilst waiting for a disk copy to finish (24gb off a slow USB disk). Today I’m doing some serious disk shuffling, getting backups off one disk onto another, so I can repartition my external drive and use it to clone the Powerbook. Since my Powerbook’s combo cd drive is dead, I want a backup way of booting, and cloning seems to be the answer. In order to do this, I’ve deleted all but the most recent backups of various machines… so finding a dead database sent shivers down my spine.

Fortunately, Debian made a backup of the Postgres data directory prior to upgrading. After switching files around, this seems to be working. But it takes a nasty bump like that to remind me to not only do backups on a regular basis, but to check they can be restored from easily.

So now I’ll do what I’ve been meaning to do for ages, and get pg_dump backups instead of filesystem-level backups sorted out, and to fully automate all the backups so I have proper rotations and don’t have to think about it. I’ve already got a spare server on the end of a very fast line set up in the office (geographical displacement, since our primary is in Docklands). I just need to populate it with some huge disks and set things going.

A slightly more reassuring “restore” story: I nuked my Windows XP laptop install in favour of Windows 2003 whilst testing deployment of some of our software just before Christmas. Prior to doing so, I used the ubuntu installation also on the machine to backup the XP drive:

dd if=/dev/hda1 of=/backup/xp.img

This image included such essential stuff as my Half-Life 2 saved game, so I was somewhat nervous about being able to restore it. I’m happy to say that after four hours of disk thrashing last night (uh, yeah, I forgot to turn DMA on), it’s back up and running:

dd if=/backup/xp.img of=/dev/hda1

It will be nice when vendors and operating system developers (proprietary and open) come up with faultless simple ways of doing backups. I’m puzzled that such a critical component of computing is left to third-party solutions.

Related posts:

1. Mac Backup
2. Have you done your backup today?
3. XML vs SQL, RDBMS vs NXD
4. Enough already
5. Still no mail