Optimal remote backups with rsync over Samba

Backing up a remote Linux machine, when all you have... is Windows

(October 2007)

A little history...

The company I work for, is a small startup. By definition, startups are forced to work with limited budgets, limited resources, limited staff - which makes them perfect places for people like me. Why? Well... Let's just say that my coding skills span over a... wide range of "stuff". I have worked on almost everything you can think of, from mini OSes for tiny embedded MPC860s and the Windows device drivers that talked to them, to full fledged database-driven MFC and C++ Builder GUIs for chemical plants... I enjoy diversity, and I can suggest no better way of keeping your mental health and staying sharp, than working in a startup.

(Just make sure you have loads of patience... you'll need it :‑)

One of the roles I double (or is it triple? quadruple? :‑) for, is that of "the crazy Unix guy" (tm). Put simply, when the services offered by my company involve Unix technology, advanced text processing and/or scripting, I am frequently "called to the rescue". It is in this context that I was called to manage a USA-based Virtual Dedicated Server (VDS) coupled with a CentOS installation (i.e. a Linux distribution). The VDS is used for the hosting (web site, e-mails, etc) of our company products.

Initially, I tried to avoid as much effort as possible :‑) I opted for a small web and e-mail hosting package, that gave no shell access. Unfortunately, I quickly found out that there were lots of other sites co-hosted by our provider on the same IP address (our own). Some of them were spammers - or more probably, their users were infected by malware that sent spam. This made our e-mail domain a permanent resident of Spamhaus, which occasionally caused mails sent by us to travel to Never-never land. In the end, we were forced to move - to our very own server, with an IP address which was ours and ours alone.

That immediately took care of our Spamhaus issues. Moreover, SSH access allowed me to tinker with the machine, and setup many things I deemed to be necessary. Over the course of weeks, I fine-tuned the installation: I got acquainted with bind, and made our DNS server report SPF records, so no spammers could use our e-mail addresses as phony "From"s ; I setup Mantis-based bug trackers for our product and client support teams; Joomla for our main site; scripted and automated package updates; and lots more. And naturally, I had to devise a way to backup the server; if the real machine hosting our server crashed, I wanted to be able to re-introduce all my custom changes - with minimal effort.

Unfortunately... startups have limited resources. Our company uses a unified Windows backup strategy, so whatever data I was to collect from our VDS machine, I would have to store them in Windows machines (equipped with RAID drives, and a host of external USB disks).

Remote synchronization under Windows

As far as remote synchronization is concerned, I believe it is safe to say that there is an undisputed king: the best way to synchronize data between a remote machine and a local one is through rsync. This amazing utility identifies and transfers only the differences between files; it currently allows us to take a backup of around 3GB of data (residing inside our USA-based VDS), and synchronize it with our local mirror transferring just under 5MB (and doing so in less than one minute)! Moreover, this transfer is done over a secured, compressible SSH tunnel, so what's not to like?

My only issue was where to store the data. That is, where to store the actual files received through rsync... I only have Windows machines to work with, so a number of challenges started to crop up...

Initially, I thought about Cygwin; I performed a small test, and found the results to be ... so and so. Certainly not optimal - Cygwin uses an emulation layer to transform Unix system calls (made by rsync) to those supported by Windows, so it is slower than the real thing. Rsync-ing to Windows filesystems also meant issues with file permissions and owners: try as I might, I could never figure out how to successfully map the UNIX machine's owners (those inside the VDS) to the Windows ones of our domain. Problems would inevitably crop up when - eventually - I would need to restore something.

Performance wise, I found out that VMWARE server offered a much better alternative. I installed the free version on my Windows machine (as a Windows service, so that it runs even when I am not logged on), and proceeded to install a small Debian under it (I would have used Qemu, but didn't find any way to run it as a Windows service). I benchmarked the virtual machine I created and found out that it was running rsync at near native speeds - a much better performer than Cygwin. The tests I did, however, were run on the virtual disk that the VMWARE server offered to my Debian - that wasn't enough. I had to somehow store the files in the Windows backup server, since only then would they be backed up (copied to the external USB drives, etc).

Dancing some Samba

As it turned out, Andrew Tridgell (the creator of rsync) had also created the software that allows Unix machines to access Windows shares: Samba. All I had to do to mount the backup share was...

	root# mount -t smbfs -o username=SEMANTIX\\BackupAdministrator \
		//WindowsBackup/VDS-backup /mnt/backup

... and the /mnt/backup directory offered me a way into the automatically backed-up folders of the WindowsBackup machine.

Now, you might think that this solved the problem; I now had a Linux accessible folder to run rsync in - and I would enjoy the full native speed of my machine in running it. Unfortunately, you are forgetting that speed was only a small part of the problem. The big one remained: the permissions/owners issue; whether you run rsync under Cygwin or over a Samba-mounted folder in Linux, you still have to cope with the mappings of Unix owners and permissions to the Windows ones. Darn. And performance of rsync itself is not enough; even though rsync would be running natively, it would have to write many small files over the Samba protocol - not the best way towards achieving speed.

An automatically expanding encrypted filesystem image

Any Linux administrator worth his salt eventually learns about loop-mounted filesystems. Basically, it boils down to this: you can use the Linux loop device to transform any file/device into a fresh new device. If you use loop-AES, you can additionally encrypt the data with an industrial strength cipher:

	root# mount -o loop,encryption=aes256 BigFile /mnt/

I already knew about this... What I didn't realize for quite some time, was that I could use it to mount a Samba-mounted BigFile!

	root# mount -t smbfs -o lfs,username=SEMANTIX\\BackupAdministrator \
		//WindowsBackup/VDS-backup /mnt/windows
	root# mount -o loop,encryption=aes256 \
		/mnt/windows/BigFile /mnt/backup

Now that solves many issues... The /mnt/backup folder will be used as rsync's destination, and since the actual filesystem inside BigFile can be any Linux supported fs (Ext3, ReiserFS, etc), UNIX permissions and owners will be kept just fine!

Caveat:By default, Samba mounting has a file size limitation of 2GB (or 4GB, not sure). This is why the "lfs" option is used in mounting, to allow for larger files.

Then again, how big should this (Windows hosted) BigFile be? Possible hard drive extensions in our VDS must be accounted for...

Well, how about anticipating all possible sizes, without wasting a single bit of real hard drive space? Unbeknownst to many people, sparse files come to our rescue: They are considered common-place in the UNIX world, being in use for decades; Microsoft OSes started supporting them with the introduction of NTFS.

Sparse files are files whose real allocation needs are fulfilled only when data are written inside them. If you try to read from places you haven't written before, you get zeroes - and these zeroes don't really occupy any space in your hard drive. If only one sector of data (512 bytes) gets written at some offset inside a sparse file, then a sector is all the sparse file will reserve from the filesystem - not the size reported by the filesystem!

All that is required to create a 150GB sparse file under Windows is this:

(From within a Cygwin command prompt)
    dd if=/dev/zero of=BigFile bs=1M count=1 seek=150000

This command will execute in 1 second, and it will only reserve 1MB of real hard drive space. Real storage will grow as needed, when data are written inside BigFile.

At this point, all the necessary components are there:

• Create an automatically expanding sparse BigFile under the Windows share:

  dd if=/dev/zero of=BigFile bs=1M count=1 seek=150000

• Mount the Windows share under the Linux machine:

  root# mount -t smbfs -o lfs,username=SEMANTIX\\BackupAdministrator \
  		//WindowsBackup/VDS-backup /mnt/windows

• losetup and format a Linux filesystem inside this file (optionally, using encryption: losetup -e aes256):

  root# losetup /dev/loop0 /mnt/windows/BigFile
  root# mkreiserfs /dev/loop0

• Finally, mount it, and run rsync:

  root# mount /dev/loop0 /mnt/backup
  root# cd /mnt/backup
  root# rsync -avz root@hosting.machine.in.US:/ ./
  

 

The perfect backup - browsing in history

So... optimal rsync execution, permission/owner preservation inside a native Linux filesystem, automatic storage space extension in the Windows server (as required by our data), and optional encryption: our USA-based Linux server is perfectly mirrored inside our Greece-based Windows backup server.

But this can be improved even more... Navigateable directories with daily snapshots can be created, and - again - almost no hard drive space needs to be paid for it!

This is accomplished through hard links. Try this under any Unix (or Cygwin):

bash$ dd if=/dev/zero of=/tmp/file1 bs=1K count=10
bash$ cp -al /tmp/file1 /tmp/file2
bash$ ls -la /tmp/file1 /tmp/file2
-rw-r--r--  2 owner group 10240 Oct 14 12:14 /tmp/file1
-rw-r--r--  2 owner group 10240 Oct 14 12:14 /tmp/file2

Did you notice the "2", in the second column, for both output lines?
It means that there are two "names" for the data inside these files: in UNIX parlance, these are two directory entries pointing to the same inode, the same "real data" in the filesystem. If you remove one of them,

bash$ rm -f /tmp/file1
bash$ ls -l /tmp/file2

...the other one can still be used to access the data:

-rw-r--r--  1 owner group 10240 Oct 14 12:14 /tmp/file2

Notice that the "link count" column, is now 1. Only if you remove this file as well, will the data really go to Never-never land.

How can this be put to use in our backups? Simple: Since hard links take almost no space, a "free" local mirror of the previous backup can be taken into another directory (using hardlinks), and THEN, rsync will work on one of the two copies, leaving the hard links in the other one to point to the old data:

root# mount /dev/loop0 /mnt/backup
root# cd /mnt/backup
root# rm -rf OneBeforeLast
root# cp -al LastBackup OneBeforeLast
root# cd LastBackup
root# rsync -avz --delete root@hosting.machine.in.US:/ ./

The "cp -al" creates a zero-cost copy of the data (using hardlinks, the only price paid is the one of the directory entries, and ReiserFS is well known for its ability to store these extremely efficiently). Then, rsync is executed with the --delete option: meaning that it must remove from our local mirror all the files that were removed on the server - and thus creating an accurate image of the current state.

And here's the icing on the cake: The data inside these files are not lost! They are still accessible from the OneBeforeLast/ directory, since hard links (the old directory entries) are pointing to them!

In plain terms, simple navigation inside OneBeforeLast can be used to examine the exact contents of the server as they were BEFORE the last mirroring.

Naturally, this history bookkeeping can be extended beyond two images. Personally, I created a script that keeps a little over two weeks worth of daily backups - and is run automatically from cron, at 3:00am every day, performing actions like these:

#!/bin/bash
rm -rf backup.16
mv backup.15 backup.16
mv backup.14 backup.15
...
mv backup.2 backup.3
cp -al backup.1 backup.2
rsync -avz --delete root@hosting.machine.in.US:/ backup.1/

Results - some thoughts

Current statistics show that our instantly navigateable backups of the last 15 days (for 3GB of remote data) are only costing...

• real storage space on the Windows backup server of 3.05GB
• ...and the transfer price of less than 5MB - on average - per rsync!

Am I not right, in calling this the perfect backup?

If you are to keep anything in mind from the above, it is this:

The power of UNIX lies in NOT trying to hit all birds with one stone; each utility does one job, and it does it well. The amazing overall functionality we have developed (not existing, as a whole - at least to my knowledge - in any commercial offering) was obtained by "glue"-ing functionality of different components together. That's how the Unix world works: by weaving components together:

• I didn't need any special effort to automate the backups and execute them at 3:00am, since cron does this job well.
• SSH was perfect in tunneling securely and compressing the delta stream generated by rsync.
• The loop device driver allowed me to work with any file on any device
• Samba gave me the way to access Windows filesystems
• Sparse files and hard links allowed me to lessen the required storage space by orders of magnitude.

An example from the future: when ZFS arrives on the Linux scene, I will also add compression in the backed up data - as well as automatic detection and correction of hardware data errors - and it will cost me NOTHING: instead of formating the BigFile as ReiserFS, I will format it as ZFS.

In fact, I could already add compression of backed up data in many places; I could ask the Windows server (through Explorer) to keep the BigFile as a compressed one; or I could apply any of the many FUSE based compressible filesystems to the mounted ReiserFS image (much better).

The possibilities are endless - in the open source world, at least.

---

Index

CV

Updated: Sat Oct 8 12:33:59 2022