Comments on: lazyweb: help me back up to a shell account

By: Rich

Rich — Sun, 08 Oct 2006 04:40:17 +0000

Awesome! Thanks. You can sign up here, or list rich+dreamhost at lafferty dot ca as the person that referred you (in normal email address form, of course).

By: sherm

sherm — Sun, 08 Oct 2006 04:10:17 +0000

Almost on topic – I think I’m going to sign up for Dreamhost and ditch my overpriced VPS thing that I don’t need.

Do you do their affiliate thing? Give me a code or something (?) and I’ll hook you up…

By: Rich

Rich — Thu, 05 Oct 2006 22:45:20 +0000

Thanks — but those won’t store ownership information. With only a single account at the far end, ownerships need to be stored as metadata, rather than just in the inode at the far end.

Brackup is looking promising, though, once I write an sftp target for it and get it storing ownership in metadata.

By: _lj_sucks_

_lj_sucks_ — Thu, 05 Oct 2006 20:11:37 +0000

rsync -avE on OS X.

On Linux, I’ve been using rdiff-backup.

By: Rich

Rich — Wed, 04 Oct 2006 16:30:34 +0000

The problem with a big loopback disk image is that then you have a single file on the Dreamhost end. I considered doing that, but then I realized that if I wanted to take advantage of their Netapp .snapshots then I’d need individual files (or else I’d have to deal with snapshots of a single 100s-of-GBs file.

Incidentally you can avoid your “same usernames” problem with rsync’s –numeric-ids option.

But even if brackup doesn’t work I don’t think this’ll be too hard to get working. Working elegantly might be another story, but still.

By: brianenigma

brianenigma — Wed, 04 Oct 2006 08:33:15 +0000

At home, I originally got the Incremental and Individual Files parts with rsync over ssh. (-r –rsh=ssh –link –perms –owner –times –delete and optionally –size-only) As someone else mentioned, it needs to be run as root for the ownership stuff to work correctly (plus, the same usernames and/or username-to-id mappings need to be on both machines or the files are owned by numeric user ids that don’t exist in /etc/passwd.)

I then changed the process to be more Metadata-Friendly by creating a sparse disk image (it’s an OS X thing that’s roughly equivalent to a loopback filesystem with a nice GUI tool and mounter), accessing that over Samba, then using the same rsync command, but on the local filesystem tree instead of over ssh. This fixes issues with keeping usernames/ids in sync and running as root as well as the metadata thing.

Not that I’ve done this, but I would imagine that if you could create a similar loopback disk image on the Dreamhost server, access that over a remote protocol (FTP won’t do it, DAV might, Samba certainly would, as would NFS), you would have the same thing. Adding encryption is relatively easy (well, easy-ish for being a Linux fs driver, from what I remember back when I last played around with it), using an encrypted loopback device. The image on the remote share is mounted through the loopback, then a local rsync is run to synchronize the files.

On the other hand, I had never heard of Brackup until just now, and that’s looking pretty nifty.

By: dossy

dossy — Wed, 04 Oct 2006 08:07:15 +0000

Mmm, Dreamhost as storage for brackup … delicious!

By: Rich

Rich — Wed, 04 Oct 2006 06:04:48 +0000

Oh. Okay! And here I thought I was just being a smartass. I’ll check that out. :-)

By: brad

brad — Wed, 04 Oct 2006 05:57:53 +0000

Yes, brackup does all of this. Except ownership metadata… it does keep permissions metadata. Adding ownership would be possible, if you run it as root.