charles skipped what i see as a highly critical question, personal
backup.
my life is on a 13" macbook air, all data, mail back decades (i do not
save all mail), etc. the whole drive is encrypted, my main reason for
moving to lion.
i have two time machine drives, one at home and one i carry on the
road. both are encrypted.
belt and braces, i also use unison to sync my laptop's home data to a
server in colo. it goes to a freebsd geli, i.e. encrypted, partition.
all keys and other critical private data are in a text file on my laptop
encrypted with gpg, which i use emacs crypt++ to access. that key file,
a bunch of x.509 certs, ... are copied to an ironkey usb.
For my laptops, nearline field storage includes my laptop's drive and a
portable external drive. Online and nearline home storage is a network
attached storage array running proprietary X-RAID (like RAID-5) with a
hot-spare drive. All my machines (desktops, servers and laptops) are set
to perform regular backups to the NAS. Offline backups are done to a
series of external USB HDs that are rotated into place for nightly
incremental and weekly full backups. Current retention schema is 4 weeks
of backups with a one week offsite physical rotation (performed monthly
to a safety deposit box). I'm at the moment trying to figure out a good
way for doing streaming backups to an offsite DC.
I found the "mobile account/portable home directory" feature in os x server
to be very useful for wife and kids (powerbooks). They get backed up and don't
realize. If they crash a drive or if i upgrade a machine I just log them back
in and resynch the machine. no more lost homework.
We used to use DVD's for off-site backup, but that's not been the best
of solutions. I've been experimenting with external hard drives but
I am less comfortable with them; I've seen too many drives fail. The
idea of letting them sit for awhile and praying they spin up later
bothers me. On the other hand, running Unison to a server someplace
else has obvious benefits and some downsides too. Backups remain a
tricky problem to get right.
We used to use DVD's for off-site backup, but that's not been the best
of solutions. I've been experimenting with external hard drives but
I am less comfortable with them; I've seen too many drives fail. The
idea of letting them sit for awhile and praying they spin up later
bothers me. On the other hand, running Unison to a server someplace
else has obvious benefits and some downsides too. Backups remain a
tricky problem to get right.
... JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
For backups I use a combination of:
1) Sync server to a single purpose windows XP box in a different building
(why windoze will become apparent shortly)
2) Continuous dribble of remote backup from XP backup to backblaze (doesn't
support server OS for obvious reasons)
3) Monthly or so backup of really critical files to microsd cards in safety
deposit box. They are inexpensive and very small, so I just buy a new one
for each backup and let them pile up in the box.
It's not perfect but works for me, and I can use backblaze to remotely
"recover" a file if access to my server is down
When my parents finally got broadband years ago and I wired their house, I
loaded Linux on an old x86 PC and stashed it in a corner of their garage.
rsync over an ssh tunnel runs nightly from a Linux server in my house (about
90 miles away) to theirs. I don't sync everything, but it gives me comfort
knowing that my family photos and such are secure somewhere else. It gives
the added advantage of letting them browse our family photo archive locally.
Yeah. I've been using external USB terabyte disks, which work OK but
are irritatingly flaky.
I keep thinking that this is what tape is for, but every time I look
at AIT or LTO tapes and jukeboxes, they seem to be about a generation
behind the disks in capacity or so expensive that I can buy ten TB
disks for the cost of a five tape jukebox.
- NAS box with a proper RAID-5 disk-set
(or just one of your servers that has enough disk space)
use this to do your backups to with your favorite tool
(duplicity, rsync, etc) at the times that you think is useful.
(And one can create a TimeMachine box using the netatalk package)
This is your 'always online' backup, containing the last X revs
Just plug in a disk there once in a while and rsync to it.
One could keep multiple disks with the same data for multiple
different revisions or when using rsync hardlinks even all revisions.
As one can get SATA disks at 2TB or 3TB one will still have to chunk
up the NAS box into pieces most of the time but that is all
scriptable (filesystem labels are useful to detect which disk
is there to automate that selection even further or heck put a file
on it describing the contents).
Presto, off-line backups, just like a tape solution but much much faster
and much much cheaper and easily expandable/upgradeable to new bigger
harddisk models to grow along with your RAID box and other needs.
Don't forget to store those offline backups far away from the NAS box
and to of course properly secure all of it so that if somebody
physically steals it or hack the thing remotely they don't have a copy
of your whole live, google/facebook and other such companies would hate
to sink to the second spot of who owns your data...
I can understand that. That's why I use a rotation and retention system
as well as two NASes syncing to one-another with the backups being
performed from the second NAS. I treat them like I used to treat tape.
The onsites contain nightly incrementals as well as weekly full dumps.
These drives are rotated every week. Every month a copy of the latest
full dump is copied to another set of drives which are sent to offsite
storage. I keep two copies offsite. To implement this, I need a
minimum of nine USB drives. I actually have 10 so I can have an extra
one onsite as a hardware replacement-spare.
My home backups are somewhat large and not yet offsite due to their size. (~4.7TB).
This is due to both purchased digital media storage and photography. These are sufficiently large that the problem is harder to resolve, as nobody "makes" a 5TB drive I can ship to a colo. Drive failures in the "backups" host also become painful to work with as I'm cheap so the ZFS pool isn't 100% mirrored.
Some machines use netatalk plus the "defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1" hack.
I've considered just subbing to backblaze as it's "cheap" on a single-host basis, but need something closer to ~5-7TB plus some room for growth (maybe 250G-500G/year). I have considered just trenching copper/fiber (see other thread) to a neighbor and placing the host there, but it's not much geographical diversity, plus if the neighbor moves it becomes fun to re-explain what you are doing. (worse if they're non-techie).
The biggest problem I've seen is with a "cloud"/"tubes" provider, my upload speed puts the 4.7TB initial sync somewhere around 145 days (assuming 3Mb/s upload).
Is anyone aware of a solution for this that is sensible $$$ without rolling my own (i'm estimating about 2-3k to do this...)? And preferably costs maybe $ or $$?
Stretching the definition of "cloud" (why not, everyone else has), you can include SmugMug, Flixter, Picassa, etc. For $20-$40/year (yes, _year_), some of these allow infinite uploads.
Moreover, while you only get a few MB/s per session, you can open multiple simultaneous uploads. I've filled my upstream pipe uploading things - which still takes days, but not 100s of days.
Anyway, this is one way to get an off-site backup, lower your b/w requirements on your personal server, and even get someone else to do things like make thumbnails, multiple sizes / magnifications, allow (paid) prints to be made, etc., etc.
I wrestled with this same problem when I started using Backblaze, and yes,
my initial full backup took about six months, but it did get done.
Backblaze doesn't have a way to prioritize, but the way I dealt with it was
to exclude the media directories from the backup for the first month or so
while the more important stuff got backed up, then drop their exclusion and
let them start trickling up.
The 500G/year of new data shouldn't be a problem, since that's under 2G/day,
which should not be a problem to push up over a decent connection.
I didn't worry during the initial six months since I also have some backups
made to bare 2tb drives in my safety deposit box (I use one of those desktop
adapters you can drop bare drives in).
All backups pass through the windows XP box since that's the gateway to
backblaze. It has 4x2tb drives on it. No raid or anything since it is a
copy itself, just 4 separate volumes. All of the "most important stuff"
lives on C so if I had to switch backup services I can exclude D/E/F from
backup until C is done. To keep power usage reasonable drives are slower
RPM and also set to spin down when idle 15 mins.
> My home backups are somewhat large and not yet offsite due to their size.
> (~4.7TB).
>
We (NAC) run a rather large ZFS thing to sell cheap 'scratch space.' When I say large, I think it surpasses well over 100 TB at this point.
So for me, it was easy. At home, my stuff spins on disk (nice big green drives, so relatively low power too) on a Win2008 Server box. So, I bought a very cool product, "Super Flexible File Synchronizer" which is unbelievably cool for what it costs. It can copy from anything, to anything, and has a gazillion options. I have what is called "synthetic backups" enabled, which is a real time (it watches at the OS to disk level), as when it sees a file change, it gets copied over the other side in a few moments. It does zip-before-copy (with encryption), it does revisioning, it does delayed deletes, etc..
--As of August 13, 2011 2:12:24 PM +0900, Randy Bush is alleged to have said:
charles skipped what i see as a highly critical question, personal
backup.
--As for the rest, it is mine.
Personal system: Important files are on the fileserver, on a RAID-Z volume. It's backed up nightly using Tarsnap; the keyfiles to that are on two machines, and two USB drives. (Including an Ironkey in the firesafe.)
Tarsnap is worth looking at if you are looking at an offsite backup for small-scale use: On a per-GB basis, it's not especially cheap, but it bills on a per-byte basis, so if you aren't storing hundreds of GB, it's cheap.
On the other hand, running Unison to a server someplace
On the other hand, running Unison to a server someplace