Hi everyone, I'm backing up my system on a more regular basis and am trying to fine tune the files that I backup. I am looking for advice on what NOT to bother to backup on the /home directory. I am using rsync (took a long time and lots of trials to figure out the man page - and still don't know 90% of it) and presently have the following on my exclude_list.txt: (Note: multiple items shown on one line are just for readability, each line in the file only has one item) tmp* TMP* .cache* cache* Cache* CACHE* *CACHE *Cache *cache .cookies* cookies* Trash Trash* TRASH* Junk* junk* .gvfs Backups backups Crash* .xsession-errors* .macromedia .thumbnails .mozilla/firefox/*/thumbnails *.corrupt minidumps .local/share/gvfs* I'd love to exclude things that perhaps one would never use from a backup to rebuild a system after an accidental clean wipe of all data. Similarly, any recommendations of what I should back up outside of /home? I am thinking of things like /etc/fstab, files that would make it easier to recover from a crash or to upgrade a distro. Thanks for any advice. John.
On 23 December 2016 at 12:11, John Moniz via talk <talk@gtalug.org> wrote:
I'm backing up my system on a more regular basis and am trying to fine tune the files that I backup. I am looking for advice on what NOT to bother to backup on the /home directory.
I am using rsync (took a long time and lots of trials to figure out the man page - and still don't know 90% of it) and presently have the following on my exclude_list.txt: (Note: multiple items shown on one line are just for readability, each line in the file only has one item)
tmp* TMP* .cache* cache* Cache* CACHE* *CACHE *Cache *cache .cookies* cookies* Trash Trash* TRASH* Junk* junk* .gvfs Backups backups Crash* .xsession-errors* .macromedia .thumbnails .mozilla/firefox/*/thumbnails *.corrupt minidumps .local/share/gvfs*
I'd love to exclude things that perhaps one would never use from a backup to rebuild a system after an accidental clean wipe of all data.
Similarly, any recommendations of what I should back up outside of /home? I am thinking of things like /etc/fstab, files that would make it easier to recover from a crash or to upgrade a distro.
Thanks for any advice.
If you only have very limited space for backups, that looks like a pretty good list (very similar to one I had a few years back). If you can afford it, I'd recommend just backing up your entire /home/ (and rsync is a great way to do it). It's easier because A) you don't have to maintain that list of exceptions, and B) you don't find out AFTER your HD crashes that you really did need that .cookies folder to get back into your bank's account page. That latter is just a random example: if you don't back it all up, you'll find out that you needed something that's no longer there. Several further recommendations: - if you have a directory full of Linux ISOs (as I suspect many of us do), that's a good directory to not back up ... unless you custom-build them - back up /etc/ in its entirety - back up /var/ if you have the space - /var/ is usually small, although Apt, Docker, and possibly your web server are given to seriously fattening it up - the options I use for local copies with rsync are: "--verbose --recursive --update --times --progress --stats --delete --perms --hard-links --links" (not necessarily right for you, but may help) - use the long-form options in scripts: it's a LOT easier to read and remember what you were trying to achieve - keep at least three rotating backups, preferably one off-site - keep your backups on encrypted media if you can -- Giles http://www.gilesorr.com/ gilesorr@gmail.com
On 12/23/2016 12:11 PM, John Moniz via talk wrote:
Hi everyone,
I'm backing up my system on a more regular basis and am trying to fine tune the files that I backup. I am looking for advice on what NOT to bother to backup on the /home directory.
I am using rsync (took a long time and lots of trials to figure out the man page - and still don't know 90% of it) and presently have the following on my exclude_list.txt: (Note: multiple items shown on one line are just for readability, each line in the file only has one item)
tmp* TMP* .cache* cache* Cache* CACHE* *CACHE *Cache *cache .cookies* cookies* Trash Trash* TRASH* Junk* junk* .gvfs Backups backups Crash* .xsession-errors* .macromedia .thumbnails .mozilla/firefox/*/thumbnails *.corrupt minidumps .local/share/gvfs*
I'd love to exclude things that perhaps one would never use from a backup to rebuild a system after an accidental clean wipe of all data.
Similarly, any recommendations of what I should back up outside of /home? I am thinking of things like /etc/fstab, files that would make it easier to recover from a crash or to upgrade a distro.
Thanks for any advice.
Take a look at rdiff-backup I prefer it over rsync because it keeps a history as diffs from the current backup. Backups are not just disaster recovery sometimes you need to go back a week to find that file you accidentally deleted. As a rule I do full backups because I have been burned by backing up just the "critical" information only to find that there was other information pushed all over the system. Database backups need to be handled somewhat special because most databases do not guarantee that the binary file backup will not be corrupted if the server is active while the data is being backed up. I will exclude things like /tmp and /var/tmp but I make sure I keep the /var/log/* files because the log files are the first thing someone who compromises your system will delete. -- Alvin Starr || voice: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
My recommendations... 1. Backup entire disk to another disk, verbatim. That is, dd if=/dev/sdX of=/dev/sdY bs=10M First, you don't have to waste time figuring out what to back up. Second, if disk fails, you can just swap the disks, and copy over only the new files since last dd. 2. Backup entire /home to another disk. That is, mount /home/sdZ /mnt/home rsync -aHxv -SAX /home/ /mnt/home/ --delete --exclude='/*/.gvfs' Again, I would copy the whole things. If you have somethings that you don't want to backup, then you can put them in /home/dont_backup, and add it to exclude list, --exclude='/dont_backup' 3. If your /home is on BTRFS filesystem, you can take snapshots, like btrfs subvolume snapshot -r home home--$(date +%F) Harddisk is "cheap", and snapshot is "cheaper". -- William On Fri, Dec 23, 2016 at 12:11:15PM -0500, John Moniz via talk wrote:
Hi everyone,
I'm backing up my system on a more regular basis and am trying to fine tune the files that I backup. I am looking for advice on what NOT to bother to backup on the /home directory.
I am using rsync (took a long time and lots of trials to figure out the man page - and still don't know 90% of it) and presently have the following on my exclude_list.txt: (Note: multiple items shown on one line are just for readability, each line in the file only has one item)
tmp* TMP* .cache* cache* Cache* CACHE* *CACHE *Cache *cache .cookies* cookies* Trash Trash* TRASH* Junk* junk* .gvfs Backups backups Crash* .xsession-errors* .macromedia .thumbnails .mozilla/firefox/*/thumbnails *.corrupt minidumps .local/share/gvfs*
I'd love to exclude things that perhaps one would never use from a backup to rebuild a system after an accidental clean wipe of all data.
Similarly, any recommendations of what I should back up outside of /home? I am thinking of things like /etc/fstab, files that would make it easier to recover from a crash or to upgrade a distro.
Thanks for any advice.
John. --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
I use backintime for my local backups. It keeps history of files, so even if you accidentally alter a file and discovers a month later, you can recover the last version. If you have a VPS somewhere, like me, you can use syncthing or owncloud too. They are very good for backups. I keep a local backintime backup, rsync to a raspberry pi every night, and sync it to the VPS every week. On Dec 23, 2016 5:21 PM, "William Park via talk" <talk@gtalug.org> wrote:
My recommendations...
1. Backup entire disk to another disk, verbatim. That is, dd if=/dev/sdX of=/dev/sdY bs=10M
First, you don't have to waste time figuring out what to back up. Second, if disk fails, you can just swap the disks, and copy over only the new files since last dd.
2. Backup entire /home to another disk. That is, mount /home/sdZ /mnt/home rsync -aHxv -SAX /home/ /mnt/home/ --delete --exclude='/*/.gvfs'
Again, I would copy the whole things. If you have somethings that you don't want to backup, then you can put them in /home/dont_backup, and add it to exclude list, --exclude='/dont_backup'
3. If your /home is on BTRFS filesystem, you can take snapshots, like btrfs subvolume snapshot -r home home--$(date +%F)
Harddisk is "cheap", and snapshot is "cheaper". -- William
Hi everyone,
I'm backing up my system on a more regular basis and am trying to fine tune the files that I backup. I am looking for advice on what NOT to bother to backup on the /home directory.
I am using rsync (took a long time and lots of trials to figure out the man page - and still don't know 90% of it) and presently have the following on my exclude_list.txt: (Note: multiple items shown on one line are just for readability, each
On Fri, Dec 23, 2016 at 12:11:15PM -0500, John Moniz via talk wrote: line
in the file only has one item)
tmp* TMP* .cache* cache* Cache* CACHE* *CACHE *Cache *cache .cookies* cookies* Trash Trash* TRASH* Junk* junk* .gvfs Backups backups Crash* .xsession-errors* .macromedia .thumbnails .mozilla/firefox/*/thumbnails *.corrupt minidumps .local/share/gvfs*
I'd love to exclude things that perhaps one would never use from a backup to rebuild a system after an accidental clean wipe of all data.
Similarly, any recommendations of what I should back up outside of /home? I am thinking of things like /etc/fstab, files that would make it easier to recover from a crash or to upgrade a distro.
Thanks for any advice.
John. --- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
On 12/23/2016 02:21 PM, William Park via talk wrote:
My recommendations...
1. Backup entire disk to another disk, verbatim. That is, dd if=/dev/sdX of=/dev/sdY bs=10M
First, you don't have to waste time figuring out what to back up. Second, if disk fails, you can just swap the disks, and copy over only the new files since last dd. Generally this is a bad idea if your copying a live file system because you end up with a disk image that is corrupted to some extent.
2. Backup entire /home to another disk. That is, mount /home/sdZ /mnt/home rsync -aHxv -SAX /home/ /mnt/home/ --delete --exclude='/*/.gvfs'
Again, I would copy the whole things. If you have somethings that you don't want to backup, then you can put them in /home/dont_backup, and add it to exclude list, --exclude='/dont_backup'
If your keeping rsync backups make sure you have some way to rotate the backups so you have more than just 1. A single backup only works when the oops/problem... happened since the last backup and you catch it before the next backup.
3. If your /home is on BTRFS filesystem, you can take snapshots, like btrfs subvolume snapshot -r home home--$(date +%F)
Harddisk is "cheap", and snapshot is "cheaper".
LVM snapshots are also handy. It will be nice when btrfs becomes the linux default file system because the snapshot feature is just so damn handy. -- Alvin Starr || voice: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
On Fri, Dec 23, 2016 at 02:32:17PM -0500, Alvin Starr via talk wrote:
On 12/23/2016 02:21 PM, William Park via talk wrote:
My recommendations...
1. Backup entire disk to another disk, verbatim. That is, dd if=/dev/sdX of=/dev/sdY bs=10M
First, you don't have to waste time figuring out what to back up. Second, if disk fails, you can just swap the disks, and copy over only the new files since last dd. Generally this is a bad idea if your copying a live file system because you end up with a disk image that is corrupted to some extent.
Yes. I do telinit 1 mount -o remount,ro / before running 'dd'. -- William
On 12/23/2016 03:56 PM, William Park via talk wrote:
On Fri, Dec 23, 2016 at 02:32:17PM -0500, Alvin Starr via talk wrote:
On 12/23/2016 02:21 PM, William Park via talk wrote:
My recommendations...
1. Backup entire disk to another disk, verbatim. That is, dd if=/dev/sdX of=/dev/sdY bs=10M
First, you don't have to waste time figuring out what to back up. Second, if disk fails, you can just swap the disks, and copy over only the new files since last dd. Generally this is a bad idea if your copying a live file system because you end up with a disk image that is corrupted to some extent. Yes. I do telinit 1 mount -o remount,ro / before running 'dd'.
Remounting read only will work but that is not what you said to do. You will need to make sure all file systems are remounted read only. It will work but could have you taking your system offline for quite a while to do the full copy if you have something like a 10TB drive. -- Alvin Starr || voice: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
On Fri, Dec 23, 2016 at 04:13:48PM -0500, Alvin Starr via talk wrote:
Yes. I do telinit 1 mount -o remount,ro / before running 'dd'.
Remounting read only will work but that is not what you said to do. You will need to make sure all file systems are remounted read only. It will work but could have you taking your system offline for quite a while to do the full copy if you have something like a 10TB drive.
Our audience is GTALUG members, so they know what we're talking about. :-) I have everything in one filesystem. I do backup of that. Then, I take snapshots of the backup. -- William
On 12/23/2016 04:58 PM, William Park via talk wrote:
On Fri, Dec 23, 2016 at 04:13:48PM -0500, Alvin Starr via talk wrote:
Yes. I do telinit 1 mount -o remount,ro / before running 'dd'. Remounting read only will work but that is not what you said to do. You will need to make sure all file systems are remounted read only. It will work but could have you taking your system offline for quite a while to do the full copy if you have something like a 10TB drive. Our audience is GTALUG members, so they know what we're talking about. :-) I have everything in one filesystem. I do backup of that. Then, I take snapshots of the backup. William, some of us GTALug members are nothing more than home desktop users. ;-)
On 16-12-23 03:56 PM, William Park via talk wrote:
telinit 1 mount -o remount,ro / before running 'dd'.
The only problem with doing a dd to backup is that it takes a lot longer than doing an rsync. It also makes the machine unavailable for use during the backup. Not a problem if you schedule the backup to run overnight when you would not be using the machine. -- Cheers! Kevin. http://www.ve3syb.ca/ |"Nerds make the shiny things that distract Owner of Elecraft K2 #2172 | the mouth-breathers, and that's why we're | powerful!" #include <disclaimer/favourite> | --Chris Hardwick
On Fri, 23 Dec 2016 12:11:15 -0500 John Moniz via talk <talk@gtalug.org> wrote:
Hi everyone,
I'm backing up my system on a more regular basis and am trying to fine tune the files that I backup. I am looking for advice on what NOT to bother to backup on the /home directory.
John, I back up my entire /home directory every night. I also back up /etc, /root and /usr/local, but very much less frequently. My backup device is a 4TB hard drive. Periodically, I burn my /home backup to Blu-ray. If I have to recover something, I use my latest backup. I have written a backup script that uses tar. KISS. -- Howard Gibson hgibson@eol.ca howard.gibson@teledyne.com jhowardgibson@gmail.com http://home.eol.ca/~hgibson
On 12/23/2016 04:11 PM, James Knott via talk wrote:
On 12/23/2016 12:11 PM, John Moniz via talk wrote:
I'd love to exclude things that perhaps one would never use from a backup to rebuild a system after an accidental clean wipe of all data. Well, you could probably pass on /dev/null. ;-)
I always back up /dev/null because it never takes up any space. :-)
On 23/12/16 12:11 PM, John Moniz via talk wrote:
Hi everyone,
I'm backing up my system on a more regular basis and am trying to fine tune the files that I backup. I am looking for advice on what NOT to bother to backup on the /home directory.
I am using rsync (took a long time and lots of trials to figure out the man page - and still don't know 90% of it) and presently have the following on my exclude_list.txt: (Note: multiple items shown on one line are just for readability, each line in the file only has one item)
tmp* TMP* .cache* cache* Cache* CACHE* *CACHE *Cache *cache .cookies* cookies* Trash Trash* TRASH* Junk* junk* .gvfs Backups backups Crash* .xsession-errors* .macromedia .thumbnails .mozilla/firefox/*/thumbnails *.corrupt minidumps .local/share/gvfs*
I'd love to exclude things that perhaps one would never use from a backup to rebuild a system after an accidental clean wipe of all data.
Personally, I wouldn't be comfortable with so many wildcards in an rsync exclude. I compiled mine through trial and error, running manual backups frequently at first and finding directories that constantly had new stuff to backup with names like tmp or cache and excluding those. Here's my rsync exclude file (might be a little out of date, as I constructed in a few years back on an Ubuntu machine, now using Debian): # These files are not necessary to backup *.swp .cache/ .config/banshee/covers/ .evolution/cache/ .gnash/ .gnome2/epiphany/favicon_cache/ .gnome2/epiphany/mozilla/epiphany/Cache/ .gvfs/ .liferea_1.8/cache/ .local/share/Trash/files/ .macromedia/ .mozilla/firefox*/Profiles/*.default/Cache/ .mozilla/firefox*/Profiles/*.default/*.sqlite .mozilla/firefox*/Profiles/*.default/weave/ Trash.msf .mythtv/themecache/ .pulse/ .thumbnails/ .icedove/*.default/Cache/ .icedove/*.default/ImapMail/ .wine/ .wapi .xchat2/scrollback/ Now, some things in there are conscious decisions, like I'm excluding my IceDove/Thunderbird ImapMail folder because I don't want to constantly rsync my cache of my ImapMail folders -- I have proper backups directly from my mail server instead. I also have a bunch of other custom folders excluded on any given machine, usually ~/Downloads/ or some kind of directory where I may download large files like ISOs which I have no need or desire to be going through my backup system.
Similarly, any recommendations of what I should back up outside of /home? I am thinking of things like /etc/fstab, files that would make it easier to recover from a crash or to upgrade a distro.
Here's the script I put in /usr/local/bin/backup to run every hour or so on all my laptops/N900: #!/bin/sh START=`date +%s` LOCAL_HOST='192.168.2.160' REMOTE_HOST='myhome.domain.tld' DEST_DIR="backups/thinkpad-x60" # this would be different on each client # Try to connect to host locally, otherwise use remote connection if ssh -q $LOCAL_HOST exit; then HOST=$LOCAL_HOST else HOST=$REMOTE_HOST fi echo "========== Backup to $HOST ==========" date DESTINATION="$HOST:$DEST_DIR" # Backup home directory (Note: this is only for primary user!) echo "---------- ${HOME} ----------" rsync -e ssh -avz --delete --delete-excluded --exclude-from=${HOME}/.rsync.exclude --numeric-ids --relative ${HOME} ${DESTINATION} rsync -e ssh -avz --relative --delete --exclude-from=${HOME}/.rsync.etc.exclude /etc/ ${DESTINATION}/ rsync -e ssh -avz --relative --delete /usr/local/bin/ ${DESTINATION}/ # Calculate elapsed time END=`date +%s` ELAPSEDTIME=`expr $END - $START` echo Finished at: `date` - "It took $ELAPSEDTIME seconds" I can run manually with `backup`, but I have it set to run every hour. Not I'm backing up most of /etc/ too. I have an exclude file that leaves out some stuff there (also trial and error by running the backup manually and seeing what perhaps wasn't necessary). Some important notes! Are there sensitive files you don't want bac As other people have mentioned, this just mirrors your home directory to a backup server somewhere, which is awesome, but not a real backup because you can't go back to older versions. So what I do is I use rsnapshot for versioned backups. I have a server running at my apartment, and a server running at my parents' place. - laptops and mobile devices do rsync mirrors to the local server - the remote server and the opposite place does a nightly rsnapshot of the backup directories (and other stuff on the servers) This way, my laptops/mobiles mirror ~hourly to my living room server, but if anything ever went wrong, I could go back to the last 7 days, last 4 weeks or last 3-6 months in the rsnapshot backup. And with that being at my parents' place, it's also in a separate physical location in the even of fire, flood, theft, etc. (If there was some kind of nuclear bomb or natural disaster or something that took out physical locations across Toronto, then I wouldn't be covered, but I also figure I'd have bigger problems to worry about...) I used to backup /var, but I don't bother from laptops these days. I can rebuild a machine in a couple hours. But from servers, I definitely backup many specific subfolders in /var/ -- e.g. /var/lib/bind # if running bind /var/lib/mailman # if running mailman /var/spool/cron/crontabs/ /usr/local/ For MySQL or PostgreSQL or LDAP, I use the tools automysqlbackup, autopostgresqlbackup and autoldapbackup respectively, which create versioned local dumps on the filesystem, and then add the daily backup directory to rsnapshot to have remote backups. Maybe I should blog about my setup in more detail... let me know if you have any other questions! I'm just shy of 10 years using this backup infrastructure, and it's served me alright.
participants (9)
-
Alvin Starr -
Blaise Alleyne -
Giles Orr -
Howard Gibson -
James Knott -
John Moniz -
Kevin Cozens -
Mauro Souza -
William Park