Time Machine Exclusions
So Time Machine is a pretty convenient way to backup your machine, and I use it to backup my laptop to an external FireWire drive. Although Time Machine backs up the "whole system," I assumed there had to be some exclusions, such as cache files or /dev, for example. After a short bit of digging, I discovered that these paths are specified in a standard property list at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist
The full list is some 57 items, and is available below. Besides the expected cache items, the list includes some items I thought interesting:
/home */Library/Logs /Users/Guest /Library/Safari/Icons.db /.Spotlight-V100
Of these, the only one I find odd is the exclusion of logs. If your system goes haywire such that you restore it completely from a backup, it might be nice to see what went wrong.
[Update] Fixed the path to the StdExclusions.plist file.
Here's the full list of the excluded paths:
Contents Excluded
The contents of these paths are excluded, but the directories themselves are preserved as they are required for a successful restore.
/Volumes /Network /automount /.vol /tmp /cores /private/tmp /private/Network /private/tftpboot /private/var/automount /private/var/run /private/var/tmp /private/var/vm /private/var/db/dhcpclient /private/var/db/fseventsd /Library/Caches /Library/Logs /System/Library/Caches /System/Library/Extensions/Caches
Paths Excluded
These directories and their contents are excluded completely.
/.Spotlight-V100 /.Trashes /.fseventsd /.hotfiles.btree /Backups.backupdb /Desktop DB /Desktop DF /Network/Servers /Previous Systems /Users/Shared/SC Info /Users/Guest /dev /home /net /private/var/db/Spotlight /private/var/db/Spotlight-V100
User Paths Excluded
These directories and their contents are excluded per-user. The path is relative to the user's home folder.
Library/Application Support/MobileSync Library/Application Support/SyncServices Library/Caches Library/Logs Library/Mail/Envelope Index Library/Mail/AvailableFeeds Library/Mirrors Library/PubSub/Database Library/PubSub/Downloads Library/PubSub/Feeds Library/Safari/Icons.db Library/Safari/HistoryIndex.sk
38 comments:
menneke at 2007-11-15 16:59:06 -0500
It's even worse than that. Today I did a full restore of my boot disk (by booting from the installation CD and selecting Restore in the Tools menu), and after that Apache2 keeps spurning out error messages that it can find its log files. Indeed, there is no Apache2 subdirectory anymore in the /var/logs directory and apparently Apache can't create it...
Devin Lane at 2007-11-16 02:01:39 -0500
Is your /var a symlink to /private/var? The log directory for apache2 (at least according to TimeMachine) is /private/var/log/apache2, is that the directory that your apache2 is trying to create? Furthermore, are you using the built in apache, or a copy obtained elsewhere?
jackr at 2008-01-18 01:14:08 -0500
Same problem here. Yes, "/var -> private/var". Apache, postfix, and I think some others fail after restore, due to missing directories they're not willing to recreate. Indications here: http://discussions.apple.com/thread.jspa?messageID=6367763#6367763 suggest this is an intermittent problem: some restores work, some fail. creepy.
jackr at 2008-01-18 01:43:29 -0500
I see that StdExclusions.plist "COntentsExcluded" lists /provate/var/log. It also lists /private/var/log/apache2, as if that directory might be backed up and then restored, but looking into my backup volume, I don't see it anywhere. I suspect that the exclusion of the contents of /private/var/log/ is excluding even the directories for things like /private/var/log/apache2.
Bug at 2008-02-05 01:02:59 -0500
Well, at least they had the decency to put them in a plist for us sysadmins...not that these exclusions are a big deal for ma+pa on their Time Capsule. I think they leave out the logs because they're supposed to be backed up for a week or two anyway, albeit not on a secondary drive.
terris at 2008-02-13 22:57:36 -0500
/home WTF? Am I missing something because I'm thinking Apple is stoopid.
Matt at 2008-03-05 00:49:10 -0500
Terris-- /Users has all the account info.
Barry K. Nathan at 2008-03-29 20:01:21 -0400
The file is actually at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist (the "Contents" part of the path is missing from this blog post).
Matthew Miller at 2008-04-19 07:46:33 -0400
I just deleted the StdExclusions.plist. My backup is 600mb bigger. I am going to replace my HD on Monday, we'll see when I restore it if I'll have all my cache's and DB's.
Matthew Miller at 2008-04-19 07:49:43 -0400
Update 1: I looked at my Time Machine Backup, all those paths/caches/DB's now appear in my Time Machine Backup
Luke Dorny at 2008-04-24 04:02:42 -0400
I just commented out all the exclusions in the plist file and the resulting backup.inProgress is currently 58GB of data. But I'd be wrong since it's rewriting all the symlinks to the correct files. Right? I'm assuming that this is the size of the (usually) excluded files on my system. Wow. See progress image here: http://flickr.com/photos/luxuryluke/2438550718/
David Stearns at 2008-07-19 19:10:29 -0400
I am installing a new hard drive in a few days on a titanium running leopard. I have my firewire back up drive partitioned with a full copy of leopard installed as a boot drive in the smaller partition but not updated. When I do a "restore" with time machine to the new drive can I use the boot partition instead of the os x leopard disk in the dvd drive? Thanks, David
Devin Lane at 2008-07-19 19:21:41 -0400
David: You can, but it'll be less smooth. You can install Leopard on the new hard drive, then from Macbuddy (the setup assistant) import users/applications/files from a time machine backup. Personally, when I did a restore after a dead hard drive, I found using the Leopard DVD to be much easier. My "install" was just the time machine restore; after my system rebooted, I was using my computer pre-hard-drive-crash.
David Stearns at 2008-07-20 10:21:53 -0400
Thanks, I planned on a time machine restore, was just wondering if I still needed the boot disc or if I could boot to the firewire and do the time machine restore. When I travel I would like to know I can just leave the disk at home and rely on the firewire drive entirely. David
Devin Lane at 2008-07-20 16:13:48 -0400
David: Unless you image the install DVD onto your firewire drive, you won't be able to install Leopard at all.
David Stearns at 2008-07-20 17:06:27 -0400
I installed it from the install disc to the drive. 20 gig partitioned for the os only. It worked just fine. David
David Stearns at 2008-07-20 17:17:11 -0400
I just held the option key and rebooted. I works just like it should. I couldn't open safari in the internal laptop drive, it would crash on open, but opens fine using the firewire drive version. When I did the install I used all my same user name and passwords, after the install it would pick the firewire as the main boot drive so I had to change that in system preferences. other than that no hang ups.
Devin Lane at 2008-07-20 18:00:15 -0400
David: I misunderstood your original post. You must boot off the Installer DVD or a partition imaged from this DVD in order to perform a time machine restore instead of a standard installation. After the OS is installed, you can import users/applications/settings from the time machine backup with MacBuddy, but you'll need enough space on that partition to fit all this data. You cannot use an installation of Mac OS X to perform a time machine restore on another drive.
David Stearns at 2008-07-20 22:17:18 -0400
thank you, that is what I was going to attempt. you saved me a lot of head aches and time. David
AL at 2008-08-19 10:03:29 -0400
Sidenote FYI: Your iTunes authentication information is stored in /Users/Shared/SC Info You can remove or comment out that line in the StdExclusions.plist so that it'll get backed up. I've actually been bypassing my iTunes authentications for years by manipulating the 'SC Info.sidb' file in that directory.
Simon at 2008-12-09 21:24:43 -0500
To get apache working after a complete restore you need to create the apache2 log folder. cd /private/var/log sudo mkdir apache2
iconz113 at 2009-01-26 13:58:02 -0500
I just need to know can I potentially lose functionality of something if i do a full time machine restore because of the exclusions?
rcfa at 2009-02-26 14:42:08 -0500
The reasons logs must be excluded is efficiency. TimeMachine is only reasonably quick and efficient because it only has to write changes. Logs change all the time, and the amount of useful backup history could be massively shortened if log files were backed up, since some of them are verbose, and constantly change, so they would be backed up every hour, a full new copy. It's the same problem you have when you don't exclude rw disk images that are used often: just mounting the disk image and unmounting it will modify it, and an entire e.g. 4GB Windows virtual machine disk image will get backed up. Once per hour! What TimeMachine really needs is one more category of files: history limited backup Right now we can just back up, or exclude. What we need is a facility that says which files should be kept in the backup history, and which should not. Typically for most files it would be sufficient to keep a current version, and ONE previous version. This would limit log files two two iterations, the System to the current OS version and the one before (allowing to restore to a known working system after a software upgrade that causes instability). Only actual user data you need a long history for, and the less history we keep for things such as system related files and the actual OS, the more space there is for user data. I submitted feedback and bug reports to Apple along these lines, but the more people do so, the more seriously Apple will take the issue.
Neil Paisnel at 2009-04-18 12:36:24 -0400
Wish I had read this earlier. I put all my users on a second HDD inside my Mac Pro I created a user called 'home'....could not get TM to back it up. Worked out that it must have been excluded, but would have saved a lot of agro if I had seen 'home' was excluded earlier !
contra at 2009-05-17 17:59:41 -0400
I favorited it. The last line is so true. I also wish life would be much easier...Thanks for sharing.
arkadas-arkadas ara at 2009-06-18 12:50:41 -0400
thnx verry nice page
mini storage hk at 2009-07-11 07:02:33 -0400
thanks for sharing
Charlotte at 2009-07-23 19:29:25 -0400
thanks so much for sharing. its gives me peace of mind (before i actually restore) to know what is truly excluded. do you have any info on excluding certain folders while restoring from a time machine drive? I'm looking to do a fresh install and only wanted to restore certains parts from my external I use for time machine. any info would be most appreciated...
wendi at 2009-08-23 15:34:43 -0400
I used Time Capsule to back up from my iMac. I no longer have the iMac and would like to restore the iMac files from Time Capsule to my MacBook. I understand that I have to wipe the MacBook clean before I can even restore from Time Capsule. This worries me in case there is a problem with restoring my iMac back up to my MacBook. Also, I tried to get into my iMac back up from my MacBook and find I cannot. If I could I would only need to restore my iMac Address Book contents since I migrated most other files from my iMac to my MacBook prior to getting rid of my iMac (now wiped clean.)
WiseMax at 2009-11-17 19:00:52 -0500
To wendi: First of all, why do that?... Hardware is different... I suggest you leave the OSX alone, don't restore any of it. Better yet, restore only personal files and applications (no need to restore all of them). This might be safer and faster, I guess... To All: I'm so sorry I didn't read befor about the excluded iTunes files... Could save me some worries, we'll see. I'm at the moment restoring my 320 GB HD in my 15" MacBook Pro that became too small to a new 500 HD. Well, it started about 4+ hours ago and is still at 33% - it still says it will be more 7 hours to go :-( it looks like 8 or 9 are possible, a total of 11-12 hours for 230 GB of system and data load... Really, not very fast.... but what matters is that it works. Will come back here to tell the rest of the story tomorrow, LOL!... Regards, WiseMax
colonies at 2010-01-29 14:35:19 -0500
thanks to the autor
PLC programming at 2010-02-08 22:28:31 -0500
Good posts on this blog make me read this blog again and again i liked this blog very much This blog has got all the useful stuff that i wanted to know about.
NewInfo at 2010-03-31 22:06:24 -0400
I know this is two and a half years after this was posted, but it seems to get referenced several places. For those wondering, at some point Apple got enough complaints to fix the problems listed here. First, they've removed the following from "Contents Excluded" /private/var/log /private/var/log/apache2 /private/var/log/cups /private/var/log/fax /private/var/log/ppp /private/var/log/sa /private/var/log/samba /private/var/log/uucp /private/var/spool And they've created a fourth category in the list: FileContentsExcluded -- paths where we need to capture entire subtree folder layout to restore disk structure, but don't want to backup contained files /private/var/log /private/var/spool/cups /private/var/spool/fax /private/var/spool/uucp
Furniture Indonesia at 2010-04-29 23:44:47 -0400
-good info indeed!
Lisa Peet at 2010-05-04 16:23:08 -0400
Thanks for sharing
TC at 2010-05-14 18:19:23 -0400
In regards to NewInfo's comment about Apple's changes to the exclusions list. This change seems to have occurred in 10.6, so if you are running 10.5.x the initial listing is still valid.
Gus at 2010-05-20 20:29:49 -0400
I have read a few postings and articles regarding Time Machine exclusions, and I have not been able to figure out what has gone wrong since my upgrade to 10.6. If anyone happens upon this post and has some input, I would greatly appreciate it. To avoid hassles with FileVault, I don't use it. Instead, at login I mount an encrypted disk image that I use to hold all client data and I do not store the password in my keychain. Under Leopard, Time Machine would backup the files inside this mounted volume just like any other drive/volume. I could exclude the folder where the disk image itself was stored (though backing up a sparse image could partially address this under Snow Leopard) so as not to have massive backups, and I could rely on Time Machine to make backups of the individual files, revisions, etc. inside the volume. Under Snow Leopard, Time Machine seems to not even try to backup the contents of the mounted volume. I cannot quite figure out why or how this is excluded. Maybe I am doing something else wrong that I am not detecting. I did, for a variety of reasons (e.g., CrashPlan, etc.), make a new encrypted volume using a sparse image but I don't see how that would cause the problem. How does Time Machine know to backup external drive volumes that are mounted and yet not to backup volumes that are mounted from a disk image? Any thoughts? Thanks very much.
Aurelio Jargas at 2010-07-28 15:03:46 -0400
Here on Snow Leopard, I see that the User Paths Excluded changed a bit: Library/Application Support/SyncServices/data.version Library/Caches Library/Logs Library/Mail/Envelope Index Library/Mail/AvailableFeeds Library/Mirrors Library/PubSub/Database Library/PubSub/Downloads Library/PubSub/Feeds Library/Safari/Icons.db Library/Safari/WebpageIcons.db Library/Safari/HistoryIndex.sk