Time Machine Exclusions
So Time Machine is a pretty convenient way to backup your machine, and I use it to backup my laptop to an external FireWire drive. Although Time Machine backs up the “whole system,” I assumed there had to be some exclusions, such as cache files or /dev, for example. After a short bit of digging, I discovered that these paths are specified in a standard property list at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist
The full list is some 57 items, and is available below. Besides the expected cache items, the list includes some items I thought interesting:
/home */Library/Logs /Users/Guest /Library/Safari/Icons.db /.Spotlight-V100
Of these, the only one I find odd is the exclusion of logs. If your system goes haywire such that you restore it completely from a backup, it might be nice to see what went wrong.
[Update] Fixed the path to the StdExclusions.plist file.
Here’s the full list of the excluded paths:
Contents Excluded
The contents of these paths are excluded, but the directories themselves are preserved as they are required for a successful restore.
/Volumes /Network /automount /.vol /tmp /cores /private/tmp /private/Network /private/tftpboot /private/var/automount /private/var/log /private/var/folders /private/var/log/apache2 /private/var/log/cups /private/var/log/fax /private/var/log/ppp /private/var/log/sa /private/var/log/samba /private/var/log/uucp /private/var/run /private/var/spool /private/var/tmp /private/var/vm /private/var/db/dhcpclient /private/var/db/fseventsd /Library/Caches /Library/Logs /System/Library/Caches /System/Library/Extensions/Caches
Paths Excluded
These directories and their contents are excluded completely.
/.Spotlight-V100 /.Trashes /.fseventsd /.hotfiles.btree /Backups.backupdb /Desktop DB /Desktop DF /Network/Servers /Previous Systems /Users/Shared/SC Info /Users/Guest /dev /home /net /private/var/db/Spotlight /private/var/db/Spotlight-V100
User Paths Excluded
These directories are and their contents are excluded per-user. The path is relative to the user’s home folder.
Library/Application Support/MobileSync Library/Application Support/SyncServices Library/Caches Library/Logs Library/Mail/Envelope Index Library/Mail/AvailableFeeds Library/Mirrors Library/PubSub/Database Library/PubSub/Downloads Library/PubSub/Feeds Library/Safari/Icons.db Library/Safari/HistoryIndex.sk
It’s even worse than that. Today I did a full restore of my boot disk (by booting from the installation CD and selecting Restore in the Tools menu), and after that Apache2 keeps spurning out error messages that it can find its log files. Indeed, there is no Apache2 subdirectory anymore in the /var/logs directory and apparently Apache can’t create it…
Comment by menneke — November 15, 2007 @ 4:59 pm
Is your /var a symlink to /private/var? The log directory for apache2 (at least according to TimeMachine) is /private/var/log/apache2, is that the directory that your apache2 is trying to create? Furthermore, are you using the built in apache, or a copy obtained elsewhere?
Comment by Devin Lane — November 16, 2007 @ 2:01 am
Same problem here. Yes, “/var -> private/var”. Apache, postfix, and I think some others fail after restore, due to missing directories they’re not willing to recreate. Indications here:
http://discussions.apple.com/thread.jspa?messageID=6367763#6367763
suggest this is an intermittent problem: some restores work, some fail. creepy.
Comment by jackr — January 18, 2008 @ 1:14 am
I see that StdExclusions.plist “COntentsExcluded” lists /provate/var/log. It also lists /private/var/log/apache2, as if that directory might be backed up and then restored, but looking into my backup volume, I don’t see it anywhere. I suspect that the exclusion of the contents of /private/var/log/ is excluding even the directories for things like /private/var/log/apache2.
Comment by jackr — January 18, 2008 @ 1:43 am
Well, at least they had the decency to put them in a plist for us sysadmins…not that these exclusions are a big deal for ma+pa on their Time Capsule.
I think they leave out the logs because they’re supposed to be backed up for a week or two anyway, albeit not on a secondary drive.
Comment by Bug — February 5, 2008 @ 1:02 am
/home
WTF?
Am I missing something because I’m thinking Apple is stoopid.
Comment by terris — February 13, 2008 @ 10:57 pm
Terris– /Users has all the account info.
Comment by Matt — March 5, 2008 @ 12:49 am
The file is actually at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist (the “Contents” part of the path is missing from this blog post).
Comment by Barry K. Nathan — March 29, 2008 @ 8:01 pm
I just deleted the StdExclusions.plist. My backup is 600mb bigger. I am going to replace my HD on Monday, we’ll see when I restore it if I’ll have all my cache’s and DB’s.
Comment by Matthew Miller — April 19, 2008 @ 7:46 am
Update 1: I looked at my Time Machine Backup, all those paths/caches/DB’s now appear in my Time Machine Backup
Comment by Matthew Miller — April 19, 2008 @ 7:49 am
I just commented out all the exclusions in the plist file and the resulting backup.inProgress is currently 58GB of data. But I’d be wrong since it’s rewriting all the symlinks to the correct files. Right?
I’m assuming that this is the size of the (usually) excluded files on my system. Wow.
See progress image here: http://flickr.com/photos/luxuryluke/2438550718/
Comment by Luke Dorny — April 24, 2008 @ 4:02 am
I am installing a new hard drive in a few days on a titanium running leopard. I have my firewire
back up drive partitioned with a full copy of leopard installed as a boot drive in the smaller partition but not updated. When I do a “restore” with time machine to the new drive can I use the boot partition instead of the os x leopard disk in the dvd drive? Thanks, David
Comment by David Stearns — July 19, 2008 @ 7:10 pm
David:
You can, but it’ll be less smooth. You can install Leopard on the new hard drive, then from Macbuddy (the setup assistant) import users/applications/files from a time machine backup. Personally, when I did a restore after a dead hard drive, I found using the Leopard DVD to be much easier. My “install” was just the time machine restore; after my system rebooted, I was using my computer pre-hard-drive-crash.
Comment by Devin Lane — July 19, 2008 @ 7:21 pm
Thanks, I planned on a time machine restore, was just wondering if I still needed the boot disc or
if I could boot to the firewire and do the time machine restore. When I travel I would like to know I
can just leave the disk at home and rely on the firewire drive entirely. David
Comment by David Stearns — July 20, 2008 @ 10:21 am
David:
Unless you image the install DVD onto your firewire drive, you won’t be able to install Leopard at all.
Comment by Devin Lane — July 20, 2008 @ 4:13 pm
I installed it from the install disc to the drive. 20 gig partitioned for the os only. It worked just fine.
David
Comment by David Stearns — July 20, 2008 @ 5:06 pm
I just held the option key and rebooted. I works just like it should. I couldn’t open safari in the internal laptop drive, it would crash on open, but opens fine using the firewire drive version.
When I did the install I used all my same user name and passwords, after the install it would pick
the firewire as the main boot drive so I had to change that in system preferences. other than that
no hang ups.
Comment by David Stearns — July 20, 2008 @ 5:17 pm
David:
I misunderstood your original post. You must boot off the Installer DVD or a partition imaged from this DVD in order to perform a time machine restore instead of a standard installation. After the OS is installed, you can import users/applications/settings from the time machine backup with MacBuddy, but you’ll need enough space on that partition to fit all this data.
You cannot use an installation of Mac OS X to perform a time machine restore on another drive.
Comment by Devin Lane — July 20, 2008 @ 6:00 pm
thank you, that is what I was going to attempt. you saved me a lot of head aches and time. David
Comment by David Stearns — July 20, 2008 @ 10:17 pm
Sidenote FYI:
Your iTunes authentication information is stored in /Users/Shared/SC Info
You can remove or comment out that line in the StdExclusions.plist so that it’ll get backed up.
I’ve actually been bypassing my iTunes authentications for years by manipulating the ‘SC Info.sidb’ file in that directory.
Comment by AL — August 19, 2008 @ 10:03 am
To get apache working after a complete restore you need to create the apache2 log folder.
cd /private/var/log
sudo mkdir apache2
Comment by Simon — December 9, 2008 @ 9:24 pm
I just need to know can I potentially lose functionality of something if i do a full time machine restore because of the exclusions?
Comment by iconz113 — January 26, 2009 @ 1:58 pm
The reasons logs must be excluded is efficiency.
TimeMachine is only reasonably quick and efficient because it only has to write changes.
Logs change all the time, and the amount of useful backup history could be massively shortened if log files were backed up, since some of them are verbose, and constantly change, so they would be backed up every hour, a full new copy. It’s the same problem you have when you don’t exclude rw disk images that are used often: just mounting the disk image and unmounting it will modify it, and an entire e.g. 4GB Windows virtual machine disk image will get backed up. Once per hour!
What TimeMachine really needs is one more category of files:
history limited backup
Right now we can just back up, or exclude. What we need is a facility that says which files should be kept in the backup history, and which should not.
Typically for most files it would be sufficient to keep a current version, and ONE previous version. This would limit log files two two iterations, the System to the current OS version and the one before (allowing to restore to a known working system after a software upgrade that causes instability).
Only actual user data you need a long history for, and the less history we keep for things such as system related files and the actual OS, the more space there is for user data.
I submitted feedback and bug reports to Apple along these lines, but the more people do so, the more seriously Apple will take the issue.
Comment by rcfa — February 26, 2009 @ 2:42 pm
Wish I had read this earlier.
I put all my users on a second HDD inside my Mac Pro
I created a user called ‘home’….could not get TM to back it up.
Worked out that it must have been excluded, but would have saved a lot of agro if I had seen ‘home’ was excluded earlier !
Comment by Neil Paisnel — April 18, 2009 @ 12:36 pm
I favorited it. The last line is so true. I also wish life would be much easier…Thanks for sharing.
Comment by contra — May 17, 2009 @ 5:59 pm
thnx verry nice page
Comment by arkadas-arkadas ara — June 18, 2009 @ 12:50 pm