Time Machine Exclusions
So Time Machine is a pretty convenient way to backup your machine, and I use it to backup my laptop to an external FireWire drive. Although Time Machine backs up the “whole system,” I assumed there had to be some exclusions, such as cache files or /dev, for example. After a short bit of digging, I discovered that these paths are specified in a standard property list at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist
The full list is some 57 items, and is available below. Besides the expected cache items, the list includes some items I thought interesting:
/home */Library/Logs /Users/Guest /Library/Safari/Icons.db /.Spotlight-V100
Of these, the only one I find odd is the exclusion of logs. If your system goes haywire such that you restore it completely from a backup, it might be nice to see what went wrong.
[Update] Fixed the path to the StdExclusions.plist file.
Here’s the full list of the excluded paths:
Contents Excluded
The contents of these paths are excluded, but the directories themselves are preserved as they are required for a successful restore.
/Volumes /Network /automount /.vol /tmp /cores /private/tmp /private/Network /private/tftpboot /private/var/automount /private/var/log /private/var/folders /private/var/log/apache2 /private/var/log/cups /private/var/log/fax /private/var/log/ppp /private/var/log/sa /private/var/log/samba /private/var/log/uucp /private/var/run /private/var/spool /private/var/tmp /private/var/vm /private/var/db/dhcpclient /private/var/db/fseventsd /Library/Caches /Library/Logs /System/Library/Caches /System/Library/Extensions/Caches
Paths Excluded
These directories and their contents are excluded completely.
/.Spotlight-V100 /.Trashes /.fseventsd /.hotfiles.btree /Backups.backupdb /Desktop DB /Desktop DF /Network/Servers /Previous Systems /Users/Shared/SC Info /Users/Guest /dev /home /net /private/var/db/Spotlight /private/var/db/Spotlight-V100
User Paths Excluded
These directories are and their contents are excluded per-user. The path is relative to the user’s home folder.
Library/Application Support/MobileSync Library/Application Support/SyncServices Library/Caches Library/Logs Library/Mail/Envelope Index Library/Mail/AvailableFeeds Library/Mirrors Library/PubSub/Database Library/PubSub/Downloads Library/PubSub/Feeds Library/Safari/Icons.db Library/Safari/HistoryIndex.sk
It’s even worse than that. Today I did a full restore of my boot disk (by booting from the installation CD and selecting Restore in the Tools menu), and after that Apache2 keeps spurning out error messages that it can find its log files. Indeed, there is no Apache2 subdirectory anymore in the /var/logs directory and apparently Apache can’t create it…
Comment by menneke — November 15, 2007 @ 4:59 pm
Is your /var a symlink to /private/var? The log directory for apache2 (at least according to TimeMachine) is /private/var/log/apache2, is that the directory that your apache2 is trying to create? Furthermore, are you using the built in apache, or a copy obtained elsewhere?
Comment by Devin Lane — November 16, 2007 @ 2:01 am
Same problem here. Yes, “/var -> private/var”. Apache, postfix, and I think some others fail after restore, due to missing directories they’re not willing to recreate. Indications here:
http://discussions.apple.com/thread.jspa?messageID=6367763#6367763
suggest this is an intermittent problem: some restores work, some fail. creepy.
Comment by jackr — January 18, 2008 @ 1:14 am
I see that StdExclusions.plist “COntentsExcluded” lists /provate/var/log. It also lists /private/var/log/apache2, as if that directory might be backed up and then restored, but looking into my backup volume, I don’t see it anywhere. I suspect that the exclusion of the contents of /private/var/log/ is excluding even the directories for things like /private/var/log/apache2.
Comment by jackr — January 18, 2008 @ 1:43 am
Well, at least they had the decency to put them in a plist for us sysadmins…not that these exclusions are a big deal for ma+pa on their Time Capsule.
I think they leave out the logs because they’re supposed to be backed up for a week or two anyway, albeit not on a secondary drive.
Comment by Bug — February 5, 2008 @ 1:02 am
/home
WTF?
Am I missing something because I’m thinking Apple is stoopid.
Comment by terris — February 13, 2008 @ 10:57 pm
Terris– /Users has all the account info.
Comment by Matt — March 5, 2008 @ 12:49 am
The file is actually at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist (the “Contents” part of the path is missing from this blog post).
Comment by Barry K. Nathan — March 29, 2008 @ 8:01 pm
I just deleted the StdExclusions.plist. My backup is 600mb bigger. I am going to replace my HD on Monday, we’ll see when I restore it if I’ll have all my cache’s and DB’s.
Comment by Matthew Miller — April 19, 2008 @ 7:46 am
Update 1: I looked at my Time Machine Backup, all those paths/caches/DB’s now appear in my Time Machine Backup
Comment by Matthew Miller — April 19, 2008 @ 7:49 am
I just commented out all the exclusions in the plist file and the resulting backup.inProgress is currently 58GB of data. But I’d be wrong since it’s rewriting all the symlinks to the correct files. Right?
I’m assuming that this is the size of the (usually) excluded files on my system. Wow.
See progress image here: http://flickr.com/photos/luxuryluke/2438550718/
Comment by Luke Dorny — April 24, 2008 @ 4:02 am
I am installing a new hard drive in a few days on a titanium running leopard. I have my firewire
back up drive partitioned with a full copy of leopard installed as a boot drive in the smaller partition but not updated. When I do a “restore” with time machine to the new drive can I use the boot partition instead of the os x leopard disk in the dvd drive? Thanks, David
Comment by David Stearns — July 19, 2008 @ 7:10 pm
David:
You can, but it’ll be less smooth. You can install Leopard on the new hard drive, then from Macbuddy (the setup assistant) import users/applications/files from a time machine backup. Personally, when I did a restore after a dead hard drive, I found using the Leopard DVD to be much easier. My “install” was just the time machine restore; after my system rebooted, I was using my computer pre-hard-drive-crash.
Comment by Devin Lane — July 19, 2008 @ 7:21 pm
Thanks, I planned on a time machine restore, was just wondering if I still needed the boot disc or
if I could boot to the firewire and do the time machine restore. When I travel I would like to know I
can just leave the disk at home and rely on the firewire drive entirely. David
Comment by David Stearns — July 20, 2008 @ 10:21 am
David:
Unless you image the install DVD onto your firewire drive, you won’t be able to install Leopard at all.
Comment by Devin Lane — July 20, 2008 @ 4:13 pm
I installed it from the install disc to the drive. 20 gig partitioned for the os only. It worked just fine.
David
Comment by David Stearns — July 20, 2008 @ 5:06 pm
I just held the option key and rebooted. I works just like it should. I couldn’t open safari in the internal laptop drive, it would crash on open, but opens fine using the firewire drive version.
When I did the install I used all my same user name and passwords, after the install it would pick
the firewire as the main boot drive so I had to change that in system preferences. other than that
no hang ups.
Comment by David Stearns — July 20, 2008 @ 5:17 pm
David:
I misunderstood your original post. You must boot off the Installer DVD or a partition imaged from this DVD in order to perform a time machine restore instead of a standard installation. After the OS is installed, you can import users/applications/settings from the time machine backup with MacBuddy, but you’ll need enough space on that partition to fit all this data.
You cannot use an installation of Mac OS X to perform a time machine restore on another drive.
Comment by Devin Lane — July 20, 2008 @ 6:00 pm
thank you, that is what I was going to attempt. you saved me a lot of head aches and time. David
Comment by David Stearns — July 20, 2008 @ 10:17 pm
Sidenote FYI:
Your iTunes authentication information is stored in /Users/Shared/SC Info
You can remove or comment out that line in the StdExclusions.plist so that it’ll get backed up.
I’ve actually been bypassing my iTunes authentications for years by manipulating the ‘SC Info.sidb’ file in that directory.
Comment by AL — August 19, 2008 @ 10:03 am
To get apache working after a complete restore you need to create the apache2 log folder.
cd /private/var/log
sudo mkdir apache2
Comment by Simon — December 9, 2008 @ 9:24 pm
I just need to know can I potentially lose functionality of something if i do a full time machine restore because of the exclusions?
Comment by iconz113 — January 26, 2009 @ 1:58 pm
The reasons logs must be excluded is efficiency.
TimeMachine is only reasonably quick and efficient because it only has to write changes.
Logs change all the time, and the amount of useful backup history could be massively shortened if log files were backed up, since some of them are verbose, and constantly change, so they would be backed up every hour, a full new copy. It’s the same problem you have when you don’t exclude rw disk images that are used often: just mounting the disk image and unmounting it will modify it, and an entire e.g. 4GB Windows virtual machine disk image will get backed up. Once per hour!
What TimeMachine really needs is one more category of files:
history limited backup
Right now we can just back up, or exclude. What we need is a facility that says which files should be kept in the backup history, and which should not.
Typically for most files it would be sufficient to keep a current version, and ONE previous version. This would limit log files two two iterations, the System to the current OS version and the one before (allowing to restore to a known working system after a software upgrade that causes instability).
Only actual user data you need a long history for, and the less history we keep for things such as system related files and the actual OS, the more space there is for user data.
I submitted feedback and bug reports to Apple along these lines, but the more people do so, the more seriously Apple will take the issue.
Comment by rcfa — February 26, 2009 @ 2:42 pm
Wish I had read this earlier.
I put all my users on a second HDD inside my Mac Pro
I created a user called ‘home’….could not get TM to back it up.
Worked out that it must have been excluded, but would have saved a lot of agro if I had seen ‘home’ was excluded earlier !
Comment by Neil Paisnel — April 18, 2009 @ 12:36 pm
I favorited it. The last line is so true. I also wish life would be much easier…Thanks for sharing.
Comment by contra — May 17, 2009 @ 5:59 pm
thnx verry nice page
Comment by arkadas-arkadas ara — June 18, 2009 @ 12:50 pm
thanks for sharing
Comment by mini storage hk — July 11, 2009 @ 7:02 am
thanks so much for sharing. its gives me peace of mind (before i actually restore) to know what is truly excluded.
do you have any info on excluding certain folders while restoring from a time machine drive? I’m looking to do a fresh install and only wanted to restore certains parts from my external I use for time machine. any info would be most appreciated…
Comment by Charlotte — July 23, 2009 @ 7:29 pm
I used Time Capsule to back up from my iMac. I no longer have the iMac and would like to restore the iMac files from Time Capsule to my MacBook. I understand that I have to wipe the MacBook clean before I can even restore from Time Capsule. This worries me in case there is a problem with restoring my iMac back up to my MacBook. Also, I tried to get into my iMac back up from my MacBook and find I cannot. If I could I would only need to restore my iMac Address Book contents since I migrated most other files from my iMac to my MacBook prior to getting rid of my iMac (now wiped clean.)
Comment by wendi — August 23, 2009 @ 3:34 pm
To wendi:
First of all, why do that?…
Hardware is different… I suggest you leave the OSX alone, don’t restore any of it.
Better yet, restore only personal files and applications (no need to restore all of them). This might be safer and faster, I guess…
To All:
I’m so sorry I didn’t read befor about the excluded iTunes files… Could save me some worries, we’ll see.
I’m at the moment restoring my 320 GB HD in my 15″ MacBook Pro that became too small to a new 500 HD. Well, it started about 4+ hours ago and is still at 33% – it still says it will be more 7 hours to go :-( it looks like 8 or 9 are possible, a total of 11-12 hours for 230 GB of system and data load… Really, not very fast…. but what matters is that it works. Will come back here to tell the rest of the story tomorrow, LOL!…
Regards,
WiseMax
Comment by WiseMax — November 17, 2009 @ 7:00 pm
I don’t normally comment on blogs, however I have to say that I rather enjoyed yours as it was informative. I´ve bookmarked your blog and hope to explorer it further when I have a little more time. Keep up the good work. Well back to my dreaming of Panama or back to the books – I wonder which one is going to win out. :)
Comment by Scott Thibaut Hancock — December 1, 2009 @ 7:57 am
thanks to the autor
Comment by colonies — January 29, 2010 @ 2:35 pm
Good posts on this blog make me read this blog again and again i liked this blog very much This blog has got all the useful stuff that i wanted to know about.
Comment by PLC programming — February 8, 2010 @ 10:28 pm
Do you use wordpress on your blog? I like the look and feel of your blog, I’ve bookmarked your page.
Darci Challenger
Comment by Darci Challenger — March 6, 2010 @ 4:19 am
i am new here,so i want to know more about this site’s information.
Comment by Term Papers — March 8, 2010 @ 10:46 am
It is glad to see this blog, it is good and detailed fun to read this, nice informative blog, Thanks for share this article.
Comment by College Research Papers — March 9, 2010 @ 6:22 am