Time Machine Exclusions

So Time Machine is a pretty convenient way to backup your machine, and I use it to backup my laptop to an external FireWire drive. Although Time Machine backs up the “whole system,” I assumed there had to be some exclusions, such as cache files or /dev, for example. After a short bit of digging, I discovered that these paths are specified in a standard property list at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist

The full list is some 57 items, and is available below. Besides the expected cache items, the list includes some items I thought interesting:

/home
*/Library/Logs
/Users/Guest
/Library/Safari/Icons.db
/.Spotlight-V100

Of these, the only one I find odd is the exclusion of logs. If your system goes haywire such that you restore it completely from a backup, it might be nice to see what went wrong.

[Update] Fixed the path to the StdExclusions.plist file.

Here’s the full list of the excluded paths:

Contents Excluded

The contents of these paths are excluded, but the directories themselves are preserved as they are required for a successful restore.

/Volumes
/Network
/automount
/.vol
/tmp
/cores
/private/tmp
/private/Network
/private/tftpboot
/private/var/automount
/private/var/run
/private/var/tmp
/private/var/vm
/private/var/db/dhcpclient
/private/var/db/fseventsd
/Library/Caches
/Library/Logs
/System/Library/Caches
/System/Library/Extensions/Caches

Paths Excluded

These directories and their contents are excluded completely.

/.Spotlight-V100
/.Trashes
/.fseventsd
/.hotfiles.btree
/Backups.backupdb
/Desktop DB
/Desktop DF
/Network/Servers
/Previous Systems
/Users/Shared/SC Info
/Users/Guest
/dev
/home
/net
/private/var/db/Spotlight
/private/var/db/Spotlight-V100

User Paths Excluded

These directories and their contents are excluded per-user. The path is relative to the user’s home folder.

Library/Application Support/MobileSync
Library/Application Support/SyncServices
Library/Caches
Library/Logs
Library/Mail/Envelope Index
Library/Mail/AvailableFeeds
Library/Mirrors
Library/PubSub/Database
Library/PubSub/Downloads
Library/PubSub/Feeds
Library/Safari/Icons.db
Library/Safari/HistoryIndex.sk

Posted in Software | 40 Comments

40 Responses to Time Machine Exclusions

  1. menneke says:

    It’s even worse than that. Today I did a full restore of my boot disk (by booting from the installation CD and selecting Restore in the Tools menu), and after that Apache2 keeps spurning out error messages that it can find its log files. Indeed, there is no Apache2 subdirectory anymore in the /var/logs directory and apparently Apache can’t create it…

  2. Devin Lane says:

    Is your /var a symlink to /private/var? The log directory for apache2 (at least according to TimeMachine) is /private/var/log/apache2, is that the directory that your apache2 is trying to create? Furthermore, are you using the built in apache, or a copy obtained elsewhere?

  3. jackr says:

    Same problem here. Yes, “/var -> private/var”. Apache, postfix, and I think some others fail after restore, due to missing directories they’re not willing to recreate. Indications here:
    http://discussions.apple.com/thread.jspa?messageID=6367763#6367763
    suggest this is an intermittent problem: some restores work, some fail. creepy.

  4. jackr says:

    I see that StdExclusions.plist “COntentsExcluded” lists /provate/var/log. It also lists /private/var/log/apache2, as if that directory might be backed up and then restored, but looking into my backup volume, I don’t see it anywhere. I suspect that the exclusion of the contents of /private/var/log/ is excluding even the directories for things like /private/var/log/apache2.

  5. Bug says:

    Well, at least they had the decency to put them in a plist for us sysadmins…not that these exclusions are a big deal for ma+pa on their Time Capsule.

    I think they leave out the logs because they’re supposed to be backed up for a week or two anyway, albeit not on a secondary drive.

  6. terris says:

    /home

    WTF?

    Am I missing something because I’m thinking Apple is stoopid.

  7. Matt says:

    Terris– /Users has all the account info.

  8. Barry K. Nathan says:

    The file is actually at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist (the “Contents” part of the path is missing from this blog post).

  9. Matthew Miller says:

    I just deleted the StdExclusions.plist. My backup is 600mb bigger. I am going to replace my HD on Monday, we’ll see when I restore it if I’ll have all my cache’s and DB’s.

  10. Matthew Miller says:

    Update 1: I looked at my Time Machine Backup, all those paths/caches/DB’s now appear in my Time Machine Backup

  11. Luke Dorny says:

    I just commented out all the exclusions in the plist file and the resulting backup.inProgress is currently 58GB of data. But I’d be wrong since it’s rewriting all the symlinks to the correct files. Right?
    I’m assuming that this is the size of the (usually) excluded files on my system. Wow.
    See progress image here: http://flickr.com/photos/luxuryluke/2438550718/

  12. I am installing a new hard drive in a few days on a titanium running leopard. I have my firewire
    back up drive partitioned with a full copy of leopard installed as a boot drive in the smaller partition but not updated. When I do a “restore” with time machine to the new drive can I use the boot partition instead of the os x leopard disk in the dvd drive? Thanks, David

  13. Devin Lane says:

    David:

    You can, but it’ll be less smooth. You can install Leopard on the new hard drive, then from Macbuddy (the setup assistant) import users/applications/files from a time machine backup. Personally, when I did a restore after a dead hard drive, I found using the Leopard DVD to be much easier. My “install” was just the time machine restore; after my system rebooted, I was using my computer pre-hard-drive-crash.

  14. Thanks, I planned on a time machine restore, was just wondering if I still needed the boot disc or
    if I could boot to the firewire and do the time machine restore. When I travel I would like to know I
    can just leave the disk at home and rely on the firewire drive entirely. David

  15. Devin Lane says:

    David:

    Unless you image the install DVD onto your firewire drive, you won’t be able to install Leopard at all.

  16. I installed it from the install disc to the drive. 20 gig partitioned for the os only. It worked just fine.
    David

  17. I just held the option key and rebooted. I works just like it should. I couldn’t open safari in the internal laptop drive, it would crash on open, but opens fine using the firewire drive version.
    When I did the install I used all my same user name and passwords, after the install it would pick
    the firewire as the main boot drive so I had to change that in system preferences. other than that
    no hang ups.

  18. Devin Lane says:

    David:

    I misunderstood your original post. You must boot off the Installer DVD or a partition imaged from this DVD in order to perform a time machine restore instead of a standard installation. After the OS is installed, you can import users/applications/settings from the time machine backup with MacBuddy, but you’ll need enough space on that partition to fit all this data.

    You cannot use an installation of Mac OS X to perform a time machine restore on another drive.

  19. thank you, that is what I was going to attempt. you saved me a lot of head aches and time. David

  20. AL says:

    Sidenote FYI:

    Your iTunes authentication information is stored in /Users/Shared/SC Info
    You can remove or comment out that line in the StdExclusions.plist so that it’ll get backed up.

    I’ve actually been bypassing my iTunes authentications for years by manipulating the ‘SC Info.sidb’ file in that directory.

  21. Simon says:

    To get apache working after a complete restore you need to create the apache2 log folder.

    cd /private/var/log
    sudo mkdir apache2

  22. iconz113 says:

    I just need to know can I potentially lose functionality of something if i do a full time machine restore because of the exclusions?

  23. rcfa says:

    The reasons logs must be excluded is efficiency.
    TimeMachine is only reasonably quick and efficient because it only has to write changes.
    Logs change all the time, and the amount of useful backup history could be massively shortened if log files were backed up, since some of them are verbose, and constantly change, so they would be backed up every hour, a full new copy. It’s the same problem you have when you don’t exclude rw disk images that are used often: just mounting the disk image and unmounting it will modify it, and an entire e.g. 4GB Windows virtual machine disk image will get backed up. Once per hour!

    What TimeMachine really needs is one more category of files:
    history limited backup

    Right now we can just back up, or exclude. What we need is a facility that says which files should be kept in the backup history, and which should not.

    Typically for most files it would be sufficient to keep a current version, and ONE previous version. This would limit log files two two iterations, the System to the current OS version and the one before (allowing to restore to a known working system after a software upgrade that causes instability).

    Only actual user data you need a long history for, and the less history we keep for things such as system related files and the actual OS, the more space there is for user data.

    I submitted feedback and bug reports to Apple along these lines, but the more people do so, the more seriously Apple will take the issue.

  24. Neil Paisnel says:

    Wish I had read this earlier.

    I put all my users on a second HDD inside my Mac Pro
    I created a user called ‘home’….could not get TM to back it up.

    Worked out that it must have been excluded, but would have saved a lot of agro if I had seen ‘home’ was excluded earlier !

  25. contra says:

    I favorited it. The last line is so true. I also wish life would be much easier…Thanks for sharing.

  26. arkadas-arkadas ara says:

    thnx verry nice page

  27. Charlotte says:

    thanks so much for sharing. its gives me peace of mind (before i actually restore) to know what is truly excluded.

    do you have any info on excluding certain folders while restoring from a time machine drive? I’m looking to do a fresh install and only wanted to restore certains parts from my external I use for time machine. any info would be most appreciated…

  28. wendi says:

    I used Time Capsule to back up from my iMac. I no longer have the iMac and would like to restore the iMac files from Time Capsule to my MacBook. I understand that I have to wipe the MacBook clean before I can even restore from Time Capsule. This worries me in case there is a problem with restoring my iMac back up to my MacBook. Also, I tried to get into my iMac back up from my MacBook and find I cannot. If I could I would only need to restore my iMac Address Book contents since I migrated most other files from my iMac to my MacBook prior to getting rid of my iMac (now wiped clean.)

  29. WiseMax says:

    To wendi:

    First of all, why do that?…

    Hardware is different… I suggest you leave the OSX alone, don’t restore any of it.

    Better yet, restore only personal files and applications (no need to restore all of them). This might be safer and faster, I guess…

    To All:

    I’m so sorry I didn’t read befor about the excluded iTunes files… Could save me some worries, we’ll see.

    I’m at the moment restoring my 320 GB HD in my 15″ MacBook Pro that became too small to a new 500 HD. Well, it started about 4+ hours ago and is still at 33% – it still says it will be more 7 hours to go :-( it looks like 8 or 9 are possible, a total of 11-12 hours for 230 GB of system and data load… Really, not very fast…. but what matters is that it works. Will come back here to tell the rest of the story tomorrow, LOL!…

    Regards,
    WiseMax

  30. colonies says:

    thanks to the autor

  31. PLC programming says:

    Good posts on this blog make me read this blog again and again i liked this blog very much This blog has got all the useful stuff that i wanted to know about.

  32. NewInfo says:

    I know this is two and a half years after this was posted, but it seems to get referenced several places. For those wondering, at some point Apple got enough complaints to fix the problems listed here.

    First, they’ve removed the following from “Contents Excluded”
    /private/var/log
    /private/var/log/apache2
    /private/var/log/cups
    /private/var/log/fax
    /private/var/log/ppp
    /private/var/log/sa
    /private/var/log/samba
    /private/var/log/uucp
    /private/var/spool

    And they’ve created a fourth category in the list:

    FileContentsExcluded
    – paths where we need to capture entire subtree folder layout to restore disk structure, but don’t want to backup contained files

    /private/var/log
    /private/var/spool/cups
    /private/var/spool/fax
    /private/var/spool/uucp

  33. Lisa Peet says:

    Thanks for sharing

  34. TC says:

    In regards to NewInfo’s comment about Apple’s changes to the exclusions list. This change seems to have occurred in 10.6, so if you are running 10.5.x the initial listing is still valid.

  35. Gus says:

    I have read a few postings and articles regarding Time Machine exclusions, and I have not been able to figure out what has gone wrong since my upgrade to 10.6. If anyone happens upon this post and has some input, I would greatly appreciate it.

    To avoid hassles with FileVault, I don’t use it. Instead, at login I mount an encrypted disk image that I use to hold all client data and I do not store the password in my keychain.

    Under Leopard, Time Machine would backup the files inside this mounted volume just like any other drive/volume. I could exclude the folder where the disk image itself was stored (though backing up a sparse image could partially address this under Snow Leopard) so as not to have massive backups, and I could rely on Time Machine to make backups of the individual files, revisions, etc. inside the volume.

    Under Snow Leopard, Time Machine seems to not even try to backup the contents of the mounted volume. I cannot quite figure out why or how this is excluded. Maybe I am doing something else wrong that I am not detecting. I did, for a variety of reasons (e.g., CrashPlan, etc.), make a new encrypted volume using a sparse image but I don’t see how that would cause the problem.

    How does Time Machine know to backup external drive volumes that are mounted and yet not to backup volumes that are mounted from a disk image?

    Any thoughts? Thanks very much.

  36. Here on Snow Leopard, I see that the User Paths Excluded changed a bit:

    Library/Application Support/SyncServices/data.version
    Library/Caches
    Library/Logs
    Library/Mail/Envelope Index
    Library/Mail/AvailableFeeds
    Library/Mirrors
    Library/PubSub/Database
    Library/PubSub/Downloads
    Library/PubSub/Feeds
    Library/Safari/Icons.db
    Library/Safari/WebpageIcons.db
    Library/Safari/HistoryIndex.sk

  37. Well, life was never easy. You have only driven in the point further.

  38. Ahmad says:

    I read a lot of interesting posts here. Probably you spend a lot of
    time writing, i know how to save you a lot of time, there is an online
    tool that creates high quality, SEO friendly posts in seconds, just type in google –
    laranitas free content source

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>