mac.column.ted: Leopard still holds some small surprises

Ted Landau

May 2008

Leopard is a surprising cat. Here it is, more than half year after Mac OS X 10.5's release, and I'm still discovering new features. I'm not talking about the big time features that are well publicized on Apple's Web site. Rather, I am referring to those much smaller, tucked-away features ? ones that tend to appeal more to "techies" than the mass market. These are features that Apple barely mentions in its PR, if it takes note of them at all. Of course, that's probably not unrelated to why it took me so long to discover some of them.

I now have a list of about a half-dozen of these quite recent discoveries (at least they were recent for me). For example, just last week, I came across a new "Reset Permissions and ACLs" feature in Reset Password, a utility accessed by booting from the Leopard Install DVD. Here are two other items from my list that I found particularly intriguing:

xattr and extended attributes

Briefly, extended attributes are a UNIX feature ? first introduced to Mac OS X in Tiger. In Tiger, extended attributes were largely limited to Mac OS X Server. So don't be surprised if you never heard about them back then. With Leopard, extended attributes have come into full bloom for both the server and client OS. [Giving credit where it is due, the basics of Leopard's extended attribute capabilities were documented as far back as October 2007, such as in this ars technica review.]

Loosely defined, attributes are metadata that are linked to individual files and folders. For example, the creation date of a file is an attribute of the file. Extended attributes are metadata that go beyond the standard set associated with items of a particular kind.

In Leopard, one such extended attribute assists in protecting your Mac from malicious software. You're probably more than familiar with the following warning message: "{name of program} is an application which was downloaded from the Internet. Are you sure you want to open it?" This message appears the first time you launch a program after having downloaded it. This is a security precaution, designed to have you confirm that the program is indeed one that you recognize and intended to launch. If you select to open it, the warning message should no longer appear on subsequent launches.

How, you may wonder, does Mac OS X know that a given application has never been launched before? It knows because Mac OS X (Safari specifically, which is why the message does not appear if you instead download from Firefox, for instance) assigns an extended attribute quarantine flag to the downloaded item. When the flag is detected on launch, the message appears. The flag is cleared after you first open the application, which is why the message doesn't appear again.

More generally, suppose you want to know if a given file has any extended attributes. You can do this easily enough, using Terminal. Here's how:

1. Use the cd command to go to the directory that contains the file or folder of interest
   (such as cd ~/Downloads to go to your Downloads folder).
2. Type: ls -l

Assuming you have an assortment of stuff in your Downloads folder, you will get output that looks something like this:

Take a look at the leftmost column of data. This shows the read-write-execute (rwx) permissions for each item. In particular, notice that there is an @ character at the end of the permissions listings for some items. This character means that the item contains at least one extended attribute. Given that we are in the Downloads folder, it is a good bet that the quarantine flag is an extended attribute here.

To find out for certain if the quarantine attribute is present, you can list all the extended attributes for a given file. To do so, use the xattr command. This command is not well documented, but it's pretty simple to use. For example, if I wanted to see the extended attributes for the Mac_DLS78.dmg file (which is for Dymo Label Software), I would enter the following command:

xattr -l Mac_DLS78.dmg

Here are the results:

For our purposes, we can ignore all the output except the item at the very bottom, the one that starts with com.apple.quarantine. This confirms that the item does indeed contain a quarantine attribute.

By the way, if I were to double-click and open the disk image (.dmg) file, the warning message would not appear. This is because the .dmg file is not itself an application. However, any application contained within the image "inherits" the image's quarantine attribute. Thus, if I mount the image and launch its Dymo Label Software application, the warning message does appear.

Now, suppose I want to eliminate the flag from the image file, so as to avoid getting any warnings. Once again, we use the xattr command to do this. Type:

xattr -d com.apple.quarantine Mac_DLS78.dmg

If I were to now re-enter the xattr -l command, the quarantine item would no longer appear in the output. And, if I were to mount the image and launch the Dymo Label application, the warning message would no longer pop-up.

At this point, you may be asking, why bother? Why not just wait for the warning message to appear, click OK, and clear the flag? Isn't that a lot simpler than hassling with the xattr command? Yes, it is. However, there are a few situations where the xattr command can come in handy.

I had one such situation crop up recently, after I installed some services into the /Library/Services folder. For some reason, each and every time I later selected one of these items from a Services menu, I would get the quarantine warning message. In other words, the flag refused to clear. I have a theory as to how this could have been avoided (but I won't bore you with it). At this point, the simplest solution was to use the xattr command to get rid of the attribute. I did so ? and the messages never appeared again!

I tripped over another such situation after following the links from a TidBITs article to the CHDK (Canon Hackers Developers Kit) wiki. From here, you can download software that you copy onto a memory card for your Canon point-and-shoot camera. This, in turn, adds new features to the camera (such as the ability to save photos in RAW format). However, this wiki page notes that the hack may not succeed if you are using a Mac ? because the downloaded files contain a quarantine attribute which prevents the Canon camera from loading the software.

The solution is to delete the attribute using xattr. The wiki page also offers an alternative solution. The relevant software downloads as a zip file. Normally, you would just double-click the file to have Mac OS X expand it. However, this causes the quarantine flag to be attached to the expanded items. If you instead use a third-party utility, such as Unarchiver, the zip file expands without attaching the quarantine flag to the expanded items. Problem solved. Although the wiki did not say this, I assume that using a browser other than Safari would also avoid the problem.

Sparse Bundle images

Sparse bundle images are another new feature in Leopard ? one you may be using even without realizing it.

Image files are often used for software downloaded from the Internet (as in the Dymo Label image cited in the previous section). When you double-click an image, it mounts almost as if you were mounting an external drive. You can then access the contents of the image file. If it is a writable image, you can even add content to it. While all images function in a similar fashion, there are actually several different types, each with its own pros and cons. Different types may also use different name extensions ? the most common one being .dmg. You can see a partial list of the different image types by setting up to create an image file yourself. To do so:

1. Launch Disk Utility.
2. Choose New > Blank Disk Image... from the File menu.
3. From the dialog that appears, select the Image Format popup menu.
4. From here, we are interested only in the bottom two options: sparse disk image and sparse bundle disk image. Choose the latter option to create a new image file in the sparse bundle format. The file name will have a .sparsebundle extension by default.

[To see a much longer list of image types, select File > New > Disk Image from Folder..., choose a folder and select the Image Format popup menu.]

Sparse disk images were available in Tiger; it is only the sparse bundle format that is new to Leopard. Both types share a similar and very useful attribute: The disk image can expand beyond its initial size, as you add more content to it. This allows you to create a functional disk image even if you don't know ultimately how much data you will be putting on it.

So why was the bundle image format added in Leopard? Because there was a significant problem with plain sparse images. A sparse image is essentially a single file. When backing up your drive, a backup utility thus sees the image as a single file, regardless of how many files are stored within the image. Further, any addition or subtraction you make to the image (such as adding even a measly 5K text document) registers the image as a modified file. This means that, if an image file were 1GB in size, the entire 1GB would need to be recopied to a backup each time the image was modified, even if the only change to the image was a 5K file addition. Not very efficient. And unnecessarily time consuming.

The sparse bundle format avoids this dilemma. Essentially, the bundle format divides the content of the image file into smaller separable bands. The image still appears as a single file in the Finder. However, it is actually a package. If you select Show Package Contents from the image's contextual menu in the Finder, you will find a bands folder containing the individual band segments (as shown in the figure below). Each band, at least in my testing, was 8MB or less. Assuming your backup software recognizes and works correctly with the bundle format, only the modified bands are copied over when backing up the image. This means that backing up the aforementioned 1GB image, with a 5K file addition, would require copying only 8MB or less!

[You can learn more about the workings of the sparse and sparse bundle formats, by accessing the manual page for the hdiutil command.]

Apple, in Disk Utility's Help pages, recommends using the sparse bundle format whenever you want to create "a blank disk image for storage." Indeed, Apple takes its own advice and uses the new format for FileVault (rather than the sparse image format used by FileVault in Tiger).

Similarly, and of particular interest here, Time Machine may also use the new sparse bundle format. It turns out that, if you back up your hard drive over a network (such as to a Time Capsule or to a USB hard drive attached to an AirPort Extreme Base Station), the backup is created and saved as a sparse bundle image. Using this image format is apparently necessary to avoid problems that Time Machine would otherwise have when trying to make incremental backups over a network.

In contrast, Time Machine does not use any image format when backing up to a drive connected directly to a Mac, via a USB or FireWire port. In this case, the individual files are copied directly and can be viewed as such in the Finder (inside a Backups.backupdb folder).

What if you've been using Time Machine to back up to a network drive and later decide to connect the drive directly to your Mac? Will Time Machine continue to work with it? As far as I can tell, no. For starters, Time Machine will not automatically recognize that this is your backup drive. You will need to reselect it first, using the Change Disk... option in the Time Machine System Preferences pane. Even after you do, Time Machine will not work properly with the sparse bundle image and you won't see any backup data listed when you open the Time Machine application. The best solution at this point is to reformat the backup drive and start over.

Note: I have seen some discussions on the Web, querying whether it is possible to avoid the need to start over by instead extracting the data from the image file and replacing the image file with the extracted data. However, I have not read of any success in these attempts. Similarly, it appears doubtful that you can take a Time Machine drive connected directly to your Mac and have it recognize your existing backup data if you move it to an AirPort Extreme. Again, your best bet when making these sorts of changes is to start over.

To send comments regarding this column directly to Ted, click here. To get Ted's latest book, Take Control of Your iPhone, click the link.

Resources