Lesson 26. Maintaining Local Volumes and Files

Time	This lesson takes approximately 1 hour to complete.
Goals	Describe the advantages and disadvantages of HFS Plus- and UFS-formatted volumes Use df and du to determine and monitor disk space usage Describe how Mac OS X handles aliases and symbolic links Monitor and troubleshoot file-system issues from the command-line interface using tools such as df, du, diskutil, drutil, and disktool Describe factors that influence disk space needs

Mac OS X supports several file-system types. In this lesson, the first of four focusing on file services, you will learn how the differences between these types can affect the experience of users and system administrators. For example, Mac OS X stores metadata differently on different file-system types, which affects the appearance and behavior of files.

Mac OS X has inherited features from UNIX, as well as from earlier versions of the operating system. Sometimes the UNIX features and Apple features are similar in purpose but different in behavior. This lesson covers symbolic links and aliases, which are also similar in purpose yet are different enough to possibly cause confusion.

Mac OS X keeps track of mounted volumes by means of disk arbitration, another topic focused on in this lesson, and you’ll learn to update that information from the command line. You will also learn several command-line tools for monitoring and troubleshooting file systems.

Understanding File-System Formats

Mac OS X supports several file-system formats, whose characteristics are outlined in the following table.

• Hierarchical File System Plus (HFS Plus): This is the file-system type used on Mac OS 9 and earlier systems. It is also known as Mac OS Extended File System. In Mac OS X version 10.2, you could enable journaling on new or existing HFS Plus volumes using Disk Util. As of Mac OS X v10.3 and going forward, newly created HFS Plus file systems have journaling enabled by default.

Occasionally, HFS Plus causes problems with UNIX programs and shell scripts that assume case sensitivity. You can format a new HFS Plus volume as case-sensitive by using Disk Utility or diskutil from Terminal. Many third-party Mac applications do not work correctly with case-sensitive file systems, however, so you generally don’t want to use a case-sensitive file system, especially on a Mac OS X boot volume.

• UNIX File System (UFS): This is similar to other UNIX file systems in that it is case-sensitive. The problem with UFS on Mac OS X is that it does not support forked files. For that reason, UFS is not recommended as the root volume on Mac OS X. Another feature of UFS is that Mac OS 9 cannot see UFS file systems. This can be an advantage because Mac OS 9 is less secure.

• MS-DOS: Mac OS X v10.3 and up also support the Windows disk format, MS-DOS (FAT). This feature allows you to use an external drive or flash memory device that was formatted for a Windows computer on your Mac OS X computer. Another Windows disk format, NTFS, is supported in read-only mode. Like UFS, Windows file systems do not support forked files.

• Other formats: Mac OS X supports the Andrew File System (AFS), a common networked file system that is particularly popular among educational institutions, with the help of a package available from openafs.org. It also supports the standard ISO 9660 format for CDs and DVDs.

Identifying Resource and Data Forks

Files on Mac OS X can be composed of two distinct halves called forks. Much more popular in the OS 9 days, resource forks were used to contain objectlike information such as icons, strings, and small chunks of code, while the data fork was used for unclassified raw data such as that involved in composing an image file. In Mac OS X, most files keep their data exclusively in the data fork, but resource forks are still used by older applications, and even by Mac OS X to store elements such as custom file icons.

Copying Forked Files on HFS Plus

When you use the graphical user interface to move files around, the File Manager built in to Mac OS X handles forked files automatically.

As of Mac OS X v10.4, most command-line utilities, such as cp, mv, and tar, now handle forked files properly. Previous versions of Mac OS X required command-line utilities that were designed for use with forked files. Two such commands—ditto (with the –rsrcFork option) and CpMac—were specifically written to handle forked files correctly. Though it is no longer necessary to use these utilities, ditto is still included on all Mac OS X systems. You must install the Developer Tools package to get CpMac, which will then be placed in the /Developer/Tools/ directory.

The following figure shows what happens when you copy files within an HFS Plus file system or between two such file systems using cp, tar, ditto –rsrcFork, or CpMac.

You can use a number of commands to check whether a file has a resource fork. If you need to check only for the existence of a resource fork, you can simply use the ls command. Normally you look at a long file listing with the –l option:

ls –l MyFile
–rw–r– – r– – 1 david david 3624 Oct 20 22:02 MyFile

In the line above, the data fork is 3,624 bytes. You can examine the resource fork by looking at what acts like a hidden object inside the file, as follows:

ls –l MyFile/..namedfork/rsrc
–rw–r– – r– – 1 david david 9928 Oct 20 22:02 MyFile/..namedfork/rsrc

Although it’s redundant, you can also use this method to look specifically at the data fork, like this:

ls –l MyFile/..namedfork/data
–rw–r– – r– – 1 david david 3624 Oct 20 22:02 MyFile/..namedfork/data

You can dump the contents of a resource fork by using the developer tool DeRez. Note that the Mac OS 9 application program SimpleText appears to be empty when you look at it with ls. All the content of this file is in the resource fork. Note that the output of DeRez is the same whether the file has a resource fork or a shadow file (shadow files are discussed in the next section).

Copying Forked Files to Other File Systems

When you use the Finder to move forked files to a file system that does not support forks, such as UFS, the File Manager puts the resource fork information into shadow files. A shadow file has the same name as the original file, with ._ prepended to its name. When you copy the file back to an HFS Plus file system, the File Manager re-creates the specific file forks.

The same happens when you use ditto –rsrcFork or CpMac to copy files to and from a non–HFS Plus file system.

Copying Forked Files With Non-Mac Applications

In Mac OS X versions prior to v10.4, ordinary UNIX command-line utilities, such as cp and mv, were not designed to handle file forks or shadow files. When you used cp to copy forked files or files with shadow files, you lost the resource fork or shadow file. This was true whether the destination was HFS Plus or another file system. DeRez will also report that the copy has no resource fork and ls lists no shadow files.

Similarly, applications on Windows computers were not designed to handle file forks or shadow files. This can be an issue if, for example, you transfer files back and forth between your Mac OS X computer and a Windows computer using a flash memory storage device. If you use the Finder or CpMac to copy the files to the device, the files will preserve their shadow files. But if you rename the files or move them around while the device is connected to the Windows computer, the shadow files are lost.

In some cases, you won’t notice any problems if the resource fork is missing. In other cases, however, you may notice missing file information. Fonts may be absent from a document. Mac OS X might not know what application to use to open the file. In most cases, a file’s custom icon will be replaced with a more generic icon.

Aliases and Symbolic Links

Sometimes you want users to be able to access a file by a different name, or from a different file-system location than the one in which it actually resides. You may have a script or program that depends on a file-system hierarchy that’s different from what you have. You may have multiple versions of a document or folder and want a link called Latest to always point to the most current one.

You can use two types of files on Mac OS X to access files and folders indirectly. One is an alias, which you create in the Finder. The other is a symbolic link, which you create using the command line. Symbolic links are common on UNIX systems. Classic applications handle symbolic links correctly.

Some examples of symbolic links in Mac OS X exist because of an extra folder, called private, which was held over from NEXTSTEP. private contains other directories like etc, var, tmp, and Network. Mac OS X uses symbolic links for each of these folders and so the operating system thinks these folders actually exist at the root level of the boot volume, not one level deep inside private, where they actually exist.

Mac OS X makes liberal use of symbolic links. You can locate links using the find command:

find / –type l –ls

Aliases and symbolic links both act as pointers to files and folders, but their behavior is different. At the very basic level, the Finder understands both aliases and symbolic links but UNIX via the Terminal has no concept of aliases so they show up as a zero byte length file.

Using Aliases

You create an alias to a file or folder using Make Alias from the Finder’s File menu. The following figure shows an alias to the folder AFP548. The arrow on the folder icon indicates it is an alias. If you use Get Info to display more information about the alias, you see that Kind is listed as Alias and that Original shows the location of the folder to which it points.

Move Original Files

If you move the original folder somewhere else—in this case, into the folder called EditedWebSites—and use Get Info to display information about the alias again, you’ll see that the information has changed. Now Original shows the new location of the original folder. Mac OS X automatically updates aliases when the original file or folder is moved. Keep in mind that dragging a file or folder to the Trash is not deleting—it’s moving it to the Trash. The file still exists, and an alias will continue to point to it.

Alias data is stored in the resource fork of a file. You can use the techniques discussed earlier to look at the resource fork of an alias. To see the raw data stored about an alias, issue the following command:

/Developer/Tools/DeRez  YourAliasFile –only alias

You can also get data using the strings command:

strings YourAliasFile/..namedfork/rsrc

You’ll see some of the path information in the output.

Delete and Replace Original Files

If, instead of moving the original folder, you delete it from the computer completely (for example, by using rm from Terminal), Get Info will show that the alias no longer points to anything. If you re-create the folder in the original location, the alias will point to the new folder.

In Mac OS X v10.1, alias resolution was based upon an HFS Plus identifier called the file ID. Starting with Mac OS X v10.2, alias resolution attempts to resolve the file pathname before attempting to resolve the alias using the file ID. This behavior corrects a problem in Mac OS X v10.1 that prevented aliases from working as expected on volumes restored from a backup copy.

Creating Symbolic Links

Symbolic links originated in the UNIX world. You create a symbolic link using ln with the –s option. In the following figure, a symbolic link is created to AFP548Site. A long listing shows what the link points to. Note that the file mode begins with the lowercase letter l.

$ ln -s AFP584Site AFP548Site_link

$ ls -l
drwxr-xr-x 2 david david 68 13 Jan 10:10 AFP584Site
lrwx------ 1 david david 8 13 Jan 10:12 AFP548Site_link -> AFP584Site

$ ls -l AFP548Site/
total 56
-rw-r--r-- 1 david david   722 13 Jan 10:16 MyDoc.rtf
-rw-r--r-- 1 david david 21190 13 Jan 10:17 MyPhoto.jpg
$ ls -l AFP548Site_link/
total 56
-rw-r--r-- 1 david david   722 13 Jan 10:16 MyDoc.rtf
-rw-r--r-- 1 david david 21190 13 Jan 10:17 MyPhoto.jpg

Move Original Files

When you look at a symbolic link in the Finder, it looks just like an alias. In fact, if you use Get Info to display more information, you will see that Kind is listed as Alias and that Original shows the location of the original folder. But something different happens when you move the original file somewhere else, as illustrated by the following figure.

First of all, the link’s icon changes (it may take a Finder refresh or a log out to see the icon change). Additionally, when you use Get Info, you see that it still points to a file that does not exist and it cannot be reassociated with another original here, as the button Select New Original cannot be chosen.

Symbolic links are not updated when you move the original file or folder. A symbolic link that no longer points to anything is called a dangling symbolic link.

Re-create an Original Path

A symbolic link is set to a fixed path when it is created. If you move or delete the original file and then place another file at the original location, the symbolic link will point to the new file. This can be useful, because you do not have to re-create the link when you replace the file. This is very useful when you update the files upon which the links rely.

Comparing Alias and Symbolic Links

The following table summarizes the differences between aliases and symbolic links. It lists how each one is created, how it appears in the Finder and the command line, and what happens when you move the file or directory pointed to.

Mounting and Disk Arbitration

The disk arbitration daemon, diskarbitrationd, monitors disk connections and maintains a disk arbitration table, which is a list of disks available to graphical user interface applications such as the Finder. When you insert a disk or mount a file system from the Finder, diskarbitrationd probes the disk to determine its file-system format, checks the disk for issues, mounts the file system, and updates the disk arbitration table.

Using Commands

There are several commands that permit you to manage disk mounts and troubleshoot mounting issues.

disktool

Sometimes you need to update the disk arbitration table manually by using the command disktool –r. Although disktool is deprecated for most of its uses, it is still useful for refreshing the disk arbitration daemon. Use this command after mounting a disk drive or disk image from the command line. Although you can use disktool for much more, it’s best to use diskutil for most features. Most features of disktool require root access to the volume. The disktool command has no man page. For more information, type disktool with no options.

diskutil

The diskutil command provides functionality similar to that of the Mac OS X application Disk Utility. You can use diskutil to mount and diskutil to unmount local disk volumes from the command line, so they disappear and reappear in the Finder. It can also be used for many other disk-related operations such as partitioning and erasing disks, conducting RAID maintenance, ejecting media, and much more. For more information, type man diskutil.

drutil

The drutil command lets you interact with attached CD and DVD burners. For example, you can use the tray command to open, close, and eject media. The command

drutil tray close –drive 1

closes the tray of the first burning device seen, if possible. The man page for drutil explains each of its options in great detail.

mount and umount

If you mount or unmount a local volume using the mount and umount commands, diskarbitrationd does not detect the disk. You typically use the mount and umount commands only for network volumes. In some cases, however, you might need to mount a local volume with mount to pass mount options that are not available with diskutil or disktool. For example, when you boot into single-user mode, you type /sbin/mount –uw / to mount the root volume read-write. You should refresh disk arbitration with disktool –r after using the mount or umount command. You can use mount with no options to obtain a list of mounted volumes.

For more information, see the man pages for mount, mount_afp, mount_ftp, mount_hfs, mount_msdos, mount_nfs, mount_ntfs, mount_smbfs, mount_webdav, and umount.

Mounting With Files

Command-line tools are not the only way to deal with local mounts, though. Next we’ll look at a couple of files that deal with mounted volumes.

volinfo.database

Mac OS X distinguishes between native HFS Plus volumes, which are present at startup, and removable volumes. The system keeps a list of native volumes in the file /var/db/ volinfo.database. Each volume in the list has a unique 64-bit volume identifier.

fstab

Mac OS X supports an fstab file so that you can mount file systems in particular locations at system startup. The format of the Mac OS X fstab file, however, is slightly different from that of other UNIX systems. Because of disk arbitration, the device files on Mac OS X are not static. You cannot create an fstab file to mount /dev/disk0s1 at /, because you have no guarantee that the mounted disk will be disk0s1 the next time the computer reboots. Instead, use the device label; for example:

LABEL=Disk2 /Disk2 hfs rw 0 0

Monitoring and Troubleshooting

This section describes some techniques for monitoring file systems and for correcting problems related to file systems. You will learn some techniques for dealing with dot files in folders you copy to Windows computers. You will learn how to use fsck to check a file system for issues and how to determine why a mounted local volume cannot be ejected. You will also learn two command-line utilities for determining file space utilization.

Dot Files in Windows

As you learned earlier in this lesson, Mac OS X folders sometimes contain shadow files. In addition, they often contain .DS_Store files. The Finder creates a file called .DS_Store in every folder it visits. The Finder uses the .DS_Store files to keep track of folder view options, icon positions, and other visual information about folders. You can safely delete these files.

When you copy a folder from your Mac OS X computer to a Windows computer, shadow files and .DS_Store files become visible, as the following figure illustrates, because the Windows graphical user interface does not hide filenames that begin with a period (dot).

If you want to delete the .DS_Store files before you transfer folders to a Windows computer, you can use a command in Terminal. For example, you could run the command

sudo find / –name ".DS_Store" –depth –exec rm {} ;

You could substitute –name "._*" in the command line to delete shadow files, but remember that you might want the shadow files if you copy the folder back to a Mac OS X computer.

Sometimes when you copy files back and forth between Mac OS X and Windows, files lose their custom icons. You can restore a file’s custom icon if you have another file that has the same icon. Just copy and paste the icon from one Get Info window to another.

If you do not want these files to show up, add the following line to the Global section of the /etc/smb.conf file:

hide dot files = yes

The next time you start the computer or restart Windows file sharing, these files will not appear to Windows users.

Check for File-System Issues With fsck

A crash, power failure, or other sudden shutdown can render a nonjournaled file system unusable. The startup script, rc, runs fsck during system startup to repair such a situation.

If the volume is HFS or HFS Plus, the fsck command invokes fsck_hfs. By default, the fsck_hfs command does not check journaled HFS Plus file systems. Occasionally, however, a journaled HFS Plus file system becomes unusable during normal usage. You can use fsck_hfs –f to force the command to check and repair a journaled HFS Plus file system with the following commands:

$ sudo fsck_hfs /dev/disk0s2
fsck_hfs: Volume is journaled. No checking performed.
fsck_hfs: Use the -f option to force checking.
$ sudo fsck_hfs -f /dev/disk0s2
** Checking HFS Plus volume.
** Checking Extents Overflow file.

To get fsck_msdos to run on a flash memory device, connect the device, then use the umount command to unmount it.

$ sudo fsck_msdos /dev/disk2s1
** Phase 1 - Read FAT
** Phase 2 - Check Cluster Chains
** Phase 3 - Checking Directories
** Phase 4 - Checking for Lost Files

The fsck command invokes fsck_msdos on MS-DOS file systems.

Getting Information About Open Files

Sometimes when you try to eject a disk, you get an error message such as “The disk ‘Data’ is in use and could not be ejected,” as shown in the following figure. The best way to resolve this problem is to determine whether a file on the disk is open and, if so, close the file from the application or utility that opened it. If you don’t know which application or utility is using a file on the disk, you can find out by using the lsof command.

daves-mac:~ david$ lsof /dev/disk1s2

Run lsof as root (using sudo) to see processes you do not own. You can use grep to find processes using files on a specific volume. For example, if Preview is using file /Volume/ Data/MyDir/MyFile.pdf, the output of

sudo lsof | grep "/Volumes/Data"

might contain the following line:

Preview 721 student17 54r VREG 14,15 448581 559 26 /Volumes/Data – – MyDir/MyFile.pdf

The command fs_usage is useful for showing which files an application uses. Type

sudo fs_usage | grep Application

where Application is the name of the application you are interested in. You will see all file-system activity related to that application, including the pathnames of files while they are being read or written.

Alternatively, to show files used by a specific process, use ps or Activity Monitor to get the process identifier (PID) of an executing application. Then type

sudo fs_usage pid

where pid is the PID of that application.

Getting Disk Usage Information

Use the df command to view disk capacity and the amount of free and available disk space on file systems. By default, values display in 512-byte blocks. Use the –h option to display values in a format that is easier to read, with unit suffixes such as kilobytes and megabytes instead of 512-byte blocks. Use the –l option to restrict output to local disks.

$ df -lh
Filesystem        Size      Used      Avail      Capacity        Mounted on
/dev/disk0s2     37G      13G       24G         36%               /
/dev/disk2s2     88G      44G       44G         50%              /Volumes/HD2

Use the du command to view the file-system block usage for each file specified as an argument. By default, every file in the hierarchy is listed under each folder argument. Use the –s option to get summary totals for folders. Use the –h option to display values in a format that is easier to read.

$ du -sh *
660K   Documents
108M   Library
8.0K   Movies

Disk space needs vary, depending on the types of files users store on disk. Movies, for example, can consume much more disk space than other types of files.

Computers shared by multiple users need more home folder space if home folders are local. You can use shared volumes for home folders. You can also use shared volumes for files that are accessed by multiple users instead of providing individual copies to each user.

Some jobs or businesses require more work space. Video files, for instance, might require more disk space.

What You’ve Learned

• The HFS Plus Mac OS X volume type differs from UFS and other supported types, particularly with respect to the handling of file forks.

• Mac OS X supports both aliases and symbolic links.

• The disk arbitration daemon keeps track of available disks.

• Use the find command to remove dot files that would show up on Windows machines connected to your file share.

• lsof will show you what processes are using currently open files.

• du and df are important tools for determining disk utilization.

References

Apple Knowledge Base Documents

The following Knowledge Base documents (located at www.apple.com/support) provide further information about local volumes and files.

Document 8647, “File System Specifications and Terms”

Document 25557, “Mac OS X: Mac OS Extended Format - Volume and File Limits”

Lesson Review

1. List some important differences between HFS Plus and UFS.

2. Is it safe to delete all of those dot files Windows users see when you save a file on a Windows file server?

3. What does diskarbitrationd do when you connect a USB storage device?

4. What is the main difference between an alias and a symbolic link?

5. What does the df command do?

Answers

1. HFS Plus is normally case-preserving but not case-sensitive. (A case-sensitive version is available.) HFS Plus provides native support for files with resource and data forks, and it is journaled by default on Mac OS X v10.3 and later. UFS is case-sensitive. It provides no native support for forked files. Journaling is not available for UFS.

2. No. Windows does not hide dot files the way most UNIX systems do. If the files are .DS_Store files, they are safe to delete as they contain only the icon position and other layout information for the Finder. However, resource fork shadow files (those that begin with ._) may not be safe to delete, as they may contain data belonging to the other half of that file.

3. diskarbitrationd probes the disk to determine its file-system format, checks the disk for unusable files, mounts the file system, and updates the disk arbitration table.

4. When you move the original file, the alias is updated to point to the new location. The symbolic link is not updated and is left dangling.

5. The df command displays disk capacity and the amount of free and available disk space on file systems.