File System Objects

Modern UNIX file systems support the following types of file system objects:

• Regular Files (S_IFREG)

• Directories (S_IFDIR)

• Character Devices (S_IFCHR)

• Block Devices (S_IFBLK)

• Named Pipes (S_IFIFO)

• Sockets (S_IFSOCK)

• Symbolic Links (S_IFLNK)

The C macro names given within parentheses are provided by the include file <sys/stat.h> (see stat(2)). You'll see more of these in Chapter 6, "Managing Files and Their Properties."

Regular Files

A regular file is generally what is most important to users of a system. It stores the data that the user wants to retrieve and work with at a later time. The UNIX file system presents this data as a continuous stream of bytes.

A regular file consists of any number of data bytes, from zero to some maximum number. This is an important distinction to note, since many file systems, including CP/M and DOS, will present only multiples of a particular block size. This forces the DOS operating system to adopt the ^Z character as a marker for the end of a text file. Without this marker byte, it is otherwise impossible to have a file logically contain 3 bytes or 300 bytes, for example. However, UNIX has no such restriction, since it is logically able to present a file of any byte length.

Another feature of the UNIX file system is that the programmer can work with the file logically. There is no longer any need for the program to care about the block size in use by the underlying file. This permits the program to seek to any offset within the file and read any number of bytes, which may or may not span multiple disk blocks. For operating systems in 1969, this was a radical concept.

A regular file is identified by ls(1) as follows:

$ ls -l /etc/hosts
-rw-r--r--  1 root  wheel  112 Feb 19 11:07 /etc/hosts
$

The first character of the ls(1) output is a - (hyphen) to indicate that /etc/hosts is a regular file.

Directories

You cannot have more than one file in a file system without a directory. The first version of DOS created files only under the root directory. However, when a file was opened, this directory was searched to see if the file existed and where it was physically allocated.

The second step is important to the operating system in question.

UNIX supports a hierarchical file system, which allows directories to contain subdirectories. This allows the file system to be subdivided into logical groups of files and other file system objects. Can you imagine how difficult UNIX would be to use if the FreeBSD 3.4-Release contained all of its 60,014 (or more) files under the root directory?

Early releases of UNIX permitted directories to be read and written like regular files. Over time, several problems with this open concept emerged:

• Program errors or accidental writes to a directory could cause the loss of several files.

• New file systems supported different directory structure entries.

• Long filename support made it inconvenient to work directly with directory entries.

The first point illustrates one of the big weaknesses of early directory management. It was possible to lose the contents of an entire directory by accidentally overwriting the directory. The following command used to create havoc:

$ echo OOPS >directory
						

If directory was the name of a directory, this command would overwrite its contents, causing UNIX to lose track of all the files that it managed. Even worse, it usually meant that the space occupied by the files in that directory was lost, since the file system was not notified of any deletion. The following shows a modern response to this problem:

$ mkdir testdir
$ echo STUFF >testdir/file
$ ls -l testdir
total 1
-rw-r--r--  1 myid  megrp  5 Apr 15 15:16 file
$ echo OOPS >testdir
testdir: Is a directory.
$ ls -l testdir
total 1
-rw-r--r--  1 myid  megrp  5 Apr 15 15:16 file
$

The example creates a test directory and file and then attempts to overwrite the directory. The response from the UNIX kernel is that testdir: Is a directory.

When all file object names were limited to 14 characters, as they were in the earlier days, it was simple to work with the directories using direct file reads and writes. However, as different directory formats emerged and long filename support was introduced, this method proved to be unsafe and inconvenient.

For all of the reasons listed here, modern UNIX provides a set of library routines to search and manage directories. These will be covered in Chapter 7, "Directory Management."

A directory is identified by ls(1), as the following example illustrates:

$ ls -dl /etc
drwxr-xr-x  14 root  wheel  2048 Apr  5 01:47 /etc
$

The first character shown by ls(1) is the letter d, indicating that /etc is a directory. Note especially the use of the -d option in the ls(1) command line. Without this option, ls(1) will attempt to list the contents of the directory, rather than the directory itself.

Block Devices

A block device is within a class of devices that work with fixed block sizes. A disk drive is a good example of this type of device. While the operating system permits you to logically read and write to your regular files using any transfer size, the operating system must read and write the disk device in terms of disk blocks of a fixed size.

Although disk devices get faster and larger each year, they are still slow when compared to the speed of the CPU. In addition to slow data transfer, disk rotation and head seek latencies add to the overall wait time involved in a disk I/O operation. Consequently, block devices are buffered with a disk cache by the UNIX kernel.

The disk cache will usually retain the most recently used disk blocks, but cache algorithms vary in order to achieve different performance goals. Because disk cache dramatically improves the performance of the file system, all file systems tend to be mounted using the block device.

Block devices can be readily identified by the ls(1) command as follows:

$ mount
/dev/wd0s2a on / (ufs, local, writes: sync 4505 async 92908)
/dev/wd0s2e on /usr (ufs, local, writes: sync 6924 async 118551)
procfs on /proc (procfs, local)
$ ls -l /dev/wd0s2a
brw-r-----  1 root  operator    0, 0x00030000 Feb 19 11:05 /dev/wd0s2a
$

The mount(8) (on many systems mount(1M)) command is used to find out what block devices have been used. Then the device /dev/wd0s2a was chosen in this example. The first character, shown by ls(1) in this example, is the letter b, indicating that /dev/wd0s2a is a block device.

Block devices are not necessarily representative of the entire disk. In most cases, these represent a disk partition so that an error in file system software cannot corrupt another partition. Additionally, each block device within the system usually has a corresponding character device as well. Block and character devices are also referred to as block raw devices and character raw devices, respectively.

When applied to a device, the word "raw" indicates that the disk space and structure are not managed. The raw device does not maintain a structure of files and directories within it. This is the job of file system software. Similarly, a database manages tables and rows within a raw device.

The cache feature of block devices may seem to suggest that a block device should be a good candidate for a database. This is usually not the case, however, since the database engine has its own custom cache algorithms that are tuned to the way that the database accesses the disk device. For this reason, database engines like Oracle, Sybase, and Informix usually perform better with the corresponding character device. This is one of the reasons that raw (character) device access to disks and partitions is being added to the Linux 2.4 kernel.

Character Devices

Character devices are a class of devices that work with various byte-sized inputs and outputs. These generally work with variable lengths of data, but not necessarily so (disks will insist on fixed block sizes). Your terminal (or pseudo-tty) is a special form of character device. As you type characters at your keyboard on the console, the operating system must read the characters and make them available to the program that is currently reading input from the terminal. This differs from the way that block devices work, in that the amount of data input is often small or variable in length.

QIC (Quarter-Inch Cartridge) tapes are another example of character devices. Tape devices will accept a program's idea of a record (within limits) and write a physical record to tape matching that size.

A character device is easily identified by the ls(1) command as shown below:

$ ls -l /dev/tty
crw-rw-rw-  1 root  wheel    1,   0 Apr 15 14:56 /dev/tty
$

The device /dev/tty is always known to the current session as your terminal device (the actual device name is different). The first character shown in the ls(1) output is c, telling you that this is a character device.

The mouse attached to the console is another example (FreeBSD):

$ ls -l /dev/sysmouse
crw-------  1 root  wheel   12, 128 Feb 19 11:05 /dev/sysmouse
$

Here again, you can see that the mouse is considered a character device.

Disks are also accessible using UNIX character devices. The same disks can be accessed using the corresponding block device that you read about earlier. However, character raw devices (for disks) are often provided to the database engines. Database engines manage the performance of disk I/O better than the block device cache because of their intimate knowledge of the data structures being used by the database.

By convention, the character raw device name of a block device usually has the letter r in front of it. See the following FreeBSD example:

$ mount
/dev/wd0s2a on / (ufs, local, writes: sync 4505 async 92982)
/dev/wd0s2e on /usr (ufs, local, writes: sync 6926 async 118585)
procfs on /proc (procfs, local)
$ ls -l /dev/rwd0s2a
crw-r-----  1 root  operator    3, 0x00030000 Feb 19 11:05 /dev/rwd0s2a
$

The mount(8) command was used to discover the block device names. Note that the ls(1) command adds the letter r to the device name to arrive at the character raw device name of /dev/rwd0s2a for the root mount. The first character of the ls(1) output shows the letter c, confirming that this is a character device.

Named Pipes (FIFOs)

In the period between 1970 and 1972, Doug McIlroy at BTL would sketch out how he would like to connect processes by saying " who into cat into grep. " In 1972, Ken Thompson finally said, "I'm going to do it!" Overnight Ken worked to implement the pipe concept. Ken also had to rework many of the tools because, at the time, the tools did not support the idea of standard input—they read from files named on the command line instead. UNIX, starting with Third Edition, was forever changed that night. The pipe feature was so well accepted that anyone who had seen it would not give it up.

Pipes are now routinely used on the command line under UNIX for all sorts of purposes, using the | pipe (vertical bar) symbol. These are anonymous pipes, since they exist only between the processes that are communicating with each other. They disappear from the system when both ends of the pipe become closed.

It is also possible to create a named pipe that exists in the file system. These are also known as FIFOs, since data that is written first in is first out of the pipe. The following shows a simple example:

$ mkfifo myFIFO
$ ls -l
total 0
prwxr-xr-x  1 myid  mygrp  0 Apr 15 16:55 myFIFO
$ ls -l >myFIFO &
$ tr '[a-z]''[A-Z]'<myFIFO
TOTAL 0
PRWXR-XR-X  1 MYID  MYGRP  0 APR 15 16:55 MYFIFO
[1] 77637 Exit 0              ls -l >myFIFO
$

The example illustrates how the ls(1) command was able to redirect its output into the FIFO myFIFO (ls was placed into the background so that another command could be started in the same session). Then the tr(1) command was started to accept input from myFIFO, translating all lowercase letters into uppercase.

Notice also that the first letter of the ls(1) output is the letter p. This is how FIFO file system objects can be identified.

Sockets

The socket was a Berkeley University concept that found its way into 4.1BSD and 4.2BSD implementations of UNIX circa 1982. Sockets permit processes on one UNIX host to communicate over a network with processes on a remote host. Sockets can also be used to communicate with other processes within the same host. (The BSD lpr(1) command does this to accept output for spooling to a printer.)

Local sockets can also exist within the file system. This is the type of socket that can be used only between processes within the same host. If you have the PostgreSQL database installed on your FreeBSD system, you might have a socket like this one:

$ ls -l /tmp/.s.PGSQL.5432
srwxrwxrwx  1 postgres  wheel  0 Mar  7 04:43 /tmp/.s.PGSQL.5432
$

The example shows that the ls(1) command identifies the socket with the starting letter s.

Sockets that connect to remote systems, however, do not appear anywhere in the file system.

Symbolic Links

UNIX has supported linked files for quite some time. However, the symbolic link is a relatively new concept by UNIX standards. It was added to address the limitations of the normal link, sometimes now referred to as the "hard link."

Normally, files can be linked only when both links are contained on the same file system. On some systems, the /usr file system is different from other parts of the root file system. An attempt to create a link on a file system that is different from the file being linked will fail:

$ ln /etc/hosts /usr/me/work/my_link
ln: /home/me/work/my_link: Cross-device link
$

The UNIX kernel tells us that these two ends of a would-be link are on different devices. The symbolic link makes it possible to overcome this limitation:

$ ln -s /etc/hosts /usr/me/work/my_link
$ ls -dl my_link
lrwxr-xr-x  1 me   mygrp  10 Apr 15 17:22 my_link -> /etc/hosts
$

Note that the ln(1) command shown here uses the -s option to request a symbolic link, as if to say, "If you list the contents of my_link you will see the contents of your /etc/hosts file." The ls(1) command output for the symbolic link shows a starting letter l.

Symbolic links work around the original problem with hard links because they are actually a special kind of file that contains a pathname. When the UNIX kernel sees that it is a symbolic link, the kernel reads this special file to find out what the real pathname is. However, it is possible that the pathname listed is yet another symbolic link. The UNIX kernel will return the error ELOOP if the symbolic link is a circular reference or simply has too many indirect references. Chapter 6 will examine symbolic links further.

Special Files

While you may not have realized it, you already know about special files. These are file system objects that allow access to devices. Here are some of the examples that you have seen already:

These are special files because they represent only the actual device in question (FreeBSD see intro(4)). It is only by convention that you tend to find these devices in the /dev directory. They could be placed in other directories.

Another important quality about special files is that their existence does not imply that the device or its driver support exists. For example, on a FreeBSD 3.4 Release system you might list a device:

$ ls -l /dev/da0
brw-r-----  1 root  operator    4, 0x00010002 Feb 19 11:05 /dev/da0
$ ls -l /dev/rda0
crw-r-----  1 root  operator   13, 0x00010002 Feb 19 11:05 /dev/rda0
$

The example shows a SCSI disk block and character device. Yet, if you were to switch to root to access this device, you would see the following:

# dd if=/dev/da0 of=/dev/null
dd: /dev/da0: Device not configured
#

The dd(1) command is told that the device is not configured (on the particular system on which it was tried). The file system object /dev/da0 is just a placeholder that informs the kernel what device you want access to, if this special file is accessed.

Harking back to an earlier example

$ mount
/dev/wd0s2a on / (ufs, local, writes: sync 4505 async 92982)
/dev/wd0s2e on /usr (ufs, local, writes: sync 6926 async 118585)
procfs on /proc (procfs, local)
$ ls -l /dev/rwd0s2a
crw-r-----  1 root  operator    3, 0x00030000 Feb 19 11:05 /dev/rwd0s2a
$

The /dev/rwd0s2a device is listed as the disk device (partition) for use by the root file system. You can also access this same device with another special file, if you create one of your own:

$ mknod /usr/me/work/root c 3 0x30000
mknod: /usr/me/work/root: Operation not permitted
$ su -
Password:
# cd /usr/me/work
# mknod root c 3 0x30000
# ls -l root
crw-r--r--  1 root  mygrp    3, 0x00030000 Apr 15 18:03 root
# rm root
# exit
$

The mknod(1) command requires root access (note the failed first attempt). As root, the mknod(1) command was used to create an entirely new special file /usr/me/work/root, which even used a different filename. Once that is created, you will find that you could access the same device by using either /dev/rwd0s2a or /usr/me/work/root (but I wouldn't advise that you do anything with your root file system!).

The special file root in the example was deleted also. Did that make the device disappear? No. Not only is the special file /dev/rwd0s2a still available, even if that entry was deleted, you could always re-create it with the mknod(1).

The special file entry specifies three pieces of information:

• Block or character device (b or c)

• The major number for the device

• The minor number for the device

The major number (3 in the example above) indicates what type of device it is (based upon the kernel configuration). The minor number can be as simple as the value zero, or it can reference a particular unit within a set. For example, a minor number of 2 might choose a second partition of the disk drive, and a minor number of 0 might reference the entire disk drive.

Minor numbers can also include bit flags. Some character devices such as tape drives have a bit set to indicate that the tape drive should be rewound upon close. In all cases, special file major and minor numbers are very kernel specific. You cannot use the same special files saved on an HPUX UNIX platform and restore them to an AIX 4.3 platform. This would be a recipe for disaster!

Special files are given attention here because they are important for those system programmers who want to take up daunting challenges such as writing database engines. The writer of any new device support must also be keenly interested in the special device entry for the hardware device.

Some device entries are pseudo devices. They don't actually represent hardware, but specialized kernel services. One pair of such devices under FreeBSD is the /dev/kmem and /dev/mem devices (see mem(4)). With the correct permissions, it is possible to inspect kernel memory through these special files. For example, a writer of a ps(1) command could choose to work through kernel structures this way (there are better ways).