The UNIX system has been in wide use for over 30 years and has helped to define many areas of computing. Although numerous individuals and organizations have contributed (and still contribute) to the development of the UNIX system, this book primarily concentrates on the BSD thread of development.
• Bell Laboratories, which invented UNIX
• The Computer Systems Research Group (CSRG) at the University of California at Berkeley, which gave UNIX virtual memory and the reference implementation of TCP/IP
• The FreeBSD project, the NetBSD project, and the OpenBSD project, which continue the work started by the CSRG
• The Darwin operating system at the core of Apple's OS X. Darwin is based on FreeBSD.
The first version of the UNIX system was developed at Bell Laboratories in 1969 by Ken Thompson as a private research project to use an otherwise idle PDP-7. Thompson was joined shortly thereafter by Dennis Ritchie, who not only contributed to the design and implementation of the system, but also invented the C programming language. The system was completely rewritten into C, leaving almost no assembly language. The original elegant design of the system [Ritchie, 1978] and developments of the first 15 years [Ritchie, 1984a; Compton, 1985] have made the UNIX system an important and powerful operating system [Ritchie, 1987].
Ritchie, Thompson, and other early UNIX developers at Bell Laboratories had worked previously on the Multics project [Peirce, 1985; Organick, 1975], which had a strong influence on the newer operating system. Even the name UNIX is merely a pun on Multics; in areas where Multics attempted to do many tasks, UNIX tried to do only one task but do it well. The basic organization of the UNIX filesystem, the idea of using a user process for the command interpreter, the general organization of the filesystem interface, and many other system characteristics come directly from Multics.
Ideas from various other operating systems, such as the Massachusetts Institute of Technology's (MIT's) CTSS, also have been incorporated. The fork operation to create new processes comes from Berkeley's GENIE (SDS-940, later XDS-940) operating system. Allowing a user to create processes inexpensively led to using one process per command rather than commands being run as procedure calls, as is done in Multics.
The first major editions of UNIX were the Research systems from Bell Laboratories. In addition to the earliest versions of the system, these systems include the UNIX Time-Sharing System, Sixth Edition, commonly known as V6, which in 1976 was the first version widely available outside of Bell Laboratories. Systems are identified by the edition numbers of the UNIX Programmer's Manual that were current when the distributions were made.
The UNIX system was distinguished from other operating systems in three important ways.
- It was written in a high-level language.
- It was distributed in source form.
- It provided powerful primitives normally found in only those operating systems that ran on much more expensive hardware.
Most of the system source code was written in C rather than in assembly language. The prevailing belief at the time was that an operating system had to be written in assembly language to provide reasonable efficiency and to get access to the hardware. The C language itself was at a sufficiently high level to allow it to be compiled easily for a wide range of computer hardware, without its being so complex or restrictive that systems programmers had to revert to assembly language to get reasonable efficiency or functionality. Access to the hardware was provided through assembly-language stubs for the 3 percent of the operating-system functions—such as context switching—that needed them. Although the success of UNIX does not stem solely from its being written in a high-level language, the use of C was a critical first step [Kernighan & Ritchie, 1978; Kernighan & Ritchie, 1989; Ritchie et al., 1978]. Ritchie's C language is descended [Rosler, 1984] from Thompson's B language, which was itself descended from BCPL [Richards & Whitby-Strevens, 1980]. C continues to evolve [Tuthill, 1985; ISO, 1999].
The second important distinction of UNIX was its early release from Bell Laboratories to other research environments in source form. By providing source, the system's founders ensured that other organizations would be able not only to use the system, but also to tinker with its inner workings. The ease with which new ideas could be adopted into the system always has been key to the changes that have been made to it. Whenever a new system that tried to upstage UNIX came along, somebody would dissect the newcomer and clone its central ideas into UNIX. The unique ability to use a small, comprehensible system, written in a high-level language, in an environment swimming in new ideas led to a UNIX system that evolved far beyond its humble beginnings. Though recipients of the source code had to be licensed, campuswide licenses were cheaply available to universities. Thus, many people became versed in the way that UNIX worked, setting the stage for the open-source world that would follow.
The third important distinction of UNIX was that it provided individual users with the ability to run multiple processes concurrently and to connect these processes into pipelines of commands. At the time, only operating systems running on large and expensive machines had the ability to run multiple processes, and the number of concurrent processes usually was controlled tightly by a system administrator.
Most early UNIX systems ran on the PDP-11, which was inexpensive and powerful for its time. Nonetheless, there was at least one early port of Sixth Edition UNIX to a machine with a different architecture: the Interdata 7/32 [Miller, 1978]. The PDP-11 also had an inconveniently small address space. The introduction of machines with 32-bit address spaces, especially the VAX-11/780, provided an opportunity for UNIX to expand its services to include virtual memory and networking. Earlier experiments by the Research group in providing UNIX-like facilities on different hardware had led to the conclusion that it was as easy to move the entire operating system as it was to duplicate UNIX's services under another operating system. The first UNIX system with portability as a specific goal was UNIX Time-Sharing System, Seventh Edition (V7), which ran on the PDP-11 and the Interdata 8/32 and had a VAX variety called UNIX/32V Time-Sharing, System Version 1.0 (32V). The Research group at Bell Laboratories has also developed UNIX Time-Sharing System, Eighth Edition (V8); UNIX Time-Sharing System, Ninth Edition (V9); and UNIX Time-Sharing System, Tenth Edition (V10). Their 1996 system is Plan 9.
After the distribution of Seventh Edition in 1978, the Research group turned over external distributions to the UNIX Support Group (USG). USG had previously distributed internally such systems as the UNIX Programmer's Work Bench (PWB), and had sometimes distributed them externally as well [Mohr, 1985].
USG's first external distribution after Seventh Edition was UNIX System III (System III) in 1982, which incorporated features of Seventh Edition, of 32V, and also of several UNIX systems developed by groups other than the Research group. Features of UNIX/RT (a real-time UNIX system) were included, as were many features from PWB. USG released UNIX System V (System V) in 1983; that system is largely derived from System III. The court-ordered divestiture of the Bell Operating Companies from AT&T permitted AT&T to market System V aggressively [Bach, 1986; Wilson, 1985].
USG metamorphosed into the UNIX System Development Laboratory (USDL), which released UNIX System V, Release 2 in 1984. System V, Release 2, Version 4 introduced paging [Jung, 1985; Miller, 1984], including copy-on-write and shared memory, to System V. The System V implementation was not based on the Berkeley paging system. USDL was succeeded by AT&T Information Systems (ATTIS), which distributed UNIX System V, Release 3, in 1987. That system included STREAMS, an IPC mechanism adopted from V8 [Presotto & Ritchie, 1985]. ATTIS was succeeded by UNIX System Laboratory (USL), which was sold to Novell in 1993. Novell passed the UNIX trademark to the X/OPEN consortium, giving the latter sole rights to set up certification standards for using the UNIX name on products. Two years later, Novell sold UNIX to The Santa Cruz Operation (SCO).
The most influential of the non-Bell Laboratories and non-AT&T UNIX development groups was the University of California at Berkeley [DiBona et al., 1999]. Software from Berkeley was released in Berkeley Software Distributions (BSD)—for example, as 4.4BSD. Berkeley was the source of the BSD name, and their distributions were the first distinct identity for the BSD operating system. The first Berkeley VAX UNIX work was the addition to 32V of virtual memory, demand paging, and page replacement in 1979 by William Joy and Ozalp Babaoglu, to produce 3BSD [Babaoglu & Joy, 1981]. The reason for the large virtual-memory space of 3BSD was the development of what at the time were large programs, such as Berkeley's Franz LISP. This memory-management work convinced the Defense Advanced Research Projects Agency (DARPA) to fund the Berkeley team for the later development of a standard system (4BSD) for DARPA's contractors to use.
A goal of the 4BSD project was to provide support for the DARPA Internet networking protocols, TCP/IP [Comer, 2000]. The networking implementation was general enough to communicate among diverse network facilities, ranging from local networks, such as Ethernets and token rings, to long-haul networks, such as DARPA's ARPANET.
We refer to all the Berkeley VAX UNIX systems following 3BSD as 4BSD, although there were really several releases: 4.0BSD, 4.1BSD, 4.2BSD, 4.3BSD, 4.3BSD Tahoe, and 4.3BSD Reno. 4BSD was the UNIX operating system of choice for VAXes from the time that the VAX first became available in 1977 until the release of System V in 1983. Most organizations would purchase a 32V license but would order 4BSD from Berkeley. Many installations inside the Bell System ran 4.1BSD (and replaced it with 4.3BSD when the latter became available). A new virtual-memory system was released with 4.4BSD. The VAX was reaching the end of its useful lifetime, so 4.4BSD was not ported to that machine. Instead, 4.4BSD ran on the newer 68000, SPARC, MIPS, and Intel PC architectures.
The 4BSD work for DARPA was guided by a steering committee that included many notable people from both commercial and academic institutions. The culmination of the original Berkeley DARPA UNIX project was the release of 4.2BSD in 1983; further research at Berkeley produced 4.3BSD in mid-1986. The next releases included the 4.3BSD Tahoe release of June 1988 and the 4.3BSD Reno release of June 1990. These releases were primarily ports to the Computer Consoles Incorporated hardware platform. Interleaved with these releases were two unencumbered networking releases: the 4.3BSD Netl release of March 1989 and the 4.3BSD Net2 release of June 1991. These releases extracted nonproprietary code from 4.3BSD; they could be redistributed freely in source and binary form to companies that and individuals who were not covered by a UNIX source license. The final CSRG release requiring an AT&T source license was 4.4BSD in June 1993. Following a year of litigation (see Section 1.3), the free-redistributable 4.4BSD-Lite was released in April 1994. The final CSRG release was 4.4BSD-Lite Release 2 in June 1995.
The UNIX system is also a fertile field for academic endeavor. Thompson and Ritchie were given the Association for Computing Machinery Turing award for the design of the system [Ritchie, 1984b]. The UNIX system and related, specially designed teaching systems—such as Tunis [Ewens et al., 1985; Holt, 1983], XINU [Comer, 1984], and MINIX [Tanenbaum, 1987]—are widely used in courses on operating systems. Linus Torvalds reimplemented the UNIX interface in his freely redistributable Linux operating system. The UNIX system is ubiquitous in universities and research facilities throughout the world, and is ever more widely used in industry and commerce.
The CSRG incorporated features from not only UNIX systems but from other operating systems. Many of the features of the 4BSD terminal drivers are from TENEX/TOPS-20. Job control (in concept—not in implementation) is derived from TOPS-20 and from the MIT Incompatible Timesharing System (ITS). The virtual-memory interface first proposed for 4.2BSD, and finally implemented in 4.4BSD, was based on the file-mapping and page-level interfaces that first appeared in TENEX/TOPS-20. The current FreeBSD virtual-memory system (see Chapter 5) was adapted from Mach, which was itself an offshoot of 4.3BSD. Multics has often been a reference point in the design of new facilities.
The quest for efficiency was a major factor in much of the CSRG's work. Some efficiency improvements were made because of comparisons with the proprietary operating system for the VAX, VMS [Joy, 1980; Kashtan, 1980].
Other UNIX variants have adopted many 4BSD features. AT&T UNIX System V [AT&T, 1987], the IEEE POSIX.1 standard [P1003.1, 1988], and the related National Bureau of Standards (NBS) Federal Information Processing Standard (FIPS) have adopted the following.
• Job control (Chapter 2)
• Reliable signals (Chapter 4)
• Multiple file-access permission groups (Chapter 6)
• Filesystem interfaces (Chapter 8)
The X/OPEN Group (originally consisting of only European vendors but now including most U.S. UNIX vendors) produced the X/OPEN Portability Guide [X/OPEN, 1987] and, more recently, the Spec 1170 Guide. These documents specify both the kernel interface and many of the utility programs available to UNIX system users. When Novell purchased UNIX from AT&T in 1993, it transferred exclusive ownership of the UNIX name to X/OPEN. Thus, all systems that want to brand themselves as UNIX must meet the X/OPEN interface specifications. To date, no BSD system has ever been put through the X/OPEN interface-specification tests, so none of them can be called UNIX. The X/OPEN guides have adopted many of the POSIX facilities. The POSIX.1 standard is also an ISO International Standard, named SC22 WG15. Thus, the POSIX facilities have been accepted in most UNIX-like systems worldwide.
The 4BSD socket interprocess-communication mechanism (see Chapter 11) was designed for portability and was immediately ported to AT&T System III, although it was never distributed with that system. The 4BSD implementation of the TCP/IP networking protocol suite (see Chapter 13) is widely used as the basis for further implementations on systems ranging from AT&T 3B machines running System V to VMS to embedded operating systems such as VxWorks.
The CSRG cooperated closely with vendors whose systems are based on 4.2BSD and 4.3BSD. This simultaneous development contributed to the ease of further ports of 4.3BSD and to ongoing development of the system.
Much of the Berkeley development work was done in response to the user community. Ideas and expectations came not only from DARPA, the principal direct-funding organization, but also from users of the system at companies and universities worldwide.
The Berkeley researchers accepted not only ideas from the user community but also actual software. Contributions to 4BSD came from universities and other organizations in Australia, Canada, Europe, Japan, and the United States. These contributions included major features, such as autoconfiguration and disk quotas. A few ideas, such as the fcntl system call, were taken from System V, although licensing and pricing considerations prevented the use of any code from System III or System V in 4BSD. In addition to contributions that were included in the distributions proper, the CSRG also distributed a set of user-contributed software.
An example of a community-developed facility is the public-domain time-zone-handling package that was adopted with the 4.3BSD Tahoe release. It was designed and implemented by an international group, including Arthur Olson, Robert Elz, and Guy Harris, partly because of discussions in the USENET news-group comp.std.unix. This package takes time-zone-conversion rules completely out of the C library, putting them in files that require no system-code changes to change time-zone rules; this change is especially useful with binary-only distributions of UNIX. The method also allows individual processes to choose rules rather than keeping one ruleset specification systemwide. The distribution includes a large database of rules used in many areas throughout the world, from China to Australia to Europe. Distributions are thus simplified because it is not necessary to have the software set up differently for different destinations, as long as the whole database is included. The adoption of the time-zone package into BSD brought the technology to the attention of commercial vendors, such as Sun Microsystems, causing them to incorporate it into their systems.
Up through the release of 4.3BSD Tahoe, all recipients of BSD had to first get an AT&T source license. That was because the BSD systems were never released by Berkeley in a binary-only format; the distributions always contained the complete source to every part of the system. The history of the UNIX system, and the BSD system in particular, had shown the power of making the source available to the users. Instead of passively using the system, they actively worked to fix bugs, improve performance and functionality, and even add completely new features.
With the increasing cost of the AT&T source licenses, vendors that wanted to build stand-alone TCP/IP-based networking products for the PC market using the BSD code found the per-binary costs prohibitive. So they requested that Berkeley break out the networking code and utilities and provide them under licensing terms that did not require an AT&T source license. The TCP/IP networking code clearly did not exist in 32/V and thus had been developed entirely by Berkeley and its contributors. The BSD-originated networking code and supporting utilities were released in June 1989 as Networking Release 1, the first freely redistributable code from Berkeley.
The licensing terms were liberal. A licensee could release the code modified or unmodified in source or binary form with no accounting or royalties to Berkeley. The only requirements were that the copyright notices in the source file be left intact and that products that incorporated the code include in their documentation that the product contained code from the University of California and its contributors. Although Berkeley charged a $1000 fee to get a tape, anyone was free to get a copy from somebody who already had it. Indeed, several large sites put it up for anonymous FTP shortly after it was released. Though the code was freely available, several hundred organizations purchased tapes, which helped to fund the CSRG and encouraged further development.
With the success of the first open-source release, the CSRG decided to see how much more of BSD they could spring free. Keith Bostic led the charge by soliciting people to rewrite the UNIX utilities from scratch based solely on their published descriptions. Their only compensation would be to have their name listed among the Berkeley contributors next to the name of the utility that they rewrote. The contributions started slowly and were mostly for the trivial utilities. But as the list of completed utilities grew, and Bostic continued to hold forth for contributions at public events such as Usenix, the rate of contributions continued to grow. Soon the list crossed 100 utilities, and within 18 months nearly all the important utilities and libraries had been rewritten.
The kernel proved to be a bigger task because it could not easily be rewritten from scratch. The entire kernel was reviewed, file by file, removing code that had originated in the 32/V release. When the review was completed, there were only six remaining kernel files that were still contaminated and that could not be trivially rewritten. While consideration was given to rewriting those six files so that a complete kernel could be released, the CSRG decided to release just the less controversial set. The CSRG sought permission for the expanded release from folks higher up in the university administration. After much internal debate and verification of the methods used for detecting proprietary code, the CSRG was given permission to do the release.
The initial thought was to come up with a new name for the second freely redistributable release. However, getting a new license written and approved by the university lawyers would have taken many months. So, the new release was named Networking Release 2, since that could be done with just a revision of the approved Networking Release 1 license agreement. This second greatly expanded freely redistributable release began shipping in June 1991. The redistribution terms and cost were the same as the terms and cost of the first networking release. As before, several hundred individuals and organizations paid the $1000 fee to get the distribution from Berkeley.
Closing the gap from the Networking Release 2 distribution to a fully functioning system did not take long. Within six months of the release, Bill Jolitz had written replacements for the six missing files. He promptly released a fully compiled and bootable system for the 386-based PC architecture in January 1992, which he called 386/BSD. Jolitz's 386/BSD distribution was done almost entirely on the net. He simply put it up for anonymous FTP and let anyone who wanted it download it for free. Within weeks he had a huge following.
Unfortunately, the demands of keeping a full-time job meant that Jolitz could not devote the time needed to keep up with the flood of incoming bug fixes and enhancements to 386/BSD. So within a few months of the release of 386/BSD, a group of avid 386/BSD users formed the NetBSD group to pool their collective resources to help maintain and later enhance the system. By early 1993 they were doing releases that became known as the NetBSD distribution. The NetBSD group chose to emphasize the support of as many platforms as possible and continued the research-style development done by the CSRG. Until 1998, their distribution was done solely over the net with no distribution media available. Their group continues to target primarily the hard-core technical users.
The FreeBSD group was formed a few months after the NetBSD group with a charter to support just the PC architecture and to go after a larger and less technically advanced audience, much as Linux had done. They built elaborate installation scripts and began shipping their system on a low-cost CD-ROM in December 1993. The combination of ease of installation and heavy promotion on the net and at major trade shows, such as Comdex, led to a large, rapid growth curve. FreeBSD quickly rose to have the largest installed base of all the Networking Release 2-derived systems.
FreeBSD also rode the wave of Linux popularity by adding a Linux emulation mode that allows Linux binaries to run on the FreeBSD platform. This feature allows FreeBSD users to use the ever growing set of applications available for Linux while getting the robustness, reliability, and performance of the FreeBSD system.
In 1995, OpenBSD spun off from the NetBSD group. Their technical focus was aimed at improving the security of the system. Their marketing focus was to make the system easier to use and more widely available. Thus, they began producing and selling CD-ROMs, with many of the ease of installation ideas from the FreeBSD distribution.
In addition to the groups organized to freely redistribute systems originating from the Networking Release 2 tape, a company, Berkeley Software Design Incorporated (BSDI), was formed to develop and distribute a commercially supported version of the code. Like the other groups, it started by adding the six missing files that Bill Jolitz had written for his 386/BSD release. BSDI began selling its system, including both source and binaries, in January 1992 for $995. It began running advertisements touting its 99 percent discount over the price charged for System V source plus binary systems. Interested readers were told to call 1-800-ITS-UNIX.
Shortly after BSDI began its sales campaign, it received a letter from UNIX System Laboratory (USL) (a mostly owned subsidiary of AT&T spun off to develop and sell UNIX) [Ritchie, 2004]. The letter demanded that BSDI stop promoting its product as UNIX and in particular that it stop using the deceptive phone number. Although the phone number was promptly dropped and the advertisements changed to explain that the product was not UNIX, USL was still unhappy and filed suit to enjoin BSDI from selling its product. The suit alleged that the BSDI product contained USL proprietary code and trade secrets. USL sought to get an injunction to halt BSDI's sales until the lawsuit was resolved claiming that it would suffer irreparable harm from the loss of its trade secrets if the BSDI distributions continued.
At the preliminary hearing for the injunction, BSDI contended that it was simply using the sources being freely distributed by the University of California plus six additional files. BSDI was willing to discuss the content of any of the six added files but did not believe it should be held responsible for the files being distributed by the University of California. The judge agreed with BSDI's argument and told USL that it would have to restate its complaint based solely on the six files or the case would be dismissed. Recognizing that it would have a hard time making a case from just the six files, USL decided to refile the suit against both BSDI and the University of California. As before, USL requested an injunction on the shipping of Networking Release 2 from the university and of the BSDI products.
With the impending injunction hearing just a few short weeks away, preparation began in earnest. All the members of the CSRG were deposed, as was nearly everyone employed by BSDI. Briefs, counterbriefs, and counter-counterbriefs flew back and forth between the lawyers. The staff of the CSRG turned from writing code to writing several hundred pages of material that found its way into various briefs.
In December 1992, Dickinson R. Debevoise, a United States District Judge in New Jersey, heard the arguments for the injunction. Although judges usually rule on injunction requests immediately, he decided to take it under advisement. On a Friday about six weeks later, he issued a 40-page opinion in which he denied the injunction and threw out all but two of the complaints [Debevoise, 1993]. The remaining two complaints were narrowed to recent copyrights and the possibility of the loss of trade secrets. He also suggested that the matter should be heard in a state court system before being heard in federal court.
The University of California took the hint and rushed to California state court the following Monday morning with a countersuit against USL. By filing first in California, the university had established the locale of any further state court action. Constitutional law requires all state filing to be done in a single state to prevent a litigant with deep pockets from bleeding an opponent dry by filing 50 cases against them, one in each state. The result was that if USL wanted to take any action against the university in state courts, it would be forced to do so in California rather than in its home state of New Jersey.
The university's suit claimed that USL had failed in its obligation to provide due credit to the university for the use of BSD code in System V as required by the license that it had signed with the university [Linzner & MacDonald, 1993]. If the claim were found to be valid, the university asked that USL be forced to reprint all its documentation with the appropriate due credit added, to notify all its licensees of its oversight, and to run full-page advertisements in major publications such as the Wall Street Journal and Fortune magazine, notifying the business world of its inadvertent oversight.
Soon after the filing in state court, USL was bought from AT&T by Novell. The CEO of Novell, Ray Noorda, stated publicly that he would rather compete in the marketplace than in court. By the summer of 1993 settlement talks had started. Unfortunately, the two sides had dug in so deep that the talks proceeded very slowly. With some further prodding by Ray Noorda on the USL side, many of the sticking points were removed, and a settlement was finally reached in January 1994. The result was that three files were removed from the 18,000 that made up Networking Release 2, and a few minor changes were made to other files. In addition, the university agreed to add USL copyrights to about 70 files, although those files continued to be freely redistributed.
The newly blessed release was called 4.4BSD-Lite and was released in June 1994 under terms identical to those used for the Networking releases. Specifically, the terms allow free redistribution in source and binary form, subject only to the constraint that the university copyrights remain intact and that the university receive credit when others use the code. Simultaneously, the complete system was released as 4.4BSD-Encumbered, which still required recipients to have a USL source license.
The lawsuit settlement also stipulated that USL would not sue any organization using 4.4BSD-Lite as the base for its system. So all the extant BSD groups—BSDI, NetBSD, and FreeBSD—had to restart their code base with the 4.4BSD-Lite sources into which they then merged their enhancements and improvements. While this reintegration caused a short-term delay in the development of the various BSD systems, it was a blessing in disguise, since it forced all the divergent groups to resynchronize with the three years of development that had occurred at the CSRG since the release of Networking Release 2.
The money received from the 4.4BSD-Encumbered and 4.4BSD-Lite releases was used to fund a part-time effort to integrate bug fixes and enhancements. These changes continued for two years until the rate of bug reports and feature enhancements had died down to a trickle. The final set of changes was released as 4.4BSD-Lite Release 2 in June 1995. Most of the changes incorporated into 4.4BSD-Lite Release 2 eventually made it into the other systems' source bases.
Though the license term requiring that due credit be given to the university had been extremely helpful in the lawsuit, the university agreed to drop it following the final release. As many people began using the BSD-style copyrights for their own code, the proliferation of due-credit clauses in open-source software became difficult to determine and unmanageably large. By agreeing to drop the due-credit clause, the university hoped to set an example for others using its license. Over time, and with a lot of effort from the BSD community, the due-credit clause has been dropped from many of the open-source programs that use the BSD-style license.
Following the release of 4.4BSD-Lite Release 2, the CSRG was disbanded. After 15 years of piloting the BSD ship, it was time to let others with fresh ideas and boundless enthusiasm take over. While it might seem best to have a single centralized authority overseeing the system development, the idea of having several groups with different charters ensures that many different approaches will be tried and that there is no single point of failure. Because the system is released in source form, the best ideas can easily be picked up by other groups. Indeed, cross-pollination of ideas between open-source projects is common.
Running an open-source project is different from running traditional software development. In traditional development, the staff are paid, so it is possible to have managers and a system architect that set schedules and direct the programmers' activities. With open source, the developers are volunteers. They tend to be transient, usually doing a project or two before finding some other activity on which they prefer to spend their free time. They cannot be directed because they only work on what interests them. Because their jobs, families, and social lives often take precedence over their work on the project, it is impossible to put together schedules. Finally, there is no paid staff to fill the management role of traditional development. Thus, a successful open-source-development project must be self-organizing and set up to gracefully handle a high turnover of its active developers.
The development model used by FreeBSD (as well as NetBSD and OpenBSD) was first set in motion by the CSRG [McKusick et al., 1989]. The CSRG was always a small group of software developers. This resource limitation required careful software-engineering management. Careful coordination was needed not only of the CSRG personnel but also of members of the general community who contributed to the development of the system. Certain outside developers had permission to modify the master copy of the system source directly. People given access to the master sources were carefully screened beforehand but were not closely supervised. Everyone committing changes to the system source received notification of all changes, allowing everyone to be aware of changes going into the system. Everyone was required to have any nontrivial changes reviewed by at least one other person before committing them to the tree. This model allowed many lines of development to proceed concurrently while still keeping the project coherent.
The FreeBSD project is organized in much the same way as the CSRG. The entire FreeBSD project, including all the source code, documentation, bug reports, mailing-list archives, and even administrative data, is maintained in a publicly readable source-code-control system. Anyone may view the source code and existing bug reports, track progress on fixing bugs, and post bug reports. Anyone may join and participate in the numerous FreeBSD mailing lists. There are three groups of people that directly work on FreeBSD: developers, committers, and the core team.
There are 3000 to 4000 developers, each of whom works on some part of the system such as maintaining the FreeBSD kernel, continuing development of the 1000 core FreeBSD utilities, writing FreeBSD documentation, and updating other open-source software in the FreeBSD ports collection. Developers are able to access the source-code repository, but they are not permitted to change it. Instead they must work with a committer or file a problem report to get their changes added to the system.
There are currently 300 to 400 committers. Like the developers, most of them specialize in some part of the system. Unlike the developers, they are permitted to make changes to those parts of the source-code repository in which they have been authorized to work. All nontrivial changes should be reviewed by one or more other committers before being checked into the source tree. Most committers are doing work of their own as well as reviewing and committing the work of several developers.
Nomination for advancement from developer to committer is done by the existing committers. Most commonly a developer will be nominated by the committer with whom he has been working. The nomination, along with a description and evaluation of past work and an initial scope of new work, is sent to the core team for approval.
At the center of the project is the core team. The core team is composed of nine people who are elected every two years. The candidates for the core team come from the committers and the committers elect the core team. The core team acts as the final gatekeepers of the source code. They monitor what is being committed and resolve conflicts if two or more committers cannot agree on how to solve a particular problem. The core team also approves developers to be advanced to being committers and in (rare circumstances) temporarily or permanently evicts someone from the committer group. The usual reason for departure from the committer group is inactivity (making no changes to the system for more than a year).
The development structure of the FreeBSD project is directly derived from the one that we had established at the CSRG. Both the CSRG and FreeBSD use a central source-code-controlled repository. The FreeBSD core team is analogous to the CSRG staff. The FreeBSD committers are much like the people to whom Berkeley gave accounts on the CSRG development machine that allowed them to commit changes to the CSRG sources. And the FreeBSD developers are similar to the people that contributed to Berkeley, but they did not have accounts on the CSRG development machine.
The FreeBSD project has made some important improvements. First, they recognize that even the most dedicated programmer will eventually burn out, lose interest, or otherwise decide to move on. There must be some way to let these people gracefully step aside rather than letting their inattention create a void at a critical point in the project. So unlike the CSRG model of having staff that were dictators for life, FreeBSD went to an elected core that is answerable to the committers. A core member who is burned out can decide (or be persuaded) not to run for reelection when his or her term ends. Core members who are not serving the interest of the committers will not be reelected. Equally important, active and energetic people have plenty of opportunity to move up through the ranks. Because the core team is elected, people rise into that rank because their peers actively working on the project feel that they should have the job. This approach works better than advancing because you are good buddies with somebody at the top. It also ensures that the core team is made up of those who are good at communicating with others, an important skill to have in that position.
Another significant improvement made by the FreeBSD project is to automate many tasks and set up remote mirrors of the source-code repository, Web site, and bug reports. These changes have allowed the project to support many more contributors than would have been possible under the CSRG model. The FreeBSD project has also managed to become much less United States-centric by welcoming developers from around the world, including active people in Japan, Australia, Russia, South Africa, Denmark, France, Germany, and the United Kingdom, to name just a few of the countries with active FreeBSD development.
The CSRG used to release new versions of the system about every two years. Changes to these distributions were rare, typically only small and critical security-or stability-related changes. Between versions, the CSRG would do test releases to gain experience with the new features that were being developed.
The FreeBSD project has greatly expanded on the CSRG distribution scheme. At any point in time there are two FreeBSD distributions. The first is the "stable" release that is intended to be used in production environments. The second is the "current" release that represents the current state of the FreeBSD system and is intended for use by developers and users needing the latest features.
The stable release changes slowly, and the changes are limited to fixing bugs, improving performance, and adding incremental hardware support. The stable system is released three to four times per year, although users wishing to upgrade more often can download and install the latest stable code as frequently as they need to do so (for example, after a major security patch has been made). The stable version of FreeBSD is analogous to the CSRG major-version releases except that they are more actively updated and are made available to the users. Like the stable release, snapshots of the current release are created every few months. However, most users of the current release update much more frequently (daily updates are common). By having mirrored copies of the stable and current distributions available throughout the world, the FreeBSD project allows its worldwide user base to stay up to date much more easily than was possible with the CSRG distributions.
About every two years, the current branch is forked to create a new stable release. Once the new stable branch has proven to be reliable enough for production use, work largely ceases on the old stable branch and production users switch over to the new stable release. The mainline development continues on the current branch. Nearly all changes are made first to the current branch. Only after a change has been tested in the current branch and proven to work in that environment is it merged-from-current (MFC-ed) to the stable release.
One advantage that the CSRG long had over the FreeBSD project was that the CSRG was part of the University of California at Berkeley. Since the university is a nonprofit organization, contributions made to the CSRG were tax-deductible to the contributor. Some people at the FreeBSD project had long felt that they should find a way to let contributors to the project get a tax deduction. A few years ago, they set up the FreeBSD Foundation, which after three years of good nonprofit work, was granted 501(c)3 status by the United States taxing authorities. This certification means that contributions made to the FreeBSD Foundation can be deducted from United States federal and state taxes in the same way as a contribution made to the university can be deducted. The ability to get a tax deduction has markedly increased the volume of monetary contributions to the FreeBSD project, which has enabled them to fund development of parts of the system that are tedious to create but necessary and important.
Over the past 10 years the FreeBSD project has grown at a steady but sustainable pace. Although Linux has attracted a mass following, FreeBSD continues to hold its place in the high-performance-server space. Indeed, Linux has helped to evangelize the viability of open source to the corporate marketplace, and FreeBSD has ridden on its coattails. It is far easier to convince your management to switch from Linux to FreeBSD than it is to convince them to move from Microsoft's Windows to FreeBSD. Linux has also supplied a steady stream of developers for FreeBSD. Until recently Linux had no central source-code repository, so to contribute you had to work for a Linux distributor or you had to get the ear of a member of the small set of people who could get changes put into the system. The much more egalitarian and merit-based organization of the FreeBSD project has provided a steady influx of high-quality developers. The typical new committer to the FreeBSD project is in their mid- to late 20s and has been programming Linux or other open-source projects for a decade. These people have enough experience and maturity that they are quickly able to become effective contributors to the project. And the mentoring inherent in the progression of developer to committer ensures that by the time someone has the right to directly commit code to the FreeBSD tree, they understand the style and code-clarity guidelines that are critically important to preserving the quality, robustness, and maintainability of FreeBSD.
The goal of the FreeBSD project is to provide software that may be used for any purpose and without strings attached. Many of the developers have a significant investment in the code (and project) and certainly do not mind a little financial compensation occasionally, but they certainly do not insist on it. They believe that their first and foremost mission is to provide code to any and all comers, for whatever purpose, so that the code gets the widest possible use and provides the greatest possible benefit [Hubbard, 2004].
References
AT&T, 1987.
, The System V Interface Definition (SVID), Issue 2, American Telephone and Telegraph, Murray Hill, NJ, January 1987.
Babao
lu &Joy, 1981. Ö. Babao
lu & W. N. Joy, "Converting a Swap-Based System to Do Paging in an Architecture Lacking Page-Referenced Bits," Proceedings of the Eighth Symposium on Operating Systems Principles,pp. 78-86, December 1981.
Bach, 1986.
M. J. Bach, The Design of the UNIX Operating System, Prentice-Hall, Englewood Cliffs, NJ, 1986.
Comer, 2000.
D. Comer, Internetworking with TCP/IP Volume 1, 4th ed., Prentice-Hall, Upper Saddle River, NJ, 2000.
Comer, 1984. D. Comer, Operating System Design: The Xinu Approach, Prentice-Hall, Englewood Cliffs, NJ, 1984.
Debevoise, 1993.
D. Debevoise, Civ. No. 92-1667, Unix System Laboratories Inc. vs. Berkeley Software Design Inc., http://sco.tuxrocks.com/Docs/USL/Doc-92.html, March 3, 1993.
DiBona et al., 1999.
C. DiBona, S. Ockman, & M. Stone, Open Sources: Voices from the Open Source Revolution,pp. 31-46, Chapter 2—Twenty Years of Berkeley Unix: From AT&T-Owned to Freely Redistributable, http://www.oreilly.com/catalog/opensources/book/kirkmck.html, ISBN 1-56592-582-3, O'Reilly & Associates, Inc., Sebastopol, CA 95472, 1999.
Ewensetal., 1985.
P Ewens, D. R. Blythe, M. Funkenhauser, & R. C. Holt, "Tunis: A Distributed Multiprocessor Operating System," USENIX Association Conference Proceedings,pp. 247-254, June 1985.
Holt, 1983.
R. C. Holt, Concurrent Euclid, the UNIX System, and Tunis, Addison-Wesley, Reading, MA, 1983.
Hubbard, 2004.
J. Hubbard, "A Brief History of FreeBSD," FreeBSD Handbook, section 1.3.1, http://www.freebsd.org/doc/en_US.IS08859-1/books/handbook/history.html, March 2004.
ISO, 1999.
, "ISO/IEC 9899 Programming Language C Standard," ISO 9899, can be ordered from http://www.iso.org, December, 1999.
Joy, 1980.
W. N. Joy,"Comments on the Performance of UNIX on the VAX," Technical Report, University of California Computer System Research Group, Berkeley, CA, April 1980.
Jung, 1985.
R. S. Jung,"Porting the AT&T Demand Paged UNIX Implementation to Microcomputers," USENIX Association Conference Proceedings, pp. 361-370, June 1985.
Kashtan, 1980.
D. L. Kashtan, "UNIX and VMS: Some Performance Comparisons," Technical Report, SRI International, Menlo Park, CA, February 1980.
Kernighan & Ritchie, 1978.
B. W. Kernighan & D. M. Ritchie, The C Programming Language, Prentice-Hall, Englewood Cliffs, NJ, 1978.
Kernighan & Ritchie, 1989. B. W. Kernighan & D. M. Ritchie, The C Programming Language, 2nd ed., Prentice-Hall, Englewood Cliffs, NJ, 1989.
Linzner & MacDonald, 1993.
J. Linzner & M. MacDonald, University of California at Berkeley versus Unix System Laboratories Inc., http://cm.bell-labs.com/cm/cs/who/dmr/bsdi/930610.ucb_complaint.txt, June 1993.
McKusick et al., 1989.
M. K. McKusick, M. Karels, & K. Bostic,"The Release Engineering of 4.3BSD," Proceedings of the New Orleans Usenix Workshop on Software Management, pp. 95-100, April 1989.
Miller, 1978.
R. Miller, "UNIX—A Portable Operating System," ACM Operating System Review, , no. 3, pp. 32-37, July 1978.
Miller, 1984.
R. Miller,"A Demand Paging Virtual Memory Manager for System V," USENIX Association Conference Proceedings, pp. 178-182, June 1984.
Organick, 1975.
E. I. Organick, The Multics System: An Examination of Its Structure, MIT Press, Cambridge, MA, 1975.
P1003.1, 1988. P1003.1,
IEEE P1003.1 Portable Operating System Interface for Computer Environments (POSIX), Institute of Electrical and Electronic Engineers, Piscataway, NJ, 1988.
Peirce, 1985.
N. Peirce, "Putting UNIX in Perspective: An Interview with Victor Vyssotsky," UNIX Review, , no. 1,p. 58, January 1985.
Presotto & Ritchie, 1985.
D. L. Presotto & D. M. Ritchie,"Interprocess Communication in the Eighth Edition UNIX System," USENIX Association Conference Proceedings, pp. 309-316, June 1985.
Richards & Whitby-Strevens, 1980.
M. Richards & C. Whitby-Strevens, BCPL: The Language and Its Compiler, Cambridge University Press, Cambridge, U.K., 1980, 1982.
Ritchie, 1978.
D. M. Ritchie, "A Retrospective," Bell System Technical Journal, , no. 6, pp. 1947-1969, July-August 1978.
Ritchie, 1984a.
D. M. Ritchie,"The Evolution of the UNIX Time-Sharing System," AT&T Bell Laboratories Technical Journal, , no. 8, pp. 1577-1593, October 1984.
Ritchie, 1987. D. M. Ritchie, "Unix: A Dialectic," USENIX Association Conference Proceedings, pp. 29-34, January 1987.
Ritchie, 1984b.
D. M. Ritchie, "Reflections on Software Research," Comm ACM, , no. 8, pp. 758-760, 1984.
Ritchie, 2004.
D. M. Ritchie, Documents on Unix System Laboratories Inc. versus Berkeley Software Design Inc., http://cm.bell-labs.com/cm/cs/who/dmr/bsdi/bsdisuit.html, March 2004.
Ritchie et al., 1978.
D. M. Ritchie, S. C. Johnson, M. E. Lesk, & B. W. Kernighan,"The C Programming Language," Bell System Technical Journal, , no. 6, pp. 1991-2019, July-August 1978.
Rosler, 1984.
L. Rosler, "The Evolution of C—Past and Future," AT&T Bell Laboratories Technical Journal, , no. 8, pp. 1685-1699, October 1984.
Tanenbaum, 1987.
A. S. Tanenbaum, Operating Systems: Design and Implementation, Prentice-Hall, Englewood Cliffs, NJ, 1987.
Tuthill, 1985.
B. Tuthill, "The Evolution of C: Heresy and Prophecy," UNIX Review, , no. 1, p. 80, January 1985.