20. Futures: Dangers and Opportunities

History is not over. Unix will continue to grow and change. The community and the tradition around Unix will continue to evolve. Trying to forecast the future is a chancy business, but we can perhaps anticipate it in two ways: first, by looking at how Unix has coped with design challenges in the past; second, by identifying problems that are looking for solutions and opportunities waiting to be exploited.

20.1 Essence and Accident in Unix Tradition

To understand how Unix’s design might change in the future, we can start by looking at how Unix programming style has changed over time in the past. This effort leads us directly to one of the challenges of understanding the Unix style—distinguishing between accident and essence. That is, recognizing traits that arise from transient technical circumstances versus those that are deeply tied to the central Unix design challenge—how to do modularity and abstraction right while also keeping systems transparent and simple.

This distinction can be difficult, because traits that arose as accidents have sometimes turned out to have essential utility. Consider as an example the ’Silence is golden’ rule of Unix interface design we examined in Chapter 11; it began as an adaptation to slow teletypes, but continued because programs with terse output could be combined in scripts more easily. Today, in an environment where having many programs running visibly through a GUI is normal, it has a third kind of utility: silent programs don’t distract or waste the user’s attention.

On the other hand, some traits that once seemed essential to Unix turned out to be accidents tied to a particular set of cost ratios. For example, old-school Unix favored program designs (and minilanguages like awk(1)) that did line-at-a-time processing of an input stream or record-at-a-time processing of binary files, with any context that needed to be maintained between pieces carried by elaborate state-machine code. New-school Unix design, on the other hand, is generally happy with the assumption that a program can read its entire input into memory and thereafter randomly access it at will. Indeed, modern Unixes supply an mmap(2) call that allows the programmer to map an entire file into virtual memory and completely hides the serialization of I/O to and from disk space.

This change trades away storage economy to get simpler and more transparent code. It’s an adaptation to the plunging cost of memory relative to programmer time. Many of the differences between old-school Unix designs in the 1970s and 1980s and those of the new post-1990 school can be traced to the huge shift in relative costs that today makes all machine resources several orders of magnitude cheaper relative to programmer time than they were in 1969.

Looking back, we can identify three specific technology changes that have driven significant changes in Unix design style: internetworking, bitmapped graphics displays, and the personal computer. In each case, the Unix tradition has adapted to the challenge by discarding accidents that were no longer adaptive and finding new applications for its essential ideas. Biological evolution works this way too. Evolutionary biologists have a rule: “Don’t assume that historical origin specifies current utility, or vice versa”. A brief look at how Unix adapted in each of these cases may provide some clues to how Unix might adapt itself to future technology shifts that we cannot yet anticipate.

Chapter 2 described the first of these changes: the rise of internetworking, from the angle of cultural history, telling how TCP/IP brought the original Unix and ARPANET cultures together after 1980. In Chapter 7, the material on obsolescent IPC and networking methods such as System V STREAMS hints at the many false starts, missteps, and dead ends that preoccupied Unix developers through much of the following decade. There was a good deal of confusion about protocols,¹ and about the relationship between intermachine networking and interprocess communication among processes on the same machine.

¹ For a few years it looked like ISO’s 7-layer networking standard might compete successfully with TCP/IP. It was promoted by a European standards committee politically horrified at the thought of adopting any technology birthed in the bowels of the Pentagon. Alas, their indignation exceeded their technical acuity. The result proved overcomplicated and unhelpful; see [Padlipsky] for details.

Eventually the confusion was cleared up when TCP/IP won and BSD sockets reasserted Unix’s essential everything-is-a-byte-stream metaphor. It became normal to use BSD sockets for both IPC and networking, older methods for both largely fell out of use, and Unix software grew increasingly indifferent to whether communicating components were hosted on the same or different machines. The invention of the World Wide Web in 1990–1991 was the logical result.

When bitmapped graphics and the example of the Macintosh arrived in 1984 a few years after TCP/IP, they posed a rather more difficult challenge. The original GUIs from Xerox PARC and Apple were beautiful, but wired together far too many levels of the system for Unix programmers to feel comfortable with their design. The prompt response of Unix programmers was to make separation of policy from mechanism an explicit principle; the X windowing system established it by 1988. By splitting X widget sets away from the display manager that was doing low-level graphics, they created an architecture that was modular and clean in Unix terms, and one that could easily evolve better policy over time.

But that was the easy part of the problem. The hard part was deciding whether Unix ought to have a unified interface policy at all, and if so what it ought to be. Several different attempts to establish one through proprietary toolkits (like Motif) failed. Today, in 2003, GTK and Qt contend with each other for the role. While the debate on this question is not over in 2003, the persistence of different UI styles that we noted in Chapter 11 seems telling. New-school Unix design has kept the command line, and dealt with the tension between GUI and CLI approaches by writing lots of CLI-engine/GUI-interface pairs that can be used in both styles.

The personal computer posed few major design challenges as a technology in itself. The 386 and later chips were powerful enough to give the systems designed around them cost ratios similar to those of the minicomputers, workstations, and servers on which Unix matured. The true challenge was a change in the potential market for Unix; the much lower overall price of the hardware made personal computers attractive to a vastly broader, less technically sophisticated user population.

The proprietary-Unix vendors, accustomed to the fatter margins from selling more powerful systems to sophisticated buyers, were never interested in this wider market. The first serious initiatives toward the end-user desktop came out of the open-source community and were mounted for essentially ideological reasons. As of mid-2003, market surveys indicate that Linux has reached about 4%–5% share there, closely comparable to the Apple Macintosh’s.

Whether or not Linux ever does substantially better than this, the nature of the Unix community’s response is already clear. We examined it in the study of Linux in Chapter 3. It includes adopting a few technologies (such as XML) from elsewhere, and putting a lot of effort into naturalizing GUIs into the Unix world. But underneath the themed GUIs and the installation packaging, the main emphasis is still on modularity and clean code—on getting the infrastructure for serious, high-reliability computing and communications right.

The history of the large-scale desktop-focused developments like Mozilla and OpenOffice.org that were launched in the late 1990s illustrates this emphasis well. In both these cases, the most important theme in community feedback wasn’t demand for new features or pressure to make a ship date—it was distaste for monster monoliths, and a general sense that these huge programs would have to be slimmed down, refactored, and carved into modules before they would be other than embarrassments.

Despite being accompanied by a great deal of innovation, the responses to all three technologies were conservative with regard to the fundamental Unix design rules—modularity, transparency, separation of policy from mechanism, and the other qualities we’ve tried to characterize earlier in this book. The learned response of Unix programmers, reinforced over thirty years, was to go back to first principles—to try to get more leverage out of Unix’s basic abstractions of streams, namespaces, and processes in preference to layering on new ones.

20.2 Plan 9: The Way the Future Was

We know what Unix’s future used to look like. It was designed by the research group at Bell Labs that built Unix and called ’Plan 9 from Bell Labs’.² Plan 9 was an attempt to do Unix over again, better.

² The name is a tribute to the 1958 movie that has passed into legend as “the worst ever made”, Plan 9 from Outer Space. (The legend is, unfortunately, incorrect, as the few who have seen an even worse stinkeroo from 1966 called Manos: The Hands of Fate can attest.) Documentation, including a survey paper describing the architecture, along with complete source code and a distribution that installs on PCs, can be readily found with a Web search for the phrase ’Plan 9 from Bell Labs’.

The central design challenge the designers attempted to meet in Plan 9 was integrating graphics and ubiquitous networking into a comfortable Unix-like framework. They kept the Unix choice to mediate access to as many system services as possible through a single big file-hierarchy name space. In fact, they improved on it; many facilities that under Unix are accessed through various ad-hoc interfaces like BSD sockets, fcntl(2), and ioctl(2) are in Plan 9 accessed through ordinary read and write operations on special files analogous to device files. For portability and ease of access, almost all device interfaces are textual rather than binary. Most system services (including, for example, the window system) are file servers containing special files or directory trees representing the served resources. By representing all resources as files, Plan 9 turns the problem of accessing resources on different servers into the problem of accessing files on different servers.

Plan 9 combined this more-Unix-than-Unix file model with a new concept: private name spaces. Every user (in fact, every process) can have its own view of the system’s services by creating its own tree of file-server mounts. Some of the file server mounts will have been manually set up by the user, and others automatically set up at login time. So (as the Plan 9 from Bell Labs survey paper points out) “/dev/cons always refers to your terminal device and /bin/date to the correct version of the date command to run, but which files those names represent depends on circumstances such as the architecture of the machine executing date”.

The single most important feature of Plan 9 is that all mounted file servers export the same file-system-like interface, regardless of the implementation behind them. Some might correspond to local file systems, some to remote file systems accessed over a network, some to instances of system servers running in user space (like the window system or an alternate network stack), and some to kernel interfaces. To users and client programs, all these cases look alike.

One of the examples from the Plan 9 survey paper is the way FTP access to remote sites is implemented. There is no ftp(1) command under Plan 9. Instead there is an ftpfs fileserver, and each FTP connection looks like a file system mount. ftpfs automatically translates open, read, and write commands on files and directories under the mount point into FTP protocol transactions. Thus, all ordinary file-handling tools such as ls(1), mv(1) and cp(1) simply work, both underneath the FTP mount point and across the boundaries with the rest of the user’s view of the namespace. The only difference the user (or his scripts and programs) will notice is retrieval speed.

Plan 9 has much else to recommend it, including the reinvention of some of the more problematic areas of the Unix system-call interface, the elimination of superuser, and many other interesting rethinkings. Its pedigree is impeccable, its design elegant, and it exposes some significant errors in the design of Unix. Unlike most efforts at a second system, it produced an architecture that was in many ways simpler and more elegant than its predecessor. Why didn’t it take over the world?

One could argue for a lot of specific reasons—lack of any serious effort to market it, scanty documentation, much confusion and stumbling over fees and licensing. For those unfamiliar with Plan 9, it seemed to function mainly as a device for generating interesting papers on operating-systems research. But Unix itself had previously surmounted all these sorts of obstacles to attract a dedicated following that spread it worldwide. Why didn’t Plan 9?

The long view of history may tell a different story, but in 2003 it looks like Plan 9 failed simply because it fell short of being a compelling enough improvement on Unix to displace its ancestor. Compared to Plan 9, Unix creaks and clanks and has obvious rust spots, but it gets the job done well enough to hold its position. There is a lesson here for ambitious system architects: the most dangerous enemy of a better solution is an existing codebase that is just good enough.

Some Plan 9 ideas have been absorbed into modern Unixes, particularly the more innovative open-source versions. FreeBSD has a /proc file system modeled exactly on that of Plan 9 that can be used to query or control running processes. FreeBSD’s rfork(2) and Linux’s clone(2) system calls are modeled on Plan 9’s rfork(2). Linux’s /proc file system, in addition to presenting process information, holds a variety of synthesized Plan 9-like device files used to query and control kernel internals using predominantly textual interfaces. Experimental 2003 versions of Linux are implementing per-process mount points, a long step toward Plan 9’s private namespaces. The various open-source Unixes are all moving toward systemwide support for UTF-8, an encoding actually invented for Plan 9.³

³ The tale of how UTF-8 was born involves Ken Thompson, Rob Pike, a new Jersey diner, and a frenzied overnight hack <http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt>.

It may well be that over time, much more of Plan 9 will work its way into Unix as various portions of Unix’s architecture slide into senescence. This is one possible line of development for Unix’s future.

20.3 Problems in the Design of Unix

Plan 9 cleans up Unix, but only really adds one new concept (private namespaces) to its basic set of design ideas. But are there serious problems with those basic design ideas? In Chapter 1 we touched on several issues that Unix arguably got wrong. Now that the open-source movement has put the design future of Unix back in the hands of programmers and technical people, these are no longer decisions we have to live with forever. We’ll reexamine them in order to get a better handle on how Unix might evolve in the future.

20.3.1 A Unix File Is Just a Big Bag of Bytes

A Unix file is just a big bag of bytes, with no other attributes. In particular, there is no capability to store information about the file type or a pointer to an associated application program outside the file’s actual data.

More generally, everything is a byte stream; even hardware devices are byte streams. This metaphor was a tremendous success of early Unix, and a real advance over a world in which (for example) compiled programs could not produce output that could be fed back to the compiler. Pipes and shell programming sprang from this metaphor.

But Unix’s byte-stream metaphor is so central that Unix has trouble integrating software objects with operations that don’t fit neatly into the byte stream or file repertoire of operations (create, open, read, write, delete). This is especially a problem for GUI objects such as icons, windows, and ’live’ documents. Within a classical Unix model of the world, the only way to extend the everything-is-a-byte-stream metaphor is through ioctl calls, a notoriously ugly collection of back doors into kernel space.

Fans of the Macintosh family of operating systems tend to be vociferous about this. They advocate a model in which a single filename may have both data and resource ’forks’, the data fork corresponding to the Unix byte stream and the resource fork being a collection of name/value pairs. Unix partisans prefer approaches that make file data self-describing so that effectively the same sort of metadata is stored within the file.

The problem with the Unix approach is that every program that writes the file has to know about it. Thus, for example, if we want the file to carry type information inside it, every tool that touches it has to take care to either preserve the type field unaltered or interpret and then rewrite it. While this would be theoretically possible to arrange, in practice it would be far too fragile.

On the other hand, supporting file attributes raises awkward questions about which file operations should preserve them. It’s clear that a copy of a named file to another name should copy the source file’s attributes as well as its data—but suppose we cat(1) the file, redirecting the output of cat(1) to a new name?

The answer to this question depends on whether the attributes are actually properties of filenames or are in some magical way bundled with the file’s data as a sort of invisible preamble or postamble. Then the question becomes: Which operations make the properties visible?

Xerox PARC filesystem designs grappled with this problem as far back as the 1970s. They had an ’open serialized’ call that returned a byte stream containing both attributes and content. If applied to a directory, it returned a serialization of the directory’s attributes plus the serialization of all the files in it. It is not clear that this approach has ever been bettered.

Linux 2.5 already supports attaching arbitrary name/value pairs as properties of a filename, but at time of writing this capability is not yet much used by applications. Recent versions of Solaris have a roughly equivalent feature.

20.3.2 Unix Support for GUIs Is Weak

The Unix experience proves that using a handful of metaphors as the basis for a framework is a powerful strategy (recall the discussion of frameworks and shared context in Chapter 13). The visual metaphor at the heart of modern GUIs (files represented by icons, and opened by clicking which invokes some designated handler program, typically able to create and edit these files) has proven both successful and long-lived, exerting a strong hold on users and interface designers ever since Xerox PARC pioneered it in the 1970s.

Despite considerable recent effort, in 2003 Unix still supports this metaphor only poorly and grudgingly—there are lots of layers, few conventions, and only weak construction utilities. A typical reaction from a Unix old hand is to suspect that this reflects deeper problems with the GUI metaphor itself.

We quoted Brian Kernighan and Mike Lesk to similar effect in Chapter 11. But the inquiry can’t stop with indicting the GUI, because with all its flaws there is tremendous demand for GUIs from end users. Supposing we could get the metaphor right at the level of the design of user interactions, would Unix be capable of supporting it gracefully?

The answer is: probably not. We touched on this problem in considering whether the bag-of-bytes model is adequate. Macintosh-style file attributes may help provide the mechanism for richer support of GUIs, but it seems very unlikely that they are the whole answer. Unix’s object model doesn’t include the right fundamental constructs. We need to think through what a really strong framework for GUIs would be like—and, just as importantly, how it can be integrated with the existing frameworks of Unix. This is a hard problem, demanding fundamental insights that have yet to emerge from the noise and confusion of ordinary software engineering or academic research.

20.3.3 File Deletion Is Forever

People with VMS experience, or who remember TOPS-20 often miss these systems’ file-versioning facilities. Opening an existing file for write or deleting it actually renamed it in a predictable way including a version number; only an explicit removal operation on a version file actually erased data.

Unix does without this, at a not inconsiderable cost in user irritation when the wrong files get deleted through a typo or unexpected effects of shell wildcarding.

There does not seem to be any foreseeable prospect that this will change at the operating system level. Unix developers like clear, simple operations that do what the user tells them to do, even if the user’s instructions could amount to commanding “shoot me in the foot”. Their instinct is to say that protecting the user from himself should be done at the GUI or application level, not in the operating system.

20.3.4 Unix Assumes a Static Filesystem

Unix has, in one sense, a very static model of the world. Programs are implicitly assumed to run only briefly, so the background of files and directories can be assumed static during their execution. There is no standard, well-established way to ask the system to notify an application if and when a specified file or directory changes. This becomes a significant issue when writing long-lived user-interface software which wants to know about changes to the background.

Linux has file- and directory-change notification features,⁴ and some versions of BSD have copied them, but these are not yet portable to other Unixes.

⁴ Look for F_NOTIFY under fcntl(2).

20.3.5 The Design of Job Control Was Badly Botched

Apart from the ability to suspend processes (in itself a trivial addition to the scheduler which could be made fairly inoffensive) what job control is about is switching a terminal among multiple processes. Unfortunately, it does the easiest part—deciding where keystrokes go—and punts all the hard parts, like saving and restoring the state of the screen, to the application.

A really good implementation of such a facility would be completely invisible to user processes: no dedicated signals, no need to save and restore terminal modes, no need for the applications to redraw the screen at random times. The model ought to be a virtual keyboard that is sometimes connected to the real one (and blocks you if you ask for input when it isn’t connected) and a virtual screen which is sometimes visible on the real one (and might or might not block on output when it’s not), with the system doing the multiplexing in the same way it multiplexes access to the disk, the processor, etc... and no impact on user programs at all.⁵

⁵ This paragraph is based on a 1984 analysis by Henry Spencer. He went on to note that job control was necessary and appropriate for POSIX.1 and later Unix standards to consider precisely because it oozes its way into every program, and hence has to be thought about in any application-to-system interface. Hence, POSIX’s endorsement of a misdesign, while proper solutions were “out of scope” and hence were not even considered.

Doing it right would have required the Unix tty driver to track the entire current screen state rather than just maintaining a line buffer, and to know about terminal types at kernel level (possibly with help from a daemon process) so it could do restores properly when a suspended process is foregrounded again. A consequence of doing it wrong is that the Unix kernel can’t detach a session, such as an xterm or Emacs job, from one terminal and reattach it to another (which could be of a different type).

As Unix usage has shifted to X displays and terminal emulators, job control has become relatively less important, and this issue does not have quite the force it once did. It is still annoying that there is no suspend/attach/detach, however; this feature could be useful for saving the state of terminal sessions between logins.

A common open-source program called screen(1) solves several of these problems.⁶ However, since it has to be called explicitly by the user, its facilities are not guaranteed to be present in every terminal session; also, the kernel-level code that overlaps with it in function has not been removed.

⁶ There is a project site for screen(1) at http://www.math.fu-berlin.de/~guckes/screen/.

20.3.6 The Unix API Doesn’t Use Exceptions

C lacks a facility for throwing named exceptions with attached data.⁷ Thus, the C functions in the Unix API indicate errors by returning a distinguished value (usually –1 or a NULL character pointer) and setting a global errno variable.

⁷ For nonprogrammers, throwing an exception is a way for a program to bail out in the middle of a procedure. It’s not quite an exit because the throw can be intercepted by catcher code in an enclosing procedure. Exceptions are normally used to signal errors or unexpected conditions that mean it would be pointless to try to continue normal processing.

In retrospect, this is the source of many subtle errors. Programmers in a hurry often neglect to check return values. Because no exception is thrown, the Rule of Repair is violated; program flow continues until the error condition manifests as some kind of failure or data corruption later in execution.

The absence of exceptions also means that some tasks which ought to be simple idioms—like aborting from a signal handler on a version with Berkeley-style signals—have to be performed with code that is complex, subject to portability glitches, and bug-prone.

This problem can be (and normally is) hidden by bindings of the Unix API in languages such as Python or Java that have exceptions.

The lack of exceptions is actually an indicator of a problem with larger immediate implications; C’s weak type ontology makes communication between higher-level languages implemented in it problematic. Most of the more modern languages, for example, have lists and dictionaries as primary data types—but, because these don’t have any canonical representation in the universe of C, attempting to pass lists between (say) Perl and Python is an unnatural act requiring a lot of glue.

There are technologies that address the larger problem, such as CORBA, but they tend to involve a lot of runtime translation and be unpleasantly heavyweight.

20.3.7 ioctl(2) and fcntl(2) Are an Embarrassment

The ioctl(2) and fcntl(2) mechanisms provide a way to write hooks into a device driver. The original, historical use of ioctl(2) was to set parameters like baud rate and number of framing bits in a serial-communications driver, thus the name (for ’I/O control’). Later, ioctl calls were added for other driver functions, and fcntl(2) was added as a hook into the filesystem.

Over the years, ioctl and fcntl calls have proliferated. They are often poorly documented, and often a source of portability problems as well. With each one comes a grubby pile of macro definitions describing operation types and special argument values.

The underlying problem is the same as ’big bag of bytes’; Unix’s object model is weak, leaving no natural places to put many auxiliary operations. Designers have an untidy choice among unsatisfactory alternatives; fcntl/ioctl going through devices in /dev, new special-purpose system calls, or hooks through special-purpose virtual file systems that hook into the kernel (e.g., /proc under Linux and elsewhere).

It is not clear whether or how Unix’s object model will be enriched in the future. If MacOS-like file attributes become a common feature of Unix, tweaking magic named attributes on device drivers may take over the role ioctl/fcntl now have (this would at least have the merit of not requiring piles of macro definitions before the interface could be used). We’ve already seen that Plan 9 , which uses the named file server or filesystem as its basic object, rather than the file/bytestream, presents another possible path.

20.3.8 The Unix Security Model May Be Too Primitive

Perhaps root is too powerful, and Unix should have finer-grained capabilities or ACLs (Access Control Lists) for system-administration functions, rather than one superuser that can do anything. People who take this position argue that too many system programs have permanent root privileges through the set-user-ID mechanism; if even one can be compromised, intrusions everywhere will follow.

This argument is weak, however. Modern Unixes allow any given user account to belong to multiple security groups. Through use of the execute-permission and set-group-ID bits on program executables, each group can in effect function as an ACL for files or programs.

This theoretical possibility is very little used, however, suggesting that the demand for ACLs is much less in practice than it is in theory.

20.3.9 Unix Has Too Many Different Kinds of Names

Unix unified files and local devices—they’re all just byte streams. But network devices accessed through sockets have different semantics in a different namespace. Plan 9 demonstrates that files can be smoothly unified with both local and remote (network) devices, and all of these things can be managed through a namespace that is dynamically adjustable per-user and even per-program.

20.3.10 File Systems Might Be Considered Harmful

Was having a file system at all the wrong thing? Since the late 1970s there has been an intriguing history of research into persistent object stores and operating systems that don’t have a shared global filesystem at all, but rather treat disk storage as a huge swap area and do everything through virtualized object pointers.

Modern efforts in this line (such as EROS⁸) hint that such designs can offer large benefits including both provable conformance to a security policy and higher performance. It must be noted, however, that if this is a failure of Unix, it is equally a failure of all of its competitors; no major production operating system has yet followed EROS’s lead.⁹

⁸ http://www.eros-os.org/

⁹ The operating systems of the Apple Newton, the AS/400 minicomputer and the Palm handheld could be considered exceptions.

20.3.11 Towards a Global Internet Address Space

Perhaps URLs don’t go far enough. We’ll leave the last word on possible future directions of Unix to Unix’s inventor:

20.4 Problems in the Environment of Unix

The old-time Unix culture has largely reinvented itself in the open-source movement. Doing so saved us from extinction, but it also means that the problems of open source are now ours as well.

One of these is how to make open-source development economically sustainable. We have reconnected with our roots in the collaborative, open process of Unix’s early days. We have largely won the technical argument for abandoning secrecy and proprietary control. We have thought of ways to cooperate with markets and managers on more equal terms than we ever could in the 1970s and 1980s, and in many ways our experiments have succeeded. In 2003 the open-source Unixes, and their core development groups, have achieved a degree of mainstream respectability and authority that would have been unimaginable as recently as the mid-1990s.

We have come a long way. But we have a long way to go yet. We know what business models might work in theory, and now we can even point at a sporadic handful of successes that demonstrate that they work in practice; now we have to show that they can be made to work reliably over a longer term.

It’s not necessarily going to be an easy transition. Open source turns software into a service industry. Service-provider firms (think of medical and legal practices) can’t be scaled up by injecting more capital into them; those that try only scale up their fixed costs, overshoot their revenue base, and starve to death. The choices come down to singing for your supper (getting paid through tips and donations), running a corner shop (a small, low-overhead service business), or finding a wealthy patron (some large firm that needs to use and modify open-source software for its business purposes).

In total, the amount of money spent to hire software developers can be expected to rise, for the same reasons that mechanics’ hourly wages go up as the price of automobiles drops.¹⁰ But it is going to become more difficult for any one individual or firm to capture a lot of that spending. There will be many programmers who are well off, but fewer millionaires. This is actually a sign of progress, of inefficiencies being competed out of the system. But it will represent a big change in climate, and probably means that investors will lose what little interest they have left in funding software startups.

¹⁰ For a more complete discussion of this effect, see The Magic Cauldron in [Raymond01].

One important subproblem related to the increasing difficulty of sustaining really large software businesses is how to organize end-user testing. Historically, the Unix culture’s concentration on infrastructure has meant that we have not tended to build programs that depended for their value on providing a comfortable interface for end-users. Now, especially in the open-source Unixes that aim to compete directly with Microsoft and Apple, that is changing. But end-user interfaces need to be systematically tested with real end users—and therein lie some challenges.

Real end-user testing demands facilities, specialists, and a level of monitoring that are difficult for the distributed volunteer groups characteristic of open-source development to arrange. It may be, therefore, that open-source word processors, spreadsheets, and other ’productivity’ applications have to be left in the hands of large corporate-sponsored efforts like OpenOffice.org that can afford the overhead. Open-source developers consider single corporations to be single points of failure and worry about such dependencies, but no better solution has yet evolved.

These are economic problems. We have other problems of a more political nature, because success makes enemies.

Some are familiar. Microsoft’s ambition for an unchallengeable monopoly lock on computing made the defeat of Unix a strategic goal for the company in the mid-1980s, five years before we knew we were in a fight. In mid-2003, despite having had several growth markets it was counting on largely usurped by Linux, Microsoft is still the wealthiest and most powerful software company in the world. Microsoft knows very well that it must defeat the new-school Unixes of the open-source movement to survive. To defeat them, it must destroy or discredit the culture that produced them.

Unix’s comeback in the hands of the open-source community, and its association with the freewheeling culture of the Internet, has made it newer enemies as well. Hollywood and Big Media feel deeply threatened by the Internet and have launched a multipronged attack on uncontrolled software development. Existing legislation like the Digital Millennium Copyright Act has already been used to prosecute software developers who were doing things the media moguls disliked (the most notorious cases, of course, involve the DeCSS software that enables copying of encypted DVDs). Contemplated schemes like the so-called Trusted Computing Platform Alliance and Palladium threaten¹¹ to make open-source development effectively illegal—and if open source goes down, Unix is very likely to go down with it.

¹¹ See the TCPA FAQ <http://www.cl.cam.ac.uk/~rja14/tcpa-faq.html> for a rather hair-raising summary of the possibilities by a noted security specialist.

Unix and the hackers and the Internet against Microsoft and Hollywood and Big Media. It’s a struggle we need to win for all our traditional reasons of professionalism, allegiance to our craft, and mutual tribal loyalty. But there are larger reasons this struggle is important. The possibilities of politics are increasingly shaped by communication technology—who can use it, who can censor it, who can control it. Government and corporate control of the content of the nets, and of what people can do with their computers, is a severe long-term threat to political freedom. The nightmare scenario is one in which corporate monopolism and statist power-seeking, always natural allies, feed back into each other and create rationales for increasing regulation, repression, and criminalization of digital speech. In opposing this, we are the warriors of liberty—not merely our own liberty, but everyone else’s as well.

20.5 Problems in the Culture of Unix

Just as important as the technical problems with Unix itself and the challenges consequent on its success are the cultural problems of the community around it. There are at least two serious ones: a lesser challenge of internal transition, and a greater one of overcoming our historical elitism.

The lesser challenge is that of friction between the old-school Unix gurus and the new-school open-source crowd. The success of Linux, in particular, is not an entirely comfortable phenomenon for a lot of older Unix programmers. This is partly a generational problem. The raucous energy, naïvete and gleeful zealotry of the Linux kids sometimes grates on elders who have been around since the 1970s and (often rightly) consider themselves wiser. It’s only exacerbated by the fact that the kids are succeeding where the elders failed.

The greater problem of psychology only became clear to me after spending three days at a Macintosh developer conference in 2000. It was a very enlightening experience to be immersed in a programming culture with assumptions diametrically opposed to those of the Unix world.

Macintosh programmers are all about the user experience. They’re architects and decorators. They design from the outside in, asking first “What kind of interaction do we want to support?” and then building the application logic behind it to meet the demands of the user-interface design. This leads to programs that are very pretty and infrastructure that is weak and rickety. In one notorious example, as late as Release 9 the MacOS memory manager sometimes required the user to manually deallocate memory by manually chucking out exited but still-resident programs. Unix people are viscerally revolted by this kind of maldesign; they don’t understand how Macintosh people could live with it.

By contrast, Unix people are all about infrastructure. We are plumbers and stonemasons. We design from the inside out, building mighty engines to solve abstractly defined problems (like “How do we get reliable packet-stream delivery from point A to point B over unreliable hardware and links?”). We then wrap thin and often profoundly ugly interfaces around the engines. The commands date(1), find(1), and ed(1) are notorious examples, but there are hundreds of others. Macintosh people are viscerally revolted by this kind of maldesign; they don’t understand how Unix people can live with it.

Both design philosophies have some validity, but the two camps have a great deal of difficulty seeing each other’s points. The typical Unix developer’s reflex is to dismiss Macintosh software as gaudy fluff, eye-candy for the ignorant, and to continue building software that appeals to other Unix developers. If end-users don’t like it, so much the worse for the end users; they will come around when they get a clue.

In many ways this kind of parochialism has served us well. We are the keepers of the Internet and the World Wide Web. Our software and our traditions dominate serious computing, the applications where 24/7 reliability and minimal downtime is a must. We really are extremely good at building solid infrastructure; not perfect by any means, but there is no other software technical culture that has anywhere close to our track record, and it is one to be proud of.

The problem is that we increasingly face challenges that demand a more inclusive view. Most of the computers in the world don’t live in server rooms, but rather in the hands of those end users. In early Unix days, before personal computers, our culture defined itself partly as a revolt against the priesthood of the mainframes, the keepers of the big iron. Later, we absorbed the power-to-the-people idealism of the early microcomputer enthusiasts. But today we are the priesthood; we are the people who run the networks and the big iron. And our implicit demand is that if you want to use our software, you must learn to think like us.

In 2003, there is a deep ambivalence in our attitude—a tension between elitism and missionary populism. We want to reach and convert the 92% of the world for whom computing means games and multimedia and glossy GUI interfaces and (at their most technical) light email and word processing and spreadsheets. We are spending major effort on projects like GNOME and KDE designed to give Unix a pretty face. But we are still elitists at heart, deeply reluctant and in many cases unable to identify with or listen to the needs of the Aunt Tillies of the world.

To nontechnical end users, the software we build tends to be either bewildering and incomprehensible, or clumsy and condescending, or both at the same time. Even when we try to do the user-friendliness thing as earnestly as possible, we’re woefully inconsistent at it. Many of the attitudes and reflexes we’ve inherited from old-school Unix are just wrong for the job. Even when we want to listen to and help Aunt Tillie, we don’t know how—we project our categories and our concerns onto her and give her ’solutions’ that she finds as daunting as her problems.

Our greatest challenge as a culture is whether we can outgrow the assumptions that have served us so well—whether we can acknowledge, not merely intellectually but in the sinew of daily practice, that the Macintosh people have a point. Their point is made in more general, less Mac-specific way in The Inmates Are Running the Asylum [Cooper], an insightful and argumentative book about what its author calls interaction design that (despite occasional crotchets) contains a good deal of hard truth that every Unix programmer ought to know.

We can turn aside from this; we can remain a priesthood appealing to a select minority of the best and brightest, a geek meritocracy focused on our historical role as the keepers of the software infrastructure and the networks. But if we do this, we will very likely go into decline and eventually lose the dynamism that has sustained us through decades. Someone else will serve the people; someone else will put themselves where the power and the money are, and own the future of 92% of all software. The odds are, whether that someone else is Microsoft or not, that they will do it using practices and software we don’t much like.

Or we can truly accept the challenge. The open-source movement is trying hard to do so. But the kind of sustained work and intelligence we have brought to other problems in the past will not alone suffice. Our attitudes must change in a fundamental and difficult way.

In Chapter 4 we discussed the importance of throwing away limiting assumptions and discarding the past in solving technical problems, suggesting a parallel with the Zen ideas of detachment and ’beginner’s mind’. We have a larger kind of detachment to work on now. We must learn humility before Aunt Tillie, and relinquish some of the long-held prejudices that have made us so successful in the past.

Tellingly, the Macintosh culture has begun to converge with ours—MacOS X has Unix underneath, and in 2003 Mac developers are (albeit with a struggle in some cases) making the mental adjustment to learn the infrastructure-focused virtues of Unix. Our challenge will be, reciprocally, to embrace the user-centered virtues of the Macintosh.

There are other signs that the Unix culture is shedding its insularity as well. One is the convergence that seems to be going on between the Unix/open-source community and the movement called “agile programming”.¹² We noted in Chapter 4 that Unix programmers have seized happily on the concept of refactoring, one of the preoccupations of the agile-programming thinkers. Refactoring, and other agile concepts like unit-testing and design around stories, seem to articulate and sharpen practices that have heretofore been widespread but only implicit in the Unix tradition. The Unix tradition, on the other hand, can bring groundedness and the lessons of long experience to the agile-programming party. As open-source software gains market share it is even conceivable that these cultures will fuse, much as the old-time Internet and early Unix cultures did after 1980.

¹² For an introduction to agile programming, see the Agile Manifesto <http://agilemanifesto.org/>

20.6 Reasons to Believe

The future of Unix is full of difficult problems. Would we truly want it any other way?

For more than thirty years we have thrived on challenges. We pioneered the best practices of software engineering. We created today’s Internet and Web. We have built the largest, most complex, and most reliable software systems ever to exist. We outlasted the IBM monopoly and we’re making a run against the Microsoft monopoly that is good enough to deeply frighten it.

Not that everything has been triumph by any means. In the 1980s we nearly destroyed ourselves by acceding to the proprietary capture of Unix. We neglected the low end, the nontechnical end users, for far too long and thereby left Microsoft an opening to grossly lower the quality standards of software. Intelligent observers have pronounced our technology, our community, and our values to be dead any number of times.

But always we have come storming back. We make mistakes, but we learn from our mistakes. We have transmitted our culture across generations; we have absorbed much of what was best from the early academic hackers and the ARPANET experimenters and the microcomputer enthusiasts and a number of other cultures. The open-source movement has resurrected the vigor and idealism of our early years, and today we are stronger and more numerous than we have ever been.

So far, betting against the Unix hackers has always been short-term smart but long-term stupid. We can prevail—if we choose to.