Chapter 7

The Past, Present, and Future of Patterns

“Prediction is hard, especially about the future.”

Yogi Berra, Philosopher in the Baseball Hall of Fame

Philosopher of science James Burke is fond of noting that history rarely happens in the right order or at the right time, but the job of a historian is to make it appear as if it did. One benefit of writing a series of Pattern-Oriented Software Architecture (POSA) books is we have the opportunity to revisit our earlier predictions, summarize what actually occurred, and examine why it occurred the way it did.

This chapter revisits our 1996 forecasts on ‘where patterns will go’ that appeared in the first volume of the POSA series, A System of Patterns [POSA1]. We discuss the directions that patterns actually took during the past four years, analyze where patterns are now, and revise our vision about their future given the benefit of hindsight.

7.1 What Has Happened in the Past Four Years

In A System of Patterns [POSA1] we predicted how we thought patterns would evolve. Four years later we iterate through the same parts of the corresponding chapter in this book and summarize what actually transpired. We reference much of the relevant work during this period, although our list is not exhaustive. Additional references are available at http://hillside.net/patterns/.

Patterns

In 1996 we predicted that many patterns for many software development domains would be (re)discovered and documented during the next several years. We expected that these patterns would fill many gaps in the pattern literature. That part of our forecast has certainly come true, although gaps remain in the literature, particularly in distributed, fault-tolerant, transactional, real-time, and embedded systems, middleware, and applications.

The body of pattern literature has grown significantly during the past four years. For example, the PLoP and EuroPLoP pattern conferences in the USA and Europe have prospered and expanded to several new venues, including ChiliPLoP in the USA and KoalaPLoP in Australia. These conferences are based on a ‘writers workshop’ format [POSA1] [CW99]. In a writers workshop authors read each others’ patterns and pattern languages and discuss their pros and cons to help improve content and style.

In the past fours years books have appeared containing patterns on a wide range of subjects, including:

• Edited collections of patterns from many areas of software development [PLoPD3] [PLoPD4] [Ris98] [Ris00a] [Ris00b].
• A seminal book on analysis patterns for the health care and corporate finance domains from Martin Fowler [Fow97a].
• This book—Patterns for Concurrent and Networked Objects—which focuses on the architecture and design patterns for concurrent and networked middleware frameworks and application software.
• Software development process patterns [Amb99a] [Amb99b] and patterns for configuration management [BMT99].
• Patterns for particular programming languages, such as Smalltalk [Beck97] and Java [Grand98] [Grand99], or specific to particular middleware, such as CORBA [MM97].
• So-called ‘anti-patterns’ [BMBMM98] [BMT99], which present common ‘solutions’ to software development problems that seem reasonable, but that fail to work for various reasons.

It is important for us to note that these books differ in their focus and their quality. However, it is beyond the scope of this chapter to critique and review all these publications—we leave this to your judgement and on-line book reviews, such as those at amazon.com. Nevertheless, the volume of publications during the past four years reveals a growing interest in patterns in the software research and development communities

Many of the most widely-applied patterns are derived from or used to document frameworks [GoF95] [POSA1] [Sch96] [RKSOP00]. These patterns can be viewed as abstract descriptions of frameworks that help developers reuse their software architectures [JML92]. Frameworks can be regarded from another perspective as reifications of patterns that enable direct design and code reuse [HJE95] [HS98]. Experience has shown that mature frameworks exhibit high pattern density [RBGM00].

Pattern Systems and Pattern Languages

As the awareness of patterns has grown in the software community, an increasing amount of work has focused on pattern languages [Cope96] [Gab96] [Cope97]. As early as 1995 leading authors in the pattern community began documenting collections of patterns for specific software development domains [PLoPD1], particularly telecommunications [DeBr95] [Mes96] [ACGH+96] [HS99b]. These patterns were more closely related than the stand-alone patterns published earlier. In fact, some patterns were so closely related that they did not exist in isolation. The patterns were organized into pattern languages in which each pattern built upon and wove together other patterns in the language.

Many of the published pattern languages appear in [Beck97] [PLoPD2] [PLoPD3] [PLoPD4]. As we demonstrate in Section 6.2, A Pattern Language for Middleware and Applications, the patterns in this book also form a pattern language, or more precisely the core of a pattern language, for developing distributed object computing middleware, applications, and services. Our prediction that work on pattern languages would expand has therefore also come true. Again, however, significant gaps remain in the literature, due in large part to the effort required to write comprehensive pattern languages.

There has been work on classifying patterns and organizing them into categories [Tichy98], pattern systems [POSA1], and the comprehensive Pattern Almanac appearing in [Ris00a]. In general, however, pattern languages have become more popular than pattern systems and pattern catalogs during the past four years. The patterns community has found pattern languages to be the most promising way to document sets of closely-related patterns. Section 6.4, Pattern Languages versus Pattern Systems provides further justification and motivation for the gradual transition from pattern catalogs and systems to pattern languages.

Methods and Tools

In 1996 we predicted that work on methods and tools for using patterns effectively would expand, despite the scepticism of some experienced developers [POSA1]. The software research community has worked steadily on this topic, yielding methods for identifying patterns in existing systems [SMB96], as well as many tools for applying patterns to forward [BFVY96] [FMW97] [SRZ99] and reverse engineer software [Ban98] [PK98] [Mic00] [PC00] [PSJ00].

Research on automating software tools for patterns has not yet impacted the software tools community significantly, however, though there has been some recent progress. For example, notations for documenting patterns have been integrated into UML [BRJ98]. Similarly, tools are emerging that generate code automatically from patterns described via meta-models [AONIX99] [MTOOL99].

Algorithms and Data Structures

We predicted a growing interest in the application of patterns to the analysis of the relationships between algorithms, data structures, and patterns. We expected that this effort would be followed by the documentation of algorithms and data structures as patterns, using a specialized pattern form.

To date there have been several papers [Nuy96] and textbooks [FF97] [Preiss98] that integrate patterns from [GoF95] into undergraduate algorithms and data structures courses. In general, however, the focus of this work has been on undergraduate education, rather than a direct contribution to the body of literature from the pattern research community.

Formalizing Patterns

Over the past four years, much research has been conducted to formalize patterns [LBG97] [MK97] [LS98] [Mik98] [BGJ99] [EGHY99]. This work has not yet impacted the pattern community or software practitioners directly, however.

One reason for the lack of impact is that a pattern is a schema for solving a set of related problems, so it can be implemented repeatedly without necessarily being exactly the same each time. In contrast, formalisms aim to capture a particular concept as precisely as possible. Consequently, to capture the inherent variation in patterns, the formalisms used to specify patterns are often large and complex, which makes them hard to understand and use. Unfortunately this defeats many of the benefits of patterns, which are most effective when they enhance—rather than impede—communication among software development team members.

7.2 Where Patterns are Now

Today the pattern community has advanced significantly compared to where it was four years ago. In particular, the body of documented concrete patterns and pattern languages is larger and more diverse than it was four years ago—and it is still growing.

However, different groups in the community prefer different pattern paradigms, including:

• The top-down theoretically-oriented approach inspired by Christopher Alexander’s view of patterns [Ale79] and pattern languages [AIS77] [Cope96] [Cope97].
• The more bottom-up engineering-oriented approach pioneered by the Gang-of-Four [GoF95] and POSA work [POSA1].
• The approach advocated by authors of ‘special-purpose’ patterns, such as anti-patterns [BMBMM98].

In addition to a better understanding of what patterns are in theory, the patterns community now has a better understanding of how patterns work in practice [Vlis98a] [Bus00b]. By 1996 patterns had been applied opportunistically by isolated pockets of researchers and developers [GR96] [HJE95] [John92] [Cope95] [McK95] [SchSt95] [Maf96] [Sch96] [KCC+96]. In the intervening years patterns have been applied more systematically to production systems throughout a wide range of topics and domains in the software industry, including:

• Application frameworks [FS97] [BGHS98] [FJS99a] [FJS99b] [FJ99].
• Real-time CORBA middleware [SC99] [PRS00], high-performance Web servers [Sch97], network management [KTB98], concurrency control, and synchronization [SPM99].
• Input and output processing [Har97] [HS99b] [Tow99].
• Performance measurement [NLN+99] [GP97] and optimization [PRS+99].
• Security [YB99] and dependability [Maf96] [SPM98] [ACGH+96].
• Empirical studies that evaluate the pros and cons of applying patterns [PUPT97].
• The Java programming language [AG98] [GJS96], its standard libraries [Sun00a] [FHA99] and third-party frameworks [IBM98], its virtual machine specification [LY99] and implementation [Sun00b], and its development community [GG99].

Further evidence of the growing impact of patterns is the extent to which developers today not only use them in their designs implicitly—which they did long before patterns were described in books—but how they now apply patterns explicitly in their daily work. Patterns provide a common vocabulary that researchers and developers now use to exchange ideas concisely in design reviews, problem resolutions, training on system-specific features, and documentation of software architectures.

Naturally, it is important to guard against over-enthusiasm, where developers apply patterns blindly wherever possible. While patterns can improve design flexibility and (re-)usability, they may also incur additional costs. Implementing certain patterns may, for example, require more initial programming or design effort, or may yield more code to maintain. Some pattern implementations involve extra indirection that can yield performance overhead on certain platforms. Developers and managers must therefore always evaluate the benefits and liabilities of patterns carefully before using them, particularly if a simple class or function is sufficient to meet the requirements!

7.3 Where Patterns are Going

In the future we expect interest in patterns to grow substantially, as researchers and mainstream software projects continue adopting pattern-oriented paradigms, methods, and processes. This section describes our vision for each major topic area reviewed above, and outlines the key issues we foresee in future research. We also add several new categories—software development processes and education—to reflect promising directions in the patterns community not covered in [POSA1].

Patterns

The patterns literature with the most impact in the past four years has focused on concrete patterns and pattern languages, often derived from object-oriented application frameworks [John97]. Looking ahead, we expect that future work in the pattern community will expand upon this tradition. For example, the next generation of object-oriented applications and frameworks will embody patterns explicitly. Patterns will also continue to be used to document the form and content of frameworks [SFJ96]. Other key topics and domains that will benefit from concrete pattern mining include the following:

• Distributed objects. Many patterns associated with middleware and applications for concurrent and networked objects have been documented during the past decade [SC99] [Lea99a], including those in this book. A key next step is to document the patterns for distributed objects, extending earlier work to focus on distribution topics, such as remote service location [JK00] and partitioning, naming and directory services, load balancing, dependability, and security.

  An increasing number of distributed object computing systems, for example, must provide high levels of dependability to client programs and end-users. With the adoption of the CORBA Fault Tolerance specification [OMG99g] and ORBs that implement this specification, developers will have more opportunities to capture their experience in the form of patterns for fault-tolerant distributed object computing, than they had in the past [IM96] [ACGH+96] [NMM00].

• Real-time and embedded systems. An increasing number of computing systems are embedded, including automotive control systems and car-based applications, control software for factory automation equipment [BGHS98], avionics mission computing [HLS97], and hand-held computing devices. Many of these systems are subject to stringent computing resource limitations, particularly memory footprint and time-constraints.

  Developing high-quality real-time and embedded systems is hard and remains somewhat of a ‘black art’. Relatively few patterns have been published in this area as a result [NW98] [Bus98] [Lan98] [Lea94]. We expect the body of patterns for real-time and embedded systems to grow in the future. A driving force will be the maturation of object technology, together with the development tools, methods, and techniques that make it successful (including patterns), that will be applied increasingly in these domains.

• Mobile systems. Wireless networks are becoming pervasive and embedded computing devices are become smaller, lighter, and more capable. Thus, mobile systems will soon support many consumer communication and computing needs. Application areas for mobile systems include ubiquitous computing, mobile agents, personal assistants, position-dependent information provision, remote medical diagnostics and teleradiology, and home and office automation. In addition, Internet services, ranging from Web browsing to on-line banking, will be accessed from mobile systems.

  Mobile systems present many challenges, such as managing low and variable bandwidth and power, adapting to frequent disruptions in connectivity and service quality, diverging protocols, and maintaining cache consistency across disconnected network nodes. We expect that experienced mobile systems developers will document their expertise in pattern form to help meet the growing demand for best software development practices in this area.

• Business transaction and e-commerce systems. Many business information systems, such as accounting, payroll, inventory, and billing systems, are based on transactions. The rules for processing transactions are complex and must be flexible to reflect new business practices and mergers. Business systems must also handle increasingly large volumes of transactions on-line, as discussed in the Example section of the Leader/Followers pattern (447).

  The advent of e-commerce on the Web is exposing many business-to-business systems directly to consumers. Despite the importance of these systems, relatively little has been written about their analysis, architecture, or design patterns. We expect the body of patterns on transactions and e-commerce to grow, based on the preliminary work of [John96] [Fow97a] [KC97].

• Quality of service for commercial-off-the-shelf (COTS)-based distributed systems. Distributed systems, such as streaming video, Internet telephony, and large-scale interactive simulation systems [BHG00] [FJS99b], have increasingly stringent quality of service (QoS) requirements. Key QoS requirements include network bandwidth and latency, CPU speed, memory access time, and power levels. To reduce development cycle-time and cost, such distributed systems are increasingly being developed using multiple layers of COTS hardware, operating systems, and middleware components.

  Historically, however, it has been hard to configure COTS-based systems that can simultaneously satisfy multiple QoS properties, such as security, timeliness, and fault tolerance. As developers and integrators continue to master the complexities of providing end-to-end QoS guarantees, it is essential that they document the successful patterns to help others configure, monitor, and control COTS-based distributed systems that possess a range of interdependent QoS properties [ZBS97].

• Reflective middleware. This term describes a collection of technologies designed to manage and control system resources in autonomous or semi-autonomous distributed application and systems. Reflective middleware techniques enable dynamic changes in application behavior by adapting core software and hardware protocols, policies, and mechanisms with or without the knowledge of applications or end-users [KRL+00]. As with distributed system QoS, patterns will play a key role in documenting best practices that can help to ensure the effective application of reflective middleware-based applications.
• Optimization principle patterns. Much of the existing patterns literature has focused on software quality factors other than performance. Although this may be acceptable in domains where non-functional requirements, such as usability or extensibility, are paramount, other domains—particularly distributed and embedded real-time systems—value efficiency, scalability, predictability, and dependability above many other software qualities.

  In these domains, therefore, we expect to see an increasing focus on optimization principle patterns that document rules for optimizing complex software systems [NLN+99] [PRS+99]. Yet there is some debate in the patterns community whether optimization principle patterns are really patterns, or just principles [Bus00b].

In addition to the pattern mining activities described above, we expect to see increased publication of edited collections of concrete patterns related to particular application domains. For example, Linda Rising has edited a special issue of the IEEE Communication Magazine [IEEE99] and an edited book collection [Ris00b] that focus on the patterns and pattern languages in the telecommunication domain.

Pattern Languages

In addition to documenting stand-alone patterns, we expect to see continued codification of patterns into full-fledged pattern languages. We also expect to see existing stand-alone patterns [GoF95] [POSA1] and compound patterns [Vlis98c] integrated into current or new pattern languages, as discussed in Section 6.2, A Pattern Language for Middleware and Applications.

Conversely, as more experience is gained implementing larger architectural patterns, such as Reactor (179) and Proactor (215), we expect that smaller patterns, such as ‘Event Demultiplexor’ or ‘Continuation-passing Event-driven System’, will emerge. These patterns can help to improve the concrete realization of the larger architectural patterns by ‘decomposing’ them into smaller patterns. This decomposition process is yet another way to define pattern languages for the types of systems that follow the structure defined by architectural patterns that are being decomposed.

The trend towards pattern languages stems from the confluence of two forces:

• As more patterns are documented, the coverage of patterns in certain domains is gradually becoming more complete.
• As the patterns in a domain become complete it is more likely that other patterns in these domains can build upon or expose relationships to existing patterns [Ris00a].

This synergism occurs because as new patterns are documented in a domain they often address issues or sub-problems related to existing patterns’ problems. As these relationships become explicit, many patterns in a domain can be connected to form pattern languages.

The transition from patterns to pattern languages is ambitious and non-trivial [Vlis98b]. A key impediment to the growth of pattern languages is the effort required to document them. Pattern languages are easier to read and apply than they are to write because their authors must both:

• Be experts on the technical topics in a particular domain and
• Devote considerable time to documenting their expertise in a comprehensible form that identifies all patterns in the language and integrates each individual pattern into the overall language

Fortunately, individual patterns and compound patterns [Vlis98c] are increasingly being documented by the patterns community, based largely on abstractions from common frameworks [Rie97]. As a result, more material is becoming available to weave into pattern languages, alleviating the need to write everything from scratch.

New generations of authors are also gradually emerging from the writers workshops held at venues, such as the PLoP family of conferences. Over time we hope that these authors will help fill key gaps in the existing pattern language literature.

Experience Reports, Methods, and Tools

As noted in Section 7.2, Where Patterns are Now, many reports on experiences using patterns for industrial software development have now been published. We expect the number of such reports still to grow, in particular in domains where no such reports are available today, including electronic medical imaging, real-time aerospace control systems, and global Internet e-commerce systems.

This accumulation of documented experience will then allow researchers and developers to identify general, domain-independent core principles for using patterns effectively. Many of these principles have only been understood and applied implicitly in earlier projects. With a sufficient number of experience reports available, we expect the pattern community will distill and document these common core principles explicitly to help increase developers’ potential for applying patterns successfully.

We also expect these core principles will be woven together to form a ‘method’ or set of principles for pattern-based software development [Bus00a]. This method or set of principles will complement and complete existing software development methods, such as the Unified Software Development Process [JBR99], and documentation ‘methods’, such as UML [BRJ98]. The UML models of today largely denote the what and when of software designs. The pattern-oriented methods and principles of tomorrow will also help to justify why certain models are suitable for resolving key challenges in a particular domain.

We believe the most successful pattern-oriented methods and principles will be created ‘bottom-up’, by generalizing from the collective experience of expert pattern users and software architects. This inductive process has a better chance of success than approaches that try to define pattern-oriented methods and principles ‘top-down’. Similarly, when experience-based methods and principles for pattern-based software development become available, we expect that they will be codified into software development tools. Such tools should see wider adoption than at present, because they will automate good practices that are derived from experience.

Pattern Documentation

We expect books will remain the primary media for disseminating concrete information on patterns. However, we anticipate increased use of the Web, and Web-based protocols and tools, such as HTML and XML, to make it easier to document patterns and connect them into pattern languages. Prior efforts have had some success within isolated contexts, such as the Portland Pattern Repository in Ward Cunningham’s WikiWikiWeb [PPR]. The state-of-the-practice in this area may advance as a result of three trends:

• The growing ubiquity of Web access
• The advent of low-cost and high-quality electronic book viewers
• More effective e-commerce methods to remunerate authors

Formalizing Patterns and Pattern Languages

Research on formalizing the concept of patterns will certainly continue, and be extended by formalisms to capture more comprehensive pattern languages. It is unclear, however, whether such work will have greater impact in the future than it does currently. We do expect that useful work on formalizing particular instances of patterns in a given system or context will occur to document specific pattern implementations.

Certain formal techniques, such as design-by-contract [Mey97], can help to prevent improper use of a pattern implementation within a particular application. Focusing on formalizing instances of patterns therefore can be of practical use in production software development, to help improve system design and implementation quality.

Software Development Processes and Organizations

Much work on pattern languages has focused on improving software development processes and organizations [Cope95] [Cock97] [Ris98] [Har96]. Some software development teams have applied these patterns to help improve the quality of their workplace [Gab96] and their productivity. This work is also beginning to influence the academic software process research community. We expect to see more integration between patterns and software development processes and organizations over the next few years.

Patterns appear to be particularly useful for documenting ‘lightweight’ software development processes, such as open-source [OS99] and eXtreme Programming (XP) [Beck99] [KiLe00] processes. In addition, processes based on refactoring [Opd92] [FBBOR99] have been applying patterns successfully to help restructure existing software and enhance its modularity, maintainability, and reusability.

Education

In 1996 we did not foresee the significant effort that has arisen to document so-called ‘pedagogical patterns’ [PPP00] [Bir00]. Pedagogical patterns aim at capturing the principles of educating people effectively. Moreover, patterns are now widely used to teach software design and programming in undergraduate curricula [Lea99b] [Sch99] [LG00] [TM00] [Wal00]. They are also used to train and retrain software professionals [Ant96].

In the short term we expect that many undergraduate courses will apply and teach patterns to convey key points of good practice and sound principles of software architecture and design [Ken98]. Patterns are now commonplace in many professional software development organizations. Substantial experience has therefore accrued on how to apply patterns successfully. Academic institutions will hopefully codify this experience to help educate the next generation of software development professionals in patterns and their effective documentation and application. When these students graduate and start their professional careers, patterns will become even more pervasive than they are today.

Our Long-Term Vision

Looking much further ahead, we believe that knowledge of patterns, pattern languages, and framework components will ultimately progress to the point where software developers will possess methods and tools similar to those of biologists. After centuries of experimentation and modeling, biologists have totally or partially deciphered the genetic code of some organisms. Guided by this knowledge, biologists are now beginning to understand those basic elements within DNA sequences that are shared between organisms, as well as those that make each one unique.

Biologists are applying this information to develop new techniques for manipulating DNA, to provide cures for hereditary conditions, such as the desease Cystic Fibrosis. In this context scientists are both:

• Discovering core properties of DNA sequences and
• Inventing new techniques for manipulating these sequences safely and ethically

Both biological systems and software systems are highly complex. When decomposed into their basic components, such as DNA sequences and genes or patterns and pattern languages, however, they can be understood and controlled more readily. We believe that as core software patterns and pattern languages are understood and reified in the form of reusable components and frameworks, larger-scale complex software systems can be developed more predictably.

We expect that contemporary research and development on patterns and pattern languages will ultimately yield breakthrough discoveries of core software properties. In turn, these ‘software DNA’ discoveries will spur the invention of new processes, methods, tools, components, frameworks, architectures, and languages. These artifacts will enable us to engineer the complexity of large-scale software more effectively than is possible with the current state-of-the-art.

7.4 A Parting Thought on Predicting the Future

As the Baseball Hall of Fame’s resident philosopher Yogi Berra so eloquently stated: ‘prediction is hard, especially about the future’. Not all of the predictions we made in 1996 became true, particularly our forecast that use of pattern-oriented software tools would become widespread. Our revised predictions of the future in this chapter include a healthy dose of uncertainty. In particular, our current forecasts are based upon:

• Our knowledge of the pattern community
• Its ongoing and planned interests and research activities—insofar as we are aware of them and
• The future research directions we are guiding or that we consider fruitful

We expect many of our predictions to come true. The future of patterns and pattern languages may take directions however, such as pattern-based model checking, that we have not imagined today. Consider the forecasts in this chapter as one possible vision among many in the ongoing dialogue on patterns and their future, and not as the prophesies of infallible oracles.

When the next book in the POSA series is published in several years, we will revisit these forecasts, as we have revisited our 1996 forecasts from A System of Patterns [POSA1]. We will then know what really happened and will hopefully have the benefit of hindsight to explain why!

Credits

Thanks to the external reviewers who gave us comments on this section, including Joe Bergin, Bob Hanmer, Neil Harrison, Doug Lea, Ossama Othman, Carlos O’Ryan, Dirk Riehle, Linda Rising, John Vlissides, and Roger Whitney.