What is the agile methodology good for? The main practices, with benefits and shortcomings

The most well-known characteristic of agility is its iterative, time-boxed approach. Software is developed in cycles of a fixed length, most often two or three weeks long. It is an iron rule that an iteration is not prolonged. Instead, if a feature cannot be completed in that timeframe, it is dropped and taken up again in the subsequent cycle. The goal of an iteration is always a feature that is completely “done.” In development, this means a piece of software that is fully tested and can be shipped.

This iterative approach, on one hand, helps in shaping the users’ and stakeholders’ requirements. A customer representative is present in all major agile meetings. The customer is not only invited to, but even responsible for, giving his input to each iteration planning. On the other hand, the short cycles are a way to respond to the accelerated changes in the business and technology environment. This allows development projects with a time horizon of two or three years to catch up with these changes along the way at much shorter time intervals.

Agile thinking accepts that an application system, designed one or two years ago, no longer matches the actual needs when it is eventually rolled out. The agile approach provides the means for continuous modifications that are necessary because of the way users’ work has changed, the company has been reorganized, or the product and service portfolio looks different now.

The short development cycles are accompanied by strict governance on quality. One of the most popular prejudices about agile methods is that “Agile means anarchy.” Actually, the opposite is true. Although the agile methodology emphasizes self-organizing teams, it has firm principles in place. Test automation and continuous integration are central concepts in agile projects. The worst blame for an agile developer is that she or he “broke the build.” The ultimate goal is to have the software in a fully deployable state after each iteration (ideally, but not always achieved in practice, even throughout the iteration).

This principle—keeping the system always in a deployable state—creates a natural limit on the amount of (temporary) havoc that the developers may wreak on the system between two deliveries. This is sensible, but it also creates a scaling problem for complex features. It is hard to implement a complex feature so that it “fits into one iteration.” Complexity comes in three flavors:

Type 1 complexity can be addressed by decomposing the system into components that are as loosely coupled as possible. Each component is developed by one scrum team. Coordination between the scrum teams is handled by a so-called scrum of scrums. This is a coordination board, scheduled usually once a week, where the key representatives of individual scrum teams (for instance, scrum masters and product owners; see the following discussion) meet and discuss integration issues, conflicts, and so forth. This is a standard and proven way to run large agile programs.

Handling complexity of types 2 and 3 is more tricky, since decomposition is not so straightforward if you cannot distribute the problem on several scrum teams. Following the agile paradigm strictly, teams usually solve issue 2 by slicing complex features in such a way that they can be implemented across several iterations. Problem 3 is often addressed by deliberately oversimplistic solutions that fit into one iteration, including all integration work.

Both approaches can have negative consequences with regard to architectural integrity: an unnatural system partitioning in the first case, “refactoring hell” (an endless succession of refactoring operations) in the second. Here, the agile methodology requires deliberate architectural planning to achieve a proper balance between long-term considerations and quick wins.

This is part of the tricky relationship between agile methods and architecture, which we will discuss later in more detail. Another aspect is that the core concept of agile techniques is a cross-functional, self-organized team. The team takes common responsibility for delivering the product. Each team member must fulfill this responsibility in an end-to-end fashion. There are only two predefined, exposed roles:

Beyond these two, specialist roles such as architect are subject to some debate in the agile community. Some authors frown on such dedicated roles in the team, a sign of the tension between the architectural “wisdom of the few” and the agile notion that “the team is everything.”

The agile team favors direct communication over written documents. This is in line with common sense: Humans communicate most effectively and efficiently when they come together in a trustful atmosphere, expressing their insights and opinions with words, drawings, symbols, gestures, facial expression, voice modulation, and so forth. Agile has several elements promoting a directly perceivable experience of the development process.

One is the daily scrum round, another is the wall on which user stories are put as small yellow sticky notes instead of rows in the project manager’s Excel sheet. User stories in the Ongoing part of the wall are an ever-present incentive to get the task done. In the same way, moving such a sticky note from Ongoing into the Done quadrant (in front of the team during the scrum meeting) gives a special sense of satisfaction to an agile team member.⁹ This way, the entire software development process is implicitly visible and traceable to all the stakeholders.

As sensible as focus on direct communication may be, it also gives rise to some skepticism about agile methods with regard to proper written documentation. Agile proponents seem to hear it frequently—at least that would explain the many voices claiming that the opposite is true (in a more or less unnerved tone¹⁰) and that being agile doesn’t mean writing code without documentation.

The truth is, as we know from our own experience, that in agile methods it is just easier to “forget” documentation than in waterfall methods, where the written document is a mandatory process step. In an agile project, explicit care must be taken to ensure proper documentation—for example, by stating it as part of the condition of satisfaction of a user story or in the definition of done. With an effort, agile projects can produce excellent documentation, as opposed to poorly managed waterfall projects that can come up with hundreds of pages of irrelevant gibberish.

In summary, agile values and principles essentially bring customer centricity, the human touch, professional integrity, and a sense of belongingness into the work culture. At the same time, it demands a high degree of trust and accountability among all team members and stakeholders. It does not suffice to “do” agile methods; the organization must to some extent “be” agile to fully implement this methodology and to benefit from it.

Silver bullet, or special tool for the innovator niche? Agile in the enterprise world

Despite the frustration of both IT executives and professionals with the waterfall methodology and its “big bang—big crash” integration shock, it is still the predominant project execution model in the enterprise world.¹¹ Agile methodology has gained a foothold there, but it still resides in a niche. Common wisdom (outside the agile community) considers it to be mainly suited for innovative projects with limited cost pressure,¹² although there is some indication that adoption of agile methods generally results in a productivity gain.¹³ All in all, agile techniques meet several obstacles in the enterprise world:

Agile techniques often enter an enterprise through the back door, as a niche methodology used in handpicked innovation or pilot projects. Despite the obstacles, there is a lot of enterprise interest in agile methods, indicating that the adoption rate will further increase beyond that niche.

Agile and architecture: An antagonism?

There is an objection that we should look at in more detail: agility and architecture, does that fit? Doesn’t Principle 11 in the Agile Manifesto claim: “The best architectures (…) and designs emerge from self-organizing teams” (Beck et al., 2001), declaring something as specialized and top-down as EA obsolete? Does it make sense to apply something to EA that actually despises architectural planning?

Indeed, some authors advocate an architectural practice in Agile projects that can be called lightweight at best, nonexistent at worst. They express a lot of mistrust toward the traditional architecture paradigm. “Software architecture has a history of excesses that in part spurred the reaction called Agile,” write Coplien and Bjørnvig (2010, p. 85). Following their concept of design means streamlining architecture to the point of nonexistence: “We will strive for an architecture delivered as [application programming interfaces] APIs and code rather than duplicating that information in documents. While the interface is the program, the code is the design” (p. 85).¹⁶ Such a definition of architecture may work for small teams and simple applications, but it doesn’t scale up to larger and more complex systems.

With no intention of bashing an otherwise fine book, we believe that Coplien and Bjørnvig is a showcase of the naïve view of EA that some agile apologists seem to have.¹⁷ In contrast, other authors propose a modern, balanced way of thinking about agile architecture. Breivold, Sundmark, Wallin, and Larsson (2010) could not find proof for many of the concerns associated with agile methods and architecture. Leffingwell (2011, p. 386) introduced the phrase intentional architecture¹⁸ as an extension of the purely “emergent architecture” from Agile Manifesto Principle 11:

 Even minor, system-level redesigns can cause substantial rework for large numbers of teams, some of whom would otherwise not have to refactor their module. It is one thing for one team to redesign their code based on lessons they have learned; it’s quite another to require ten teams to do so based on another team’s lessons learned. (…) For developers, architects, and businesspeople who have experience building large-scale systems and portfolios (…) a solely emergent architecture is counter to our experience.

The looming refactoring paralysis as a consequence of a purely emergent architecture is only one side of the coin. Another possible objection against agile methods is that the processes in EA, and in the enterprise generally, are simply not operating with a time window of the typical sprint length of three weeks. This, of course, is true. But it is at closer inspection not a counter-argument against the application of agile principles to EA—just the opposite. The long process cycles add to EA’s lack of transparency and promote a silo mentality. Agile techniques can help here.

Having looked at the relationship between agile methods and architecture as a concept, let’s also inspect the architect as a person in the agile world. The status of this role is, at first sight, a bit unclear. Agile techniques put emphasis on an empowered team and a spirit to get things done. Therefore, architects are often viewed as notorious residents of the ivory tower. “Things are usually easier said than done, and software architects are notoriously good at coming up with things to say,” writes Timothy High (Monson-Haefel, 2009, p. 102).

In addition, any specialization is viewed with a certain skepticism. In an ideal agile world, every team member should be involved in every essential development task, especially in the design of the system. This is manifested in the “all hands on deck” paradigm.

An enterprise architect, on the other hand, has a planning horizon of years (instead of just the next two sprints) and performs a highly specialized task that is very far away from designing and coding the end product. That somehow makes her a legal alien in the agile world: eyed with skepticism, tolerated, but not really at home.

The agile architect is therefore a highly adaptable being—responsible for the design but at the same time carefully aware that the true decision power should be in the hands of the team. Johnston (2010) writes:

 In an agile development team many people will contribute to these [architectural] things. The agile architect will help all team members to contribute to the architecture, taking good ideas from everyone and making them part of a coherent whole. The architect may also adapt ideas originating elsewhere, but without losing the team’s ownership of the solution.

Kruchten (2004), in an admittedly not completely serious post,¹⁹ formulates:

 The architect doesn’t talk, he acts. When this is done, the team says, “Amazing: we did it, all by ourselves!”

 When the architect leads, the team is hardly aware that he exists.

Somehow this evokes the notion of a white magician, pulling the strings in the background on behalf of the higher good of architectural order and clarity. This idealized image feels a little bloodless and quixotic. However, the question remains whether this is really a sign for natural antagonism between agile methods and the architect’s profession. There is no more consensus and clarity in the image of an architect in EA, either. This role somehow oscillates between the Machiavellian master of political power games (as some enterprise architects would like to see themselves) and the overstrained inhabitants of the EA ivory tower you sometimes meet in real life.

Principle 1: Eliminating waste

Lean thinking starts with the definition of value created by the process under consideration, whether industrial production, software development, or architecture creation. This value needs to be defined from a customer viewpoint. Enterprise architecture can be seen as a product; from that perspective, the value of EA lies in the artifacts and services created by the EA group in interaction with other players in the enterprise. We discussed these artifacts and services in Chapter 3, “What Enterprise Architects Do: Core Activities of EA.”

Based on this value definition, every step in the product creation process can be divided into two categories: value adding and non value adding. In an ideal lean world, the value stream contains only value-adding steps and therefore would have an efficiency of 100%. The realistically achievable efficiency is much less than that.²¹ All non value-adding steps are considered Muda (the Japanese term for waste) in the lean philosophy. Eliminating waste means working toward an ideal value stream and is therefore the core principle of lean thinking.

Already in the manufacturing-focused Toyota Production System, the interpretation of waste goes way beyond defective pieces and leftover raw material that needs to be discarded. Waste also comprises unnecessary processing steps (i.e., work applied without need), inventory, and waiting. Poppendieck and Poppendieck (2003, 2007) adapted the seven kinds of waste originally described by Ohno (1988) to the software world and provided their interpretation of each waste type for software projects. We have added our interpretation of waste in EA to this compilation, as listed in Table 7-1.²²

Table 7-1 The Seven Types of Waste

Some authors have added other waste types; Liker (2004), for instance, also lists “unused creativity” of employees as the eighth type of waste. This certainly makes sense, especially when you analyze waste in the IT and architecture field. However, we will cover the broad participation of all EA stakeholders in the architecture decision processes at length in Chapter 8, “Inviting to Participation: EA 2.0.” Therefore we will leave it at the original seven types in this chapter.

Some of the waste types are transferable to software creation and EA in a pretty straightforward way (waiting, defects). For others, a bit of creative thinking is needed. The work pieces that are created in manufacturing correspond to the features of a software system. This explains the interpretation of in-process inventory and overproduction to partially done work and extra features in software development. In EA, overproduction can be translated to over-architecting. In that case, the architects create more architecture than needed or make it more elaborate and detailed than justified by the need at hand.

Transportation and motion mean, in the production process of tangible goods, a physical movement of half-finished pieces from one plant to another or within a machine tool. In software creation, the “plants” are essentially development teams, and the “machines” are the individual developers. So, waste here means time and traction lost in knowledge transfer and handover procedures between teams and by overloading a developer with too many parallel tasks. This explains the translation for both the software and the EA area to handoffs and task switching.

Extra processing, finally, is interpreted in two different ways in Poppendieck and Poppendieck (2003) and (2007). Both make sense, therefore they are both listed here. Extra processing means applying manufacturing steps to a work piece that are either overdone or not required at all. That matches a wide range of debatable extra processes in software creation, from the beloved daily status reports in times of crisis, over specifications written but not read, to elaborate QA checklists that are never seriously evaluated. In Poppendieck and Poppendieck (2007), extra processing is translated into relearning, which describes the unnecessary rebuilding of competences and skills not retained in the team from earlier work experience.

For EA, we interpret extra processing as redundant processes, which fits better with the procedural character of EA. Other than in the case of over-architecting, the right piece of architecture is created on the proper level of detail—but some steps in the architecture creation process are wasteful, such as performing architecture reviews that are too fine-grained or are done just to demonstrate authority (“I just want to have a say on it”) without any value added.

Several tools are used in lean manufacturing for detecting waste in the production processes. The most prominent one is the value stream mapping. The processing steps are noted as a flow diagram, using a dedicated notation that allows us to mark the type of operation, storage, movement, or delay. We will look into waste detection tools for EA more closely in the section “Building Block 1: Get rid of waste by streamlining architecture pocesses.”

Principle 2: Build quality in

The statement Build quality in expresses a very similar attitude to the agile methodology’s emphasis on persistent quality control. The common notion between the two is that quality is not added as an afterthought by a quality assurance phase at the end of the project (with all its undesirable side effects). Instead, quality is ensured on a daily basis as part of the actual software creation. Lean software development draws support from agile techniques with regard to the measures for achieving this daily quality control, such as continuous integration, test-driven development, and refactoring.

The quality of architecture artifacts is usually assured by peer reviews, stakeholder signoffs, and approvals. A mentality to ensure quality during creation, instead of (or in addition to) milestone-based acceptance, requires a continuous and broad participation of all members of the IT community, down to the development teams on the ground. The EA participation concepts described in Chapter 8 can help in dealing with quality assurance.

Principle 3: Create knowledge

The Create knowledge principle describes the focus on building and retaining knowledge about the complex processes of software creation and about the domain for which the software is designed. Lean thinking involves constantly monitoring and fine-tuning the production process. Faults are considered a normal part of a lightweight trial-and-error approach and seen as an opportunity to learn, as long as a strict policy for immediate fault correction is in place (see Principle 2: Build quality in).

Lean thinking regards the popular myths of getting things right the first time and a zero-defect mentality as counterproductive, since they tend to create a risk-averse and uncreative atmosphere.²³ Instead, lean thinking tries to create an environment that allows a sequence of small decisions that can be easily verified and taken back with little cost if needed.

Principle 4: Defer commitment

Defer commitment means that decisions are taken as late as possible to keep one’s options open. This allows being open to change, just like the “welcoming change” attitude in agile thinking. It also induces a simple design (read “as simple as possible”), by avoiding a premature preference for one out of several possibilities for architecting a solution.

The principle also implies using abstraction as a means to switch to a different solution in some modules of the architecture later on. The effect that is perhaps valued most by practitioners is that the architect has more time to learn about the subject and usually has to deal with less uncertainty when she defers commitment to the last responsible moment.

Principle 5: Deliver fast

The principle Deliver fast expresses the high value of a quick response to new or changed customer requirements in lean thinking. On one side, the ability of fast delivery is a result of the flexibility created by all lean principles; a lightweight process is easier to set up than a heavyweight one and will therefore create quicker results. On the other hand, this principle specifically expresses that lean thinking regards speed as a value in itself. Not only is IT quick-changing; also outside IT, in the customer’s domain, situations change rapidly. An IT department or supplier that is able to react fast to new input simply delivers better value.

It should be noted, however, that fast doesn’t mean hacking. Speed is primarily related to quick decision making and must not conflict with quality; see also Principle 2: Build quality in. At face value, deliver fast may also sound contradictory to Principle 4: Defer commitment. Again, this is not so. Deferring commitment simply means that decisions should be made as late as needed (to keep all options open) but not so late that they endanger a quick response to customer demands.

An important tool to achieve fast delivery is creation of a seamless flow in the manufacturing process, usually combined with the definition of takt time. The German word takt is used throughout the lean literature as a loanword, meaning clock cycle. Each operation should fall into the same beat, guaranteeing a steady manufacturing flow without waste due to waiting or delays.

Another key concept for fast delivery is the pull principle. In manufacturing, pulling means to produce goods on demand only, triggered by an order created upstream in the value stream. This is a precondition for the production pipe to have an ideal flow (another key lean term) of intermediate work items. The opposite concept is push—producing goods based on a forecast of future demands and “pushing” them downstream in the production process in the hope that they are available in sufficient numbers once the need for them arises. Pulling means short lead times and fast order processing in small quantities. Pushing, on the other hands, implies large batch sizes and long storage times. No wonder that lean manufacturing prefers pull over push.

Transferring the pull concept from manufacturing to software development means that features should only be developed (and architecture documents only be written) when a concrete demand exists. The level of detail and refinement should be appropriate to the actual request, avoiding proactive “gold-plating.” We will look deeper into the implementation of the pull principle in the section called “Building Block 3: Practice iterative architecture through EA Kanban.”²⁴

Principle 6: Respect people

Principle 6: Respect people deals with the human aspects in software creation. The core idea is that decisions should be taken by people who are best qualified to do so. In the agile world this principle is labeled Empower the team (Poppendieck and Poppendieck, 2003). It bears a strong resemblance to the agile thought of the team as the primary body of decision making when it comes to operational aspects of software development. Strong entrepreneurial (i.e., focused on team building) leadership, nurturing of technical expertise, and a short decision-making channels characterize a lean organization.

Another aspect of this principle—again a parallel to agile techniques—is the appreciation of stable teams, where knowledge can be built up and cultivated over a long period of time. A self-empowered organization therefore is also the ideal environment in which to implement Principle 3: Create knowledge.

Principle 7: Optimize the whole

Optimize the whole emphasizes the holistic approach in lean development. People should see the “big picture” when making decisions instead of considering only their own private silo (team, department, job description, subsystem, application, etc.). A violation of this principle leads to only locally optimized (and therefore suboptimal) solutions.

In software development, optimize the whole primarily means to consider the complete value chain, from gathering requirements down to running the developed software. This requires everyone in the software creation process to step out of the specialization cubicle. A way of thinking that goes like this:

 I’m an architect, and I do my job well if I produce as many high-quality design documents as possible, and I am measured by the number of features I design.

is not lean thinking.

Software Kanban

Software Kanban has been derived from the way workflow is organized in lean manufacturing. Its central element is the Kanban board (see an example in Figure 7-1).²⁵ The development tasks flow as task cards from left to right. Whenever there is a status change for a task—for instance, the implementation has been finished—the corresponding card is moved one cell to the right.

In Kanban, iterations are optional. The system instead focuses on enforcing the pull principle. Each cell (apart from Backlog and Live in our example) has a work in progress (WIP) limit attached to it—a maximum number of tasks that may reside in that cell. When that limit has been reached, no further task card may be moved into it. For instance, if the limit for ongoing development tasks is three, a scrum team already working on three tasks may not accept a further one. The team first must finish at least one of the ongoing tasks. This helps avoid waste from in-process inventory and delays. A team can easily judge at what time they need new tasks to work on and can pull them from the left of their cell on the board. This effectively promotes the pull concept.

When applying Kanban to EA, as we will do later, we will use the term design in progress (DIP), coined by Donald Reinartsen (1997), instead of WIP. DIP describes the maximum number of design tasks currently being worked on.

Partially done work

Partially done work, together with Overarchitecting (covered in the next subsection), is the most obvious waste area in the life of any architect. The manufacturing counterpart is In-Process Inventory: A heap of work pieces stored between two processing steps. Partially done work means that architectural artifacts—specification, models, documentation—are not completed and closed but are left in a semifinished state and stockpiled for later use.

This can be the result of narrow-minded thinking. A customer business representative once told us:

 Well, our enterprise architects are usually happy when they have some Web service available in the ESB; that’s where they stop thinking. I have to convince them that their job is not done unless the service has reached the consumer on a mobile app or through some other channel.

Probably you know this lack of end-to-end consideration from your own experience (of yourself and others).

In other cases, the architecture concepts are sound but are just put on the shelf in an unfinished state because a more important issue came up. As often as not, these half-ready specifications will never be touched again. That turns them into waste via Overarchitecting.

Navigating between these two waste types—Partially done work and Ovearchitecting—is the tightrope walk an architect must perform as part of her daily business. It is basically the chasm between too little and too much elaboration and attention to detail (or the detail work being done at the wrong time). A common sense approach to priorities, combined with solid time management, is the architect’s main tool to make the proper decision in each particular case.

It may sound curious, but partially done work can also manifest itself as too much information. An unstructured clutter of data, concepts, and ideas is of no use to the reader. In a similar way, a model without a view appropriate for the target audience—one that the reader is used to and can understand—is not ready to be used.

Another aspect of this waste type is the possible lack of a clear information owner. If many actors work on the same piece of information and no crystal-clear rules on document storage and archiving are in place, a plethora of parallel document versions exists—scattered across multiple inboxes and C: drives, many of them with minor changes and additions.

Table 7-3 lists the interpretation of this waste type in manufacturing, software development in general, and architecture in general as well as examples for wasteful activities in the main EA activities EA-1 to EA-8, as introduced in Chapter 3. As can be seen from these examples, waste due to partially done work is often associated with doing things halfheartedly. It can be the result of an underempowered or understaffed EA group, but it might just as well indicate some suboptimal EA processes that put too little stress on completion and results.

Table 7-3 Waste Due to Partially Done Work

Overarchitecting

As indicated, the waste types Partially done work and Overarchitecting are to some extent two sides of the same coin (the attention to detail is in some way mismatched). The manufacturing counterpart to Overarchitecting is Overproduction. Transferred to EA, this means that an enterprise architect performs activities and produces documents that no one needs. Or, if the EA’s work actually meets a demand, the documents’ level of detail is such that no one will read them. Sending out all information to everyone (i.e., pushing instead of pulling information) also is a flavor of overarchitecting. Essentially, the EA effort is invested into the wrong activities and initiatives. Table 7-4 lists examples of this waste type.

Table 7-4 Waste Due to Overarchitecting

The strong presence of overarchitecting can indicate the control freak mindset of the Guardians of Wisdom type of EA organization, which we portrayed in the EA caricatures in Chapter 1. In any case, overdoing architecture efforts will create waste and can even have harmful consequences.

Redundant processes

Waste due to Redundant processes means that, although the architectural work may have the proper focus, it requires too much effort. The manufacturing counterpart is Extra processing—the right work piece is being worked on at the right time, but too many processing steps are used on it. As Table 7-5 outlines, this type of waste can manifest itself as too much bureaucracy or in general as an ineffective EA or IT organization.

Table 7-5 Waste Due to Redundant Processes

Meetings (if not properly moderated) or excessive reviews are a relatively harmless nuisance but can pile up into a veritable time sink. A more serious source of waste is a strictly hierarchical chain of command in which technical decisions are by principle made on a high management level, where the technical implications are no longer understood. Such an organization requires a lot of communication across the ranks to properly brief the decision makers. Another very frequent case of unnecessary effort by Redundant processes is the creation of proprietary technology platforms, where slightly customized standard frameworks would suffice.

Handoffs

Waste due to Handoffs is the counterpart to transporting a work piece between different machines in manufacturing. Passing on partially completed tasks to someone else will always create friction. But even communication (or the lack of it) between stakeholders in the EA process is a potential source of waste by handoff. If the dialogue between enterprise architects, business managers, management boards, IT management, IT development teams, and implementation partners is disrupted, the EA process will not work properly.

Table 7-6 lists several examples of the Handoffs waste type. Many of them are, in some way or another, associated with a silo attitude. The chief mechanic mindset described the EA caricatures in Chapter 1 is one example of a lack of proper communication between stakeholders. The “over-the-fence” mentality of passing implementation projects over to partners without proper interaction along the way is another.

Table 7-6 Waste Due to Handoffs

Waste due to Handoffs is also created if the EA organization gives up too much responsibility to partners. In general, partnering and outsourcing can have critical side effects. In one of our projects, a mobile network operator had outsourced its network. As a consequence, it had problems accessing key data about network failures, which the operator needed for enhancement planning. The problem was that, by accessing the failure data, the operator also would have gained insight into the individual fault resolution performance of the partners’ monitoring personnel. The network operator was not in the least interested in this information but was not allowed to have access for data privacy reasons.

But even a less drastic way of collaborating with partners can cause serious extra effort due to handoffs. When consultancy companies perform application modernization initiatives on behalf of their client’s IT organizations, they often own the architecture of the whole solution. After finishing, they leave the scene—with the client’s architects scratching their heads because they do not understand the architecture decisions that have been made.

Task switching

In manufacturing, motion means the movement of a work piece within one machine (as opposed to transportation, where it is moved between different machines or production sites). Transferred to the abstract principles of software architecture, an overly layered system exhibits too much motion.³⁰

In terms of software and architecture processes, motion means Task switching. A human brain is by nature ill suited for parallel processing. The damaging effect of information overload and too many concurrent activities on organizational effectiveness is well known (see, for instance, Spiro, 2011). Enterprise architects are no exceptions to this rule. Table 7-7 lists many examples in which the architect can be overloaded by a multitude of parallel tasks in her daily work.

Table 7-7 Waste Due to Task Switching

Sometimes task switching can grow out of control if one of the manifold side activities of an enterprise architect prevails to an extent that it doesn’t leave room for much else. In that case, there is actually not much task switching anymore (apart from the one big switch to the predominant side activity). However, the enterprise architect is prone to deteriorate to a program or project manager or to a universal Mr. Fix-It for technology issues no one else can solve.

Delays

Other than in manufacturing, the intermediate products of information design processes—specifications, concepts, models—are nearly always perishable goods. They lose their value when kept on the shelf too long. Therefore, Delays are always harmful in EA. Unfortunately, they are also a very common waste. All stakeholders in the EA process (with the exception of the enterprise architects themselves) do have other main priorities than the creation of EA. In addition, these stakeholders mostly belong to the ranks of middle and higher managers, whose schedules are always packed. This makes the EA processes prone to holdups.

Table 7-8 lists some examples, such as waiting for appointments, information, review feedback, or approvals. In some cases, it is not the EA organization that has to wait: When a project wants to use a certain technology but the EA group has not made up its mind yet, another delay is created.

Table 7-8 Waste Due to Delays

Defects

Defects in general are considered waste in lean thinking. The Build quality in principle demands that production processes should be set up in a way that no defects occur in the first place. Lean software development proponents even go so far to claim that if you detect faults in final testing, something is wrong.³¹

For architecture, this principle is hard to implement. It can be as flawed as any other human product or service—maybe even more so due to its fuzzy and forward-pointing nature, which makes verification difficult. Architectural defects in general, as listed in Table 7-9, comprise all kinds of misguided design, from a plain misunderstanding of the business requirements to more sophisticated flaws such as resume-driven design.³²

Table 7-9 Waste Due to Defects

Enterprise architecture in particular offers a rich choice of options for building in flaws and defects—from politically motivated technology decisions and pet projects down to inappropriate modeling languages. Every enterprise architect will have her own set of tales in this department; therefore, the list in Table 7-9 (like any of the other waste tables) does not claim to be comprehensive. It merely outlines some frequently occurring EA defects.

The EA waste matrix

Padma Valluri, Bank4Us enterprise architect with a mission of purging the EA processes of bureaucratic ballast, has been allocated a small budget for an external consultant. She decides to contact Sandro Poggi, an expert in lean and agile architecture. Padma has worked with Sandro in earlier projects and knows that he is the right person to support her efforts.

On Sandro’s advice, Padma organizes a workshop to jointly compile an EA waste matrix. Most of her colleagues on the EA team can free up time to participate. To get an outside view of the EA work, Padma also approaches those key IT and business representatives with whom the enterprise architects interact most. She has good standing with them. In addition, there is a mutual feeling that the interaction of business, IT, and EA has room for improvement. Therefore many of the stakeholders promise to take part in the workshop, too.

For the meeting, Sandro paints the matrix in Table 7-10, with the cells left empty, onto large sheets of paper he hangs on the meeting room walls. After a brief introduction to the lean concepts and the meaning of the seven waste types, he asks each workshop participant to write her or his personal ideas for waste removal on sticky notes. The notes should be marked with or as a priority indicator and then be pinned to the appropriate cell in the matrix. Afterward, Sandro moderates the group’s joint review of all sticky notes. The notes are reallocated to other cells if needed, duplicates removed, and the final priorities per table cell agreed on.

Table 7-10 Example of an EA Waste Matrix ( = High Potential for Waste Removal, = Medium Potential)

The outcome of the workshop is a filled-in matrix, as shown in Table 7-10. In addition, Sandro and Padma compile the notes for the detailed waste tables we saw in Tables 7-3 through 7-9. Before Padma can write a business case for a dedicated lean and agile EA project, she wants to perform one exemplary deep dive to ensure that the lean approach is indeed the right way for her EA organization. Table 7-10 helps her select the area to look at first. It seems that EA-3 (Evolving the IT Landscape) and EA-5 (Developing and Enforcing Standards and Guidelines) offer the most potential for waste removal.

After some discussion with her teammates and with Sandro, Padma picks one particular, slightly infamous project from EA-5 that ran under the name of MobileDevGuide. This exploration of mobile technologies and platforms, targeted at creating an enterprise-wide technology and development guideline for Bank4Us mobile applications, was not a badge of honor for the EA team. The project took much longer to complete than anticipated, yet none of the stakeholders was really happy with the result. Padma reasons that if the lean and agile EA approach reveals how this project would have run better, it should as well be able to help her improve the more frequently occurring EA processes.

The EA value stream

The removal of waste is, in lean methodology, inseparably linked to the analysis where value is generated. In other words, the EA value stream needs to be analyzed. Trivial as it might sound, the first step in value stream analysis is always a clear definition of the term value. Any process step either adds value or it does not. In mass production, a step is value-adding if it directly contributes to manufacturing of a work piece.

Painting a car body is value-adding; storing it to dry is not. The latter might be an example of a step that does not have a value in itself but is required by the process. Therefore, lean experts usually use the three-fold classification scheme that follows.³⁴ The goal of value stream optimization is to eliminate the NVA and minimize the R-NVA process steps.

What is now value-adding in terms of EA activities? There is no general answer to this question. The creation of architecture documents and models can be value-adding—or not, depending on whether there is a true demand for them (as opposed to the documents that are produced but never read). The same applies to the definition of architecture processes and the creation of architecture views (such as executive summaries or presentation slides). Communication, learning, or measures to improve employee satisfaction are usually not directly value-adding but can fall into either the R-NVA or the NVA category.

As a conclusion, the definition of EA value depends on the context of the creation process and on the specific goal defined for it. Therefore the first step should always be the precise definition of a process goal. The sidebar contains a useful template for goal definition.

In the goal template, the parts in pointed brackets should be replaced by a specific work result and the appropriate time constraints. The fragments without defects and in an efficient way are not negotiable; they should, without exception, always be part of the goal. The timeliness criterion means “at the right time” and not necessarily “as fast as possible.” As we saw in the “Seven Wastes of EA” subsection, information is a perishable good that should not be stockpiled.

Padma and Sandro now ponder how they can analyze the MobileDevGuide value stream. Sandro points out that the initial EA value stream analysis should be performed again as a group exercise. All major stakeholders in the process need to be present. Padma should consider upstream and downstream actors as well—representatives from the business lines who gave the requirements for MobileDevGuide but also the guideline consumers, such as project managers and architects from the IT department. This might even involve implementation partners from different companies.

In the workshop itself, Sandro prefers using wallpaper, pens, and sticky notes instead of drawing tools. The market offers a rich set of flowcharting tools as well as dedicated lean value stream mapping editors, which can be used to capture and refine the workshop results afterward. When moderating the value stream analysis workshop, Sandro plans to use the process activity mapping, design structure matrix, and pipeline response matrix tools.

Process activity mapping

Lean process experts use process activity mapping to analyze the concrete process flow and search for improvement potential. There is no fully standardized terminology and notation for it. We lean toward the version by McManus (2005),³⁶ with slight adaptations to the specific situation in EA processes.

A value stream map is created by “put[ting] yourself in the position of a design as it progresses from concept to launch” (Womack and Jones, 1996, p. 16). The map depicts the flow of an EA document or model through different kinds of processing steps. The notation is listed in Figure 7-2. Broad arrows symbolize major flow of information or intermediate work products; the fine arrow stands for contributing information, such as review feedback. The process nodes can be:

This list is not comprehensive. An exhaustive best-practice guide for process activity mapping could easily fill a book of its own.³⁸ The lack of standardization leaves room for individual customizing of the notation if it makes sense for a specific process. One sensible enhancement, for instance, is to use swim lanes to correlate process nodes with actors.

Below the process activity diagram, a timeline can be added. It denotes phases of activity (line at bottom) versus waiting (line at top).³⁹ The timeline is usually annotated with the lengths of the respective phases. Further key performance indicators (KPIs), such as lead time or in-process time, can be added to the process nodes. However, this makes more sense for the analysis of frequently repeated tasks, where exact measurements count. In EA, the value stream map is more useful as a tool to detect potential for optimization and design improvements. A precise quantification of the improvement potential is usually neither needed nor possible, at least not without introducing additional monitoring as the first step.

Back at Bank4Us, Padma Valluri has organized a lean deep-dive workshop, moderated by Sandro Poggi, to analyze the MobileDevGuide project. The project is notorious among the Bank4Us IT crowd, which makes it a good showcase for Padma to explore the capabilities of lean EA. How could this project have been executed in a better way?

MobileDevGuide was started in the context of the Closer to Customer strategy assigned by the Bank4Us CIO (see Chapter 1). One IT maxim in this strategy was Full Multichannel Capability, but at first the company had absolutely no standards in place that covered mobile technology. Oscar Salgado, chief architect, dealt with the topic of mobility support already in our discussion of EA activity, “Monitoring the project portfolio (EA-6),” in Chapter 3.

Therefore, the EA team proposed a thorough technology and platform evaluation, targeting the creation of compulsory standards and development guidelines. Not much mobile know-how was available in the bank’s IT organization. For that reason, the allocated budget for MobileDevGuide included an unprecedented luxury: the creation of a small prototype by an offshore team of one of Bank4Us’ preferred implementation partners, managed by the bank’s IT organization.

Much to everybody’s dismay, the project turned out to be a failure. It took a full year instead of the planned six months, overrunning schedule and budget by 100%. Yet after all this effort, no one was really happy with the result. The IT crowd claimed that the guidelines were impractical and overly rigid; the business crowd was disappointed with the prototype’s capabilities. Worst of all, due to the delay several development projects could not wait for the guidelines to be finalized (or claimed they couldn’t) and started with the implementation of their mobile applications—each using a different technology, of course.

MobileDevGuide is still a bit of a political minefield. So Sandro, moderating the process activity mapping workshop, uses all his professional experience to avoid finger-pointing. As a first task, he discusses with the workshop participants which goal statement would have been appropriate for the project. They come up with the following:

 Produce the mobile application development guidelines for Bank4Us in compliance with both the business and the IT requirements and validated by a working prototype, in an efficient way and without defects, within six months.

He guides the group to map out the process on whiteboard and flipchart. The final result is shown in Figure 7-3, with the process steps numbered 1 to 9. This drawing is pretty revealing to the workshop participants. It shows that most of the project’s difficulties did not stem from personal failure but from the intrinsic waste in the creation process. In the evaluation phase of the workshop, Sandro and Padma note down the following insights:

All in all, the findings indicate a fair amount of waste—at least delays, redundant processes, partially done work, and handover (with a potential for the other waste types as well). Padma and Sandro decide to evaluate the results from the other analysis tools first before drawing final conclusions.

The Design Structure Matrix (DSM)

In addition to the process activity map, Sandro introduces a second tool in the MobileDevGuide workshop. The design structure matrix (DSM) uses a very simple tabular format⁴⁰ to specifically highlight iterations and rework in the process. Figure 7-4 shows the DSM table that the participants created in the workshop.

If step k provides input to step n, the intersection of column k and row n is marked with an x. All dots below the diagonal line in the lower half of the matrix represent forward flow—intermediate work products proceeding from one processing step to the next. This is unproblematic. Dots above the line, however, denote iterations and backward flow. They are a strong indicator of waste due to redundant processes or overarchitecting.

Not surprisingly, the culprits in MobileDevGuide are Steps 2, 4, 6, and 8—the review and approval steps (and their iterations) that were the source of some rework in the project. Had the process activity map been more complex, this might have been overlooked but made visible by the DSM.

The pipeline response matrix

Sandro asks the workshop participants to create, as a third tool, a pipeline response matrix⁴¹ for the MobileDevGuide project. This is a specific view of the process activities, focusing on the ratio between the in-process time (the number of working days really spent on the task at hand) and the task inventory (the time needed to work on unrelated, parallel tasks). Each task is visualized as a square, with the horizontal edge denoting cumulative in-process time and the vertical one indicating cumulative task inventory time. The total duration is the sum of horizontal and vertical edge length. An “ideal” task in the diagram is as flat as possible to avoid waste due to delays.

Figure 7-5 shows the MobileDevGuide matrix. Let’s take a closer look at Task 1, Generalize requirements. Its overall duration was 20 working days. During that period, the enterprise architect was able to work roughly 50% of his available working time on this task (in-process time = 10 d); the second half he had to dedicate to unrelated, parallel activities (task inventory = 10 d). The underlying thought model originates from manufacturing, assuming a first-in, first-out (FIFO) processing of incoming tasks—therefore the term inventory for the architect’s other activities.

The pipeline response matrix is a good way to analyze waste due to task switching and delays. In Figure 7-5, the total in-process time is 79 days—less than half the total duration of 197 days. The rest is waiting. In other words, it might have been possible to finish the project in the planned six months if the internal processes had been better organized and less riddled with parallel processing and waiting. As a consequence, Padma and Sandro formulate the following additional findings for MobileDevGuide:

A pilot project at Bank4Us

Padma has now seen enough to consider piloting the lean and agile approach with a real EA project. She discusses with her boss, Oscar, which activity would be suitable to test the idea. Her own proposal to Oscar is to approach Ian Miller and his team of enterprise architects, who work on the application rationalization program in the Closer to Customer initiative. (We accompanied Ian and his team in describing the typical EA activities in Chapter 3.)

From the occasional meeting in the coffee kitchen, Padma knows that Ian is inundated with work. She reckons that he will gratefully accept any support that might help him better organize his project. In addition, she has a trustful relationship with Ian; she knows him as a critical yet open-minded person. If the concept works for Ian, Padma feels, she can convince her other peers as well.

In an initial meeting with Ian, Padma and Sandro establish the basic facts about the application rationalization program. Ian leads a project team of five other people who all work (more or less full-time) on the application rationalization program: two further enterprise architects from Oscar Salgado’s EA group, two dedicated business process experts, and someone from the IT organization who is focused on planning and coordinating implementation projects.

In addition, Ian’s list of key people who need to be kept informed, interviewed, or monitored for progress runs into the several of dozens: business heads who want to know what’s going on, key users who understand the crucial and undocumented traits of business processes and applications, IT program managers like Sarah Taylor (whom we met in Chapter 1), architects and managers from projects implementing parts of the transformation—the list could be continued seemingly endlessly.

So far, Ian struggles with organizing his project. The governance boards of the EA and IT organizations are not much help. They are useful in monitoring the project milestones, but they do not offer much help in steering the architecture work for a complex IT transformation program. Office space is scarce at Bank4Us; therefore Ian was not even able to seize a permanent “war room” for his team. Luckily, at least two of his team members reside on the same floor as Ian himself. But the coordination of meetings with the larger team and the main stakeholders alone takes up a substantial amount of Ian’s time.

After the meeting with Ian, Sandro and Padma design a possible architecture scrum setup for the application rationalization program, as shown in Figure 7-15. Ian acts as scrum master. Chief architect Oscar Salgado takes the role of the product owner, representing the sponsor’s intentions for the program. The team will meet every morning in Ian’s office for 15 minutes in a daily face-to-face scrum meeting, standing up in order to keep it brief and concise. Team members who are traveling participate via phone. Each participant in the scrum round answers the three typical scrum questions:

Oscar Salgado is invited to participate in the chicken role, meaning that he is allowed to listen in but should not ask questions or give comments. Any in-depth discussion is postponed to after the scrum.

In addition to the core scrum team, Sandro has designed two scrum of scrums, which will meet once a week in a fixed timeslot. The IT Project Portfolio Scrum of Scrums gathers the program managers of the IT organization involved in the program implementation. The meeting will make sure that everyone is on the same page with regard to the implementation progress, planning, and showstoppers.

The Architecture Scrum of Scrums helps supervise the architecture implementation. It provides a platform for regular exchange between the EA team and the project architects. Both scrums are coordinated by one member of Ian’s team. As the overall project manager, Ian is also invited to both meetings.

Padma likes the concept. She decides to present it to Ian and his team once she has better understood how the actors in this organization structure will interact in practice. She tells Sandro that she will require more of his time.

Levels of agile requirements

The basic building block of agile requirements management is the user story. It follows a fixed wording pattern: As a …, I want …, so that …, which associates it with a specific stakeholder or role. The so that part transports the user intention. A typical user story for a reporting system in the Bank4Us IT landscape reads like this:

 As head of compliance for the US/West region,

 I want a daily report on the main SOX compliance indicators

 so that I can report the compliance level to my regional board.

A user story is sized so that it can be implemented within one iteration (usually two to four weeks). For requirements that need more effort, agile techniques provide the more coarse-grained terms feature, epic, and theme.

These terms are not standardized, though. When you use agile methods for small or midsize software projects, epics are just a kind of large user stories.⁴⁸ In the same context, themes denote a related group of user stories. For an online job placement portal, the themes job search, CV database, and job application comprise all stories describing these particular aspects of the portal.

For use in an enterprise context, we are at liberty to interpret these terms in a broader sense so that we can apply them to the whole IT landscape instead of just one project. Some authors⁴⁹ have provided their interpretation of these agile terms for the enterprise.

Leffingwell (2011) sees an effort factor of roughly 50 between the different granularities.⁵⁰ This fits well into usual dimensions encountered in different planning horizons, as shown Figure 7-18 and Table 7-12.

It is worth noting that there is no fixed format for the higher abstraction levels described here. “Epics may be expressed in bullet form, as a sentence or two, in video, as a prototype, in a short business case, or indeed in any form of expression suitable,” writes Leffingwell (2011, p. 453), but that applies to the other levels as well. One can use the user story format if this adds value.

But it is also suitable to use any particular domain-specific format—for example, in our case, the taxonomy that EA uses to describe transformations of the IT landscape. It is even advisable to keep the descriptions concise and domain-specific. “I was once at a meeting where an (otherwise) clever agilist, pushing back on what he considered to be delaying and excessive epic-level documentation requirements, told his PMO management, ‘You can even tell us in interpretive dance; we’ll take it from there,’” writes Leffingwell (2011, p. 453)—not entirely seriously, though.

Iterations in EA

Padma Valluri has scheduled another series of meetings with Sandro Poggi, her agile architecture consultant. Together they analyze the parallels between the TOGAF framework, which the Bank4Us EA team is using, and the agile methodology. After Sandro has given her a quick introduction to agile requirements management, Padma sees the match. The main question for her is how iterations, as promoted by the agile methodology, fit TOGAF.

As already outlined in Chapter 4, TOGAF does allow, and even explicitly recommends, the use of iterations in its ADM part. Figure 7-20 displays some proposed iteration cycles. In addition, ADM users can define their own cycles without losing the TOGAF-compliancy seal. Nonetheless, the architecture development iteration will be the most common iteration type, covering phases A–F.

As Figure 7-21 shows, the iterations can be used to connect the various levels of architecture detail with each other. The ADM phases A–F on the enterprise strategic architecture level prepare and kick off architecture work on the next lower level of detail, in this case the segment architecture level, and from there down to capability (or capability increment) architecture.

TOGAF supports iterative work very well. We can nicely reconcile the ADM phases with lean and agile’s lightweight control mechanisms.

TOGAF as blueprint for EA Kanban

Padma and Sandro discuss what kind of lean and agile process would be able to organize the tasks for Ian’s team in a flexible, efficient, and TOGAF-compliant way. Sandro proposes a Kanban approach.

We have introduced software Kanban in the section “Learning from mass production: Lean software development.” The principle of Kanban is that task cards flow through a series of process steps. In software development, these steps typically form a sequence like Plan – Implementation Ongoing – Testing Ongoing – Waiting for Deployment – Live.

For EA tasks, there is a perfect match in TOGAF ADM, which already provides a detailed process model for EA activities with standardized phases. Instead of the above steps for software development, Ian and his team can use the TOGAF phases A–H (plus Preliminary phase). The rich set of predefined activities within each phase provides a blueprint for possible EA tasks.

Figure 7-22 shows the principle of an EA Kanban Board using TOGAF ADM. Requirements such as Assess TAR extension to make TR2 obsolete flow clockwise through the ADM phases (instead of from left to right, as in software Kanban), triggering phase-specific tasks (for example, Model business processes in Phase B) along the way. The Requirements Management phase in the middle is replaced by a joint backlog for all phases, marked “To be done.”

Rules for the EA Kanban board

When entering a phase, the Kanban cards are first placed in the corresponding “To be done” sector. As Figure 7-23 shows, cards in progress are placed inside the circular shape denoting a TOGAF phase, and finished tasks are placed outside the phase. After all applicable phases are completed, the cards are either moved into the top left box marked “Done” or else iterated again.

Two types of cards should be used on the Kanban board:

The following nine steps explain in detail how the Kanban cards are handled. The step descriptions are supported by Figures 7-24 through 7-26. We focus on only one TOGAF phase (B in our example) of the segment architecture. A typical epic might be Assess TAR extension to make TR2 obsolete, as handled by Ian Miller and his team. Epics and tasks are denoted by E and T, respectively.

The targets of this process are to produce a constant flow through Phases A–H to reduce the overall cycle time, increase visibility, and allow for flexible task planning. All tasks of a theme, epic, or feature must be completed before it can be moved into the next phase. However, it is possible to drop a task, just as in lean and agile software development. In that case, it is a deliberate decision to speed up cycle time instead of completing the architecture within one TOGAF phase. The whole TOGAF cycle will have to be iterated again.

This rule, together with a consequent observation of DIP limits, will prevent the EA Kanban board from overflowing with piles and piles of cards. However, the Kanban approach is not a silver bullet that solves work overload problems. It simply makes that overload plainly visible and offers a structured approach to process the card piles by and by and to avoid them in the future.

A final note on the design of the EA Kanban board: We have reused the circular shapes of the TOGAF ADM visualization for the sake of better recognition. When Kanban is used as a tool in the daily EA business, it makes sense to hang the Kanban board physically on the wall and use sticky notes for task cards. In that case, it will be more practical and space-efficient to use a rectangular design like the one depicted in Figure 7-27.

The process we have sketched here applies to a collocated EA team that can have its regular stand-up scrum meetings in front of a physical board. For distributed teams, there are manifold commercial or open-source Web-based Kanban tools that can be used as a base for a distributed EA Kanban.