What Does It Mean to “Deliver” Software and IT Services?

A product is not possible without its customers. Not only are they expected to purchase and use it once it has been built, but they can also provide input and even inspiration during the design process. This applies equally to software products and IT services. Nowadays, engineers and IT professionals are not simply handed a hundred-page requirements document followed by many days of individual coding. They must instead define their products and services in full collaboration with their customers and business stakeholders.

However, these collaborations are not “ad hoc” interactions. They have to be planned, managed, and delivered to mutual satisfaction. This is achieved through product and project management, which covers end-to-end delivery (in software, this is referred to specifically as SDLC – software delivery life cycle) – from envisioning the products through execution, customer delivery, and ongoing support and maintenance. IT professionals must be full participants in this process. They engage in customer interactions, provide their input into features that are being delivered, use customer feedback to fine-tune their deliverables, and continuously improve both the products and the way they are being built.

Yet according to a 2019 Product Management survey from Gartner, only 55% of all product launches take place on schedule [1], and the Product Development and Management Association (PDMA) points out that the product failure rate in software and IT services is 39% [2]. It is up to future software engineers and IT professionals to improve this success rate and more effectively manage expectations with their stakeholders. The purpose of this book is to equip them with the means to do so.

First, some definitions: By “delivering software products and IT services,” we mean the ability of IT professionals to properly understand, interpret, and address customer needs by providing high-quality, adequately priced products and services. We also refer to this process as “delivering customer value.”

When we use the term “building products,” we refer to the features and functionality of the software that is being built or the service being provided. This also includes peripheral areas such as marketing and pricing the product, and the professionals who specialize in this particular area of delivery are called product managers. When we refer to how these products are being built or how the services are being provided, we call this project management. However, you do not need to be a product manager or project manager to take an active role in ensuring customer success. IT professionals too have a role to play in delivering solutions to customers.

The History of Project Management as a Profession

The first recorded project in the history of project management goes back to 2570 BC, when the Great Pyramid of Giza was completed. Ancient records show that there were managers for each of the four faces of the pyramid. Each of them covered planning, execution, and control in managing their part of the project. Similarly, there are records confirming that the project managers of the Great Wall of China, constructed over 2500 years, starting in the 7th century BC and completed in the mid-1800s, used soldiers, common people, and criminals and organized them by their deliverables to complete the project [5].

It was only in the 20th century that traditional project management became a profession and a formalized school of study. In the 1910s, Henry Laurence Gantt, an American mechanical engineer and management consultant, produced a concept of a graphic representation of tasks across time. His chart – now known as a Gantt chart – clearly illustrated a project’s individual activities and deliverables and allowed visual indications of how well the work was progressing, who was responsible for it, and when and where action would be necessary to keep the project on schedule (see Figure 1-1).

A significant leap forward in professional project management happened in the 1960s when a number of new techniques were introduced by industry leaders. These included the following:

1. 1.

   Critical Path Method (CPM): This technique, developed by the DuPont Corporation, was used to define a project’s duration by sequencing all of its activities and identifying which sequences had the least amount of scheduling flexibility. The intent was to keep this sequence of activities (referred to as “the critical path”) on track. This technique allowed DuPont to manage the process of shutting down its many large chemical plants for maintenance and restarting them on time once the maintenance was complete. This technique allowed the company to save millions of dollars per shutdown project, and the Critical Path Method contributed enormously to many other large-scale infrastructure projects in the United States and elsewhere during the postwar decades and continuing to the present day.

    
2. 2.

   Work Breakdown Structure (WBS): See Figure 1-2. This approach was used by the US Department of Defense for the development of the US Navy’s Polaris nuclear missile project in the 1950s and 1960s. Polaris was a submarine-based weapon that helped usher in the Cold War. In an era before computers, WBS represented a deliverable-oriented breakdown of a project into smaller components – usually as small and as granular as possible. The Work Breakdown Structure remains a key project component since it organizes teams’ assigned work into manageable and measurable sections.

    

Throughout the decades of the 20th century, project management allowed for significant savings and greater efficiency and was highly regarded by major industry players. In the 1960s, project management was launched as an official profession, with responsibilities for organizing, planning, executing, and delivering products and services. In 1969, the Project Management Institute (PMI), a nonprofit professional organization dedicated to advancing the practice, science, and profession of project management, was launched in the United States.

Almost immediately, project management practices were applied to the emerging software engineering profession, and industry-agnostic standards were developed. In the 1980s, the PMI created a guidebook known as the Project Management Body of Knowledge (PMBOK). In Europe, a parallel project management approach was developed called the PRINCE Methodology, and this became the standard for all government and information system projects in the UK from 1989. In essence, the difference between the two is that PMI focuses on tasks (activities) first while PRINCE focuses on roles (people). However, both adhere to a discipline of effective and thorough planning and communication to ensure on-time, on-budget, and on-scope delivery. These naturally became important concepts for IT professionals in all specializations.

The seminal book on software engineering and project management, The Mythical Man-Month: Essays on Software Engineering by Fred Brooks (1), discussed the insights for managing complex software projects based on the importance of meeting timelines in software delivery. The book presented thought-provoking concepts, which are well understood nowadays, such as the notion that adding people to a delayed project only makes it slower. This statement is now known as the Brooks’ law.

As Brooks put it, “while it takes one woman nine months to make a baby, nine women can’t make a baby in one month.” This was one of the first attempts to show that IT project management is in no way similar to many other industries that require repeatable work delivered with minimum variation and maximum efficiency. Software project management is an art as much as a science and needs to be managed with a full understanding of its unique complexities and challenges.

Lean

In parallel to project management, the concept called Lean manufacturing was becoming increasingly popular in heavy industry, and it too was to become a staple of IT and influential in the development of SDLC, DevOps, and continuous testing. Lean manufacturing is based on rigorous process thinking in manufacturing, and as such, it has its roots in the construction of the Arsenal in Venice in the 1450s, where the first highly efficient production line was implemented for shipbuilding. The Arsenal employed over 1,600 people, and the production was divided into three main stages: (a) framing, (b) planking and cabins, and (c) final assembly. The Arsenal used standardized, interchangeable parts, streamlined the assembly line, and enabled minimal handling of materials during each stage of production.

In the early 20th century, Henry Ford implemented a similar streamlined and highly efficient process in car building. He modernized the car assembly line and grouped assembly activities by steps while making an inventory of parts available by each process group. This process of increasing efficiency and minimizing “waste” during production was adopted by Toyota manufacturing and allowed for the efficient mass production of high-quality cars after World War II (2). The Toyota Production System was brought to the attention of the global community by American writer and management consultant Peter Drucker. It included five important concepts (see Figure 1-3):

1. 1.

   Define value: Value is defined by the customer’s needs for a specific product. All activities that do not add value are defined as “waste” and should be avoided.

    
2. 2.

   Map value stream: A value stream is a set of processes involved in taking a specific product from raw materials to customer delivery. The process of identifying the value stream, referred to as Value Stream Mapping (VSM), is an important first step in the approach to any delivery or optimization effort.

    
3. 3.

   Optimize flow: The flow defines the sequence of processes and their efficiency in order to ensure that the work being done to deliver the product or service will flow smoothly from one step to another without interruptions or delays. As a result, time to market improves, and efficiency increases. This is a simple but extremely hard-to-implement concept. In his novel about transformative management style using the concept of flow and the flow-based theory of constraints, Eliyahu M. Goldratt (3) tells a story of a plant manager who is using the flow-based theory of constraints to improve performance and save the plant in 90 days, or it will be closed by corporate headquarters with hundreds of jobs lost. The theory of constraints is a methodology that takes a scientific approach to improvement. It is based on identifying the most important limiting factor (i.e., constraint, referred to as a “bottleneck” in manufacturing) that stands in the way of achieving a goal and then systematically improving that constraint (the “weakest link in the chain”) until it is no longer the limiting factor. This approach is based on a hypothesis that every complex system consists of multiple linked activities, one of which (the bottleneck) acts as a constraint for the entire system.

    
4. 4.

   Establish pull: There are multiple manifestations of “waste” in the delivery life cycle, and overproduction is a major root of inefficiency. Work is pulled step by step based on capacity rather than pushed to the next step in the value stream upon completion of the prior step. This ensures that spare parts, as well as manufactured items, are “pulled” by the next step in the value chain by the customer for the finished product when they need those. This is known as “just-in-time” manufacturing or delivery vs. creating expensive inventory that needs to be managed.

    
5. 5.

   Pursue perfection: Despite all the significant gains that Lean thinking brought to manufacturing, it is important to understand that Lean is not a static system. The concept of continuous improvement was at the center of Lean. Continuous reflection and retrospection are the foundation of Lean thinking. There is a well-known quote from one of Toyota’s top managers who was asked why they are keeping their production facilities open for tours and whether they are concerned that their competitors will copy their practices. The response was that the competitors would copy today’s practices, and tomorrow, Toyota will be already doing things differently. The concepts of personal accountability, respect for people, lightweight leadership, continuous improvement, and delivery of value to customers made Lean more than a project management method. It became a mindset.

    

As a result of TPS adoption, by 1986, productivity had doubled, and absenteeism was down from 25% to 3%.

 Topic for Group Discussion

Split into five groups and have each group discuss one of the concepts described before. How does it apply to IT services and software delivery? Does it resonate with you? Can you think of any real-world examples?

The concept of optimizing software development and IT service delivery over the entire value stream became the foundation of Lean software delivery. Companies applied classic Lean tools such as value stream mapping and flow-based tools to increase flow efficiency. Specifically, these included the theory of constraints (TOC), building quality “in” (test automation) to increase the efficiency of software delivery and IT services. Even more importantly, Mary and Tom Poppendieck focused on the importance of building the right products vs. building the product right. This distinction allowed for the emergence of a profession that was parallel to project management, called product management. While the project manager was responsible for “how” to deliver the project, the product manager was responsible for “what” needs to be built.

 Topic for a Group Discussion

How do you distinguish between building the right product and building the product right? Think of specific examples: defining budget for a new project, deciding on the features that address customer needs, automating testing to improve the quality of the software, moving systems to the cloud to improve reliability – which ones are examples of building the right product and which ones are examples of building the product right? Why are both concepts important? Which one do you find more important and why?

Agile

The advancement of traditional project management and the power of Lean thinking made the emergence of Agile software delivery only a matter of time. Like many great ideas, Agile was born out of frustration. In February 2001, 17 software practitioners met at the Snowbird ski resort in Utah. They were all thought leaders in software delivery. They were also all frustrated about the inefficiency of software delivery management.

In the 1990s, software projects were managed the same way as construction projects, which is to say sequentially. For example, to build a house, you first need to create and finalize designs and blueprints, prepare the construction site, pour the foundation and complete framing, then install plumbing and electrical and complete drywall, install interior trim, and so forth. You cannot start plumbing until your foundation is complete. The same project management principles were applied to software delivery – phase by phase. It took weeks to write requirements. After those were signed off, the software design phase started, which took several more weeks. Then, the development would start. It would go on for a few months, followed by testing.

During testing, bugs were found, which would require a significant amount of rework by developers and sometimes would even necessitate a redesign. When testing was completed, users would get engaged in user acceptance testing, and frequently, they would say “this is not what we envisioned” or “this is what we wanted a year ago when we started this project, but it is no longer valid,” and the whole project would have to start from scratch or just get canceled.

The 17 people in Utah challenged the most fundamental concept in this traditional, so-called “Waterfall” delivery method. Their reasoning was that in software delivery, it should take only hours, if not minutes, to alter code, and if the users were able to review the result right away, the amount of rework would significantly decrease. Software delivery is different from house construction, so why should construction rules apply to software delivery?

Meanwhile, an average project was measured in years rather than months or weeks. In some industries, such as aerospace and defense, sometimes, it took 10, 15, or even 20 years to develop a complex system before it went into production. For example, the Space Shuttle program launched in 1982 used information and processing technologies from the 1960s.

While the Waterfall model was widely adopted, alternatives were already emerging. The 17 signatories of the Agile Manifesto had already been experimenting with different frameworks that are now united under the Agile umbrella. Jeff Sutherland and Ken Schwaber invented the Scrum process in the early 1990s. The term came from the British game of rugby and referred to a team working toward a common goal. Kent Beck developed extreme programming during his work at Chrysler and published his findings in 1999.

The group of 17 practitioners did not challenge tools or techniques; they came together to solve a fundamental problem, which had a root cause in obsolete thinking. To do so, they suggested a new set of values and principles, which they called the Agile Manifesto.

They spoke about valuing individuals and interactions over processes and tools and valuing working software over comprehensive documentation. For example, why should negotiation over hundreds of pages of a requirements specification take 30–50% of project delivery time if it would just take hours to build the feature, show it to the customer, and receive direct and immediate feedback? Why would it take weeks to approve project plans, which became obsolete the moment a new requirement came in, which happened in 89–93% of all software projects? As intuitive as it seems now, this was a major breakthrough in thinking about software delivery.

They also addressed major challenges of the Waterfall software delivery, such as inconsistent utilization. In Waterfall, people were allocated to projects rather than working together continuously as a long-standing team. Allocation usually happened at the beginning of the project, meaning that testers sat idle for weeks until the coding was completed. Even worse, capacity management was done by project, so people were working on multiple projects at a time, lacking predictable work management and losing time at context switching. They were overloaded toward the end of the project while not being utilized at the beginning of it.

Nowadays, Agile is no longer specific to software delivery or IT management. Customer-centricity, self-organizing teams, and iterative delivery with a short feedback loop are equally essential in delivering value to any customer in any industry. Agile has been adopted by airlines, financial services companies, human capital divisions, leadership teams of major companies, and many others. When we talk about Agile, we think about iterative delivery of value to customers; we think about the collaboration between business, engineering, and the customer; we think about a short feedback loop to learn directly from the customer and to continuously improve the products that we build or the services we deliver. We think about teamwork based on optimizing end-to-end delivery flow while empowering people and teams. It is important to realize that Agile is not just project management – it’s a different mindset. Agile brings in a mindset of collaboration, value delivery, and continuous improvement via short feedback loops.

From this point of view, Agile is being used as an “umbrella” term for a number of different frameworks used in software delivery and IT management. (See Figure 1-4.) Some of these frameworks are listed as follows:

1. 1.

   Scrum: The most popular Agile framework. It is designed for small cross-functional software delivery teams who break their work into goals, which can be completed within timeboxed iterations. These one-to-four-week iterations are called Sprints. As mentioned earlier, Scrum takes its name from the sport of rugby, emphasizing the team-based nature of work performed.

    
2. 2.

   Extreme programming (XP): An Agile software development framework that has the goal of delivering high-quality software with high responsiveness to changing customer requirements. XP is focused on engineering practices for software development. The term “extreme programming” is based on the concept of taking the most beneficial aspects of traditional software practices, such as code review and refactoring, to extreme levels. For example, code refactoring is a beneficial software development practice, and when taken to the extreme, code is continuously being refactored. Peer reviews are beneficial practices, and when taken to the extreme, it results in the practice of “pair programming,” which is a situation in which two developers collaborate in code delivery.

    
3. 3.

   Kanban – literally translated from Japanese as “visual board” – is a Lean method of highly visual workflow management. Most teams practicing Lean use Kanban to visualize and manage their work. Kanban promotes continuous collaboration and encourages ongoing learning and improvement through workflow optimization.

    
4. 4.

   OSAM (Optum Scaled Agile Model), DSDM (Dynamic Systems Development Method), FDD (Feature-Driven Development), Crystal, and RUP (Rational Unified Process) are proprietary Agile frameworks focusing on different aspects of Agile software delivery with a lower rate of adoption.

    
5. 5.

   There are a number of competing Agile frameworks related to the adoption of Agile within large organizations with multiple software delivery teams, frequently with hundreds of people working on the same product. These frameworks include SAFe (Scaled Agile Framework), LeSS (Large-Scale Scrum), DAD (Disciplined Agile Delivery), and Scrum@Scale. We will discuss these in Chapter 10.

    

 Tip

In the traditional project management, a project is defined as a temporary endeavor undertaken to create a unique product, service, or result. A project is temporary in that it has a defined beginning and end in time and therefore defined scope and resources. Contrary to the traditional project management, in the Agile framework, we are talking about a product or a service. Products have their own life cycle from envisioning to sunsetting, but there are no predefined timelines since this is determined by customer demand.

While Lean works traditionally well in manufacturing where similar products are being delivered (e.g., car manufacturing), it was a common opinion that Waterfall is best for large projects such as airplane assembly and software delivery, while Agile works best for smaller, less risky projects. However, this opinion has been recently invalidated. Companies like Boeing, SpaceX, and Tesla use Agile successfully for complex projects with elevated risk levels, such as space shuttles, cars, and airplanes. Nokia Network adopted Large-Scale Scrum (LeSS), while UnitedHealth Group and developed its own Optum Scaled Agile Model (OSAM) based on the Scaled Agile Framework (SAFe). Other examples of successful SAFe adoption include Porshe, CVS, Deutsche Telekom, Kaiser Permanente, and US Airforce [6].

With that, let’s start the journey, but just prior to that, let’s review the topics we covered.

Key Points