1.1 Evolution of Software Development Processes

The software engineering literature has stressed the importance of software development processes and their influence on product quality for decades [2,4]. Consequently, efforts for improving software development processes have made different contributions over time. Among the landscape of contributions to software processes, we can find process models such as the Waterfall model [3] or the Spiral model [5], process standards, which are widely used in the industry, such as the ISO/IEC 12207:2008 [6], ISO/IEC 90003:2004(E) [7], model-based improvement frameworks such as CMMI [8], SPICE (ISO/IEC 15504:1998) [9], and Bootstrap [10], and, more recently, adaptive approaches for software development such as Agile and Lean methods [11]. Fig. 1 depicts a timeline with some of these important contributions to software development processes.^b It bases on the review of process models conducted by Münch et al. [2].

Software development processes have gradually evolved from heavyweight, stage-based processes, which built on the assumption that requirements are relatively stable and split the development process on distinct sequential phases based on those requirements (such as the Waterfall model [3]), to lighter/more flexible processes that try to cope with current business turbulence. As illustrated in Fig. 1, process-centric models, which were considered deterministic and repeatable [18], have given way to flexible approaches, which emphasize continuous evolution of software functionality, flexibility, speed, face-to-face communication, customer involvement, and a minimal number of lightweight artifacts.

1.2 The Emergence of Agile Software Development

The main driver behind the evolution of software processes toward Agile and Lean software development is the environment in which the software development industry works today, which is fundamentally different from the environment in which it used to work 20 years ago. On the one hand, the prevalence of software products has shaped a software-driven society. Software is all around us, from the (electronic) toothbrush that we use in the morning, to the alarm that we set-up before going to sleep. Thus, the business around software products is open to countless opportunities. On the other hand, the Age of Information,^c which is strongly based on the Internet, has made digital resources constantly available for everyone. Information flows are accelerated, global communication and networking are faster and individuals can explore their personal needs easier. Overall, software relevance, globalization, and market dynamics, characterized by rapid and unpredictable change, have shifted the software development landscape. These market features put pressure on software and software-intensive organizations to develop what Eisenhardt and Martin [20] called dynamic capabilities. Software development organizations need to be creative, innovative, and flexible, while working with incomplete information and pursuing economic efficiency. This situation is particularly relevant for organizations in domains such as web or mobile applications and services. In these contexts, applications are frequently developed in a matter of weeks or months, rather than years, as it was common when using traditional processes [1]. Yet, the emergence of trends like internet of things (IoT) will shape software development processes of more traditional industries in the near future as well.

To cope with this situation, a community of practitioners formulated the Agile Manifesto in 2001 [11]. In a 2 days meeting in the Wasatch mountains of Utah in February 2001,^d 14 independent software practitioners proposed the Agile Manifesto as an alternative to software processes that were considered too inflexible and slow to adapt to business changes. It provoked a change in the way in which the software engineering community was addressing software development processes. In essence, Agile recognizes the limitations of anticipating external business changes and provides means oriented to implement minimum viable products (MVPs) quickly as a key competitive advantage [21]. The idea is that MVPs are incrementally developed in collaboration with customers, as more information about the product is known. Contrary to traditional phase-based software development processes, Agile appraises iterative development. Traditional software development phases (e.g., requirements, development, and testing) are run in small cycles to achieve continuous delivery and feedback and, therefore, improve the company's capability to adapt to market and customer fluctuations [21]. Furthermore, Agile emphasizes the human side of the sociotechnical software development activity as a primary driver of project success [22,23]. Agile methods advocate that the key role that people plays in software development makes the development process less deterministic and repeatable than previously considered by traditional prescriptive methods [18]. Kruchten [24] explained it using the party metaphor: “Consider you organize a party this Saturday at your home. You invite a group of people, order the food, move the furniture, and you record carefully: who comes, when, who eats what, drinks what, etc., and everyone leaves having had a good time. The next Saturday, to have another great time, you decide to have the very same people, the very same food and drinks, you ask them to dress the same way, arrive at the same time, you introduce them to the same people, bring them the same glasses with the same drinks, make the same jokes,... will you be guaranteed to have a nice party again...?” In the same line, Boehm stressed that “one of the most significant contributions of the agile methods community has been to put to rest the mistaken belief that there could be a one-size-fits-all software process by which all software systems could be developed” [2]. The bad news is that, unfortunately, there is no silver bullet for software development processes. That is, there is no definitive answer to the question: what is the best software development process I should apply? However, the good news is that Agile and Lean software development offer means to cope with current software business dynamics, which can be adapted to different contexts.

Regarding to the adoption level of Agile and Lean software processes in the software development industry, what was initially considered as a fad has been progressively consolidated. Nowadays, a great extent of the software industry follows these methods. For example, the latest State of Agile Survey by VersionOne [25] found that 43% of its 3880 respondents worked in software organizations with more than 50% of their teams using Agile. Only 4% of respondents indicated that in their organization no team was using Agile. Similarly, a survey study conducted in Finland in 2012 concluded that the Finnish software industry has moved toward these methods, being the 58% of the survey's respondents working in organizational units using Agile software development [26]. Nevertheless, it is also worth mentioning that the adoption of Agile has not always been a smooth process. Indeed, Agile software development caused some initial controversy among those that saw it as an attempt to undermine the software engineering discipline [27]. Particularly, there were some initial concerns about using it as an easy excuse to avoid good engineering practices and just code up whatever comes next [27–30]. However, as we will see in the rest of the chapter, and particularly in Sections 4 and 5, Agile software development is a much more disciplined approach than it may seem at first sight.

1.3 Lean Thinking and Continuous Deployment (CD)

A few years after the formulation of the Agile Manifesto, Lean thinking (or simply Lean) started to become also the center of attention of the software development industry [15]. However, Lean did not appear as a software development approach of its own until 2003, when Poppendieck and Poppendieck introduced it in their book Lean Software Development: An Agile Toolkit [15]. One of the reasons that captured the attention of the software community upon Lean thinking was the focus of Lean on the organization as a whole [15,31–33]. Agile software development was mainly conceived to work for small software development teams [34]. However, scaling Agile methods, such as Scrum [35] or eXtreme programming (XP) [36], to operate a whole organization was not straightforward; Agile does not explicitly provide support in that sense [32,37–39]. Thus, Lean thinking was discerned as a promising solution to be combined with Agile methods. Nowadays, Lean thinking is not only applied as a way of scaling Agile, but also as well as a way of improving software development processes from a wider perspective [40].

The interest on Lean thinking, which originally emerged in the automotive industry [41], is not exclusive to the software development industry. Lean has been widely adopted in different domains, from the healthcare domain to the clothing industry. Table 1 shows some examples of benefits that are attributed to Lean in diverse domains, in terms of profitability, productivity, time-to-market, and product quality.

During last years, a vogue for rapid and continuous software engineering is emerging [46]. Rapid and continuous software engineering bases on many ideas from Agile and Lean software processes. Concretely, it refers to the organizational capability to develop, release, and learn from software in rapid parallel cycles, often as short as hours or days [46–48]. Speed is the key aspect in this trend. The aim is to minimize the step between development and deployment so that new code is deployed to production environment for customers to use as soon as it is ready. CD is the term used to refer to this phenomenon [16,49–52]. Although the concept of deploying software to customers as soon as new code is developed is not new and it is based on Agile software development principles, CD extends Agile software development by moving from cyclic to continuous value delivery. This evolution requires not only Agile processes at the team level but also integration of the complete R&D organization, parallelization, and automation of processes that are sequential in the value chain and constant customer feedback [46]. Rapid releases, continuous delivery, and CD will be widely discussed in Section 5.

2.1 Overview of Agile Literature

First, we explore the research topics in Agile software development over the past 2 decades. We divided the articles into four roughly equally large groups based on the year of publication and plotted a word comparison cloud of their titles. Fig. 2 shows the word comparison cloud. Early years (2000–07) are in black, second period (2008–11) is in gray, third period (2012–14) is in red, and the most recent era (2015–17) is in blue. The words in each quadrat are more common in their given periods than in other periods, i.e., the size of each word is determined by its deviation from the overall occurrence average. The figure was plotted with R Package “wordcloud.”

Although the early years contain several articles about Agile software development in general, e.g., Refs. [28,53,54], the Agile software development method that was the most investigated early was XP [36]. XP is an Agile method with the most doctrinaire approach by setting detailed development practices such as test-driven development and pair programming. The early success of XP as well as declining interest toward it later, may well be explained by its mandating approach to a set of development practices. Adopting it at first can be straightforward as it gives clear rules such that all the development code is done by pair programming. However, later people may realize that such ultimatums might not make sense in their context and, thus, they would turn to more flexible Agile methods. For example, for pair programming research shows that having two junior programmers can form a cost-effective pair while the same does not hold for two expert programmers [55]. During the early years, many papers about Agile manufacturing appeared [56] which addressed the manufacturing of physical products. Work in this area also influenced Agile software development as we shall explain in the next section.

In the second period (2008–11) research about Agile software development in the technological contexts of the web and service-oriented architectures increased in numbers. Thus, the scientists working on technical advances also connected their works to software process advances in Agile software development, e.g., Ref. [57]. During that time, researchers connected Agile software development to work on global and distributed software development, which addresses issues of developing the same software in multiple sites with diverse locations around the globe, e.g., Ref. [58]. You should also notice that papers connecting Agile to global and distributed software development were the first research steps toward scaling Agile beyond one colocated team. Research efforts in Agile software development increased also in the areas of process simulations, e.g., Ref. [59], linking business to Agile, e.g., Ref. [60], and in customer and other type of collaborations, e.g., Ref. [61].

In the third period (2012–14), the technical advances of cloud computing [62] and model-driven development [63] were linked with the process advances of Agile. The term “Lean” gained popularity as a way to advance Agile even further [64]. Learning and teaching Agile was studied in universities [65] but articles about learning Agile also include papers about self-reflection in Agile teams [66].

In the most recent period (2015–17), we can see the rise of word continuous, which increases in popularity due to the practices of continuous integration, CD, and continuous delivery [46,67]. These practices highlight the importance of running working software, which is integrated and delivered to clients continuously, rather than providing major updates biannually, for example. We also notice increase in popularity of Kanban that is a method for managing work and tasks in Agile software development. Systematic literature reviews (SLRs) about Agile increased in popularity in the most recent period as well. SLRs offer an overview of particular areas of Agile software development like using software metrics in Agile [68] or technical debt [69].

2.2 Trends in Agile

The previous section offered a general overview on Agile software development over the past 2 decades. For this section, we did individual searches on topics presented in previous section but also on engineering topics we knew that had been introduced with Agile software development such as refactoring. Thus, we searched for the keywords shown in Fig. 3 combined with the word “software”. Adding word software was necessary to exclude, for example, papers discussing Scrum in context of Rugby instead of Agile software development. Research in software engineering in general has increased in past 2 decades [70], so we normalized all search results by the number of papers found with the general terms of “software development” or “software engineering”. We also normalized all trend graphs to have values between 0 and 1 where max is the year with the highest normalized amount of papers.

The concepts of XP, Scrum, Lean, Kanban, DevOps, and continuous integration/deployment/delivery are shown in Fig. 3A–F. We can see that XP has been declining in research interest since 2004, Fig. 3A. Scrum, Lean, and Kanban have had good popularity since 2009 but they still appear to be gaining popularity, although the growth trends is not as steep as it has been in the past. DevOps and continuous integration/deployment/delivery seem to be gaining popularity fast and can be characterized as the hot research topics in Agile software development. These topics will be developed in Sections 5 and 6.

Fig. 3G–I shows the popularity of the concepts of technical debt and code smells that are used to describe suboptimal software engineering solutions which hinder the velocity of Agile teams. We can see that the research in both areas is still increasing. The concept of technical debt appears to be more recent than code smells. The concept of technical debt is also larger than code smells. Code smells refer to code problems only, while technical debt can refer to problems with using an old version of the build server, for example. Refactoring is about code modifications to fix code smells or technical debt or due to the need for other behavior persevering changes. The popularity of refactoring research has been stable since 2009. The concept of technical debt in the context of Agile will be developed in Section 8.

Finally, Fig. 3J–L shows the popularity of three Agile software development practices that were popularized by XP that are pair programming, test-driven development, and user stories. We can see that although the popularity of XP has declined in research literature, the popularity of pair programming has remained relatively stable since 2003, and the research interest in TDD has similarly remained stable since 2007. Research on user stories that could also be titled as Agile requirements engineering is still experiencing fast growth. So, even when XP might become extinct, some of its practices are still sparking wide interest.

2.3 Summary

This section has demonstrated popularity of many Agile software development concepts and practices in the research literature. Next sections will go deeper to the foundations of Agile and help the reader in putting these concepts together as all of these concepts are essentially solutions of the business and engineering problems companies are having while developing software-based products, services, and solutions.

3.1 Lean Thinking

Lean was born as part of the industrial renaissance in Japan after the Second World War in the 1940s. Traditional mass production systems, which are based on producing large batches of products through continuous assembly lines of interchangeable parts, were unsuitable in a Japanese market lashed by the war and characterized by low volumes. Thus, Japanese manufacturers found themselves surrounded by obstacles that forced them to develop new methods. Japanese new production methods led to what we know today as Lean or Lean thinking. However, it is interesting to notice that the term Lean was not used by Japanese manufacturers. Lean was first used by researchers from the Massachusetts Institute of Technology (MIT) when they began research under the International Motor Vehicle Program (IMVP) in 1986.^e With the aim of investigating the differences across worldwide automotive industries, IMVP researchers discovered that the Japanese auto industry was far ahead when compared to the automotive industry in America. By carefully studying Japanese methods, particularly those of Toyota under its Toyota production system (TPS), they conceived an entirely different production system, which they called Lean manufacturing [41]. Thus, Lean or Lean thinking is the English term that western researchers used to describe Japanese original production methods.

According to MIT's researchers, Lean is about “doing more with less” by ideally producing “the right things, at the right time, and in the right place” [41]. Based on fundamental industrial engineering principles, Lean thinking, as interpreted by MIT's researchers, steeps in a philosophy of maximizing value and minimizing waste. Five principles guide this philosophy [71]: value, value stream, flow, pull, and perfection.

• • Value is everything that a customer is willing to pay for. Thus, value is understood as customer value. Defining and understanding value from the perspective of the customer is the central focus of Lean thinking. Lean thinking stresses that every activity inside a company should contribute to create customer value. The goal is to make organizations deliver as much customer value as possible. Its counterpart, waste, or muda in Japanese, is everything that consumes resources but creates no customer value.
• • Value stream is the optimized end-to-end collection of actions required to bring a product/service from customer order to customer care. The activities in the value stream are classified in three groups: (1) activities that create value and, therefore, are legitimately part of the value stream; (2) activities that create no value for customers but are currently needed to manufacture the product. These activities remain in the value stream as, otherwise, the product cannot be manufactured, but the aim is to improve the production process so that they can be removed at some point; and (3) activities that clearly create no value for customers and should be immediately removed from the value stream.
• • Flow means that production activities should be organized as a continuous flow, eliminating discontinuities. Flow requires that unnecessary steps and inventories are eliminated (waste). Opposite to mass production, in which interchangeable parts are continuously produced to assemble final products, in Lean thinking, products are made using single piece flows. In a single piece flow, operations are efficient in a way that batch sizes and inventories are replaced by working on one product at a time.
• • Pull focuses on producing products only when they are really needed. Accordingly, in a pull system, an upstream process produces only when a downstream process is ready and “pulls” some more work from the upstream process. This minimizes inventories. In traditional push systems, work is produced in response to a preset schedule, regardless of whether the next process needs it at that time. Accordingly, the process is transformed from batch-and-queue to just-in-time (JIT) production.
• • Perfection, or Kaizen in Japanese, refers to the enterprise-level improvement and learning cycle. Lean aims to achieve zero waste and defects based on the concept that there is no end in the strive for perfection and learning.

MIT's five principles of Lean are probably the most popular interpretation of Lean thinking. Some voices have questioned the validity of IMPV's studies by pointing out to a certain biased interpretation of the original Japanese methods [72]. For example, Toyota and its TPS, which had a great influence on Lean thinking as interpreted by MIT's researchers, was originally called the Respect for Humanity System. The respect for humanity system was based on two main pillars: continuous improvement and respect for people [73]. Thus, value and waste were not originally the main focuses of the paradigm. When later Lean was interpreted from a western point of view, influenced by the existing western's social–economical system, value, and waste became key components of Lean thinking. Nevertheless, biased or not, it is indisputable that MIT's studies, and the five principles outcome of these studies, have vastly influenced the shape of Lean thinking as it is understood nowadays. The book The Machine That Changed the World^f [41] provides a more detailed description of the story of Lean.

Besides MIT's researchers, it is important to know that other authors have also provided slightly different interpretations of Lean thinking. Table 2 presents most popular interpretations of Lean thinking. It can be said that Lean has incrementally evolved over time [41,71,74–76]. As presented in Table 2, Onho [74] described the TPS mainly from a perspective of reducing wastes such as overproduction, waiting, transportation, overprocessing, inventory, movement, and defects. Although Onho emphasized the concepts of JIT and autonomation, many authors have taken waste as the key element of Lean thinking. For example, Cusumano referred to Lean as: “The authors [referring to the authors of The Machine That Changed the World] used the term ‘Lean’ to describe any efficient management practice that minimizes waste” [77]. In a similar vein, Shah and Ward defined Lean production as “an integrated socio-technical system whose main objective is to eliminate waste by concurrently reducing or minimizing supplier, customer, and internal variability” [78]. The excessive focus on waste has frequently created certain unfair reputation to Lean thinking among those who have seen it as an easy excuse to get rid of employees and resources. However, although the concept of waste is undoubtedly important in Lean, the 5, 14, and 13 principles of Lean thinking by Womack and Jones [71], Liker [75], and Morgan and Liker [76] respectively (see Table 2), clearly show that Lean thinking is more than only removing waste. As we will see in Section 4, the interpretation of Lean thinking in software development has been mainly taken from an Agile perspective, as Lean software development was largely built upon the values and principles formulated in the Agile Manifesto. Thus, although waste is an important concept in a software development context as well, Lean software development goes way beyond waste reduction.

3.2 Lean Manufacturing Toolkit

Lean thinking has no formal practices, but builds on any practice that supports its principles. Still, a toolkit of recommended practices, tools, and techniques, from which to choose from, have emerged over time to implement the fundamentals in practice. Table 3 summarizes those that are relevant from a software development perspective, as they have been mentioned in the context of Lean software development (and will appear later on in the next sections). For a more thorough listing of Lean terms, with examples and illustrations, the reader is referred to The Lean Lexicon created by The Lean Enterprise Institute [79].

3.3 Agility

Similar to Lean thinking, the agility movement does not has its origins in software development but in a manufacturing context. However, agility in manufacturing emerged well after Lean thinking was already in use. Concretely, it appeared in manufacturing at the beginning of the 1990s, with the publication of a report by the Iacocca Institute [80]. This report defined agility as a solution to adjust and respond to change, and satisfy fluctuant demand through flexible production. Specifically, Nagel and Dove [80] defined agility as “a manufacturing system with extraordinary capability to meet the rapidly changing needs of the marketplace, a system that can shift quickly among product modes or between product lines, ideally in real-time response to customer demand.” Christopher and Towill [81] describes how “the origins of agility as a business concept lie in flexible manufacturing systems, characterizing it as a business-wide capability that embraces organizational structures, information systems, logistics, processes, and in particular mind set.”

Similar as well to the concept of Lean, agility seems to be “highly polymorphous and not amenable to simple definition” [82]. However, two aspects are usually stressed in the literature: adaptation and flexibility. The concept of adaptation is based on the contingency theories, which are classes of the behavioral theory [83]. These theories state that there is no universal way of managing an organization. The organizing style should be adapted to the situational environmental constraints in which the organization operates. Thus, organizations are seen as open systems that must interact with their environment to be successful. Successful organizations do not keep themselves in isolation but develop their adaptation capabilities. A close related term is organizational flexibility. Organizational flexibility is considered as “the organization's ability to adjust its internal structures and processes in response to changes in the environment” [84]. The aim of agility is to achieve an adaptive and flexible organization able to respond to changes and exploit them to take advantage.

The literature highlights also four capabilities that are particularly important to achieve an Agile organization [85]:

• • Responsiveness, as the ability to identify and respond fast to changes. It does not only mean identifying changes when they happen (being reactive to changes) but also proactively sensing, perceiving, and anticipating changes to get a competitive advantage.
• • Competency, which involves all capabilities needed to achieve productivity, efficiency, and effectiveness. Competency comprehends each area in an organization, from strategic vision, appropriate technology and knowledge, product quality, cost effectiveness, operations efficiency, internal and external cooperation, etc.
• • Flexibility, which refers to the organizational capability to make changes in products and to achieve different objectives using the same facilities. Flexibility involves from product characteristics and volumes to organizational and people flexibility.
• • Speed or quickness, which relates to making tasks and activities in the shortest possible time in order to reduce time-to-market.

To achieve these capabilities, Agile organizations are usually flat organizations with few levels of hierarchy, and informal and changing authority. Regulations with respect to job description, work schedules, and overall organizational policies are also loose, and trust is given to teams to perform their tasks. Teams are self-organized and job descriptions are redefined based on need [84]. A more comprehensible description of the main attributes of agility as it originally emerged in the manufacturing domain can be found in Ref. [84].

3.4 Combining Lean and Agile in Manufacturing

As in software development, the combination of Lean and Agile also occurred in manufacturing [86]. However, in a manufacturing context, the migration occurred from Lean and functional systems to Agile and customized production [81]. That is, Lean thinking was already in place when many organizations started to look for agility. Accordingly, the shift was characterized by a strong influence of Lean thinking, mainly understood as efficiency and waste reduction. In this context, Lean and Agile seemed to have some intertwined ideas [86]. Agile's focus on flexibility and capacity to embrace change could challenge Lean's focus on overall economic contribution. For example, achieving flexibility in a manufacturing context may request important investments in machines and storages. Marson-Jones et al.’s matrix [87] was created to guide the combination of Lean and Agile in a manufacturing context. It is based on the idea of market winners and market qualifiers. The combination is guided according to the primary company's aim as follows:

• – If the primary company's aim is on reducing cost (i.e., cost is the market winner and, therefore, dominates the paradigm as primer requirement), then Lean should be the primary strategy. In this case, aspects such as quality, lead time, and service level are highly desirable but are not strictly requested (market qualifiers).
• – However, if the goal of the company is to improve its service level, Agile should take priority. In this case, cost would be a market qualifier but not a market winner.

In a similar vein, the theory of Leagility in a manufacturing domain [86,88] says that while the combination of Agile and Lean might work well, since Lean capabilities can contribute to Agile performance, the combination has to be carefully planned to prevent risks that Agile's demands may cause on Lean capabilities [86]. Specifically, time and space restrictions are considered in the manufacturing domain, limiting the combination of Agile and Lean to three scenarios:

1. 1. Different value streams, that is, different products.
2. 2. The same product but at different points in time.
3. 3. Using both paradigms at different points in the value stream by using decoupling strategies (strategies to decouple Agile and Lean).

In other words, in manufacturing, there is a tendency to suggest that although Agile and Lean can be combined, they cannot be used at the same time in the same space. Fortunately, the software domain does not have the same restrictions. Indeed, although building upon similar principles, the own characteristics of software development have made the combination of Agile and Lean for software development quite different from its combination in manufacturing. Next section develops this combination in detail.

4.1 Agile Software Development

The Agile Manifesto [11] started the Agile software development movement. Similar to Lean thinking principles, the Agile Manifesto only establishes the fundamentals of ASD. Besides, a variety of methods have emerged to implement Agile's values and principles in practice. XP [36], Scrum [35], dynamic systems development method [93], crystal [94], or feature-driven development [95] are among the popular Agile methods. In this section, we will describe Scrum and XP as they are the most widely used in industry [26], having captured also a great attention in research (see Section 2). As we will see next, while XP provides concrete software development practices, Scrum could be mostly considered as a software management approach.

4.1.3 Scrum

Nowadays, Scrum is among the most, if not the most, used Agile method [26]. Scrum was first mentioned in 1986, as a new approach to product development in a study in Harvard Business Review by Takeuchi and Nonaka [99]. Thus, the ideas behind Scrum did not originate in the software domain either, but were adapted to it. Scrum uses the rugby metaphor of players packing closely together with their heads down and attempting to gain possession of the ball (see Fig. 6) to refer to a holistic product development strategy where a development team works as a unit to reach a common goal. In Scrum, a common team goal is defined and shared among all team members, and all team members do their best to reach that common goal (e.g., gaining possession of the ball in rugby). The team is self-organized and can decide the best means to achieve the goal. Thus, the team leader does not dictate what to do next, but uses a facilitative style to help the team achieve its goal. Ideally, the team is physically colocated (reminding the concept of cell in Lean thinking), or uses close online collaboration, to facilitate communication and create a team spirit.

Scrum started to be used for software development in the early 1990s. The origins of Scrum for software development is a topic of frequent debate. Some mistakenly believe that Jeff Sutherland, John Scumniotales, and Jeff McKenna invented Scrum in 1993. However, Sutherland himself has acknowledged that their first team using Scrum at Easel Corporation took Takeuchi and Nonaka's work [99] as their inspiring source [100]. Although the team was already using an iterative and incremental approach to building software, “the idea of building a self-empowered team where everyone had the global view of the product on a daily basis” was motivated by Takeuchi and Nonaka's work [100]. Sutherland's team^h started to develop what we know today as Scrum in software development. In 1995, Sutherland introduced the Scrum team to Ken Schwaber, CEO of Advanced Development Methods, who started to use Scrum at his own company as well. Sutherland and Schwaber worked together to formalize the Scrum development process, which was first presented in a paper describing the Scrum methodology at the Business Object Design and Implementation Workshop, held as part of Object-Oriented Programming, Systems, Languages & Applications ‘95 (OOPSLA ‘95). Sutherland and Schwaber, as 2 of the 17 initial signatories of the Agile Manifesto, incorporated the ideas behind Scrum into it.

Scrum is a quite simple process framework. Rather than focusing on technical software development practices, Scrum is a method for managing the software development process. It focuses on how a software development team should organize its work to produce software in a flexible way, rather than on the technical practices that the team should apply. Fig. 7 depicts the Scrum process framework. It is composed by three main elements: the scrum team (in the middle of the figure), scrum events (in red), and scrum artifacts (in blue).

• • The product owner, the development team, and the scrum master compose the scrum team. The product owner is the responsible person for maximizing the value of the product and the work of the development team. It reminds to the Chief Engineer in Lean thinking. The development team is composed by the professionals who do the work in order to deliver a potentially releasable product increment at the end of each sprint. The scrum master is responsible for making sure that the development work adheres to the Scrum rules (roles, events, and artifacts).
• • Scrum considers five well-scheduled events: the sprint, the sprint planning, the daily Scrum meeting, the sprint review, and the sprint retrospective. A sprint is each of the increments in which the software development process is divided. The sprint planning is the meeting held at the beginning of each sprint to agree on the work that will be developed in the following sprint. During the sprint, the Scrum team meets daily during 15 min in the daily Scrum meeting to synchronize activities and create a plan for the next 24 h. At the end of the sprint, the work done during the sprint (potentially shippable increment) is inspected during the sprint review meeting. Finally, the sprint retrospective is a meeting for the Scrum team to inspect its way of working.
• • Three main artifacts are considered in Scrum: the product backlog, the sprint backlog, and the increment itself. The sprint backlog is an ordered list that contains all work to be done during the sprint (e.g., user stories, features, functions, requirements, enhancements, and fixes). The product backlog contains all the work to be done for the next product release. The increment is the sum of all product backlog items completed during a sprint (i.e., outcome of the sprint).

Next, we describe how all these pieces work together. The product is defined through product backlog items, which are usually known as user stories and features. Product backlog items are stored initially in the product backlog and incrementally transferred to the sprint backlog as the product development progresses through sprints. User stories and features describe the product's functionality from a customer perspective. The underlying strength of these backlog items is that they focus on the value that will be delivered to the user. User stories are often defined following the pattern:

• ‘As a < User >, I want to < Have > so that < Benefit >’

For example, a user story of a library system could be: “As a librarian I want to have facility of searching a book by different criteria so that I will save time to serve my customers.” In addition, each user story contains acceptance criteria that establish the criteria that the implementation of the user story must meet to be considered as implemented (as done in the Scrum's jargon). User stories contain also an estimation of how much it will take to implement the story, measured as estimated story points. The product owner is responsible of defining the user stories and the acceptance criteria for each user story. User stories are usually described at different level of detail. Particularly in large systems, where high-level product features are initially defined and, as the work progresses, elaborated in more detail through user stories [102]. User stories and features are incrementally created. Thus, it is not expected that a complete set of backlog items is created from the beginning, but the product backlog is an alive artifact that adapts to customer/business's changes. The product owner is the customer's voice to the team and, therefore, responsible for defining features/user stories and prioritizing them to maximize the value of the product. The product owner is also responsible to ensure that the product backlog is visible, transparent, and clear to everyone.

The development is carried out through increments or time boxes, called sprints (lasting between 2 and 4 weeks). Thus, products are developed iteratively and incrementally, maximizing opportunities for feedback and, therefore, increasing responsiveness and flexibility. Each sprint has a sprint goal that is defined during the sprint planning meeting, including the user stories that will be undertaken during that sprint. This is the common goal shared by everyone in the team (as in rugby). The Scrum team is self-organized and has the authority to decide on the necessary actions to complete the sprint goal, which is defined in cooperation with the product owner. Scrum teams are cross-functional, which means that the team has all skills and competences needed to meet the goal. Concerning the size of the team, scrum teams should be small enough to remain nimble but large enough to be able to complete significant work within a sprint. In practice, scrum teams are composed of between 5 and 10 members. Although there are also distributed scrum teams, the ideal case is that all team members are collocated in the same working area, as the Cell and Big-room concepts in Lean thinking.

The sprint backlog, selected from the product backlog, contains a prioritized set of user stories (or any other backlog item such as bug fixes) that will be implemented in that sprint. Once the sprint backlog is established during the sprint planning meeting, it remains unchangeable during the sprint. This means that no one is allowed to ask the development team (or any team's member) to work on a different set of requirements or tasks and, similarly, the development team is not allowed to act on what anyone else may request. After every sprint, two meetings are held. On the one hand, the sprint review meeting, to analyze the progress, and to demonstrate the current state of the product. The goal of this meeting is to show to all interested stakeholders, and particularly to customers/product owners, the potentially releasable increment that has been developed during the sprint. The key aspect in Scrum is that each sprint delivers customer value. On the other hand, a retrospective meeting is also held at the end of each sprint to reflect about the way of working. In particular, the purpose of the meeting is to reflect on how the last sprint went in order to identify (1) aspects that went well and, therefore, should be kept and (2) potential improvements that need corrections. The outcome of this meeting is an actionable plan for implementing improvements to the way the scrum team does its work. This mechanism allows self-reflection and continuous improvement (kaizen) in Scrum. In addition, every day the scrum team has a stand-up meeting of around 10–15 min, usually at the beginning of the day, in which each team member discusses what s/he did since the last stand-up meeting, whether s/he faced any obstacle in that task, and what s/he will do before the next stand-up meeting in the following day. The daily scrum meeting provides transparency on the scrum team daily work and reduces complexity. It allows the team to inspect its progress toward the sprint goal, reveals possible impediments, and promotes quick decision making and adaptation. Fig. 8 shows an example of a daily scrum meeting of a scrum team.

In this process, the scrum master ensures that the rules of Scrum are followed and is in charge to remove impediments in the work of the scrum team. For example, the scrum master can help the product owner by finding techniques for an effective management of the product backlog, or to the scrum team by facilitating scrum events. Therefore, the scrum master is a servant–leader for the scrum team rather than a manager.

As described earlier, Scrum is a fairly simple method. However, despite its simplicity, Scrum builds upon three important pillars: transparency, inspection, and adaption. Scrum provides transparency to the development process by making progress visible all the time through events such as daily scrum meetings and sprint reviews. Scrum allows frequent inspection to detect undesirable variances (e.g., changes in customers’ needs) and adapt to the new circumstances (e.g., the resulting product will not meet customer's expectations). In words of its founders, “Scrum implements the scientific method of empiricism [...] to deal with unpredictability and solving complex problems” [103]. Through small iterations, Scrum improves predictability and control risk by making decisions based on what is known.

Besides the main elements of Scrum described earlier, a number of practices have adhered to Scrum during time. Although, they are not part of the original Scrum framework, they have shown to be very useful as well in certain contexts. For example, Scrum of scrums (or meta-Scrum) are used to synchronize the work of several scrum teams when scaling Scrum from one team to several teams. Backlog grooming is a popular practice to keep the product backlog in shape by reviewing that the backlog contains appropriate items that are properly prioritized. The Agile glossary by the Agile alliance is a good resource to learn more about different terms in the context of Scrum and Agile software development in general (https://www.agilealliance.org/agile101/agile-glossary/).

4.2 Lean Software Development

Nowadays, Lean software development is certainly tied to Agile software development. However, few know that the interest of the software development industry on Lean thinking dates to early 1990s, well before the Agile Manifesto was formulated. Next, we analyze the path followed by Lean thinking within software development before and after the Agile Manifesto, which constituted a breach in the conception of Lean thinking in software development. Then, we focus on Kanban, since it is the most popular Lean technique applied in software development, and list different sources of waste in software development.

4.2.3 Kanban

Kanban is a popular Lean technique used in software development. It is defined as a signals system to implement the principle of pull^j in manufacturing. The idea behind Kanban is very simple. It focuses on using a visual signaling device to give authorization and instructions for producing items in a pull system. The signals tell an upstream process the type and quantity of products to make for a downstream process. Thus, products (or product parts) are only manufactured when needed. Kanban cards are the most common example of these signals. However, Kanban can be also implemented through other devices such as metal plates or colored balls. Fig. 9 shows an example of how Kanban cards support pull. Process A only produces when it receives a Kanban card. Similarly, Process B returns the Kanban card to Process A when it needs more materials.

This same idea has been applied in software development but in the form of a board and sticky notes, as depicted in Fig. 10. The board represents each activity in the development cycle in a column (e.g., definition of backlog items, analysis, development, acceptance testing, and deployment). The sticky notes represent user stories or tasks from the product backlog. Thus, the Kanban board visualizes the development flow of these user stories/activities. The number of items under each activity is limited by establishing work-in-progress (WIP) limits. For example, in Fig. 10, three backlog items can be under analysis at the same time, three under development, two in the phase of acceptance testing and one in deployment. Items from the backlog are moved from column to column as the development progresses. An item can be moved to the next development step only if there is room for the new item according to the WIP limits. Thus, teams are not overloaded with work and can focus on the items that provide maximum customer value. Teams can pull more work at any time, triggered by availability of resources. For example, whenever a team member is free, s/he can pull the highest priority ticket from the product backlog. Besides development activities and WIP limits, the Kanban board can include also other information such as the person/s working on each backlog item. It is up to the team to design the Kanban board that better serves its needs.

The board has the same purpose that signals in a manufacturing context, implementing pull in practice by visualizing the workflow (providing visibility to the entire software development process), and enabling the identification of problems, such as bottlenecks in the workflow (a source of waste). Moreover, WIP limits help reduce inventory and focus on the feature that is being developed [113]. Kanban provides five key benefits [113]:

1. 1. It visualizes workflow. The Kanban board visualizes the status of each activity in the development process. The team can easily see the big picture of what has been done so far and what needs to be done. Transparency provides a greater control of the workload to deliver in a continuous manner.
2. 2. It limits work-in-progress. WIP limits are used to limit the amount of work that can be pulled into each activity. It helps optimize workflow and prevent from team overloading and multitasking. Moreover, WIP limits make bottlenecks visible, helping to a better distribution of resources.
3. 3. It helps measure and manage flow. Besides providing information to eliminate bottlenecks in the entire value stream, lead times can also be easily calculated using the board. The shorter the lead-time, the greater is the value for customers and the organization.
4. 4. It makes process policies explicit. The process and the policies that the team agrees on how to manage the workflow becomes also visible through the Kanban board. For example, it is clear for everyone what kind of items take priority at a certain time in the development process (e.g., bug fixing vs new user story development).
5. 5. It supports continuous improvement in a collaborative way. The Kanban board allows for transparency in the development flow and process monitoring. Inefficiencies and bottlenecks are visible. Kanban teams need to work on those, if they want to maintain a steady flow (Kaizen culture).

Ahmad et al. [122] reviewed the scientific literature on Kanban in software development and found promising benefits when using Kanban, such as better understanding of the whole process by all stakeholders, especially developers, improved team communication, and increased customer satisfaction. However, they also found some challenges, such as integrating Kanban with Agile practices, changing the organizational culture according to Lean and applying Kanban in distributed environments. Regarding integrating Kanban and Agile, Kanban is often used in combination with Scrum in which is known as Scrumban [123]. In Scrumban, Scrum teams employ Kanban boards to visualize their development flow. Depending on the team and the situation at hand, sprints are changed by a continuous delivery flow but other Scrum's elements such as daily scrum meetings or periodic reviews are kept [123].

5.1 Need for Speed

Why does speed matter? The traditional view of software as a static item, which can be bought and owned, is not the current reality of the software industry. Software products and software-intensive systems are increasingly object of a constant transformation. Indeed, customers expect a continuous evolution of product functionality that provides additional value [126]. Speed matters because it provides a competitive advantage to satisfy customers’ needs. Speed is concerned with shortening feedback loops between a company and the users of its products. It allows companies to better understand user's behavior and adapt to business/customer changes faster. The risk of predicting what customers want is decreased by increasing feedback loops between users and companies and, thus, learning (instead of predicting) what customers want. Need-for-speed (N4S) is the name of the research program in which we have conducted part of our research activities during the last few years (http://www.n4s.fi/en/). Executed by the forefront Finnish software companies and research institutions, N4S focused on making software business in real time. Concretely, during the program, we investigated solutions for adopting a real-time experimental business model to provide capability for instant value delivery, based upon deep customer insight. In the rest of the chapter, we reveal our most relevant findings from the program. Particularly, we discuss key process elements that are important to get speed in software development and the concept of Lean start-up as an experimental model to foster innovation.

Coming back to the reasons why speed is important in software development processes, in his recent book Speed, data, and ecosystems: excelling in a software-driven world [127], Bosch reflects on five reasons why speed matters in current software business. According to Bosch,

1. 1. Speed increases the company's ability to adapt development priorities. Working in short increments (such as sprints) allows companies set-up priorities more often. Obviously, the overall product's direction cannot be changed every increment but priorities in small items such as features can be adapted.
2. 2. Speed allows accelerating the innovation cycle by a faster iteration through the phases of build-measure-learn (BML). Speed allows companies learn what new ideas would work for their customers faster than the competence. We further develop the concept of BML and the Lean startup movement, in which BML originated, in Section 7.
3. 3. Speed gives development teams confidence to go into the right direction as it enables believable progress tracking. Team satisfaction increases as team's members realize that they are able to provide customer value on a continuous basis.
4. 4. Speed allows continuous feedback on the delivery of value. It enables fast user feedback to guide the following development steps. Data collection is fundamental in this phase so that decisions are based on data rather than intuition or subjective opinions.
5. 5. Speed supports product quality. Faster and more frequent release cycles should not compromise software quality. In fact, although it may seem paradoxical, when properly applied, rapid software development favors quality. In rapid software development coding and testing are done in parallel. Integration and bug fixing are not left to the end of the process, and testing is highly automatized. Overall, speed forces organizations to maintain high quality levels all the time, as new software is continuously delivered to users.

As we described in Section 3.3, speed is one of the four main capabilities of Agility. Speed is embedded in the Agile Manifesto as well. For example, Agile principles, such as “our highest priority is to satisfy the customer through early and continuous delivery of valuable software” and “deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale” make clear reference to speed. The notion of sprints in Agile methods such as Scrum aims also to increase speed in the software development process. Instead of waiting until the end of development, customers can see progress after every sprint. Based on Agile software development principles, rapid software development increases speed to approaches where the step between development and deployment is minimized and code is immediately delivered to production environment for customers to use (and learn from customer's usage of the product). Iterative Agile and Lean software development are extended toward continuous flow of product functionality.

5.2 Continuous Delivery and Continuous Deployment

Continuous delivery and continuous deployment (CD) are at the heart of rapid software development. Intuitively, both relate to produce new fully evaluated and ready-to-deploy software revisions continuously, even many times per day. Humble and Farley [16], in their book Continuous Delivery: Reliable Software Releases through Build, Test, and deployment automation published in 2010, state that continuous delivery provides enterprises with the ability to deliver rapidly, reliably and repeatedly value to customers at low risk with minimal manual overhead. The central concept in Humble and Farley's approach is a deployment pipeline that establishes an automated end-to-end process. Automation is used to ensure that the system works at technical level, executes automated acceptance tests and, lastly, deploys to a production or staging environment. The goal of continuous delivery is to enable the delivery of new software functionality at the touch of a button, using a fully automated process. In the ideal scenario, there are no obstacles to deploying to production; there are no integrations problems, builds can be deployed to users at the push of a button.

There is certain confusion in the use of these terms: continuous delivery and continuous deployment (we refer to the last one as CD). The early scientific literature tended to use both interchangeably, as synonyms [46]. However, there are important differences between them. Humble and Farley [16] describe CD as the automatic deployment of every change to production, while continuous delivery is an organizational capability that ensures that every change can be deployed to production, if it is targeted. However, the organization may choose not to do so, usually due to business reasons. In other words, continuous delivery refers to the organizational capability to keep software always releasable. That is, software is ready to be released to its users at any time but the company decides when it will be released. On the other hand, in CD, software is automatically deployed to production when it is ready. Note that while CD implies continuous delivery, the converse is not true. Fig. 11 graphically illustrates the difference between continuous delivery and CD.

Humble and Farley argue that although CD enables releasing every good build to users, it does not always make sense. In continuous delivery, any build could potentially be released to users, but it is the business, not IT, who decides about release schedules.

Leading organizations, such as Facebook, Microsoft and IBM have actively moved toward rapid releases [52]. A plethora of evidence related to continuous delivery and deployment exists in companies’ white papers and online blogs, where practitioners have voiced their experiences, expectations and challenges in moving to this direction (e.g., Facebook,^l Microsoft,^m Google,ⁿ Adobe,^o and Tesla^p). Next section develops into the main characteristics that allow companies to increase speed in their software development processes and achieve continuous delivery/deployment.

5.3 Key Elements to Get Speed in Your Software Development Process

In the context of the Need-for-Speed project,^q we reviewed the scientific literature on CD and continuous delivery using the SLR method [46]. Our objective was to determine the underlying factors that characterize this phenomenon as they appear in the scientific literature. Next, we describe the most relevant factors that we found in our study.

1. 1. Fast and frequent releases, with preference given to shorter cycles or even continuous flow (weekly or daily deliveries). Release cycles are accelerated when moving toward rapid software development. This implies that the surrounding areas that enable fast releases are also transformed. For example, planning needs to be a continuous activity as well in order to enable a steady flow of small changes. A tighter integration between planning and execution is needed to ensure alignment between the needs of the business context and software development. The ability to release quickly does not mean that development should be rushed into without a conscious understanding of what is actually being done. The release process needs to be clear, including having a clear release management workflow and delivery workflow.
2. 2. Flexible product design and architecture. The software architecture will evolve during the product life-cycle and, therefore, needs to be able to adapt to changing conditions. The architecture must be flexible. Still, it has to be robust to allow organizations to invest resources in offensive initiatives (e.g., new functionality, product enhancements, and innovation) rather than defensive efforts (e.g., bug fixes). The architecture and design have to be highly modular and loosely coupled [128–130]. Moreover, constantly measuring and monitoring source code and architectural quality is important [130,131]. The literature provides some techniques to manage architecture aspects in rapid software development. For example, rapid architecture tradeoff analysis to accommodate rapid feedback and evaluate design options [131], quantifying architectural dependencies by combining Design Structure Matrix (DSM) and Domain Mapping Matrix (DMM) [132] and identifying and assessing risky source code areas based on diverse metrics [133]. In addition, in order to enable continuous and rapid experimentation (see fourth factor), mechanisms in software architectures for run-time variation of functionality and data collection as well as rollback mechanisms to revert changes are required. The literature offers some solutions to achieve this end. For instance, Rally Software suggests A/B testing with Feature Toggle as a technique to manage and support run-time variability of functionality [134].
3. 3. Continuous testing and quality assurance. Testing and quality assurance practices have to be performed throughout the whole development process, as new functionality is rolled out, and not just at the end of the development. Transparency on the testing process is important to avoid problems such as duplicated test effort and slow feedback loops [135]. Well-known testing techniques in Agile software development, such as test automation and continuous integration support continuous quality assurance. The concept of technical debt, which is further developed in Section 8, is also important to balance speed and stability. However, besides technical aspects, continuous testing, and quality assurance require also a culture of quality. Developers bear the responsibility for writing good code, perform thorough tests as well as support the operational use of their software [119,136,137]. As systems become larger and more complex, such culture complements test automation and allows quality to be maintained at scale. The principle is that more frequent release cycles should not compromise software quality.
   The latest advances toward managing quality in rapid software development focus on supporting quality-awareness during rapid cycles through continuous monitoring and visualization of product and process quality. For example, our work on the Q-Rapids project^r focuses on a data driven, quality-aware rapid software development framework. In this framework, quality and functional requirements (e.g., user stories or features) are managed together [138]. Thus, quality requirements are incrementally elicited and refined based on data gathered at both development time and runtime. Using data-mining techniques, project, development, runtime, and system usage data is aggregated into quality-related strategic indicators to support decision makers in steering future development cycles. A strategic dashboard aggregates the collected data into strategic indicators related to factors such as time to market, team productivity, customer satisfaction, and overall quality to help decision makers making data-driven, requirements-related decisions in rapid cycles [139].
4. 4. Rapid and continuous experimentation. In rapid software development, systematic design and execution of small field experiments guides product development and accelerates innovation. Continuous experimentation allows companies to learn customer needs and base business and design decisions on data rather than on stakeholder opinions [140]. For example, Facebook uses A/B (split) testing as an experimental approach to identify user needs [136]. In A/B testing, randomized experiments are conducted over two options (e.g., two similar features) to compare, through statistical hypothesis testing, how end-users perceive each feature [136,141]. Experimental results are continuously linked with product roadmap in order to provide guidance for planning activities [142]. Rapid experiments are at the heart of the BML innovation cycle (see Section 7).
5. 5. User involvement. The role of users is essential in rapid software development. Everything goes around providing value to users. Thus, users are continuously monitored to learn what will provide value to them. Mechanisms to involve users in the development process and collect user feedback from deliveries as early as possible (even near real-time) are essential in rapid software development. They guide design decisions and innovation. However, techniques need to be nonintrusive so that users are not stressed with continuous feedback requests. Several options can be used in this sense. For instance, similar to Facebook, Rally Software [134] uses also A/B testing with Feature Toggle as a technique to manage and support run-time variability of functionality. Facebook uses a fast upgrade of database systems to deploy experimental software builds on a large scale of machines without degrading the system uptime and availability [143]. Besides involving customers in experiments, customers are also involved in testing activities. Again, strategies applied with such purpose have to be nonintrusive. For example, dark deployment is used to deliver new features or services, which are invisible to customers and have no impact on the running system, to test system quality attributes and examine them under simulated workload in a real production environment [134,136]. Another relevant practice, canary deployment, delivers a new version of the system to a limited user population to test it under real production traffic and use. The new version is then delivered to the whole user population once it reaches a high enough quality level [134,136].
6. 6. Automation. Computers perform repetitive tasks, people solve problems. In the context of continuous delivery and deployment, the focus is on automating the entire delivery pipeline as much as possible. The goal is that deployments are predictable, routine tasks that can be performed on demand in a matter of seconds or minutes. Continuous integration automates tasks such as compiling code, running unit and acceptance tests, monitoring and validating code coverage, checking compliance with coding standards, static code analysis and automatic code review. The continuous integration practice of Agile software development is extended to release and deploy automation. For example, Facebook [136] has a tool for deployment called Gatekeeper that allows developers to turn features on and off in the code, and to select which user groups see which features. Similarly, Agarwal [144] presents an automation system to integrate configuration management, build, release and testing processes. The system enables migrating changes from one environment to another (e.g., development to production), performing a release with selected content, and upgrading software in a target environment. Humble et al. [145] recommend automating build, testing, and deployment in the early stage of the project and evolving the automation along with the application.
7. 7. DevOps. DevOps is another key element in continuous delivery and deployment because it allows achieving rapid end-to-end software development. DevOps integrates development and operations. It enables transparency and understanding of the whole development pipeline to overcome corporate constraints that often cause delays in product deliveries (e.g., handover delays and communication gaps). The concept of DevOps is further developed in next section.

6.1 The Origins of DevOps

Compared to early years of its inception, the concept of DevOps today is less ambiguous owing to the increased number of scientific studies describing its underlying characteristics and its adoption in software-intensive organizations [146–150]. DevOps emerged in 2009 as an approach for implementing CD [151]. For several years, it had remained unclear what the concept entails, despite its popularity in the software industry.

The term “DevOps” is a combination of two words, “Development” and “Operations.” As a concept, DevOps stresses on building an Agile relationship and collaboration between software development and operations, whereby operational requirements are considered early into, and throughout the software development process. The main aim of DevOps is to speed up the delivery of quality software changes more reliably and repeatedly in production environment. DevOps proposes some practices and the use of tools to automate the management of software infrastructures, which, over the years, have become complex, heterogeneous and of large-scale.

Historically, the term DevOps was coined by Patrick Debois when organizing a practitioners’ event called DevOpsDays.^s Prior to DevOpsDays, in 2008, Debois gave a presentation at the 2008 Agile conference titled: “Agile infrastructure and operations” [152] that focuses on employing Agile principles to infrastructure-related activities and projects, such as upgrading application server that has multiple infrastructural component dependencies. After his presentation, Debois observed a growing interest on the topic among other software practitioners. For instance, few months prior to the DevOpsDays event, two other pioneers of DevOps, John Allspaw and Paul Hammond, shared their experiences of multiple deployments in a presentation titled “10 deploys per day: Dev & ops cooperation at Flickr” at the 2009 Velocity conference. Together with other software practitioners, the pioneers introduced a DevOps movement that today embodies a diverse set of practices and tools crucial for speeding up the delivery process of software changes to end-users.

Much of what is addressed by DevOps movement deals with problems resulting from organizational silos between software development and operations. For many medium and large software-intensive organizations, software development and operations activities are traditionally performed by separate organizations. Typically in such context, software is designed, implemented and tested by software developers and testers, and, upon completion, the developed software is handed over to operations personnel, such as system administrators. Operations personnel then deploy and operate software in production environment. Such a setup is beneficial to facilitate required specialization and prevent mix up of software development and operations tasks in daily work of engineers. However, the separation, exacerbated by poor communication between software development and operations, has been reported to bring several serious problems, including unsatisfactory testing environments and systems being put into production before they are complete [153]. The problems intensified when software developers started to use Agile methods, but without their adoption among operations personnel [154,155]. As a result, conflicting goals of “stability vs. change” began to appear. Operations personnel, being skeptical of the impacts of frequent changes to production environment, tend to seek stability by avoiding frequent system changes to production, which software developers tend to seek with Agile methods. To facilitate effective collaboration between software development and operations, several approaches, ranging from ad-hoc and informal to formal approaches, were suggested [156]. However, more recently DevOps emerged as a solution that focuses on automating the deployment pipeline and extending Agile methods to operations, as visualized in Fig. 12 [146,147]. The deployment pipeline, visualized in Fig. 13, is the path taken by all software changes to production, including changes of software features and configuration parameters. In the deployment pipeline, though it may manifest differently across software-intensive organizations, similar patterns can be observed. A common pattern is that, the software developer performs development tasks on his/her local development environment and may performs some tests before deciding to commit the changes to team's shared development environment, where the changes are made visible to others. Upon commit, software changes are automatically compiled, tested for integration and with successful tests a build is created. The newly created software build is then passed for additional tests such as acceptance tests before being deployed to production environment, where software changes are made visible to end-users. Specific details in the level and efforts needed for test and deployment automation vary across software-intensive organizations, depending on the domain and type of application being developed.

6.2 DevOps Practices

The adoption of DevOps in software-intensive organizations is not homogenous due to factors such as target objective and starting baseline. DevOps adoption entails a series of organizational and sociotechnical practices to be employed. The organizational practices focus on organizational structures and ways in which an organization deals with the relationships and responsibilities between software development and operations. Meanwhile, the sociotechnical practices focus on engineering practices related to the development of software changes, deployment pipeline and interaction of software developers and operations personnel.

From the organizational perspective, DevOps has four common practices [157,158]:

1. a. separating development and operation with high collaboration;
2. b. separating development and operation with a facilitator between the two groups;
3. c. development team gaining more responsibilities of operations and operations existing in a small team and;
4. d. no visible operations team.

Among the four practices, the practice of separating software development and operations with high collaboration is most common in large software-intensive organizations [157,158]. The practice of separating software development and operations with a facilitator between them, and assigning more responsibilities to developers are more common in small and medium sized organizations [157,158]. However, regardless of the organization size, the practice of having developers who gain more responsibilities of operations is observed to increasingly becoming common [157,158]. This has huge implications to the working culture and division of responsibilities [136,148,159,160] A complimentary practice consists of establishing of a separate team that acts as a facilitator between the two groups. [148,159,161,162] argue for such a team because it can significantly reduce required efforts and minimize disruptions for development team. Particularly, in large organizations, the established team consists of a small number of members with multidisciplinary background including that of software development and operations. The team is responsible for standardizing deployment mechanisms such as building a deployment pipeline platform, which is to be used by software developers [148,161,162]. An important aspect to notice is that when knowledgeable software developers are given the responsibility to perform operations tasks, they tend to be given access to critical environments [159,160]. Such access allows software developers to provision by themselves different software development environments.

DevOps practices from a sociotechnical perspective focus on automating the deployment pipeline, including automatic management of software infrastructure, which consists of development and production environments. The sociotechnical DevOps practices include technical infrastructure where software development and operations activities take place and its interaction with human agents. Therefore, the deployment pipeline is the technical infrastructure where sociotechnical DevOps practices are observed. In specific, a survey of software practitioners by Cito et al. [157] reports on incorporation of practices in the following areas for DevOps: (1) software deployment and provisioning; (2) software design and implementation; (3) software troubleshooting; and (4) software monitoring and production data usage. Automation practices in the deployment process, which focus on automatic provisioning of environments, are more prominent in cloud-native software [157]. One reason is the progress in virtualization technology, which is made prevalent through cloud computing. Cloud computing provides access to computing resources that, through DevOps tools such as Puppet and Chef, software developers can use to provision and manage environment repeatedly and fast. The tools advocated in DevOps, which are used to automatically provision and manage environments, apply the concept of infrastructure-as-code, which is central to DevOps. The Infrastructure as Code concept entails a practice of using software engineering practices for managing source code such as programming language and version control, to implement and manage scripts used for environment provisioning, configuration management and system deployment [163]. In practice, the infrastructure-as-code concept is facilitated by DevOps tools like Puppet and Chef, which feature a domain-specific language used to write scripts for environment provisioning and software deployment. The benefit of using infrastructure-as-code is that it helps minimize inconsistencies and complexity of software infrastructure, which in turn helps shorten the release cycle time. We further elaborate on two sociotechnical DevOps practices as these were the focus areas in our research.

Automation of deployment process. The automation of deployment process in DevOps involves two main activities. First, an automatic provisioning of development environments, e.g., test, that are similar to the production environment. Second, the application of deployment strategies such as blue/green deployment, rolling upgrade and canary deployment, to deploy software changes to desired environments. The latter is meant to replace a manual process for deploying software and configuring environment as well as to minimize environment inconsistencies with production, as these constitute to common antipatterns in software delivery process and have impacts to the release cycle time of changes [16]. Additionally, a deployment strategy is selected and facilitated by the use of deployment automation tools. Blue/green deployment and rolling upgrade are two commonly used deployment strategies for cloud application [147]. In blue/green deployment, a number of virtual machine (VM) instances, which contain old version of the software, are maintained while in parallel the same number of VM instances, which contain an updated version of the software, are provisioned and only when they are ready, traffic is routed to the new instances [147]. In rolling upgrade deployment strategy, a small number of VM with new software version are upgraded directly to environment containing VM instances with old version and in parallel a number of old VM instances are switched off [147]. However, each strategy may also pose some reliability issues, including the amount of time needed to prepare and place a new VM to the desired environment [147]. Recently, microservice architecture has been found beneficial when considering deployment automation. Microservice architecture loosely decouples the software into collaborating services whereby each service implements a set of related functions that can be easily developed and frequently deployed, using technology such as Docker, independent of other services [164]. Despite its many benefits, microservice architecture still presents several challenges that need to be addressed in research, including complexities introduced in managing and monitoring system comprised of many different service types [165].

Monitoring. In DevOps, monitoring becomes important not just for operations personnel but also software developers. For instance at Facebook, developers in a team are responsible for monitoring and teams schedule. They rotate among themselves an “oncall” person who receives alerts of urgent issues through automated phone calls but also shields others to focus on development work [166]. Software monitoring tools, such as New Relic, allow developers to install monitoring agents to the software in order to gather real-time software and server monitoring data and metrics specified as relevant by the development team [147]. Insights from monitoring data can be used to inform and guide developers identify software failures and performance improvements as well as gain feedback from users based on software usage among other things [147]. Recent research contributions on this aspect have focused on finding effective ways to correlate runtime data to software changes during development time for such purpose, as to automate troubleshooting activity among others [167]. This is contrast to the traditional approach whereby developers and operators employ manual steps to diagnosis faults by navigating and inspecting relevant information from log statements, software metrics, and using mental models [167]. For this purpose, Cito et al. [167] have proposed an analytical approach to established explicit links that carry semantic relations between related fragments that can be individual logs, metrics, or excerpts of source code.

Finally, it is relevant to mention that DevOps practices are championed and well established in software-intensive organizations that are involved in cloud and web development [136,148,157,162]. DevOps practices are increasingly being evaluated for their feasibility in other contexts, such as embedded systems, particularly because several factors make the adoption of DevOps in such contexts challenging, e.g., unavailability of technology to enable flexible and automatic system redeployment repeatedly [159,168,169].

6.3 The Benefits and Challenges of DevOps

Benefits and challenges of DevOps have been reported in empirical studies. For example, among the most popular benefits are a reduction in release cycle time; improved quality and eased communication between software development and operations [148,161,162]. However, successful adoption of DevOps requires a mindset change, management support and shared responsibility among all stakeholders involved in the delivery of software. Support from senior management strengthens the value of and the adoption of DevOps across an organization [170]. However, the organization may encounter several challenges during the adoption of DevOps, including the need to rearchitect the software to better support DevOps in deployability, testability and loggibility [171]. Microservice architecture is one architectural design having such benefits as scalability, flexibility, and portability, and is in support of DevOps [164]. Furthermore, since in DevOps, the development team is given the responsibility to develop and operate the system that they develop, this requires the team to be equipped with necessary skills, thus, increasing the learning curve among development team members [160]. Alternatively, for some processes, their inner working may be abstracted through automation achieved in the deployment pipeline, thus, demanding a low learning curve from development team members.

7.1 Origin and Key Concepts

In 2011, Ries [172] applied the principles of Lean thinking into startup companies in his book “The Lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses.” In the book, he focused on the question how to tune a successful business case from a product idea. In his thinking, leanness was crystallized by defining waste as a product without market potential: the biggest waste is to create a product that nobody is willing to buy. Ries presented how to work in a Lean way and avoid waste by introducing a process of early learning. The key characteristics of that process are (1) earliness, (2) close customer cooperation, (3) rapid progress, (4) measurement of the value, (5) learning, (6) corrective actions, and (7) iteration.

• • Earliness refers to the need to follow Lean thinking from the very beginning of a startup. In the opposite case, the course of the company may already be fixed toward a wrong direction, and the efforts done turn out to be waste.
• • Close customer cooperation refers to the principle that the customer(s) need to be linked to the startup's doings as early and closely as possible. The measurement of the value of the product is fully based on the customers’ opinions. A real customer is the source of the feedback needed when measuring the business potential of the innovation. Ries’ book classifies customers by defining different groups of customers being relevant for a startup in different phases of its evolution path. The customer groups are defined mostly in the context of American business environment and products aimed to mass markets. In a practical situation, however, close customer relationship is easier to create in case of products aimed to specific customers instead of mass markets [173]. In any case, early and close customer cooperation is very beneficial for a startup figuring out its first product. Reasonable amount of effort should be put in finding potential customers willing to cooperate with a startup—willingness to work with a startup having only a vague idea of a product and a MVP (minimum viable product), may be low among the potential customers.
• • Rapid progress refers to a startup's need to validate the value of the product idea as rapidly as possible. Rapid progress along with earliness is a key means to avoid waste in Lean startup thinking, but the need to speed up the product development is a common trend in the software industry, as discussed earlier in Section 5. The need for a rapid progress and close customer cooperation are the drivers behind one of the key concepts of Ries’ Lean startup, building the MVP.
• • Measurement of the value of the business case is done by the customer on the basis of that MVP—and only in that way. An MVP should be at the same time easy and fast to create, but functional enough to be a valid basis for an evaluation by the targeted customer(s). What the MVP is at the end, is left fairly open. Ries’ book addresses MVPs by introducing examples. That is understandable when aiming at covering a big variety of startup with the same principles, but in a practical situation the MVP must be well figured out to fit to the innovation, to the measurements purposes, and to the overall context.
• • Learning refers to the experiences and opinions collected from the customer(s) concerning the value of the product idea. How the learning is gained, is covered in Ries’ original book by defining such terms as validated learning and actionable metrics. Validated learning is a result of validated experiments with the latest version of the MVP and the real customer(s). As the experiments are the source of validated learning, a key issue, besides the definition of the MVP, is careful planning of the experiments in which the value of the MVP is measured.
• • Corrective actions refer to the conclusions drawn from learning. The corrective actions are crystallized in Lean startup thinking into two clear alternatives, changing the idea or continuing its development, called in Ries’ terminology pivoting or persevering. The criteria for pivoting or persevering are summarized by Ries as so-called actionable metrics as an opposite of vanity metrics. The actionable metrics should be accessible and audible. The point in putting such attributes as actionable and vanity on the metrics used for decision making is to highlight courage of a startup's key persons to explore the innovation in question in a critical and objective way. The positive feelings or bride of the innovator(s) should not lead to development of products that turn out to be waste. Measuring and learning in Lean startup thinking can be seen as a means to identify difficulties and find reasons not to persevere but pivot, as studied in Ref. [174].
• • Iteration refers to the principle that the whole process of building MVP(s), measuring, and learning, has to be repeated until the aimed product gains the functionality and characteristics that guarantee customer acceptance. This principle represents the old idea of continuous and incremental development, applied in Lean startup thinking on the customer-centric validation of an innovation. The same continuous and incremental way of doing is proposed also for the growth stage of a startup, when the first product(s) has successfully been launched to the markets and the startup is seeking growth. A broader view on the practices of continuous and incremental development was presented in earlier sections.

Ries encapsulates the earlier characteristics in a simple, cyclic process containing three steps: (1) build, (2) measure, and (3) learn, BML cycle (see Fig. 14). After the three steps, the startup evaluates the results of the steps and makes a decision whether to keep the idea, persevere, or to modify it, pivot. In the former case the development of the idea to a product continues, while in the latter case the idea is modified and the build–measure–learn steps are repeated. Following this process should prevent a startup wasting its resources in developing products without market demand.

7.2 Derivatives and Further Developments

Ries’ Lean startup thinking was rapidly accepted not only by practitioners but also by the academia too. A set of variants based on similar thinking were presented. Stainert and Leifer presented fairly similar ideas in their hunter–gatherer model [175], which was further tuned by Nguyen-Duc et al. [176]. While the hunter–gatherer model was at a fairly high and philosophical level, Bosch et al. [177] and Björk et al. [178] presented in 2013 a development model called Early Stage Software Startup Development Model (ESSSDM), claimed to be an operationalization of Lean thinking on software startups (see Fig. 15).

The ESSSDM model was developed to bring the ideas of Ries closer to the practical work in software startups by addressing such areas as managing multiple product ideas in a product portfolio, validation criteria of the product ideas, and techniques to use for validating them. The ESSSDM model, with its focus on product portfolios, is not principally different from the original Lean startup thinking. A product portfolio including alternative product ideas, means validating several ideas in parallel or in series. However, a smaller startup may not have resources for parallel evaluation of several product ideas, as strongly recommended in the ESSSDM model. An idea portfolio may also be difficult from the focus and motivation perspectives: founding a company and taking the responsibility and risks tied to it requires courage [179]. A strong belief on a single idea may be a better catalyzer for the needed courage than a set of ideas, which possibly are not as credible.

7.3 Lean Startup and Other Startup Research

Starting fairly early, Lean startup thinking has been combined with other research paths on innovation and entrepreneurship. Müller and Thoring studied in 2012 differences and similarities of Lean startup model and design thinking [180]. The comparison highlighted, in an interesting way, certain characteristics of Lean startup thinking: (1) it assumes that a product vision exists, e.g., a Lean startup is driven by a product vision of the startup founders, (2) it is customer centered instead of user centered, and (3) its focus is on the business model for a product idea—not on user desirability or technical feasibility of the product.

The Lean startup thinking focuses on creating the business case and on validating product ideas from the business perspective. The customer acceptance of the idea is defined as the criterion of the waste. Practitioners having experience in product development know that the business feasibility is only one dimension of the whole. The technical feasibility and user desirability must be evaluated, too. Putting too much effort on an idea that is too expensive or too difficult to realize with existing technology, or difficult to use, is waste even when the idea has a real market potential [173]. Technical feasibility and user desirability are broad areas where the weight of different criteria varies according to the context and purpose of the product: a data communications product must provide the user with speed and reliability, an interactive product with desirable user experience, understandability and easiness of use, and a banking system with data security.

An example of a broader focus validation of the idea is the work of Hokkanen et al., combining the lean startup principles with the user experience of the targeted product [181–183]. Hokkanen's work complements the Lean startup from the user-centered view by highlighting the importance of a good user experience in an early stage—as a part of the MVP. According to her, an MVP must be mature enough and its user experience desirable enough to help selling the idea and collecting meaningful feedback from the customer(s). Following the original Lean startup thinking, she recommends early building of communication channels to real users, implementing usability tests before measuring, and applying planned methods to collect the feedback necessary for correct learning.

7.4 Summary of the Lean Startup Movement

As mentioned earlier, Lean startup thinking is vision driven—there is an innovation, the value of which has to be measured. The startup is founded to realize the innovation to a product. Research [173] has shown that many times it really is the case. A typical story of a startup is that a single person gets an idea that he/she believes in, gathers possibly a team, preferring old friends and other trusted people, and founds a company [179]. Sometimes, founding is preceded by a long period of deliberation before the final decision for a startup is made. Of course, there are also different cases, where the need for a certain product or service is clear and the business case is easier to figure out. That does not, however, decrease the value of the basic idea of the Lean startup thinking, better to evaluate than to develop waste.

The works by Bosch et al. [177], Björk et al. [178], Müller and Thoring [180], and Hokkanen et al. [181–183] are examples of how the Lean startup thinking, defined at a high abstraction level, and trying to address a big variety of different startups, needs additional stuff around the principles. However, the principles, validating before progressing, iterating, and improving continuously, seeking learning from real customers, being critical not vain, and avoiding waste, are good guidelines for any product development. The principles are broadly applicable also outside the original business-related focus, and their value is intuitively understandable. The basic message is: keep on learning—do not believe that you know the value of your idea without measuring it.

8.1 Metrics in Agile and Lean Software Development

Our SLR about the use of software metrics in Agile software development [68] revealed both similarities and differences between Agile and traditional software development. The top goals of both development approaches are similar, that is, to improve productivity and quality. The majority of the metrics are in one way or the other tied to those goals. However, we found value-based differences in means how those goals are to be reached. Traditional software development utilizes software metrics from the viewpoint of scientific management by Taylor, i.e., controlling software development from outside. The traditional measurement programs use ideas stemming from industrial product lines and manufacturing industry such as Six Sigma. Traditional metrics programs were often large-scale corporate initiatives, which emphasized following a pregiven plan. Agile principles, such as empowering the team, focusing on rapid customer value, simplicity, and willingness to embrace changes, form a contrast to the traditional measurement programs.

Fig. 16 shows the top six metrics used in Agile software development. We selected the top metrics based on the number of occurrences and the perceived importance of each metric in the sources (for more details, see Ref. [68]).

Velocity was found as the most important software metric in Agile development. Velocity indicates how fast is the team working or how much a team can achieve in a certain time-boxed period, e.g., in a sprint of 4 weeks. Velocity is used in sprint planning. Velocity is also used to determine different scopes for the sprint such as having a minimum scope of features that can be delivered for sure and having a stretch goal for the iteration. Having two different scopes can help in customer communication as the minimum can be promised and the maximum indicates all the changes that may happen during a sprint. Velocity enables finding bottlenecks in a software process where progress is slower than desired. Making process improvement or automation to these parts results in an excellent cost benefit ratio for the improvement. However, our study also found cases where attempts to maintain velocity led to cutting corners and lowered the product quality.

Effort estimate, which indicates the anticipated work cost of a particular feature or story, was found as the second most important metric. Effort estimates are important in feature prioritizations as all features with equal value can be sorted by the effort estimates to provide a sprint scope with the highest possible value. However, No Estimates movement started by Ron Jefferies, approximately says that making effort estimates can be counterproductive as the accuracy of the estimates is too poor to make the effort spent in estimation worthwhile. Nevertheless, effort estimates were often used and found important in the sources of our literature review.

Customer satisfaction was found as the third ranked metric, although it could be claimed that it should be the most important due to the customer centric nature of Agile software development. Yet, it was less frequently mentioned in our sources than Velocity or Effort estimate. Customer satisfaction is a metric with many possible operationalizations. We found several measures of customer satisfaction such as the number of change requests by the customer, net promoter score (the probability that a customer would recommend the product or producer to another potential customers), Kano analysis (quality model that distinguishes between must-be, one-dimensional, attractive, indifference, and negative quality), and postrelease defects found by the customers. We found that most successful projects measured customer satisfaction more often than other projects.

We find that the core books giving advice on Agile software development are missing some core software engineering metrics such as the defect count. In our review of industrial empirical studies, it was ranked in the fourth place indicating its importance in Agile software development. Defect count can have many uses. It may be a measure of software testing as the number of defects found per time unit in testing measures in-house software quality. Furthermore, the defects experienced in customer usage can be a measure of the software quality in the wild.

Technical debt, which is covered in more detail next, was among the top metrics used by Agile teams. Technical debt hinderers day-to-day software development and reduces velocity. In one company, a technical debt board was used to make technical debt visible in order to ensure that enough resources and motivation were given to removing the debt. The board was also used to make plans how to remove technical debt. The board made sure that developers picked the highest priority technical debt items instead of cherry picking lower priority but more interesting tasks.

Build status is an important measure in Agile software development. It is an indication of the agile principle working code. If builds are not passing, this indicates that no progress according to Agile principles has been made. Build status often encompasses quality gates in addition to the simple assertion that the software has been successfully built. It can include the results of automated unit testing. Frequently, successful build requires that certain end-to-end automated acceptance test is passed. When working on software with very high reliability requirements, even violations from static code analysis tools, such as Findbugs or Coverity, may be treated as build breakers.

We also studied the purposes of using metrics in Agile software development, which are depicted in Fig. 17.

We found that over 70% sources used metrics for planning and progress tracking of sprints or projects. Understanding and improving quality were found as a reason for using metrics in roughly one-third of the studies. Fixing process problems inspired roughly half of the studies in using metrics. Finally, motivating people was found as the reason for using metrics in 25% of the studies, for example displaying a technical debt board can motivate people to reduce technical debt. The percentage does not add up to 100% because many studies had multiple motivations in using metrics.

8.2 Technical Debt in Agile and Lean Software Development

The Agile Manifesto highlights “continuous attention to technical excellence and good design enhances agility.” Similarly, XP has “coding standards” as one of its key practices. Still, the scientific literature highlights that Agile software development appears to be more prone to technical debt compared to traditional software development [184]. This section develops into the concept of technical debt and describes different strategies that can be applied to manage it in Agile software development.