Chapter 5. Rapid Iterative Development
Long before agile was popular, technology professionals had proposed methodologies for rapid iterative development. These methodologies were often focused on the iterative nature of waterfall, but also attempted to address the challenges of developing software using traditional software development lifecycles. Another key differentiator was a focus on involving the end user throughout the development lifecycle, which previously was only accomplished by professional business analysts. Developers had little or no direct contact with the folks who were going to directly use the system. Back then, mainframe software development often took years from the point that requirements were defined to when the system was finally rolled out. Efforts to speed up the process met with mixed results. We discussed rapid application development (RAD) in Section 3.6, and much of the information aligns with the lessons we learned during that journey to improve our software development methodology. Many technology professionals looked for effective ways to quickly prototype and deliver applications more expeditiously. Although agility has many distinct advantages and capabilities, rapid iterative development is an approach that has merit even on its own.
5.1 Goals of Rapid Iterative Development
Rapid iterative development has a goal of quickly developing functionality that can be demonstrated and reviewed with all stakeholders. One reason why this approach has merit is that technology professionals often need to get up to speed when learning new technologies. Taking an iterative approach enables the developers to do this while simultaneously demonstrating functionality. Reducing the time-to-market is often an absolute necessity. Rapid iterative prototyping has long been a proven method to improve quality and productivity while often reducing costs and shortening timelines.
5.2 Why Is Rapid Iterative Development Important?
Rapid iterative development is important because it enables us to develop the code in an iterative fashion, improving our knowledge and technical capability much more quickly than would be possible using noniterative approaches. This is particularly important when learning new technologies and development frameworks that are being used by a team that has little or no prior experience with this approach. Gaining immediate input from the end user is also extremely valuable. Rapid iterative development is not necessarily agile in nature, and it was around a long time before agile became popular. We have seen situations where organizations had significant software processes in place, often due to audit and regulatory requirements specific to their industry. Trying to adopt agile development would have been disruptive and introduced far too much risk. Helping the teams adopt a rapid iterative approach was acceptable and aligned with their culture and existing processes.
Rapid iterative development reduces risk by allowing the team to build, package, and deploy in short timeframes—without necessarily having to use agile fixed timebox sprints.
5.3 Where Do I Start?
You need to start by understanding the value of embracing rapid iterative development and old-fashioned process engineering, which will enable you to tailor the methodology to your benefit. The following goals and objectives will help guide this effort. I usually begin by looking at the group’s existing processes, identifying what works well and exactly what needs to be improved. Taking an iterative approach is especially helpful when we do not fully understand what needs to be built up-front. In this case, we often build a prototype, examine it, and then build the next iteration. This iterative approach helps from many different perspectives. The first view to consider is that of the developer’s.
5.4 The Development View
Developers tackle many complex technical challenges, each of which can involve significant technical risk. Risk is not inherently bad, but must always be understood so that it can be mitigated. We have worked in many financial trading environments where some very smart folks thrived on accurately well-understood risk. However, technical risk can be a little different. Technical risk involves the use of a technology, usually for the first time, and may or may not be implemented successfully. As technology professionals, we all like to be at the forefront of the technology industry. However, being the first to tackle new ways of doing things also means that they might just not work as expected. If you have implemented a system using Java J2EE with a well-known relational database such as MySQL or Postgres, then you have a pretty good idea of how long the effort will take and some of the potential technical challenges that you might face. But if you want to implement the same system using node.js and a NoSQL database, then you might find yourself facing new and unexpected technical challenges. Tackling the unknown challenges is exactly what technical risk is all about. What you need to do is decide on a course of action that will accurately and reliably identify and mitigate technical risk.
If you are implementing a new technology, then building something very small is an excellent first step. It is wise to continuously add a few small features and verify frequently that the system works as expected. Taking an iterative approach significantly mitigates technical risk. Another way to mitigate technical risk is with efficient development practices such as controlled isolation.
5.5 Controlled Isolation
Controlled isolation helps developers be more productive and avoids costly mistakes by providing a private workspace to get work done without the distraction of changes from other developers engaged in the same development effort. In version control systems, controlled isolation is provided by workspaces that are periodically synced from the master version control system. Rapid iterative development also provides an approach that thrives on controlled isolation where developers can work on creating code to solve isolated problems without the distraction of trying to tackle too many moving parts. Rapid iterative development puts sufficient focus on getting a reasonable amount of work completed in a fast and productive way.
Controlled isolation is closely related to managing complexity.
5.6 Managing Complexity
Complexity is one of the greatest challenges in any technology development effort. As software engineers, we manage increasingly complex (and exciting) systems. The only way that we can accomplish this work successfully is to tame this complexity. Rapid iterative development helps us manage complexity by completing one manageable piece of work and then tackling the next piece in an iterative way. This approach allows us to focus on just one challenging aspect of the system at a time without having to tackle the entire system—which for most professionals would be almost impossible. Trying to tackle overly complex systems results in human errors and software defects that can be difficult to find and resolve. Another important approach to managing complexity is continuous integration.
5.7 Continuous Integration
Continuous integration is a popular industry best practice that focuses on frequently merging and integrating changes from two or more developers. The more often that you integrate changes, the easier it is to address any problems or challenges that may arise. Continuous integration is usually managed by an automated merging of changes from developers using a version control system (VCS). Continuous integration servers generally take changes committed (e.g., checked in) into the VCS, automatically compiling the code, deploying the release baseline to a test environment, and then, most importantly, running a series of automated tests. Rapid iterative development benefits greatly from continuous integration, resulting in better code and improved programmer productivity. When implementing these industry best practices, it is always best to focus on effectively managing technology risk.
5.8 It’s All About (Technology) Risk
Risk is not bad. Many organizations thrive on risk, but only if that risk is understood and managed successfully. Rapid iterative development provides an excellent approach for understanding and managing technology risk. With each iteration, specific features can be isolated, understood, and implemented successfully. What is key here is that risk must be identified and plans formed to mitigate the risk by determining the best course of action to accomplish goals. The rapid evolution of technology results in many challenges that must be addressed. This is especially important when adopting new, and often complex, technology approaches that implicitly involve technology risk.
5.9 Taming Technology
Rapid iterative development helps technology professionals implement a specific piece of technology, which helps development engineers tame what is so often bleeding-edge technology. This usually involves implementing a piece of the new technology, allowing engineering resources to get up to speed and understand how the new technology works. Taming technology also relies upon excellent architectural design.
5.10 Designing Architecture
Quality has to be built in from the beginning of the software and systems development effort. The only way to accomplish this goal is to design architectures that are fit for purpose and fit for use. The rapid iterative approach allows us to design architectures that are robust and scalable.
5.11 Conclusion
Rapid iterative development does not solve every problem and is not a substitute for more robust agile iterative approaches. But sometimes situations require a pragmatic approach that is less than ideal. Rapid iterative development provides a practical approach that should be part of your agile ALM arsenal.
Further Reading
Martin, James. (1991). Rapid Application Development. New York: Macmillan Publishing.