Chapter 14. Agile in a Non-Agile World
Being agile in a non-agile world can be difficult, and at times even seem impossible to accomplish. We have often found ourselves in organizations that insisted on a waterfall approach. What is most difficult is trying to predict things that are just not possible to ascertain up-front. Many are unaware that waterfall was originally envisioned as an iterative process because today it seems that some organizations expect their employees to be able to predict the future to a degree that is simply not reasonable. The real problem is that these are the same organizations that expect you to make the project actually conform to the plan once it has been developed and approved. Any deviations may be perceived as a lack of planning and proper management. Being agile in a non-agile world can be challenging and is fraught with its own set of risks and pitfalls. This approach is sometimes called hybrid agile.
14.1 Goals of Hybrid Agile
The goal of hybrid agile is to design a process that aligns with both agile and non-agile methodologies. This is no easy task, and many folks have criticized attempts to adopt hybrid agile approaches in software and systems development. We are not without our own concerns, but our report from the trenches is that hybrid agile is, in fact, the norm in many organizations. In every engagement, our goal is to come up with the best possible methodology, given the reality that we see on the ground in many organizations. In some cases, the goal of hybrid agile is to act as a bridge to adopting agile completely, whereas in other organizations, hybrid agile allows the organization to adopt agile principles and practices within the framework of the existing organizational structure and culture.
14.2 Why Is Hybrid Agile Important?
Hybrid agile is important because it provides a pragmatic approach to adopting agile principles and practices that can be implemented in situations that are less than ideal. We come across two common reasons why hybrid agile may be the only viable approach. The first is that you have a project that really must embrace a non-agile approach, as is the case in situations where requirements must be well defined and understood up-front. The second is a situation where agile would work just fine, but management will not agree to allow you to fully embrace agile development. This latter scenario is quite common in many of the organizations with which we consult. The degree to which your approach can adopt agile practices may vary. Hybrid agile may not seem ideal, but it is often the best approach for many real-world scenarios that you may find yourself having to manage (in a less-than-perfect situation).
In order to be successful, you need to understand where you are starting from.
14.3 Where Do I Start?
Always begin by understanding the situation and the culture in which you are trying to work. Is the business problem that you are trying to solve one in which agile is just not appropriate? Is your organization completely opposed to a non-agile methodology, making it unlikely you will be able to change senior management’s view on how projects are managed? Alternatively, you may actually be working toward winning over your management by convincing them that agile is a viable approach and simply experiencing the stretching typical during any transitional period.
After assessing current hurdles, it is time to create a plan for adopting agile in a way that is aligned with your environment and organizational culture.
14.4 Pragmatic Choices
Making tough choices is part of everyday life. Many of our colleagues write books that focus on developing software that seems best suited for a perfect world that, more often than not, seems to us to exist only in their imaginations. We find that ivory tower–designed methodologies, while theoretically very interesting, are not always easy to implement in the real world—and the somewhat messy environments—the rest of us worker bees inhabit. However, most organizations, from banks to healthcare providers to defense contractors, do indeed exist within a less-than-perfect environment, and we suggest that good software methodology can play a huge role in helping manage those situations, benefiting the organization with great software that helps drive the business to success and profitability. We note that quite a few companies do represent themselves as having a great approach to delivering software. Some of these firms do not have the regulatory and compliance requirements that we see in large banks and other real-world firms. So although their approach might work for them, it is not always portable to other companies, and even if it was, their approach might not align successfully in a different corporate culture and ecosystem. The pragmatic approach is to understand your circumstances and design a methodology using all available industry best practices that have been shown to be successful in your situation.
Being pragmatic means that you understand fully what agile principles teach and what agile development brings to the table. Some folks take a seemingly hybrid agile approach because they are just ignorant of agile principles, and that is not what we are advocating at all. In fact, it could be argued that hybrid agile requires a stronger knowledge of agile because you must understand precisely what aspects can be tailored and adjusted to the circumstances within which you are operating. Done well, we have seen some advantages to a hybrid agile approach.
14.5 The Best of Both Worlds
Waterfall does have some distinct advantages. We have heard technology managers bemoan not having spent more time understanding complex requirements up-front and wishing that their project had not taken an iterative approach so early in the lifecycle. Agile, despite having some weaknesses and notable failures, also boasts several advantages that lead teams to a great deal of success in their development efforts. Our advice is to try to understand the choices that you have and to design a software methodology that matches the environment you are working in. This might mean that you spend a little more time up-front working on developing requirements, knowing full well that they may change. Designing a life support system or the software for a nuclear power plant requires a great deal of discipline initially to define requirements. We have had engineers from both of these disciplines in our public classes embracing agile practices, but from within their own development framework, with all of its regulatory and audit requirements. Sufficient understanding of both approaches will enable you to tailor an ALM that works within your world. We also think that your methodology may necessarily grow and adapt over time, and that might actually be the best choice for the world in which you are working.
14.6 Keeping It Agile
The transition to agile can be a long journey for many companies. Keeping your approach agile means that you apply many of the principles that you expect your team to adopt when they develop code to the way in which you adopt agility itself. Changing how your team works is not likely something that can be completely transitioned overnight. Certainly training, mentoring, and coaching are extremely valuable—and often essential for your success. But, even in the best of circumstances, you may find that your transition to agility cannot be described as flipping a light switch such that you turn agile “on” or “off,” but rather a transformation that takes some time and quite a bit of effort, no doubt with a few mistakes and failures along the way.
The successful agile transformation often depends largely upon choosing the best pilot.
14.7 Establishing the Agile Pilot
Choosing the right pilot for your first agile project is an important step. Most often, we hear that it is really best to start with a new project that does not have any existing methodology “baggage” in place. We do not disagree with this choice, but note that we have come across highly motivated teams that did just fine transitioning their existing practices to agile methodology over time. You should always consider the experience and skill of team members when choosing an agile pilot. But frankly, you should especially focus on their culture in terms of collaboration and teamwork above all else. Be wary of teams where one or two members have strong dominant personalities and there is a lack of excellent teamwork, especially in terms of respect for each other.
14.8 Transitioning to Agile
We often see teams that are transitioning to agile, but since projects are already in play, they cannot easily just switch over. They may have deliverables that they have already committed to completing within a waterfall-based project. Transitioning to agile can be difficult and, in many ways, a balancing act. We have seen many situations where the senior managers insisted on deliverables that they were accustomed to receiving while the mid-level development managers were actively trying to help the teams transition to agile development. Missing a deadline will do little to increase confidence in your team and your approach, so we certainly believe that you should meet your deliverables as promised. But also start to look for opportunities to socialize and communicate your less visible transition to agile in terms of what you are doing to improve your own software development process.
For example, regularly scheduling demos of the systems that you are developing and reviewing the contents of your backlog may help enhance management’s confidence that they will get a completed product on or before your deadline. Just as communication is important within your scrum, so, too, is managing the information being delivered to the senior managers who are responsible for approving your budget.
14.9 Having a Baby
Some endeavors take a specific amount of time and cannot be divided into sprints. Having a baby is one of them. Creating certain hardware and circuit chips is another. Many of these efforts require software or firmware to be developed in parallel.
Acknowledging that some tasks just take a fixed amount of time is important. Equally important is realizing that many of us find ourselves working in environments that are not optimal. We acknowledge that we are perhaps called in more often when these challenges exist, but the elephant in the room is that most organizations have environments that are less than perfect.
14.10 The Elephant in the Room
Although many of my colleagues will not admit this, the proverbial “elephant in the room” is that many, or perhaps most, organizations are working with agile in atypical situations. We have discussed hybrid agile and that senior management may indeed place demands upon any project, which means that we will have to meet their requirements. Living in an imperfect situation is the norm, and that is exactly the world within which we must produce results.
We are often compelled to define requirements in a waterfall manner. We also may have to employ a project management strategy that has much more ceremony than one would expect from an agile process. One important governance issue is management wanting to know exactly when the project will be completed.
14.11 Are We There Yet?
Knowing when a project will be completed is a common requirement. We see many project budgets in the millions of dollars, and frankly no one is going to give you that much money without adequate governance over the project in terms of how the money will be spent, including a clear schedule on when the work will be completed.
Many development managers may feel that senior management’s requirements for accountability are unreasonable, but the truth is that many agile projects have failed. You need to take a pragmatic approach as to how and why agile projects can fail in order to avoid the most common causes.
14.12 Agile Disasters
The most common problem we have seen is code from a release iteration that was not at a point where it was really ready to be seen being delivered to a customer prematurely. Far more serious, however, is releasing code that has just not reached a state where it meets quality standards.
Larger projects may be at greater risk for disaster. There have been a number of very large-scale projects that did not meet their goals, even after spending their budget and burning through their development schedule.
Although there may be significant challenges, many developers successfully manage to operate in these situations.
14.13 Developer View
Developers are a resilient bunch, and we often see agile development being successfully conducted under even the most difficult circumstances. Agile in the trenches, even under these circumstances, often involves agile practices such as scrums, requirements tracked in user stories, and information radiators. Developers often do a great job of making their development environments agile even under the most difficult circumstances.
We also have run into circumstances where corporate rules directly impacted the agile practices that could be adopted by the team.
14.14 No Information Radiators Allowed
We worked in a bank where there were many rules and regulations. The corporate security department did not allow any work products to be left out on the desk after hours, much less any diagrams or posters related to systems architecture or other technical requirements. Agile developers often make information radiators that display important information about the project, its status, and its deliverables. In this company, we were not permitted to have any information radiators on display after hours for fear of corporate espionage. The rule was so strict that we could not leave business cards on our desk, even if we gave them to other colleagues all the time. Sometimes corporate rules can get in the way of the most basic things.
14.15 Waterfall Is Iterative, Too
We like to remind folks that waterfall as it was envisioned by Winston Royce was iterative by design. This means that the original view was that analysts tracked requirements, designed the system, coded, tested, and started from the beginning to iteratively develop a system that met the users’ requirements.
That is not to say that waterfall is not without its challenges by any means. We have seen folks have to pretend that they understand requirements before it was conceptually possible and to make estimates about new technology that were completely impossible to guess at up-front. However, the waterfall methodology itself should not be blamed for this dysfunctional behavior; rather, the way in which it has been implemented in many organizations is the real root of many problems. There are many circumstances in which defining requirements as thoroughly as possible is an absolute must-have.
14.16 Document Requirements as Much as Possible
The best approach for many projects is to define as many of the requirements as possible while simultaneously identifying risks and other open issues that just cannot be understood and specified up-front. In this regard, having more requirements defined is better, but one should never be forced to lie about what is known. This tends to be one of the biggest problems with the waterfall approach. The folks who have made Lean methodology so popular in software development have come up with a clever solution to address this dichotomy.
14.17 Last Responsible Moment
Many of our colleagues in Lean development have used the phrase “last responsible moment” to describe the practice of waiting to make decisions until all, or at least most of, the essential information is available. Waterfall often results in analysts trying to define requirements about things that they do not completely understand. Putting off the final decision until more information is available results in better decisions and better designs. Even if you are using a hybrid approach, never define requirements until you have the information that you need to specify them correctly. Taking an iterative approach and seeing prototypes of the system being constructed often enable you to make better information decisions. It also helps you identify and mitigate technology risk.
14.18 Technology Risk
Technology risk is inherent in any development effort, especially when the team is using newer technologies that have not previously been implemented by this team. The hybrid agile approach can facilitate iterative development and allow the developers to acquire some valuable experience by building prototypes. Even if the team is not embracing fixed timebox sprints, technology risk should be identified and a strategy specified for mitigating this risk. In practice, we always like to learn a new technology by iteratively developing components of the system in as simple a way as possible.
14.19 Understanding the Ecosystem
Development processes must align with the culture of the organization and the environment within which the company must compete. Understanding the ecosystem can help you identify the most appropriate approach to enhancing productivity and quality.
14.20 Mature Agile
As we discussed in Chapter 4, mature agile processes should consider the whole lifecycle and all of its requirements. Although we certainly value working software over comprehensive documentation, hybrid agile is often focused on the so-called items on the right, including comprehensive documents, processes and tools, contracts where needed, and, of course, having a pragmatic approach to planning.
14.21 Meeting IT Governance Requirements
IT governance focuses on ensuring that senior managers receive the information they need in order to make good decisions. The information could be project status, sales and profitability, or even identified risks that should be raised up the chain of command. Many companies will not adopt agile, specifically because they have an IT governance framework in place that is well established throughout the enterprise and management decides to compel each group to continue to provide information in this standard way.
14.22 Conclusion
Many technology professionals who would like to embrace agility must function within an organization that has considerable non-agile structures in place. This situation is actually more common than you might think. Sometimes non-agile processes exist in organizations that are just beginning their agile transformation. Other times, non-agile processes are part of the established fabric of the organization and are unlikely to go away anytime soon. Implementing agile practices in a non-agile environment may not seem ideal, but nonetheless, it is the world that many of us actually live in.