In the traditional software organization, the emphasis in software development is on features, bug fixing, and the plan. Development proceeds in a linear fashion, beginning with analysis and design, the development of a plan, coding, testing, fixing of problems, careful tracking against the plan, and then use by customers.
This linear (waterfall) approach to software development is still the primary method of software development taught in schools. But this is an engineering approach to software development that isn’t suited to software development as described in the Introduction.
At least part of the reason software firms are rarely able to achieve sustainability is because software developers do not receive an education that prepares them for the real world. In the real world of software development, software must be developed in a complex and changing ecosystem of technology, competitors, and customers. And it isn’t until programmers are first exposed to the commercial world that the size of the code base they work on and the size of the teams they work in become significant. This is also the first time programmers have to write code and live with it for any meaningful length of time. Not just their code either—with personnel turnover there comes an increasingly large body of code that nobody on the current team wrote.
It’s unrealistic to expect the education system to duplicate real-world conditions. This is because the education system needs to teach students their craft, which must be taught in a controlled environment so students aren’t overwhelmed. But knowing a craft is only one part of what is needed. In order to provide a bridge from the academic to the practical worlds, we need a mindset that can be used to create a culture of software development. This should be the culture of sustainable development so that teams can achieve and sustain a high pace of development over the long term, despite all the surrounding complexity, without heroics. Otherwise, it’s too easy to burn out on the profession.
Without the right mindset and culture, too many people try and fail with the methods they are taught. They then fall back on ad-hoc variants of the traditional linear approaches because the linearity provides a degree of comfort that progress is being made while also being familiar. But by being ad-hoc, there is either going to be too little structure (which is code-then-fix development) or too much, which applies too much bureaucracy in an attempt to keep the process under control. And bureaucracy tends to stifle innovation, flexibility, and change tolerance—the very properties that a dynamic and changing environment requires.
The bulk of this book, and most books on software development, are filled with practices and rules. Practices are too often treated as templates for success, when the reality is that practices must be tuned to the situation and team. And rules are a result of bureaucracy that is put in place to try to make software development more predictable. Predictability is often achieved, but at the cost of innovation and the ability to respond to customer requests. If your team implements a given set of practices but doesn’t have the right mindset or culture that supports the practices, you’re not going to get far. You will be better off than doing purely ad-hoc software development, where you will be good but not great.
Before your team talks about development practices you need to first focus on the principles you are going to follow. These principles should guide you every day in making tradeoffs and in fostering the right mindset and culture for your team and organization. The practices will follow from the principles you choose to adopt.
The difference between principles and practices is analogous to the difference between vision and features in a project. Unsuccessful projects start by defining a set of features that when put together are surmised to meet some user need. These projects are technology looking for a market, and the odds of their success are low. Successful projects, on the other hand, start with a clear user need and a vision of how to meet that need. The vision is simple and can be used to develop an initial set of features and a prototype that can be put in front of customers for feedback. The feedback is used to refine the feature set, which is then put in front of customers for more feedback, and so on. This is iterative development, as it is called in agile software development. The following are key aspects in the vision-oriented approach to product development:
A user need and vision so it is clear what is being built
Rapid refinement to adapt to change
A close relationship with users so they can provide timely feedback
Continual learning to refine the product so it best meets user needs while avoiding unnecessary features
Too many people are looking for a magic set of rules or list of practices that define the silver bullet of software development. Our education and upbringing reinforce this desire by using rules and step-by-step procedures to help us solve problems. We run into trouble, however, because we rarely recognize the role of complexity in the application of rules. Rules work best when the problem is simple but break down as complexity increases. But software development, and product development in general, is not simple.
In sustainable software development, the principles are like vision and the practices are like features. Principles define what you want to accomplish and the practices how you want to go about it. Too much of the mantra in software development is around sets of practices. We hear “follow this set of practices” or “if you aren’t following these practices, you aren’t using this technique.” If only it were that simple! These practice-oriented approaches put the cart before the horse by defining the features before the vision. While there is no doubt that in every case the practices are valuable and can be learned from, if you don’t know what you are trying to achieve, then your efforts are going to be misdirected.
The problem with practices and rules in the face of complexity, as in software development, is that it is too easy (and human) to lose sight of where you are going and what you want to accomplish, get bogged down in the details, and flail around. The most common response to flailing around is to apply bureaucracy, to define a set of strict rules and milestones that cannot be deviated from. But bureaucracy stifles innovation and is most often misguided, leading to unintended side effects. It is also inflexible and change-intolerant.
Some practice-oriented approaches provide a simple set of practices that are intended to lead to sustainable development. But, by stating their approach as a set of hard-and-fast rules, they mislead people into believing they are complete, when in fact they are not. Good teams apply these practices and can achieve success, but the chance of failure is still high because of the complexities inherent in software development and the people who develop it. Also, by stating that the practices must be adhered to, a different sort of bureaucracy or elitism develops where people say silly things like: “I won’t work more than 40 hours this week because that is one of our practices,” even though there is a critical deadline in the near future, or “We never design aspects of our architecture because we only design what we need today,” even though it is plainly obvious that a bit of extra work today would make tomorrow’s work, that everyone recognizes needs to be done, an order of magnitude easier.
Great teams recognize that what they are trying to achieve is more important than how they achieve it. They listen to the practices that are proposed to them, ask questions to understand what they are trying to accomplish, and then apply them as needed. Great teams see through the strict rule-based façade and are successful because they see rules as a solid foundation to build on. They work from a clear user need (high user value through features and high quality) and a vision (the principles) and continually learn and refine their approach (the practices) as their project progresses. This is what underlies sustainable development. To summarize, there are three elements to sustainable development:
A vision and set of guiding principles [1] that define what needs to be accomplished.
Practices that are consistent with the principles. The practices outlined in this book are not a set of rules. Instead, they are intended as building blocks that your team should refine and add to over time through continual improvement of your approach to software development.
A tight feedback loop of experimentation, learning, and refinement so that during the development of the project the project team is able to change, modify, and enhance their practices as required so the team can best fulfill the principles (vision).
Given a vision and desire to achieve sustainable software development, the following principles apply:
Continual refinement of the product and project practices.
A working product at all times.
A continual investment in and emphasis on design.
Valuing defect prevention over defect detection.
Each of these seemingly arbitrary principles contributes to sustainability. Since the rest of this book outlines practices that support these principles, let’s examine each principle in turn.
Continual refinement is a recurring theme in sustainable development. Software is complex and must evolve continuously over a long period of time in the face of changing user requirements and an evolving ecosystem of markets, partners, technologies, and competition. At the start of any project it is impossible to plan for all the required changes. Therefore, teams must plan for change and implement practices that enhance their ability to change. These practices must cover how they manage their projects, how they design their products, and also the practices used to develop the products themselves. To accommodate these changes, teams need user involvement and an emphasis on reflection, learning, and experimentation but also lightweight practices to keep themselves on track and monitor progress and risks.
Continual refinement is required for sustainable development because it defines a set of practices that establish a heartbeat for a project. This heartbeat establishes a steady rhythm that is used to keep the project going and continually monitor progress in a lightweight way.
A software product should always be in a working state. This doesn’t mean it is always functionally complete, just that it works and has high quality. The longer the product is not working, the greater the chance that quality is degrading every working minute, and the greater the cost to get the software working again before completing the project. The goal of software teams is to ship their products, and by keeping a product in a state as close as possible to being shippable, the easier it will be to ship.
A secondary meaning to this principle is that the emphasis of the team is on producing a working product and shipping it, not on producing documentation of user requirements, software designs, etc. that might lead to a product. This is a key principle of agile development: The best way to get user feedback is to give users something they can really comment on, a product that is written to fit their needs even if it is only work in progress. Documentation, even if carefully reviewed, does not elicit the same quality of feedback as something that users can actually use. Even prototypes are better than a document.
A working product is required for sustainable development because it ensures that time is spent on productive activities. Effort spent getting the product back to a working state is wasted effort and a missed opportunity to be doing valuable work.
Design is a key differentiator between products. Both good design and bad design are immediately obvious and can heavily influence purchasing decisions, but the most important aspect of design quality in terms of sustainable development is on the productivity of the development team. Design decisions and changes are made every day and by every member of the team, and the team needs to allow itself to make design changes as appropriate throughout the project. Traditional software development emphasizes trying to design the software to anticipate every possible future use at the beginning of the project or prior to the writing of any code. This does not work. Agile software development advocates designing only what you need today while not making silly decisions that close the door on future enhancements. Contrary to what some people believe, agile development actually requires a lot of design, perhaps even more than traditional approaches.
A continual emphasis on design is required for sustainable development because good design and sustainable design practices extend the life of the product by keeping the implementation in a healthy state that enhances maintainability and changeability while ensuring there is a simple long-term design vision.
Defect detection involves trying to find and fix defects after changes have been merged into the production software. Defect prevention, on the other hand, emphasizes catching defects before the changes are merged into the production software.
The key symptoms of a defect detection culture are an undue reliance on defect tracking systems, manual testing efforts, and large numbers of defects that reach customers.
In the typical defect detection organization, organizations invest a great deal in a QA or testing organization. They have elaborate defect tracking systems in which members of the organization spend countless hours sorting, prioritizing, and assigning defects for fixing. And yet regressions (defects fixed or features introduced in previous versions of the product which no longer work) are still introduced in new product versions as features are added and bug fixes are made. The quality onus is primarily on QA and testers, and in many organizations a virtual wall is constructed between developers and QA. Developers produce new versions of the software, and then throw it over the wall to QA, who test it and then accept or reject it. The software may take weeks or months in QA before the product is ready to ship to customers.
In a defect prevention culture, defect-tracking systems are still required, but their importance is greatly reduced. This is due to much lower defect counts because development teams have multiple safeguards in place to ensure that regressions are never (or rarely) introduced in already existing functionality and that new features and bug fixes are properly tested and have safeguards that ensure they will not break in the future. A heavy emphasis is placed on automated testing so that people (QA and testers) can focus on testing the product in a realistic manner, in ways that mirror what actual customers will do with the product. There are no virtual walls in a defect prevention culture because ideally the testers are part of the team, and the quality onus is spread evenly between the developers and QA and testers.
An emphasis on defect prevention is required for sustainable development to ensure the development team is highly efficient and is putting its effort into creating value for the customer, not on wasted effort. Also, by minimizing the number of defects that reach customers, development teams are able to have productive conversations with users about features and workflow because users are able to use the product in realistic ways.
Striving for Sustainable Development Is Like Juggling
I like to use the metaphor of juggling when explaining sustainable software development. Striving for sustainable development is like juggling the six balls shown in Figure 3-1. Think of the entire team juggling together, like many minds controlling one pair of arms and hands.
Figure 3-1. Software development is like juggling the four balls of sustainable development at the same time as the much larger balls that represent features and bugfixing. Instead of just juggling the two large balls, sustainable software development recognizes the genius of the AND and results in the ability to juggle all six balls at the same time, even though they are different sizes and weights.
The four small balls, each representing an element of sustainability, have a subtly different color, texture, and weight. If you drop one ball, you’re still juggling but your outcome isn’t going to be as good as it could be, and your audience (the customer) is definitely going to notice the mistake. Juggling demands you understand each ball, its shape, weight, color, and texture, and that you are able to focus on any one ball while keeping all four in focus.
The larger balls that represent features and bugfixing are completely different in shape and weight from the small balls. The complexity of the task of juggling the four small balls and two large balls usually leads teams to juggle only a few balls because that’s much easier.
The team may juggle just the large balls. This leads either to the purely ad-hoc code-then-fix approach to software development or bureaucracy, consisting of milestones, rules, and mandatory documents. Unfortunately, these approaches ultimately lead to unsustainable development.
Or the team might juggle one or more of the small balls so they don’t have to deal with the large balls. They’ll produce awesome designs or have a working product or continually refactor their code, but they won’t get anything done. And if they manage to ship and have customers, chances are the customers won’t be satisfied.
Great software teams are able to juggle all six balls. They understand that juggling the small balls comes first because they are more predictable, and this makes the task of juggling the large balls much easier. This ability to accomplish the genius of the AND in the face of the tyranny of the OR leads to sustainable development while capturing the complexities required to do so.
Sustainable software development requires a certain kind of culture. While it is hard to completely describe the desired culture, it is possible to help focus on what that culture is by using some descriptive words and phrases.
The team must be highly disciplined in its approach and goal focused. Discipline is what the team uses to ensure that all the work is done properly and with no shortcuts. For example, discipline helps teams ensure that their work is really done when they say it is.
Unfortunately, agile development has been labeled as being undisciplined by some. Yet any team, regardless of its practices, will not succeed unless it is disciplined. What I suspect is happening is that there is confusion about the difference between discipline and ceremony. Ceremony in this case refers to the required activities that must be performed in a project: for example, a requirements document before a certain milestone, milestones such as a feature freeze, weekly status meetings, design documents, or mandated beta programs. Agile teams minimize ceremony in favor of keeping to the task at hand and producing results (a working product), while more involved processes emphasize ceremony and documentation over the results. In many cases, unfortunately, people mistakenly believe that unless documents are created there is no discipline. This is a mistaken belief however, because these documents are frequently wasted effort and are no guarantee that a prescribed process is being followed.
The team must be responsible for its work and not play the part of a victim. Victims go along with anything and don’t rock the boat, then absolve themselves of any responsibility so they can lay the blame elsewhere. There are actually very few situations in life where you are truly a victim. In most cases, the victim is just as susceptible to his or her own line of thinking and unwillingness to take charge of the situation as they are to the situation. For example, if a decision is made that a responsible person knows is wrong, that person doesn’t just go along with it, he or she ensures that everyone is aware of the consequences so that the decision is either made with the right data or is changed. Victims, on the other hand, go along with the decision and then say, “I was told to…,” even though they knew it was the wrong thing.
Responsible people have courage to take on any situation and do not get swept up by events. They trust others to do the best they can and are always diligent to ensure that nothing falls between the cracks. They don’t just say, “that can’t be done.” They explain why and then help uncover the real problem and provide a viable alternative.
The team has multiple leaders who work together, and the leaders are not just the managers, team leads, or coaches. See Chapter 8 for more ideas on leadership in sustainable software development.
I also believe that an attempt to build a successful culture is enhanced by the presence of at least one level 5 leader [Collins 2001]. These are the individuals who recognize that the greatest success is when they can help others succeed and give them the opportunity to receive the accolades they deserve, who view success as a team accomplishment and failure as a personal responsibility, not an opportunity to apportion blame on others. These people are few and far between and are unfortunately rarely promoted into senior management because they often lack the charisma, high-volume talking, and massive ego and self-confidence; the traits most often associated with getting ahead in the corporate world.
Effective teams strive for the optimal balance between being visionary (thinking about the long term) and tactical (focusing on today’s problems), and this balance changes over the course of the project. In the early phases of a project there should be a heavier emphasis on being visionary, and in latter phases the emphasis should be placed on tactics related to finishing the project. Projects fail when there is too much emphasis on being visionary late in the development cycle or when there is too much emphasis on tactical issues too early in the project. Great teams are able to balance the two at each moment in the project.
In sustainable development it is important that everyone on the team shares the same sense of urgency to complete the project. If the business people are worried about the competition and the developers are more concerned about the perfect architecture, failure will result.
A shared sense of urgency is also the most important factor in building teams. It isn’t compatible personalities or team building exercises or even necessarily mutual respect that make the best teams, it’s a shared goal or aggressive but achievable deadline that the team uses as a rallying cry for getting the project done.
Collaboration is a vital success factor in sustainable development. Good teams are able to work together, embrace the differences between personalities and roles, and strive for a successful project. They also recognize that there are clear roles for each of them to play. Businesspeople don’t write software and programmers don’t write business plans. Collaboration should be like breathing; it is always there and is done almost subconsciously. And it should be done in person (as opposed to e-mail or voice mail) as much as possible because that is the most effective form of collaboration.
Great cultures recognize talents and skills as separate from each other. Too many companies are focused on skill alone and recruit only for skills. But skills can be taught, whereas talent can’t. Software development is a perfect example because people who are talented at complex problem solving make excellent programmers. Excellent programmers can learn new programming languages in a short period of time and will over time outperform their peers. And yet, most job ads in our industry still look for a certain number of years of experience programming in a particular language when what they should be looking for are people who are talented problem solvers.
Continual improvement and learning is a vital element of sustainable development. Great people and teams continually learn and refine their projects and how they approach the projects. They use short time increments (many times per year) to ensure their pace is high and lightweight and use techniques to monitor progress, keep themselves on track, and introduce changes as required.
Great teams understand that change is necessary and desirable. They also understand that change must sometimes be made in the face of conflicting or incomplete information. However, they also realize that just because a decision is made at one point in time does not mean that it can’t be or won’t be modified as new information is available. Hence, being change tolerant also means that sometimes some extra work must be done at the current time so that change tolerance is possible in the future as well.
Risk is a constant element of all projects. Effective teams are always aware of risk, are able to differentiate between real and perceived risk, and are not afraid to make decisions with the best information possible in the face of risk. They are also constantly looking for ways to reduce or alleviate risk.
You can’t always choose what you do, but you can choose how you do it [Lundin et al 2000]. Good people and teams recognize this. They find ways to make work more fun for their co-workers and customers.
Mindset and culture are more important than practices. Unfortunately, culture is not something that can be turned on and off like a light. Luckily, there is a spectrum of possible ideal cultures. What might work in one environment and with one set of team members will not necessarily work in a different environment with different people.
Creating the ideal culture takes a lot of hard work, and it takes people dedicated to making it happen—good people. The vast majority of people want to succeed and have the tools required to do so, if they are put into the right environment. There must be a shared sense of purpose and vision, and this is why there must be an effort made to getting the people with the right attributes onto the team and others off the team, or on the bus as in Good to Great [Collins 2001]. It’s how the people in the team apply themselves to creating the necessary environment that separates the poor team from the average from the good and from the great team. There is an element of luck and being in the right place at the right time with the right people. Applying the principles and practices outlined in this book and thinking about your desired culture should get your team well on the way to a thriving, positive culture.
The rest of this book describes some of the recommended practices that support the four principles of sustainable development. There is a chapter for each principle, and each chapter explains in detail the rationale behind the principle and the practices that help to make it real.