In order to achieve sustainable development, teams need a way to balance short-term requirements and long-term needs: to ship their product as soon as they can while accepting, anticipating, and even welcoming change.
Software is complex and must evolve continuously over a long period of time. Throughout its life cycle, a software project must contend with changing user requirements in an evolving ecosystem of markets, partners, technology, and competition. In this environment, just about the only thing a software team can rely on is that change is a given. While some changes can be anticipated, teams must accept that most cannot; there are no crystal balls to predict the future. Experience with a particular software ecosystem can help, but it is still no guarantee for success. Therefore, teams must adopt a mindset and related practices that help them easily adapt to change as it happens while also anticipating change. The core agile software development practice of iterative development (described below) encourages continual refinement because it is a lightweight but formalized practice that helps teams adapt to and anticipate change.
In agile development, teams work from a simple and clear vision and deal with change through frequent iterations; the goal of each iteration is to deliver something useful to customers. Iterations begin with the team’s laying a plan for upcoming iterations but with a focus on the next iteration. This planning takes into account the most recent available information such as an understanding of what the team has accomplished in past iterations (velocity, described below) customer feedback, or technology changes. The team then executes on the plan for the duration of the iteration, usually just a few weeks, delivers the software to the customer, and then takes a few moments to reflect and learn how it can further improve before planning and starting the next iteration. The agile methods keep the process as lightweight as possible so that the focus is placed on achieving results, while the tight feedback loop with customers encourages teams to do only what is most useful to the customers, which helps minimize wasted effort.
The main advantage of agile methods is that they help teams manage uncertainty and change early and throughout the project. However, while all agile projects combine aspects of anticipation and adaptation, poorly implemented agile methods can result in sloppy planning and reactive thinking [Highsmith 2004b]. Sloppy planning results when teams ignore information they already know or should know with minimal effort. Reactive thinking results when too much emphasis is placed on the team’s ability to adapt to change. The result could be a project where lots of changes are being made but the amount of actual progress is low. Hence, the challenge with agile methods is to rely on both adaptability and anticipation and to have a team that knows when and how to apply each, and in what measure.
Traditional methods of software development rely too heavily on upfront planning and anticipation and virtually ignore adaptation. These approaches stress attempting to understand all user requirements and producing a detailed design before writing any software.
Agile methods are not the exact inverse of traditional methods; the common agile statement “we have more confidence in our ability to adapt than our ability to predict the future” should not be an excuse to ignore anticipation. Instead, in agile methods there must be an understanding of good enough anticipation. For example, spending a few days at the beginning of a project understanding user requirements or spending an hour every day discussing software design are worthy activities because they can cut down on rework and even, if done right, increase the ability of the team to adapt to change. The latter topic, especially as it applies to design, is dealt with in greater detail in Chapter 6 on Design Emphasis.
The need to involve both adaptation and anticipation in agile methods is similar to the need to consider both the short and long terms. In software projects, if too much emphasis is placed on the short-term, then software quality will suffer because the team will be continually reacting to crises and not being proactive. This will result in a continual degradation of the ability to adapt to change as described in Chapter 2. Equally as dangerous is when too much emphasis is placed on the long term (i.e., anticipation); these are the projects that are continually at risk of never shipping because they are unable to deal with changing user requirements through their emphasis on planning and design.
Iterative development can be thought of as plan a little, do a little, learn a little. Iterative development is a core practice of sustainable software development because it gives the team a lightweight way to make measurable progress and to get feedback from customers as frequently and as early as possible. Getting feedback as early as possible is critical to ensuring that the system meets user requirements, that it is as simple as possible, and that there is minimal wasted effort. Minimizing wasted effort (features that users don’t care about, features that are built in unusable ways, etc.) is an important business consideration because wasted effort has a cost and is inefficient.
Iterative development is amazingly simple. The schedule is broken up into a number of short equal-length iterations (2 to 6 weeks). At the end of each iteration, the software must be in a shippable state because it is given to customers for feedback. Each iteration has a little bit of planning, some real work is done, feedback is received from customers, the team records its progress to date, and reflects on how they can improve in the future.
Iterative development allows the team to replan and reprioritize at the beginning of every iteration. Planning is done in a lightweight but very effective manner using feature cards, with one feature card per feature. A feature card is a simple 3.5”x 5” index card. Because each iteration requires customer feedback, the team is able to keep its feature cards sorted into those that are done, those that are going to be worked on in the current iteration, and those that are going to be worked on in future iterations. Uncompleted features are arranged into a number of future iterations, and as the project proceeds, features will be added, removed, and reprioritized by the customer.
In iterative development, the feature cards and their arrangement into a number of iterations is the project schedule. The cards should be posted onto a board in a place that is accessible to the team. There are no detailed project schedules (PERT or Gantt charts), user requirement documents, etc. Instead, the feature cards contain all the documentation required to begin the work: a feature description[1], a time estimate, key acceptance criteria, risk (high, medium, low), and just about anything else that is useful to the team that will fit. One way to look at feature cards is that they are collaboration enablers: The cards themselves are less valuable than the conversations (and collaboration) that result from filling them out, moving them around, and understanding them. Hence, feature cards are meant to be barely sufficient, so that the team’s emphasis is on producing a working product, not a stack of documentation.
Iterative development is important for sustainable development because it provides a regular heartbeat or checkpoint to the development effort. The regular rhythm is good for the team because it breaks the effort down into manageable pieces; it’s good for project stakeholders and customers because they know they can expect to see progress on a regular basis [Cohn 2005]. Furthermore, since the goal of iterative development is to get feedback, teams are encouraged to keep the software in a working state; if the software is unreliable, then customers won’t use it and can’t give meaningful feedback. This topic of a Working Product is the focus of Chapter 4.
Iterative development is most effective when there is broad participation in iteration planning. At the beginning of each iteration, the entire team (business people, technical staff, and customers) has a short and intense iteration planning session. In this planning session the team analyzes and sorts the current set of desired features into an iteration plan based on risk (do the risky things early) and priority (determined by the customer). Tradeoffs are made through conversations within the team during the planning session.
Cross-functional collaboration is important to sustainable development because there is no hiding risk from any of the participants. Hence, active risk management is encouraged. Also, because it is a team exercise, everyone involved understands the current product status, what still needs to be accomplished, and also what can be accomplished in the time available. And because the planning is focused on providing value to customers real tradeoffs can be made, for example, between new features and making existing features work better for customers.
Iterative development is a humbling (in a good sense!) experience for the entire team. It’s always easy to get enthused about a particular feature, but you can only introduce it once it is compared to the other features already in the plan. Collaboration helps to ensure that the right tradeoffs are made and the best possible priorities are set. There is no hiding the harsh reality that when someone wants to add a feature, in order to do so, another feature has to be removed from the plan!
A key component of iterative development is the tracking of the project’s velocity. Each feature card documents the time required to complete that feature in feature points. Feature points can be relative or absolute, though most people seem to prefer that feature points roughly equate to the number of ideal days of effort by a canonical developer. The card also documents the customer acceptance criteria; when these criteria have been met, the feature is completed. The velocity essentially is a record of the number of feature points completed in any given iteration.
Velocity is tracked throughout the project and gives an immediate indication of how well the project is (or is not) progressing. Incorporating velocity into the iteration planning sessions the team allows team members to make realistic plans for each iteration and also ask themselves difficult questions in cases where they are not making progress.
Given a record of velocity and the use of iterative development, all that team members really need to chart their progress is two charts that are easily maintained on a whiteboard or web page: velocity (by iteration) and number of feature points remaining. Some examples are shown in Figure 7-1. Although velocity cannot be compared between teams, within a team these two charts provide status of the project at a glance and a realistic gauge of when the project will complete based on the past and current progress[2].
Figure 7-1. These two charts are sufficient for most projects to document and understand the state of their project. The chart on the left is a simple line chart that shows the velocity in each iteration. The chart on the right shows the number of feature points remaining in the project. By simply projecting a straight line and using the team’s current and average velocity to date, the project team can visually determine when the product will ship with the current feature set. In this case, in iteration 7.
Iteration 0 is a short iteration (2 to 6 weeks) that the team can use to perform critical analysis work before starting actual development. Iteration 0 also helps teams identify and document longer term considerations. Iteration 0 is an excellent time to:
Ensure there is a project vision. A project vision is critical for any project, because the team must know what it is building. The vision helps the team make tradeoffs in its iteration planning meetings and also every day as members build the product.
Create an initial release plan (see below). The release plan consists of what the team thinks is the minimum required set of features documented on feature cards and arranged into a number of iterations so that each iteration contains a target total number of feature points that does not exceed the current (or estimated) velocity.
Create guiding principles for the design and the user experience. Guiding principles are described in more detail in Chapter 6.
Create a design vision, high-level architecture, and key components of the architecture as described in Chapter 6.
Write quick prototypes to understand critical features or risky areas better. Prototyping is described in Chapter 4.
In order for iterative development to work, there must be a strong notion of a release. A release is a version of a product that can be installed and used by customers in their production environments. Usually, a release requires a number of iterations to build.
Release planning is important for sustainable development because it provides the longer term view that is necessary for planning purposes. Each project or release starts with Iteration 0 (described previously), where the project vision and a plan that encompasses a number of iterations is laid out.
Once iteration 0 has been completed, the team should move into a build phase. The build phase consists of a number of iterations, with each iteration ending in a review with customers and planning for the next iteration.
During the build phase, the release plan that was created in iteration 0 must be periodically updated. Updates are required whenever the velocity changes or as the team identifies new features or determines that some features are not required. A common problem is that as new features are added to a project, other features are not dropped. This is why it is important to track the number of feature points remaining in the project, because when this happens it will be obvious that the project cannot complete on schedule given the current velocity as depicted in Figure 7-2. This is good because it forces the team to make tradeoffs frequently throughout the project as soon as a problem occurs rather than near the end of the release cycle (some call this panic early, panic often). The need to make tradeoffs also underlines why it is important to have business people involved in the planning, because as with any project, you can either change the date, the feature set, or the resources. You can’t expect to fix all three. When businesspeople are involved in the planning, the types of tradeoffs that are possible during the release plan updates are obvious to them.
Figure 7-2. A case where the release plan must be reworked because all the features can’t be completed on schedule. What has likely happened is that too many features have been added to the release plan and no features were dropped. The team must do one or more of the following: change the date, add some resources, or drop some features from the release plan.
Hold a standup meeting the same time every day that is attended by the entire team—including businesspeople. The meeting should take an absolute maximum of 15 minutes (standing helps encourage this). The daily standup meeting is important for sustainable development because it helps to get issues out in the open each day, which helps to ensure that people aren’t blocked for long periods of time and that everyone on the team is aware of what everyone else is doing. The traditional alternative is typically a weekly or monthly status meeting, and the problem with these meetings is that issues are often raised too late, that they’re too long, and that the critical topics don’t get raised.
In a daily standup meeting, each person answers three questions:
What did you do yesterday?
What are you going to do today?
Are there any obstacles to your progress?
Keep track of time and limit discussion to ensure that the meeting finishes quickly. Invariably, issues come up during the meeting that can be dealt with immediately after the scrum by those who are interested.
Standup meetings are highly effective at gathering status, getting important issues out into the open, and encouraging collaboration among the team. The most effective way to solve problems when more than one person needs to be involved is face-to-face rather than in e-mail; the standup meeting, and the collaboration that happens immediately after, is an aid to the critical issues being dealt with face-to-face immediately.
It is also important that daily standups lead to solving the problems of the moment immediately after the standup is finished. Weekly status report meetings are not nearly as effective because many problems are no longer immediate and the delay is a potential killer of team momentum. Likewise, e-mail can have a similar effect because there is a disconnect between the raising of the problem and any possible solution. Of course, there are always impromptu meetings, and some problems will still require these, but the existence of the daily standup means that team members know there is a forum where they can get these issues out in the open easily.
Retrospectives [Kerth 2001] are a critical part of sustainable software development. In order for continual refinement to be effective, the team must be able to continually learn and adjust not only its project (what the team is building) but also the methods members use (how the team works).
Teams should hold retrospective meetings at regular intervals during their projects. Ideally, this would be at the end of every iteration or the end of every second or third iteration. The purposes of these retrospectives are to provide a forum where the teams can openly question their current development processes and progress, learn from their mistakes and failures, and decide what changes they need to make to proceed effectively.
One of the key aspects of a retrospective is that you must lay out ground rules in advance. It is vital that a retrospective be a positive experience that focuses on learning and not on laying blame. Start by assuming that everyone on the team is doing the best he or she can, with the recognition that there is always room for improvement for every member of the team. Ideally, retrospectives should help uncover areas for improvement and give everyone ideas as to how to contribute more strongly to the project. But retrospectives should never be personal or antagonistic—keep those discussions private and behind closed doors.
Retrospectives are a simple mechanism that helps increase the chance of your project’s success. Frequent retrospectives during your project help ensure that course corrections and changes to your project and processes are made when you need to make them, not after it is too late. When action is required, the next retrospective should uncover what was not completed and why.
Retrospectives also serve as a useful forum to help the team bond because it gives them ownership of their processes with an eye to continual improvement. Team members aren’t being told how to do their work—they are committing to each other how they are going to work. There is a huge difference between being told and making a commitment to your peers!
A simple format for the retrospective is to first ensure that the entire team is present (including businesspeople), select a moderator (ideally neutral, such as from another team), and then answer some basic questions such as:
What worked well?
What needs to be improved?
What should we do differently?
Then, review outstanding action items from previous retrospectives and develop a new set of action items and changes that will be put in place immediately. Be sure to document your results and archive them in a central location so you can refer to them later.
Tip: Be Creative!
There are many creative things you can do to make retrospectives more productive and fun. For example, use different colored sticky notes to capture issues of differing levels of priority. Use a teddy bear or similar object to control who is speaking. Or capture an image of the screen of the product at each retrospective. It can be a lot of fun in the retrospective at the end of the project to review these images and notice how much the product has changed and evolved.
The duration of retrospectives is a variable that must be carefully monitored to ensure they do not become a burdensome overhead. Frequent checkpoint retrospectives that are held during a project should take an hour or less, while retrospectives held at the end of a major release might take a day or more, depending on how elaborate a retrospective is required.
Coaching and team development are important for continual refinement to help the team achieve the best possible results while collaborating together and with customers. Coaching is an interpersonal skill that helps people provide each other with real-time feedback so they can do their work more effectively. Real-time feedback is important for continual refinement because it ensures problems are dealt with as quickly as possible. Team development (understanding the business and professional development) are equally important because the more the team understands the business and has current skills, the better it will be at both anticipating and initiating changes in its ecosystem.
Coaching is an important interpersonal skill for every member of the team. Coaching is essentially providing individuals (people who work for you, people you report to, and people you work with), teams, and customers the real-time feedback they need to become more effective and continually improve. Coaching helps minimize friction and negative tensions while helping people work better together and focus on the desired result.
Understanding your company’s business is an important element of sustainable development. The knowledge gained through this understanding is vital in anticipating and understanding changes to the ecosystem your project is dependent on. While some would argue that software developers should focus on writing code, I believe that this too often results in undesirable side effects, such as:
Team members who don’t understand what is valuable to their customers and why, even when told. This is often because they do not understand how the customers use the product and hence what is important to them.
People who cannot listen to customers and understand what they are really asking for. Understanding what the customer is trying to do is critical because users most often ask for alterations to their current workflow only, when perhaps the problem is the workflow itself.
People who don’t understand what provides value to customers. Sometimes, work that provides value to customers is mundane, but it must be done.
People who don’t understand the economics of their business. How does your company make its money? Sometimes, customers ask for major features. If you always listen to your customers, then they will get everything for free. Your company needs to stay in business, though, and it may be more desirable to sell the customer a new product or add on to an existing product.
In companies with multiple products, especially where there is overlapping functionality, it is common for people on each team to want to add all features their customers ask for to their product when in fact only one of the products may need it. Also, where multiple products serve the needs of a single customer or market segment, there may be detrimental competition between the product teams; these teams need a major wake-up call to realize that the competition they need to be concerned with is outside their company!
People who don’t understand the economics of their markets. Too many technically brilliant products are conceived to address what may be a sexy or obvious opportunity, which unfortunately is not viable economically due to a small target market or unrealistic pricing. Other problems may be misunderstanding the technical sophistication of users and building a solution that is too complex or too simple for them. Or, developers may not understand the economic reality of their customers where, for example, end-users may not value a software product at a realistic level because they (the end-user) are faced with massive costs in other areas that dwarf any potential for software.
Teams that don’t pay attention to their development processes. I have seen too many projects, where due to a whole range of factors such as poor planning and design, sloppy programming practices, a lack of risk analysis, defect backlogs, etc., a project team is required that is too large for the size of the target market. These are projects that are quickly cancelled, because the investment does not justify the returns: i.e., why should a company invest in a project with lower returns than if it was to take the money and put it in a bank account to collect interest?
If the only value you provide to an organization is writing code, then your job can be outsourced (i.e., worked on by cheaper labor).
You need to understand your company’s markets, users, competition, and financials. You won’t always have access to all this information down to the finest level of detail, but you should have enough access to ask intelligent questions or be able to ask questions to get some information. Don’t look upon it as “management’s” responsibility to give you this information; effective communication goes both ways. People in management are almost always too immersed in the details of the business to realize what type of information you need.
My largest observation of people who work in most companies today is that they are too isolated from customers. Try to visit your customers or at least ask people who do visit customers to write up reports or give a short talk about what they learned. Bring your customers to visit you as often as you can.
One of the things I have little tolerance for is a software team that insists it needs to completely rewrite the product. Too many developers don’t understand the company’s need to stay in business. The problem, of course, is that it’s always easier to write code than to read it, especially when you didn’t write it. And it’s no fun just fixing bugs in someone else’s code. But few companies in the software industry can afford to start again from scratch. That’s why teams should be encouraged to rewrite parts of their products as required and aggressively and constantly refactor as recommended in Chapter 6. The alternative is not pretty: While a product is being rewritten, your competition is developing new features while you are effectively standing still. Even if you build in new features, you are still going to be behind simply because of the lost market share. People who get this are highly treasured by organizations (or at least should be!).
Professional development is important for both personal and project reasons [Hunt and Thomas 2000] [Hohmann 1996]. From a project standpoint, teams that are knowledgeable in as many areas as possible, even those that are seemingly unrelated to the current project, have the best chance of succeeding at sustainable development. The reason for this is simple: These teams have a varied set of skills and knowledge, a rich tool chest if you will, that can be drawn from and applied with maximal effect. Professional development is vital to ensure that the team has all the skills it needs to understand its ecosystem of markets, competition, and underlying technologies.
At a personal level, your skills are vital to your team’s success and to yours. It is your responsibility to ensure you are continually improving in your profession. You can’t rely on anyone else to tell you to take courses or read or learn. You have to be proactive. You can’t blame anyone else if your career does not progress as you would like. Read, take training courses, experiment with new programming languages and approaches to problems, and above all learn, collaborate, mentor, and teach.
Every project should collect some key metrics and find ways to make them visible to the project team. Visibility might be achieved by a simple web page or maybe even a big visible chart on a whiteboard or by the water cooler. The reason metrics are helpful is that they can, if they are presented in the right way, help the team focus on keeping important aspects of its project under control. Metrics help the team track its progress on a regular basis and get early warning signs when problems are beginning to develop. Entire books have been written on the subject of metrics, and I think sometimes people get carried away. A small set of simple metrics should suffice, and if their collection and display can be automated, then your team will be in good shape to continually monitor and improve its progress.
An overlooked aspect of metrics is that they enhance learning. By making them visible, they help drive meaningful discussions about the trends and how to change or improve the trends. If done in a positive way, these conversations help drive the collaborative problem solving that is invaluable to the team. Of course, if metrics are implemented in a negative way, such as by tracking individual instead of team metrics, then you can expect the opposite effect.
Some example metrics might be:
Charts of velocity and feature points remaining (as described above).
A chart that shows the status of all system tests, and how long they took to run. This helps to catch system and performance problems early and prevents cases where the system tests don’t run.
A graph that shows the amount of time required for nightly build(s). This helps to keep the builds as small and fast as possible, which is a critical issue to ensure developers are as productive as possible: Slow builds mean lower productivity.
A graph that shows the number of outstanding (not fixed) defects, by severity. Outstanding defects are a key indicator of sustainable development. Teams cannot carry around large backlogs of defects and should fix as they go.
A graph that shows the number of incoming defects, by severity over time. This is an indicator of how well tested the product was. High incoming rates might lead teams to set aside more time for fixing defects in the short term, or perhaps help point to areas of the product that need refactoring/replacement.
A graph that shows the number of fixed defects per week over time.
A companion to the previous two graphs (incoming and fixed defects) is a graph that shows the net gain. The net gain is essentially incoming minus fixed, which should always be around 0 and as close as possible to a flat line.
A chart that tracks the amount of source code versus test code. This gives team members visible evidence that they are adding enough tests as their source code grows.
A chart that shows defect density, in terms of number of defects (on average) per thousand lines of code.
Continual refinement is an important principle for sustainable development because it enables teams to both adapt to and anticipate change in what they are doing (their project) and how they are doing it. Teams break down their projects into short, manageable iterations. The advantage of breaking down work in this way is that complexity is much easier to manage in small slices, and immediate feedback from users provides the teams with a powerful way to change directions if they need to. In addition, teams employ regular retrospectives so that they can talk about their projects in an open and honest way. This allows them to make adjustments to their processes, which is equally important to sustainability because continual learning and improvement help ensure the project will last over the long term.
[1] It takes a lot of practice to write good feature descriptions. See [Cohn 2004] for many good ideas.
[2] Although velocity cannot be compared between teams, the temptation to do so is very real. If people don’t understand that every team and every project are different then it might be a good idea to estimate in relative points instead of days.