It is highly likely that in order for your organization or team to achieve sustainable development, a new product development culture is required. I feel this chapter is necessary because I have seen too many situations where enthusiastic people with great ideas are stymied, consciously or unconsciously, by their organizations.
Too few people consider how to introduce change to their colleagues and management and how to make the changes last. So-called change agents almost always have a great deal of enthusiasm, but they often take on the entire burden of change themselves, hit a stumbling block or opposition, and then do one of many things: retreat into a shell, lash out, get burned out, or find another job. Many organizations, usually unknowingly, have a built-in immune system that can quickly stifle change. But even in these situations change is not impossible if you understand the dynamics of the people in the organization.
Where change is required, there is the significant challenge of achieving lasting change. It’s easy for people to try out something new, but as soon as something goes wrong, the most common reaction is to revert to what was known to work in the past—even if it doesn’t really work. Lasting change requires changes to what is measured, recognized, rewarded, and reinforced. For example, many companies recognize people who work long hours and who are independent project heroes, whereas in a sustainable development culture, it would be more valuable to recognize people who find innovative ways to achieve technical collaboration on their projects and make the people around them more effective.
Achieving lasting culture change is hard. It takes a lot of determination, vision, and persistence. Principles and practices aren’t good enough because people define culture. And people are complex and unique. Therefore, it is unrealistic to expect that a sustainable software development culture can be so clearly defined that it is identical in every instance. In fact, the opposite is true; we should expect sustainable development cultures to be different simply because of the people involved, and these differences will manifest themselves through variations in how practices are applied.
There are two change scenarios of interest in this chapter: sustainable development on a new project and turning a project that is on the unsustainable path into sustainable. These situations are obviously unique and require variations in approach. This chapter is therefore broken up into three parts: the common topic of making change happen, followed by a discussion of the unique aspects of the two scenarios.
Many good books are available on how to make change happen in an organization. For reference, I’ve included some of my favorites in Appendix 4.
I wish it were possible to provide a template for making change happen. Something along the lines of: If you’re a programmer and want to change your organization to sustainable development, do this, then that, say this, etc. My personal experience with leading change has been that every organization is different, that this is particularly true of software organizations, and that software organizations are especially difficult to change. The problem I think stems from many factors, including:
Software people come from many backgrounds. I’ve worked with people who have degrees in engineering, physics, mathematics, astrophysics, computer science, psychology, sociology, English, music, business, and accounting. Many of my co-workers have had multiple degrees; some have even had no degrees.
In many organizations people who understand amazingly little about software development, sometimes appallingly so, wind up managing the organization. Note: I’m not necessarily saying this is a problem. I have worked for some managers who are excellent at knowing what they don’t know and who to ask—although I’ve also worked for managers who don’t, which is a problem. It’s difficult to find good managers who love both business and software development. Usually, people with a sales or financial background and/or people who have written a few lines of code sometime in the past are the ones who wind up in control because they tend to be more extroverted and/or charismatic than most technical people. Management is only a problem when it is bad or incompetent.
Many organizations do not discuss or try to consciously foster their culture and development practices or pay attention to key people issues such as professional development and leadership. Instead, they focus on projects, schedules, and results. While there is no question that results are important, if the bottom line is the main focus then there is a problem.
There are a lot of incredibly bright people in software organizations. On just about any topic of importance there will be no shortage of opinions. This can be both good and bad depending on the topic at hand, and it can lead to situations where creating consensus is difficult or even impossible. However, being bright in a problem-solving sense doesn’t necessarily translate into being a good communicator. Many people in software organizations are good talkers but not listeners, and some are not very good at expressing their opinions in a respectful way. Hence, while having a group of bright people may seem like a good thing, when it comes to making change happen, sometimes the fact that everyone has an opinion and that some people are poor at expressing their opinions means that resistance to change can be quite high. Or, resistance might just seem high because people are usually very good at expressing displeasure but not as proactive as they should be at providing positive feedback.
Software education lacks standards and a poor understanding of what the basics are. Most people learn more about software development on the job than they do in school because only in the workplace are they confronted with the task of maintaining a million lines of code for ten years or joining projects where a large amount of code already exists and the people who wrote it are no longer available. I also know people who have graduated from top universities in computer science who have been well schooled in theory such as NP complete problems but haven’t taken a course in compilers or computer construction!
People don’t like being told how to work. They want to be able to choose the best approach based on learning. Hence, the challenge with achieving culture change is finding ways to achieve complete buy-in and then helping and providing advice as required.
People are different. Consider this basic list of character traits: introvert, extravert, opinionated, stubborn, consensus-builder, excitable, emotional, and restrained. Software developers tend to be introverts. Sales and marketing people tend to be extraverts. Add age as a factor as well: Older and more experienced workers tend to be more jaded by past experiences (or perhaps they are just more relaxed).
Organizations are complex simply because of the mix of different people, for better and worse. What will work in one organization likely will not work in another.
Complacency. We almost always underestimate the effort required to introduce change into an organization. Complacency is often at the heart of failed change efforts because if your co-workers don’t understand the need for change they won’t actively participate.
It’s hard to admit that you have to change, too. We tend to generalize when things get uncomfortable. Driving a car is a perfect example. When asked, we consistently say that there are too many jerks on the road and that other drivers cause the problems. And yet that clearly can’t be the case because someone has to be the other driver. Few people admit that they need to improve their driving skills when actually we should all be continually striving to get better.
Try this experiment with your co-workers. Have all of them cover their eyes and ask people to raise their hands if they believe they are open to change. Then, with eyes still covered, ask people to raise their hands if they believe other people are unwilling to change. What you will invariably see is that virtually every hand will be raised in response to both questions! The implication should be obvious: that while you may think you are open to change, the impression you give to other people may be different. Change has to start with you and it is your responsibility to find ways to engage others.
None of these factors is catastrophic. They’re really just the people factors you need to keep in mind when considering how to introduce change into your organization. People, and the culture they have built for themselves, are probably the largest variable between organizations. People and culture are also the largest potential barriers to change. You have to recognize that any change challenges the comforts that have been built up over time.
In order to create change, I believe the following change factors must be present:
Leadership
A sense of urgency
Executive support
And the following change enablers enable the overall change effort:
Persistence
Training
Continual “wins”
Positive reinforcement of desired behaviors
Communication, vision, and strategy
Each of the above topics is covered in detail below.
In my experience, you’re going to get acceptance or resistance in varying degrees that usually resembles a bell curve, as illustrated in Figure 8-1. The point is that it’s very rare that you can convince everyone right away, but you can influence how the changes will be perceived and accepted by the majority of people. And most importantly, you can’t manipulate people, but you can engage them.
Figure 8-1. The reaction people have to any change initiative is most likely going to take the form of a bell curve (or normal) distribution. Some people are going to be apathetic or be openly resistant, and others are going to be enthusiastic and want to get to it. Most reactions are going to fall somewhere in between, and the factors that are going to influence whether the bulk of the people are going to resist or enthusiastically support the changes are whether there is a sense of urgency that everyone can identify with, the degree of executive support, and the involvement of key leaders at every level in the organization.
Leadership, for some reason, is a term that many people shy away from. Perhaps this is because of the negative connotation: that leaders are the senior managers in the company and that leadership is hence top-down. Certainly, many management books are written that way, with the need for change coming from the CEO or senior management, and various techniques such as strategic goal setting and coaching are employed to get the organization on board and steer the right course. The other extreme is the bottom-up approach, where change is initiated at the very lowest level of the organization and then percolates up through the hierarchy. This is often how Extreme Programming gets started in an organization, with a software team trying it out and then the practices being adopted by other teams. The common thread in both these extremes is that no single person can make change happen.
In order for any change to succeed, there must be leadership at all levels. Change starts with individual leaders. These are people who realize that change has to start with what they can control: themselves. They start change by first occupying a new space themselves, then finding ways to get other people (their co-workers, their boss, their team, and their subordinates) to want to occupy that space with them, and then helping them to do so. By this definition, there are leaders at many levels in an organization, from the programmer who prods her co-workers and team leader into using automated tests, the team leader or coach who helps his team achieve its best work and ship a product, the product manager who works with users to ensure they understand the importance of their input, to the CEO who leverages the expertise of her teams to set the business goals for the organization. The trick in making change happen is finding as many leaders as possible and at as many levels of the organizations as possible, developing and empowering them, and getting them actively involved in the change process.
Change won’t happen unless people understand and agree with the need for change. One of the most important tasks before trying to initiate change is to develop a message that clearly and emphatically states the need for change. The new way can’t just be better; it has to make a positive difference to everyone in the organization and its customers, and by extension to the company’s bottom line.
Change can happen without executive support, but the more executive support there is, the easier the change effort will be. An executive is really anyone who controls the ability to spend money and helps steer company policy—the higher in the organization, the better. Executives can help free up funds and people’s time for training; they can also help with communication and rewarding desired behaviors. A word of warning: It may also be necessary to convince executives that they need to change, too—for example, if their style is overbearing or if they are unable to collaborate.
With any change, there are going to be peaks and valleys in effort, performance, and results. You have to go in knowing that these are going to occur and that you will need continual effort over a long period of time. The peaks are going to be periods of time where it will be easy to relax and relish success too much, when what you really need is a new set of goals that push you on to even greater accomplishments. The valleys are by far the most trying periods, where it seems like everyone is ignoring the message or reverting to their old ways. It is completely natural for people to revert to the familiar during trying periods because the familiar ways are going to be more comfortable than new approaches. A common valley in a change initiative is at the beginning (remember the chemical manufacturing example in Chapter 1), where there is a dip in performance as the team deals with all the same problems using new approaches. These are the times where leaders must lead, to ensure that the organization doesn’t get too comfortable when everything is going well and overreact when it’s not.
Training is often vital to help people learn new skills and understand new methods. It can be done by bringing consultants in (external training) or using your own people (internal training). Each type of training has its own advantages and disadvantages, but the key factor is that whoever leads the training has to have credibility with the audience. External consultants should be carefully selected to ensure that your organization is maximizing its investment by having the best experts with the deepest expertise (and credibility) possible.
Internal training may seem like a luxury because it requires that people have time to prepare the course material. However, in my experience, it can be powerful and effective, especially when one of the organization’s respected leaders puts together appropriate material where he or she can pass on important experiences or dive into a topic that is relevant to the organization.
Training topics should be carefully chosen. To introduce sustainable development into an organization, I recommend a minimum of training on agile project management, hands-on technical training for technical staff, plus some kind of executive training to give an overview of the methods being introduced and the business rationale behind them.
The most important thing to aim for is to produce continual “wins.” These can be in the form of new tests, new tools, changes in defect trends, successful projects, or whatever is meaningful to your company. They are particularly vital in the early days of change and should be targeted as soon as possible, mostly to ensure that people see that change is possible and that it is recognized (via communication). You cannot expect change to happen quickly, and so you must use the continual wins to reinforce success and keep everyone aware of the fact that progress is being made.
Change requires new behaviors that must replace existing behaviors. For example, here are some of the common behavior changes required when introducing sustainable development to an organization:
Project leaders must be able to create an environment where everyone participates in iteration planning. In many organizations the role of the project leader is to identify the tasks that must be completed, collect estimates from the developers, create a schedule, and then track progress against the schedule.
Refactoring and rewriting must be fully supported by everyone in the organization.
People must learn to resist the pressure to get features done. If the product is not in a working state, what’s the point in completing the feature and adding to the problem?
In many organizations, executives or product managers have the most contact with customers and teams are insulated from the “demands” of customers. It’s almost always best to set up direct and frequent collaboration with customers.
Developers must write tests and work at defect prevention. If developers are still rewarded for the number of hours they work and features they churn out, regardless of quality, they will not embrace their role in defect prevention, and no culture change will result because they will still look to a testing or QA organization or customers to find problems.
People must collaborate. If heroes are rewarded for saving projects and acting as prima donnas, then collaboration will be less important than being a hero.
Lots of positive and well-timed communication is a critical change enabler. At every company I have worked at, communication has been identified by employees as one of the principal shortcomings of the organization or management team. I think this is a reflection of the fact that you can never communicate enough.
The purpose of communication during change is to keep people excited and in tune with the importance of the changes. It is possible to communicate too much, which is why it is very important to think about the timing of your communication. Definitely talk about the continual wins the team has achieved and recognize that over time things will begin to look drier and less exciting. Those are the points where you may need to set a new set of goals or get some new people involved or find some new ways to get the message out.
A metaphor for how to get started in turning around a project is catching up on e-mail after an extended absence such as a vacation. The way I deal with this situation is I start with what I can control, which is the e-mail I receive the day I return. This ensures that the backlog doesn’t get any worse while I work on older messages. I will first look for recent high-priority messages and deal with those. Then, as time permits, I will start by deleting (without reading) messages that are clearly junk or out of date. I will then look for mail threads (i.e., multiple messages with the same subject) and start by reading the most recent message on the thread. In most cases, I can then delete all the older messages and be caught up on the subject. By the time I’ve finished this process, very few messages will be left, and life will be back to normal because I’ll be caught up.
Moving to a culture of sustainability is obviously not the same nor is it as simple as dealing with an e-mail backlog. However, if you are able to recognize that you need to start with what you can control, know where you want to end up, and have an understanding of the steps you can take to get yourself out of the current situation, then the change scenario becomes more plausible. You can’t try to solve everything at once, and you need to think about the problem in understandable increments that keep you on the desired path. Change initiatives will fail if they attempt to boil the ocean or lack focus or a clear sense of first steps and overall vision.
Think of your organization as a supertanker; it is big and takes a long time to change direction. Given that you know where you want to end up, recognize that you can’t make the entire turn at once. Instead, what are the things you can do at any point in time that nudge the supertanker in the desired direction?
In my opinion, one of the most common mistakes made when introducing change is to use a transition plan. In my experience, they don’t work. The problem with transition plans is it is common to rationalize reasons for why the transition plan cannot be followed, such as a recession or a critical project or feature that must be completed first. Typically, transition plans are something that can be put in place later or are somehow optional. Later may never come. There must be a strong desire for change, and change often means taking a risk or being aggressive. Transition plans are rarely risky or aggressive.
An analogy that one of my colleagues came up with is losing weight. In order to lose weight, you can’t talk about the need to lose weight and the need to change your lifestyle by eating better and exercising regularly. Successful weight loss is doing it, not talking about it, starting immediately not at some time in the future or by phasing in better food and more exercise.
If you are working on a project that is on the unsustainable development curve, the most important thing you must recognize is that you can’t just “blow up” your current processes and practices. That would most likely lead to disaster. Next, you need to recognize that achieving sustainable development is going to take time, and probably more time than you think. You also have to recognize that you are going to have to make compromises and can’t strive for perfection; you likely won’t ever have the architecture you’d like or have tests in place for every single line of code. Just think about getting continually better over time, in tiny increments if need be.
If you can, start with a small team. Then, use the successes and knowledge gained in that team to get other teams involved. You might want to consider moving people between teams if you can. New people added to a highly efficient team will be able to learn the new principles quickly. However, you need to be careful when moving people from the “model” team to other teams because they can easily get buried in the day-to-day reality of their new team unless they are strong, persistent leaders/coaches or can be moved in pairs or sub-teams. You might want to think of these sub-teams as a “tiger team”, that can be added to any project and lead change by example. But these people must not think of themselves as better or superior.
Unsustainable projects, or those on the path to unsustainability, are likely going to have a few key problems that are going to need targeting. Most likely, these will include aspects such as:
Legacy code that is poorly structured and has many broken windows.
Inadequate or nonexistent test coverage.
Attitudes that result in the use of phrases like “ship to QA” or “throw it over the wall for testing.” These are indicators of a defect detection culture, where developers are not doing enough testing themselves.
Entrenched development practices and attitudes that are counter to what is required. The longer practices have been used, the harder it is to change them and the easier it is for people to go back to them at the first sign of trouble or stress.
Defect rates that are unsustainable and increasing.
Bureaucratic project management practices that emphasize planning and tracking progress against the plan.
A lack of understanding of the current development problems and visibility into their magnitude.
Assuming that you have buy-in to change and have decided on your approach (training, consultants, etc.), where do you begin? Start by analyzing the mindset of your organization and mapping it against the four principles: continual refinement, working product, defect prevention, and design emphasis. Given that your early goal is going to be building wins, the following practices can be used to produce results quickly:
You can get an errant project back on track quickly using the agile project management practices simply by beginning iterative development, tracking your velocity, using daily standup meetings, and retrospectives.
Start ruthless testing by emphasizing unit tests for new code and, if you can, by putting safeguards in place such as system level tests to catch regressions sooner.
If you aren’t already doing so, make sure your product is built nightly (completely, from scratch) and that you are using a configuration management system.
Pay attention to your design practices. If you do a lot of up-front design currently, don’t stop doing design because you don’t think it’s agile. Don’t get hung up on simple design and emergent design. Start with frequent rapid design meetings and architectural reviews.
Make sure you have a defect tracking system in place and that you are rigorously tracking defects. Then, start collecting some metrics to find out what kind of defect trend and backlog you have. This will help plan your next steps to help get defects under control.
As you become familiar with the new practices, use your retrospectives as a way to review the practices from this book (or any other good source) and introduce new practices as you see fit.
In many ways, introducing sustainable development principles and practices for a new project is going to be easier than on an established project. Teams on new projects are almost always determined to do things the right way. It’s going to be possible, for example, to employ ruthless testing by having good test coverage and building testability into your project.
However, new projects are often launched with unrealistic expectations, or they are too ambitious. If you’re going to start with new practices, I recommend you start with the keystone practices from Appendix 1 and then use your retrospectives to tune and add to your processes over time.
Another problem with new projects is that people are often trying out new practices for the first time. I have found that it’s vital to have the team talk about its practices in an open way, especially through the initial phases. For example, I worked with one team where the team embraced simple design, but a few developers had read a XP book and thought simple design meant no up-front design. This resulted in some of the code being refactored repeatedly and some obvious features not being planned for. The result was chaotic until the team realized there was a problem and members were able to communicate openly about what they wanted, so they introduced weekly design meetings until the design settled down.
Probably the most dangerous problem with new projects is that their newness rapidly fades. New projects often face immense pressure in that they need to produce a feature-complete project in as short a timeframe as possible. If there are established competitors, this can be an uphill battle. There are going to be more valleys than peaks, and a great deal of persistence and discipline is going to be required to stick with and refine the practices when the temptation is going to be to just crank out the features without considering what type of quality tradeoffs need to be made.
Every project and situation is going to be different, so it is impossible to provide a step-by-step solution to your problem. I’ve tried to outline some of the common approaches I’ve used or seen applied to various situations, particularly when dealing with a legacy product that was not developed using sustainable techniques.
Do an architectural analysis, where you identify things such as poor structure and areas with a large number of defects. Every time you modify these sections of code, even if only for a minor change, consider doing some refactoring work in addition to just making the changes, even if this means setting aside extra time to make the changes. Be sure to add tests at least for the new code, and if you can, identify the key interfaces and add tests for them.
In your iteration planning sessions, write up some cards for the refactoring work. Users, product managers, and managers are in my experience very supportive of refactoring work if they are involved in the process of helping to set overall priorities and understand, at least at a high level, the purpose of the refactoring work. The visibility that this gives to the architecture of the product is good for the entire team, while at the same time ensuring that architectural issues are weighed against the feature work that users are demanding.
Hopefully, you should now be convinced that defect backlogs are a bad thing. Introduce some visible metrics, and possibly an ambitious goal such as HP’s “10x reduction goal” [Grady 1997] to get people focused on the problem. Then, work hard toward a culture built around defect prevention and working software.
For many organizations, one of the most difficult steps in introducing agile development is getting timely feedback from users. This is an ideal problem for senior management in the company to get involved with, to set up the necessary user contacts and also to get everyone in the organization focused on the need for greater user contact.
Training is a crucial part of moving from an unsustainable to sustainable development culture. Training needs to be considered for the entire team, ideally at the same time. In the early stages, using an external source (consultant or training organization) for training is usually required, because even if someone inside the company says the same things as the consultant, the consultant will often be listened to more, simply because he or she is from outside. However, once the initial training is complete, it is highly desirable to set up an internal training program, where people inside the company are given the opportunity to train others on topics they are knowledgeable in.
An unorthodox idea that I’ve seen work is a competitive refactoring exercise. Pick a piece of code that has many broken windows in it. Then, get a bunch of people together in a conference room and do a pair programming exercise to refactor the code. Consider pairing people together who don’t work together too much. Use a projector to review the solutions that each pair came up with during the day and discuss the pros and cons of each. Optionally, an award could be given for the solution the group likes the best. This exercise is time-consuming, but it helps everyone understand the difference between good code and bad, and the result can often be applied in the live code the next day. Also, the discussions are valuable for everyone, and the whole process can be fun with an underlying thread of competition.
One factor that is often overlooked when introducing change is the impact of the physical space the team works in. If you possibly can, change the physical space that the team operates in to facilitate collaboration and reduce unnecessary interruptions. Sometimes, simple changes can make huge differences in terms of productivity, and the highly collaborative nature of agile development demands a work environment that does not inhibit collaboration and free communication. Too many organizations are penny wise, pound foolish and skimp on equipment that will increase productivity simply because the equipment is considered too expensive. Here are some ideas to consider:
Build a project room for the team or convert a meeting room into a project room. Cover the walls of the room with whiteboards and corkboards, and think of this space as the location where anyone can go to get an immediate idea of the progress of the project. The Apple II project, for example, had a war room that gave every member of the company instant insight into the project [Lynn and Reilly 2002].
Make sure there are lots of whiteboards readily available and install a projector so the team can hold productive team code and architecture reviews. Magnetic whiteboards are also an ideal place to keep the index cards for iteration and release planning.
Do people who run automated tests have fast computers? Computers are so inexpensive now that there should be no excuse to not ensuring that people have adequate computing power readily available.
Do you need to buy dedicated computers for running automated tests?
Do you use the right tools for your project?
Are you using a good version control system?
Are there SDKs, compilers, or development environments that would make your team more efficient?
If the desks available to the team inhibit collaboration through having partitions that are too high or are not large enough for pair programming, find a way to buy some new furniture that removes these problems.
I can’t stand the status quo and prefer flexibility over any other mode of work. I try to always find better ways to do things and new ways to think about problems. I do this because I thrive on learning; my first priority is getting things done, and I prefer to get things done in the most efficient manner possible. I also like the element of surprise and producing an unexpected result or an unexpected analysis. Hence, I have been involved in many change efforts in my career at the personal, team, and organization levels. Not all have been successful, of course, but I’ve been fortunate that most of them have.
In this section I’ll describe two successful changes. The first was the introduction of a controversial practice into a small development team, and the other was the introduction of agile development into a relatively large (approximately 200 people) mature software organization. I think they’re both good examples of leadership and creating positive lasting change.
The small-scale change was the introduction of the root-cause analysis practice I’ve described several times earlier in this book. At this time, I was a programmer working in a small and highly motivated team. We had a lot of good practices, and we thought we were doing pretty well. When our team leader first introduced the idea to the team of meeting once a week to review all our bugs, we resisted with all the predictable excuses: that it would take too much time, that the return on investment wouldn’t be there, etc. But our team leader had some bug history data that convinced us enough to try it out for several months.
As predicted, the first few meetings were painful and took too long. However, I think it clicked for us after about a month when we realized that our drive to fix defects properly before the next meeting was paying dividends in increased quality. Around that time, we also developed an unproven belief that if we stuck with it, our overall productivity would increase. It took a few more months, but eventually we did reach the point where we were driven to reduce our defects through prevention (we got truly serious about automated testing), we always had a working product, and we held each other to increasingly high standards. Our productivity did increase, and we were able to add more features and deal with more complexity in a team that was constant in size.
In this case, our team leader deserves the credit for helping us take the initial leap and then essentially getting out of the way as we built on the early successes. I also acknowledge him for not taking any of the credit for the success of this change, although he certainly deserved it. For that team at that particular time, this was the right change that on hindsight took us from being a good (above average) to a great team. I was a member of that team as a programmer and team leader, and I am extremely proud of what we were able to accomplish.
A few years ago the company I worked for closed down the office I worked at. The team I described in the previous example was dispersed, and we each went off and found different jobs. I joined a smaller company (approximately 400 people, about half of which are in product development) with a completely different software development culture. Its approach to software development was mostly the waterfall method, and it had been used and evolved for over a decade. I was there for about a year, left, and then came back because of the bright people, products, challenging problems, and amazing customers. When I came back, I was promoted into a new position where part of my mandate consisted of technology strategy, software architecture, and software development processes. Attention was needed in all three areas, but I believe that often process changes have the highest return, so that is where I decided to focus initially.
There was a general recognition that the software development processes were not what they should be. Some people were experimenting with different methods, but not with any noticeable success overall. There had been a few attempts to introduce new processes, but they had all failed. I spent a bit of time looking at each one and talking to the people who had been involved. The one attempt that left the biggest impression on me was one where the proposed process was meticulously documented with diagrams and loads of text. It was essentially a proposal to add more bureaucracy, with mandatory documents, sign-offs, and decision-making groups. Nobody really bought into it, for obvious reasons.
I was working with a woman from our HR group, and we decided that we wanted to bring in outside training and focus initially on project management. We started by obtaining an approved budget for some project management training. We spent a few weeks talking to consultants and various training groups. All but one of the groups we spoke to offered what I’ll call traditional training in waterfall methods: Gantt charts, PERT charts, project tracking, etc. The one exception was Jim Highsmith. At the time, I had been reading anything I could get my hands on related to software development and development processes, from the CMM to agile. The books that had gotten me the most excited were the Extreme Programming books [Beck 2004] [Auer and Miller 2001] [Jeffries et al 2000], Jim’s Adaptive Software Development [Highsmith 1999] book, and the Scrum book [Schwaber and Beedle 2001].
Initially, I wasn’t content with any of the groups we spoke to about training. The traditional project management groups made me uncomfortable because, based on my previous work experience, I had minimal confidence that what they taught worked, except perhaps for building bridges. I was intrigued by agile development but didn’t feel that any of the methods I had read about were complete. However, I realized that what really resonated with me about agile development were its principles (see www.agilemanifesto.org) and I was able to connect most of these principles to my previous job. We had a really good talk with Jim Highsmith, and he agreed to tailor a course for us that provided a balanced view across what I viewed at the time as project management and technical practices, with practices combined from ASD, XP, and Scrum.
Once we communicated our decision to provide agile development training, we encountered resistance. Mostly it was because at the time agile was new and viewed as largely untried. We were able to convince most people that the worst that could happen from such a course would be that people would have a few new ideas and skills to take away. Luckily, my boss and the majority of my peers were supportive. There was, however, one executive who was openly hostile to the idea and thought it was a waste of money and time. I spent an afternoon with him and outlined my arguments against the traditional methods (some are in the Introduction) and described my experience from my previous job. In the end, we agreed to disagree but, to his credit, he converted more to skepticism and a “wait-and-see” attitude. That experience prompted a phone call to Jim Highsmith, where we outlined a short course for the directors and select executives to introduce them to agile development and to give them a chance to ask detailed questions.
The next step was the actual training course. Our budget allowed us to initially run two 2-day courses plus a 2-hour course for directors and executives. We spent a fair bit of time identifying key people to get into the course (leaders at all levels), ensuring that we had a cross-section of all the various functions (team leaders, software developers, QA people, documentation writers, usability specialists, product managers, etc.) within our department, that the courses had a blend of vocal and quiet people, and that the skeptics were balanced by those we felt had a more open mind. Then we issued invitations and followed up if people didn’t respond. A few people said no, but luckily we had a long list.
We held a kickoff prior to the first training session where we got all the course attendees together the day before the first course for 30 minutes. We had arranged for one of the executives to be there to give a short talk, but this person had to travel at the last minute, so it was left to me. I scribbled down some notes, rehearsed beforehand, and then spent some time “on my soapbox.” Generally, I enjoy speaking in public, but I was very nervous that day because I knew this was a risk, and it was on me. I tried to speak from the heart and described my perspective on where we were and where we could be. For me, that little talk is when the change started.
The training courses went extremely well and the majority of the responses were positive. One of the reasons they went well is that quite a few people had been experimenting with approaches that were agile in spirit, but what had been lacking, perhaps, was the discipline, a framework, and a broader set of tried-and-true practices.
The next step was followup. One thing I always worry about with training courses is that it’s easy for people to get enthused during the course, but the moment they get back to their desks and are confronted with their e-mail, voice mail, feature lists, bug backlog, etc. that reality can quickly overwhelm them. Later, they’ll say they never had a chance to try out any of the new ideas. I also worry about situations where a select few people attend a course, come back enthusiastic, and then are beaten down by the pessimism of those who didn’t attend the course. Therefore, as a followup, I decided to form an agile guiding council that consisted of the more enthusiastic course participants. The purpose of the group was to get as many people as possible as quickly as possible involved in leading the change and deciding what our next steps were.
The people in the guiding council are the ones who really made the change happen. We met weekly for the first year, then bi-weekly, and then sporadically for about another year after that. The council decided to bring in more training, so we got some more budget approved. This round of training would be a few one-day courses taught by Jim Highsmith and also some technical training that we thought should focus on test-first development, simple design, and refactoring. The council worked in subteams that did things like creating a vision statement and some nifty posters and sourcing and organizing the training. We also spent a lot of time talking about next steps, challenges, and successes. For the technical training, we chose Object Mentor, which eventually also ran some other courses for us. Jim Highsmith also visited us for a day once a month or so to talk to various people and groups. He definitely helped lend credibility to the initiative, because he could speak from his experiences at other companies.
Our first real success came about two months after the initial training. One of the team leaders had left the course and essentially thought “Hmm, that was interesting. I wonder if the company is going to do anything about it?” Then, through his participation in the council and with some encouragement from his peers on the council, he decided to try it out with his team. His team was enthusiastic about the Extreme Programming books, so members built out a quick project with test-first development, etc. The quality was amazing as was the speed that the results were achieved. It definitely turned some heads and got more people interested.
Shortly after the first success, the technical training started paying dividends. Some of the programmers had been adding unit tests to their new code, and as these tests started failing when other programmers modified the code, more people started noticing. The interesting thing is that our products prior to the training did have some automated tests. We had a unit test framework, for example, but it was only used by a few individuals and only sporadically. One of my favorite stories is that we had what were essentially system-level automated tests for one of our products. But everyone ignored the tests because they always failed, and they hadn’t been updated in a long time. One day, one of the programmers called a meeting with all the team leaders for the product and, in a very nice way, outlined why the tests should be taken seriously! It was a turning point for that project.
There have been many interesting lessons learned to date. For example, we didn’t really understand the importance of the physical workspace to agile development. Pair programming turned out to be impossible on most of our desks because they are too small for two people to sit side by side. For the curious, we don’t mandate pair programming, and most of the people who do pair up only do so for an hour or two at a time. Another thing that was lacking was enough whiteboards. I’ll never forget one day when I received an e-mail saying that we would have to erase the whiteboards when customers were touring the building so they wouldn’t see anything confidential. My reply was a short NO, and luckily I managed to convince people that productivity was more important than customers seeing our building!
A few years later, I can safely say we have achieved lasting change. It isn’t uniform across all teams, because there is still a divide between our new products developed since agile was introduced and those that existed before. We have rapidly developed one amazing new product, and we’re developing other new products using the same methods. We have built up several new areas now with new desks, and we have happily converted many of our former formal sit-down meeting rooms into agile project rooms where the walls are plastered with magnetic whiteboards and corkboards. As a result, a number of large tables have been relegated to the basement to collect dust so we can stand up and move around.
The challenges for our existing projects have been quite large. The teams have adopted as many agile practices as they can, and they’ve added many new automated tests. But the amount of code that isn’t covered by automated tests is quite large, and in many cases not enough attention was paid in the past to issues such as code duplication and design, so the teams are still paying off accumulated technical debt. Still, huge strides have been made and the teams are motivated to continually improve. I hope that eventually we can close the gap with our other teams.
Implementing sustainable development in most organizations is going to require a change in the development culture. This chapter is an introduction to some techniques that can be used while outlining change factors and enablers that must be thought of. Achieving a culture change is hard and requires more effort and persistence than most people anticipate. Hence, it is important to consider what you want to achieve and how you are going to achieve it.