7. Craftsmanship
By Sandro Mancuso, April 27, 2019
Excitement. That’s what many developers felt when they first heard about Agile. For most of us, developers coming from the software factories and Waterfall mentality, Agile was the hope for emancipation. It was the hope that we would work in a collaborative environment and that our opinions would be listened to and respected. We would have better working processes and practices. We would be working in small iterations and short feedback loops. We would release our application to production regularly. We would interact with users and get their feedback. We would constantly inspect and adapt our ways of working. We would be involved at the beginning of the process. We would have daily contact with the business. In fact, we would be a single team. We would regularly discuss business and technical challenges and together agree on a way forward, and we would be treated as professionals. Business and technology would work together to produce great software products, delivering value to our companies and clients.
At the beginning, we felt Agile was too good to be true. We thought our companies would never embrace the Agile mindset, let alone the Agile practices. But most of them did, and we were positively surprised. Suddenly, everything was different. We had product backlogs and user stories instead of requirement documents. We had physical boards and burn-down charts instead of Gantt charts. We had sticky notes that we moved every morning according to our progress. There was something powerful about those sticky notes—something that provoked a deep psychological addiction. They were the representation of our agility. The more sticky notes we had on the walls, the more “Agile” we felt we were. We became a Scrum team instead of a construction team. And we didn’t have project managers anymore. We were told we did not need to be managed; our managers would now be product owners, and we would self-organize. We were told that product owners and developers would work in close collaboration, as a single team. And from now on, as the Scrum team, we were empowered to make decisions—not only technical decisions, but project-related decisions. Or so we thought.
Agile took the software industry by storm. But as in a Chinese whispers game, the original Agile ideas got distorted and simplified, arriving at companies as the promise of a process to deliver software faster. For companies and managers using Waterfall or RUP, that was music to their ears. Managers and stakeholders were excited. At the end of the day, who would not want to be Agile? Who would not want to deliver software faster? Even among the skeptical, Agile could not be rejected. If your competitors are advertising that they are Agile, but you are not, what does it make you? What would your potential customers think about you? Companies could not afford not to be Agile. And in the years that followed the Agile summit, companies all over the world embarked on their Agile transformation. The Agile Transformation Era had begun.
The Agile Hangover
Transitioning from one culture to another was not easy. Companies needed external help to transform their organization and a huge demand for a new type of professional emerged: Agile coaches. Many Agile certification schemes were created. Some certifications could be obtained by simply attending a two-day training course.
Selling Agile processes to middle managers was an easy sell—they all wanted software to be delivered faster. “Engineering is the easy part. If we fix the process, engineering will be fixed,” managers were told. “It’s always a people problem.” And they bought it. Managers manage people, and as long as they are in charge, they are happy to have their direct reports working faster.
Many companies truly benefited from their Agile transformation, and today they are in a much better place than they were before. Many of these companies can deploy software to production multiple times a day and have business and technology truly working as a single team. But that is certainly not true for many others. Managers willing to push developers to work faster are using the full transparency of the process to micromanage them. Agile coaches with neither business nor technical experience are coaching managers and telling development teams what to do. Roadmaps and milestones are being defined by managers and forced upon development teams—developers can estimate the work, but they are pushed hard to fit their estimates into the imposed milestones. It is fairly common to see projects that have all their iterations and respective user stories already defined by management for the next 6 to 12 months. Failing to deliver all story points in a sprint means developers must work harder in the next sprint to make up for the delay. Daily standup meetings become meetings where developers must report progress to product owners and Agile coaches, detailing what they are working on and when they will be done. If the product owner thinks developers are spending too much time on things like automated tests, refactoring, or pairing, they simply tell the team to stop doing it.
Strategic technical work has no place in their Agile process. There is no need for architecture or design. The order is to simply focus on the highest-priority item in the backlog and get it done as fast as possible—one highest-priority item after another. This approach results in a long sequence of iterative tactical work and accumulation of technical debt. Fragile software, the famous monoliths (or distributed monoliths for the teams trying micro services) become the norm. Bugs and operational problems are popular discussion topics during daily standup meetings and retrospectives. Releases to production are not as frequent as the business expected. Manual testing cycles still take days, if not weeks, to complete. And the hope that adopting Agile would prevent all these issues is gone. Managers blame developers for not moving quickly enough. Developers blame managers for not allowing them to do the technical and strategic work needed. Product owners do not consider themselves as part of the team and do not share the responsibility when things go wrong. The us-versus-them culture reigns dominant.
That is what we call the Agile Hangover. After years of investment in an Agile transformation, companies realized that they still have many of the problems they had before. And Agile is being blamed for that, of course.
Expectation Mismatch
Agile transformations that purely focus on process are partial transformations. While Agile coaches try to guide managers and delivery teams through the Agile processes, there is no one helping developers learn Agile technical practices and engineering. The assumption that fixing the collaboration among people would improve engineering could not be more wrong.
Good collaboration removes some of the blocks people have to do their jobs but does not necessarily make them more skilled.
There is a big expectation that comes with Agile adoption: development teams should deliver production-ready software as soon as a feature is done, or at the least at the end of each iteration. For most development teams, that is a significant change. There is no way they can achieve that without changing the way they work, and that means learning and mastering new practices. But there are a few problems. Rarely is there a budget for up-skilling developers during Agile transformations. Business does not expect developers to slow down during their Agile adoption. Most do not even know developers have to learn new practices. They were told that if they worked in a more collaborative way, developers would work faster.
Releasing software to production every two weeks requires a lot of discipline and technical skills—skills that are not commonly found in teams used to delivering software a few times per year. Things get much worse when multiple teams, with quite a few developers each, all working on the same systems, are expected to deliver features to production as they finish them. The level of mastery in technical practices and engineering must be extremely high for teams to deploy software to production multiple times a day without compromising the overall stability of the system. Developers cannot just pick an item from the top of a product backlog, start coding, and think that everything is going to be OK when they push to production. They need strategic thinking. They need a modular design that supports parallel work. They need to continuously embrace change while making sure the system can always be deployed to production. For that, they need to continuously build software that is both flexible and robust. But balancing flexibility and robustness with the need to deploy software to production continuously is extremely hard and cannot be done without the necessary engineering skills.
It is not realistic to think that teams will develop these skills simply by creating a more collaborative environment. Teams need support in acquiring these technical skills. This support can happen through a combination of coaching, training, experimentation, and self-learning. Business agility is directly related to how quickly companies can evolve their software, and that means an evolution in their engineering skills and technical practices.
Moving Apart
Of course, not every Agile transformation or company suffers from all the problems described earlier, or at least not to the same degree. In terms of business, it is fair to say that most companies that have gone through an Agile transformation, even if partially, are in a better place today. They are working in shorter iterations. The collaboration between business and technology is closer than it was before. Problems and risks are being identified earlier. Businesses are reacting faster as they get new information, truly benefiting from an iterative approach to software development. However, although companies are indeed better than they were before, the split between Agile processes and engineering are still hurting them. Most modern Agile coaches do not have enough (if any) technical skills to coach developers on technical practices, and they rarely talk about engineering. Over the years, developers started seeing Agile coaches as another layer of management: people telling them what to do instead of helping them to get better at what they do.
Are developers moving away from Agile, or is Agile moving away from developers?
The answer to this question is probably both. It seems that Agile and developers are moving away from each other. In many organizations, Agile has become synonymous with Scrum. XP, when present, is reduced to a few technical practices like TDD and Continuous Integration. There is an expectation from Agile coaches that developers apply some of the XP practices, but they do not really help or get involved in the way developers are working. Many Product Owners (or project managers) still do not feel they are part of the team and do not feel responsible when things do not go according to plan. Developers still need to negotiate hard with the business to make the necessary technical improvements to keep developing and maintaining the system.
Companies are still not mature enough to understand that technical problems are in fact business problems.
With a diminishing focus on technical skills, can Agile significantly improve software projects beyond what it did up until now? Is Agile still focused on uncovering better ways of developing software by doing it and helping others do it, as it was written in the Agile Manifesto? I’m not so sure.
Software Craftsmanship
To raise the bar of professional software development and re-establish some of the original Agile goals, a group of developers met in November 2008 in Chicago to create a new movement: Software Craftsmanship. In that meeting, similar to what happened during the Agile summit in 2001, they agreed on a core set of values and came up with a new manifesto1 that was built on top of the Agile Manifesto:
1. http://manifesto.softwarecraftsmanship.org
As aspiring Software Craftsmen, we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:
Not only working software, but also well-crafted software
Not only responding to change, but also steadily adding value
Not only individuals and interactions, but also a community of professionals
Not only customer collaboration, but also productive partnerships
That is, in pursuit of the items on the left we have found the items on the right to be indispensable.
The Software Craftsmanship manifesto describes an ideology, a mindset. It promotes professionalism through different perspectives.
Well-crafted software means code that is well designed and well tested. It is code that we are not scared to change and code that enables business to react fast. It is code that is both flexible and robust.
Steadily adding value means that no matter what we do, we should always be committed to continuously provide increasing value to our clients and employers.
A community of professionals means that we are expected to share and learn with each other, raising the bar of our industry. We are responsible for preparing the next generation of developers.
Productive partnerships means we will have a professional relationship with our clients and employers. We will always behave ethically and respectfully, advising and working with our clients and employers in the best way possible. We will expect a relationship of mutual respect and professionalism even if we need to take the initiative and lead by example.
We will look at our work not as something we need to do as part of a job but as a professional service we provide. We will take ownership of our own careers, investing our own time and money to get better at what we do. These values are not only professional values—they are personal values. Craftspeople strive to do the best job they can, not because someone is paying, but based on a desire to do things well.
Thousands of developers across the world immediately subscribed to the principles and values espoused by Software Craftsmanship. The original excitement that developers felt in the early days of Agile was not only back, but also stronger. Craftspeople, as they started calling themselves, decided not to let their movement be hijacked again. It is a developers’ movement. A movement that inspires developers to be the best they can be. A movement that inspires developers to become and see themselves as highly skilled professionals.
Ideology versus Methodology
An ideology is a system of ideas and ideals. A methodology is a system of methods and practices. An ideology defines ideals to be aimed at. One or more methodologies can be used to reach those ideals—they are a means to an end. When looking at the Agile Manifesto and the 12 principles,2 we can clearly see the ideology behind them. The main goal of Agile is to provide business agility and customer satisfaction, and that is achieved via close collaboration, iterative development, short feedback loops, and technical excellence. Methodologies like Scrum, Extreme Programming (XP), Dynamic Systems Development Method (DSDM), Adaptive Software Development (ASD), Crystal Methods, Feature-Driven Development (FDD), and other Agile methodologies are all means to the same end.
2. https://agilemanifesto.org/principles.html
Methodologies and practices are like training wheels; they are great to get people started. As with children learning to ride a bicycle, the training wheels allow them to get started in a safe and controlled way. Once they are a bit more confident, we raise the training wheels a bit so they can practice their equilibrium. We then take one of the training wheels off. And then the other. At this point, the child is ready to go on his own. But if we focus too much on the importance of the training wheels and keep them on for too long, the child gets too dependent on them and does not want them to be removed. The obsessive focus on a methodology or set of practices distracts teams and organizations from their real goals. The goal is to teach a child to ride a bicycle, not to adopt the training wheels.
In his book Agile Project Management: Creating Innovative Products, Jim Highsmith says, “Principles without practices are empty shells, whereas practices without principles tend to be implemented by rote, without judgement. Principles guide practices. Practices instantiate principles. They go hand in hand.”3 Although methodologies and practices are a means to an end, we should not deemphasize their importance. Professionals are defined by the way they work. We cannot claim to have certain principles and values when our ways of working (methods and practices) are not aligned to them. Good professionals can precisely describe how they work given a specific context. They master a wide set of practices and are able to use them according to their needs.
3. Highsmith, J. 2009. Agile Project Management: Creating Innovative Products, 2nd ed. Boston, MA: Addison-Wesley, 85.
Does Software Craftsmanship Have Practices?
Software Craftsmanship does not have practices. Rather, it promotes a perpetual search for better practices and ways of working. Good practices are good until we discover better ones to replace them. Attaching specific practices to Software Craftsmanship would make it weak and obsolete as better practices are discovered. But that does not mean that the international Software Craftsmanship community does not advocate any practices. On the contrary, since its creation in 2008, the Software Craftsmanship community considers XP the best set of Agile development practices currently available. TDD, Refactoring, Simple Design, Continuous Integration, and Pair Programming are heavily advocated by the Software Craftsmanship community—but they are XP practices, not Craftsmanship practices. And they are not the only practices. Craftsmanship also promotes Clean Code and SOLID principles. It promotes small commits, small releases, and Continuous Delivery. It promotes modularity in software design and any type of automation that removes manual and repetitive work. And it promotes any practices that improve productivity, reduce risk, and help to produce valuable, robust, and flexible software.
Craftsmanship is not only about technical practices, engineering, and self-improvement. It is also about professionalism and enabling clients to achieve their business goals. And this is an area where Agile, Lean, and Craftsmanship are in perfect alignment. All three have similar goals but tackle the problem from different but equally important and complementary perspectives.
Focus on the Value, Not the Practice
A common mistake in the Agile and Craftsmanship communities is to promote practices instead of the value they provide. Let’s take TDD as an example. One of the most common questions asked inside the Craftsmanship communities is “How do I convince my manager/colleague/team to do TDD?” That is the wrong question. The problem here is that we are offering a solution before agreeing on the problem. People will not change the way they work if they do not see the value.
Instead of pushing TDD, maybe we could start agreeing on the value of reducing the time it takes to test our entire system. How long does it take today? Two hours? Two days? Two weeks? How many people are involved? What if we could reduce it to 20 minutes? Two minutes? Maybe even 2 seconds? And what if we could do that at any time just by pressing a button? Would that give us a good return on investment? Would that make our lives easier? Would we be able to release reliable software faster?
If we agree that the answer is yes, then we can start talking about practices which could help us achieve that. TDD would be a natural choice here. For those who are not so keen on TDD, we should ask what practice they would prefer. What practice can they suggest that could bring the same or higher value according to the agreed-upon goals?
Agreeing first on the goals to be achieved is essential when discussing practices. The only thing that should not be acceptable is to reject a practice without providing a better alternative.
Discussing Practices
The discussion around practices should be done at the right level and with the right people. If we want to adopt practices that improve the collaboration between business and technology, people from business and technology should be involved in the discussion. If developers are discussing practices that would enable them to build systems in a better way, there is no reason to involve businesspeople. Business should only be involved when there is a significant impact in the cost or duration of the project.
There is a difference between re-architecting a whole monolith system into microservices and doing TDD. The former has a very significant impact on the cost and duration of the project; the latter doesn’t, as long as developers are comfortable with the technique. Whether or not developers automate their tests should not be relevant to the business. It should be even less relevant whether the automated tests are written before or after the production code. What business should care about is that the lead time from business ideas to software in production is gradually reduced. Reducing the amount of money and time spent on rework (bugs, manual processes like testing, deployment, and production monitoring) is also a business concern that should be addressed by the development teams. Reducing the cost of experimenting is also a business concern. Experimenting becomes very costly when the software is not modular and easily testable. Business and developers should have conversations about business value, not technical practices.
Developers should not ask authorization for writing tests. They should not have separate tasks for unit tests or refactoring. They should not have separate tasks for making a feature production ready. These technical activities should be factored into the development of any feature. They are not optional. Managers and developers should only discuss what is going to be delivered and when, not how. Every time developers volunteer details of how they work, they are inviting managers to micromanage them.
Are we saying that developers should hide how they work? No, not at all. Developers should be able to clearly describe how they work and the advantages of working that way to whomever is interested. What developers should not do is to let other people decide how they work. Conversations between developers and business should be about why, what, and when—not how.
Craftsmanship Impact on Individuals
Craftsmanship has a profound impact on individuals. It is common to see people separating their personal lives from their professional lives. Phrases like “I do not want to talk about work after I leave the office,” or “I have different interests in life” are said in a way that makes work look like a chore, a bad thing, or a thing you do because you have to, not because you want to. The problem of dividing our lives into multiple lives is that they are constantly in conflict. There is always that feeling that we must sacrifice one life for the sake of the other, regardless of which life we pick.
Craftsmanship promotes software development as a profession. There is a difference between having a job and having a profession. A job is a thing we do but it is not part of who we are. A profession, on the other hand, is part of who we are. When asked, “What do you do?”, a person with a job would normally say something like “I work for company X,” or “I work as a software developer.” But a person with a profession would generally say, “I am a software developer.” A profession is something we invest in. It’s something we want to get better at. We want to gain more skills and have a long-lasting and fulfilling career.
That does not mean we would not spend time with our families or that we would not have other interests in life. Rather, it means that we would find a way to balance all our commitments and interests such that we can live a single, well-balanced, and happy life. There are times when we want to give more attention to our family, our profession, or a hobby we might have. And that is totally OK. We have different needs at different times. But going to work should not be a chore when we have a profession. It is just another thing we do that gives us pleasure and fulfills us as individuals. A profession gives meaning to our lives.
Craftsmanship Impact on Our Industry
Since 2008, a growing number of Software Craftsmanship communities and conferences are being organized all over the world, attracting tens of thousands of developers. While Agile communities have an emphasis on the people and process side of software projects, the Craftsmanship communities focus more on the technical side. They have been key in promoting XP, and many other technical practices to many developers and companies around the world. It is through Software Craftsmanship communities that many developers are learning TDD, Continuous Integration, Pair Programming, Simple Design, SOLID principles, Clean Code, and Refactoring. They are also learning how to architect systems using microservices, how to automate their deployment pipelines, and how to migrate their systems to the cloud. They are learning different programming languages and paradigms. They are learning new technologies and different ways to test and maintain their applications. Developers in the Craftsmanship community are creating safe and friendly spaces where they can meet like-minded people and talk about their profession.
Software Craftsmanship communities are extremely inclusive. From the very beginning, one of the main goals of Software Craftsmanship was to bring together software developers from all backgrounds so that they could learn from each other and raise the bar of professional software development. Craftsmanship communities are technology agnostic, and all developers, regardless of their level of experience, are welcome in the meetings. The community is committed to preparing the next generation of professionals, creating various events where people joining our industry can learn essential practices to build well-crafted software.
Craftsmanship Impact on Companies
Software Craftsmanship adoption is growing. Many companies that have adopted Agile are now looking at Craftsmanship to improve their engineering capabilities. However, Software Craftsmanship does not have the same business appeal that Agile has. XP is still something that many managers do not understand or are excited about. Managers understand Scrum, iterations, demos, retrospectives, collaboration, and a fast feedback loop. But they are not so interested in programming-related techniques. For most of them, XP is about programming, not about Agile software development.
Differently from Agile coaches in the early 2000s, who had a strong technical background, most Agile coaches today cannot teach XP practices or talk to businesspeople about engineering. They cannot sit down with developers and Pair Program with them. They cannot talk about Simple Design or help with the configuration of Continuous Integration pipelines. They cannot help developers to refactor their legacy code. They can discuss neither testing strategies nor how to maintain multiple services in production. They cannot explain to businesspeople the real advantages of certain technical practices, let alone create or advise on a technical strategy.
But companies need reliable systems—systems that enable them to react quickly according to their business needs. Companies also need motivated and capable technical teams that can do a great job creating and maintaining their systems. And those are areas where Software Craftsmanship excels.
The Software Craftsmanship mindset is an inspiration for many developers. It gives them a sense of purpose, a sense of pride, and an innate willingness to do things well. Most developers, as people in general, are keen to learn and do things well—they just need support and an environment where they can thrive. Companies embracing Craftsmanship often see internal communities of practice flourish. Developers organize internal sessions where they code together, practice TDD, and improve their software design skills. They become interested in learning new technologies and modernizing the systems they work on. They discuss better ways to improve the codebase and eliminate technical debit. Software Craftsmanship promotes a culture of learning, making companies more innovative and responsive.
Craftsmanship and Agile
Some of the triggers for the creation of the Software Craftsmanship movement were related to the frustration that many developers felt with the direction Agile had taken. Because of that, some people felt that Craftsmanship and Agile were at odds with each other. People in the Craftsmanship movement who had also been part of the Agile movement criticized Agile for their excessive focus on process and lack of focus on engineering. People in the Agile movement criticized Craftsmanship for its narrow focus or lack of focus on real business and people problems.
Although there were some valid concerns coming from both sides, most of the disagreements were more related to tribalism than actual fundamental divergence of opinions. Essentially, both movements want to achieve very similar things. They both want customer satisfaction, they both desire close collaboration, and they both value short feedback loops. Both want to deliver high-quality, valuable work, and both want professionalism. In order to achieve business agility, companies need not only collaborative and iterative processes, but also good engineering skills. Combining Agile and Craftsmanship is the perfect way to achieve that.
Conclusion
At the Snowbird meeting in 2001, Kent Beck said that Agile was about the healing of the divide between development and business. Unfortunately, as the project managers flooded into the Agile community, the developers—who had created the Agile community in the first place—felt dispossessed and undervalued. So, they left to form the Craftsmanship movement. Thus, the ancient distrust continues.
And yet the values of Agile and the values of Craftsmanship are strongly aligned. These two movements should not be separate. One hopes that, one day, they’ll come together again.