Chapter 1. Art or Science?

Are you a scientist or an artist? An engineer or a craftsman? Are you a gardener or a chef? A poet or an architect?

Are you a programmer or software developer? If so, then what are you?

My response to these questions is: Yes, none of the above.

While I self-identify as a programmer, I’m a little of all of the above. And yet, none of those.

Questions like these are important. The software development industry is about 70 years old, and we’re still figuring it out. One persistent problem is how to think about it. Hence these questions. Is software development like building a house? Is it like composing a poem?

Over the decades, we’ve tried sundry metaphors, but they all fall apart. Developing software is like building a house, except when it isn’t. Developing software is like growing a garden, except when it isn’t. Ultimately, none of the metaphors fit.

I believe, though, that how we think about software development shapes how we work.

If you think that developing software is like building a house, you’ll make mistakes.

1.1 Building a house

For decades, people have likened developing software to building a house. As Kent Beck puts it:

“Unfortunately, design in software has been shackled by metaphors from physical design activities.”[5]

It’s one of the most pervasive, beguiling, and obstructive metaphors of software development.

1.1.1 The problem with projects

If you think of developing software as being similar to building a house, the first mistake you’ll make is to think of it as a project. A project has a start and an end. Once you reach the end, work is done.

Only unsuccessful software ends. Successful software endures. If you’re fortunate enough to develop successful software, when you’re done with a release you’ll move on to develop the next. This can go on for years. Some successful software lasts for decades¹.

¹ I’m writing this book in L^AT_EX - a program first released in 1984!

Once you’ve built a house, people can move in. You’ll need to maintain it, but the cost of that is only a fraction of what it cost to build it. Granted, software like that exists. Particularly in the enterprise segment, once you’ve built² an internal line-of-business application, it’s done and users are fettered to it. Such software enters maintenance mode once the project is done.

² This is a verb I aspire to never use about software development, but in this particular context, it makes sense.

But most software isn’t like that. Software that competes with other software is never done. If you’re caught in the house-building metaphor, you may think of it as a series of projects. You may plan to release the next version of your product in nine months, but to your horror find that your competitor publishes improvements every three months.

So you work hard at making the ‘projects’ shorter. When you’re finally able to deliver every three months, your competitor is on a one-month release cycle. You can see where this ends, can’t you?

It ends with Continuous Delivery[49]. That, or you’ll eventually go out of business. Based on research, the book Accelerate[29] argues convincingly that the key capability distinguishing high-performing from low-performing teams is the ability to release at the drop of a hat.

When you can do that, the notion of a software development project no longer makes sense.

1.1.2 The problem with phases

Another misconception often produced by the house-building metaphor is that software development should happen in distinct phases. When building a house, an architect first draws the plans. Then you prepare the logistics, move materials on site, and only then can you begin to build.

When this metaphor is applied to software development, you appoint a software architect whose responsibility it is to produce a plan. Only when the plan is done should development start. This perspective on software development sees the planning phase as the phase where intellectual work happens. According to the metaphor, the programming phase is like the actual construction phase of a house. Developers are seen as interchangeable workers³, basically little more than glorified typists.

³ I’ve nothing against construction workers; my beloved father was a mason.

Nothing could be farther from the truth. As Jack Reeves pointed out in 1992[86], the construction phase of software development is when you compile source code. It’s virtually free, quite unlike construction of a house. All work happens in the design phase, or as Kevlin Henney so eloquently put it:

“The act of describing a program in unambiguous detail and the act of programming are one and the same.”[42]

In software development, there’s no construction phase to speak of. This doesn’t imply that planning isn’t useful, but it does indicate that the house-building metaphor is at best unhelpful.

1.1.3 Dependencies

When you build a house, physical reality imposes constraints. You must first lay the foundations, then raise the walls, and only then can you put on the roof. In other words, the roof depends on the walls, which depend on the foundation.

This metaphor tricks people into thinking that they need to manage dependencies. I’ve had project managers who produced elaborate Gantt charts to plan a project.

I’ve worked with numerous teams, and most of them start any new development project by designing a relational database schema. The database is the foundation of most online services, and teams seem incapable of shaking the notion that you could develop a user interface before you have a database.

Some teams never manage to produce a working piece of software. After they’ve designed the database, they figure that they need a framework to go with it. So they proceed to reinvent an object-relational mapper, that Vietnam of computer science[70].

The house-building metaphor is harmful because it deceives you into thinking about software development in a particular way. You’ll miss opportunities that you don’t see because your perspective doesn’t align with reality. The reality of software development is that, metaphorically speaking, you can start with the roof. You’ll see an example of this later in this book.

1.2 Growing a garden

The house-building metaphor is wrong, but perhaps other metaphors work better. The gardening metaphor has been gaining traction in the 2010s. It’s not accidental that Nat Pryce and Steve Freeman named their excellent book Growing Object-Oriented Software, Guided by Tests[36].

This view on software development sees software as a living organism that must be tended, coaxed, and pruned. It’s another compelling metaphor. Have you ever felt that a code base has a life of its own?

It can be illuminating to view software development in this light. At least it forces a change in perspective that might shake your belief that software development is like building a house.

By viewing software as a living organism, the gardening metaphor emphasises pruning. Left to itself, a garden will grow wild. In order to extract value from a garden, a gardener must tend to it by killing weeds while aiding and supporting the desired plants. Translated into software development, it helps focusing on activities that prevent code rot, such as refactoring and deletion of dead code.

I don’t find this metaphor nearly as problematic as the house-building metaphor, but I still don’t think that it paints the whole picture.

1.2.1 What Makes a Garden Grow?

I like the gardening metaphor’s emphasis on activities that combat disorder. Just as you must prune and weed a garden, you must refactor and pay off technical debt in your code bases.

The gardening metaphor, on the other hand, says little about where code comes from. In a garden, plants grow automatically. All they need is nutrients, water, and light. Software, on the other hand, doesn’t grow automatically. You can’t just throw a computer, crisps, and soft drinks into a dark room and expect software to grow from that. You’d be lacking an important ingredient: programmers.

Code is written by someone. This is an active process, and the gardening metaphor doesn’t have much to say about it. How do you decide what to write, and what not to write? How do you decide how to structure a piece of code?

If we wish to improve the software development industry, we have to address these questions, too.

1.3 Towards engineering

Other metaphors for software development exist. For example, I already mentioned the term technical debt, which implies an accountant’s view. I’ve also touched on the process of writing code, which suggests a similarity to other kinds of authoring. Few metaphors are entirely wrong, but none are perfectly right either.

There are reasons I’ve targeted the house-building metaphor in particular. One is that it’s so pervasive. Another is that it seems so wrong as to be unsalvageable.

1.3.1 Software as a craft

I came to the conclusion that the house-building metaphor is harmful many years ago. Once you dispose of a viewpoint, you typically go looking for a new. I found it in software craftsmanship.

It seems compelling to view software development as a craft, as essentially skilled work. While you can take an education in computer science, you don’t have to. I didn’t⁴.

⁴ In case you’re wondering, I do have a university degree. It’s in economics, but apart from a stint in the Danish Ministry of Economic Affairs, I never used it.

The skills you need to work as a professional software developer tend to be situational. Learn how this specific code base is structured. Learn how to use that particular framework. Suffer through the ordeal of wasting three days troubleshooting a bug in production. Things like that.

The more you do that, the more skilled you become. If you stay in the same company, and work in the same code base for years, you may become a specialised authority, but will that help you if you decide to work somewhere else?

You can learn faster by moving from code base to code base. Try some back-end development. Do some front-end development. Perhaps try some game programming, or some machine learning. This will expose you to a wide range of problems that will accumulate as experience.

This is strikingly similar to the old European tradition of journeyman years. A craftsman like a carpenter or roofer would travel around Europe, working for a while in a place before moving on to the next. Doing so exposed them to alternative solutions to problems. It made them better at their craft.

It’s compelling to think of software developers like that. The book The Pragmatic Programmer is even subtitled From Journeyman to Master[50].

If this is true, it follows that we should structure our industry accordingly. We should have apprentices who work alongside masters. We could even organise guilds.

If it’s true, that is.

Software craftsmanship is another metaphor. I find it illuminating, but when you shine a bright light on a subject, you also produce shadows. The brighter the light, the darker the shadow, as illustrated in figure 1.1.

There’s still something missing from the picture.

1.3.2 Heuristics

My software craftsmanship years were, in some sense, a period of utter disillusionment. I saw skill as nothing but accumulated experience. It seemed to me that there was no methodology to software development. That everything depended on circumstances. That there was no right or wrong way to do things.

That programming was basically an art.

That suited me well. I’ve always liked art. When I was young, I wanted to be an artist⁵.

⁵ My oldest aspiration was to become a comics artist in the European tradition. Later, in my teenage years, I picked up the guitar and dreamt of becoming a rock star. It turned out that while I enjoyed both drawing and playing, I wasn’t particularly talented.

The problem with that viewpoint is that it doesn’t seem to scale. In order to ‘create’ new programmers, you’d have to take them on as apprentices until they have learned enough to become journeymen. From there, mastery is several more years away.

Another issue with viewing programming as an art or a craft is that it, too, doesn’t fit reality. Around 2010, it began to occur to me[105] that I was following heuristics when I programmed. Rules of thumb. Guidelines that can be taught.

At first, I didn’t make much of it. Over the years, however, I regularly found myself in positions where I was coaching other developers. When I did that, I’d often formulate reasons for writing code in particular ways.

I began to realise that I’d probably been wrong in my nihilism. That perhaps, guidelines might be the key to turning programming into an engineering discipline.

1.3.3 Earlier notions of software engineering

The notion of software engineering dates back to the late 1960s⁶. It was related to the contemporary software crisis, the dawning realisation that programming is hard.

⁶ The term may be older than that. It’s not entirely clear to me, and I wasn’t alive back then, so I don’t recall. It seems uncontroversial, though, that two NATO conferences held in 1968 and 1969 popularised the term software engineering[4].

Programmers back then actually had a good grasp of what they were doing. Many of the illustrious figures of our industry were active in those days: Edsger Dijkstra, Tony Hoare, Donald Knuth, Alan Kay. If you’d asked people back then if they thought that programming would be an engineering discipline in the 2020s, they’d probably say yes.

You may have noticed that I discuss the notion of software engineering as an aspirational goal, rather than a fact of everyday software development. It’s possible that there are pockets of actual software engineering in the world⁷, but in my experience, most software development is conducted in a different style.

⁷ NASA seems like a good bet on being one of those pockets.

I’m not alone in feeling that software engineering is still a future goal. Adam Barr puts it beautifully:

“If you’re like me, you dream of a day when software engineering is studied in a thoughtful, methodical way, and the guidance given to programmers sits atop a foundation of experimental results rather than the shifting sands of individual experience.”[4]

He explains how software engineering was well on its way, but then something happened that derailed it. What happened, according to Barr, was personal computers. They created a generation of programmers who’d taught themselves to program at home. Since they could tinker with computers in solitude, they remained largely ignorant of the body of knowledge that already existed.

This state of affairs seems to persist to this day. Alan Kay calls computing Pop Culture:

“But pop culture holds a disdain for history. Pop culture is all about identity and feeling like you’re participating. It has nothing to do with cooperation, the past or the future — it’s living in the present. I think the same is true of most people who write code for money. They have no idea where [their culture came from]”[52]

We may have squandered fifty years by making little progress on software engineering, but I think that we may have made progress in other ways.

1.3.4 Moving forward with software engineering

What does an engineer do? Engineers design and oversee the construction of things, from big structures like bridges, tunnels, skyscrapers, and power plants, to tiny objects like microprocessors⁸. They help produce physical objects.

⁸ I once had a friend who was a chemical engineer by education. After university, he became a brewer with Carlsberg. Engineers also brew beer.

Programmers don’t do that. Software is intangible. As Jack Reeves pointed out[86], since there’s no physical object to produce, construction is virtually free. Software development is principally a design activity. When we type code into an editor it corresponds to engineers drawing plans, rather than workers constructing things.

’Real’ engineers follow methodologies that usually lead to successful outcomes. That’s what we programmers want to do as well, but we have to be careful to copy only those activities that make sense in our context. When you design a physical object, real construction is expensive. You can’t just try to build a bridge, experiment with it for a while, only to decide that it’s no good, tear it down, and start over. Because real-world construction is expensive, engineers engage in calculations and simulations. It takes less time and fewer materials to calculate the strength of a bridge than it does to build it.

There’s an entire engineering discipline that relates to logistics. People engage in meticulous planning because that’s the safest and least expensive way to build physical things.

That’s the part of engineering we don’t need to copy.

But there’s plenty of other engineering methodologies that can inspire us. Engineers also do creative, human work, but it’s often structured in a framework. Specific activities should be followed by other activities. They review and sign off on each others’ work. They follow checklists[40].

You can do that as well.

That’s what this book is about. It’s a guided tour of heuristics I’ve found useful. I’m afraid it’s closer to what Adam Barr calls the shifting sands of individual experience than to a scientifically founded set of laws.

I believe that this reflects the current state of our industry. Anyone who believes that we have firm scientific evidence for anything should read The Leprechauns of Software Engineering[13].

1.4 Conclusion

If you think about the history of software development, you probably think of advances at orders of magnitudes. Yet, many of those advances are advances in hardware, not software. Still, in the last fifty years, we’ve witnessed tremendous progress in software development.

Today, we have much more advanced programming languages than fifty years ago, access to the Internet (including de-facto online help in the form of Stack Overflow), object-oriented and functional programming, automated testing frameworks, Git, integrated development environments, and so on.

On the other hand, we’re still struggling with the software crisis, although it’s debatable whether anything can be called a crisis if it’s been going on for half a century.

Despite serious efforts, the software development industry still doesn’t resemble an engineering discipline. There are some fundamental differences between engineering and programming. Until we understand that, we can’t make progress.

The good news is that you can do many of the things that engineers do. There’s a mindset, and a collection of processes you can follow.

As the science-fiction author William Gibson said:

“The future is already here — it’s just not very evenly distributed”⁹

⁹ This is one of those quotes that have a nebulous origin. It seems uncontroversial that the idea and overall phrasing is Gibson’s, but exactly when he first stated it is unclear[75].

As the book Accelerate charts, some organisations use advanced techniques today, while other lack behind[29]. The future is, indeed, unevenly distributed. The good news is that the advanced ideas are free for the taking. It’s up to you to start using them.

In the next chapter, you’ll get your first taste of concrete activities you can perform.