Preface

I can tell you how much I enjoy being a geek. I can tell you how fascinated I was with the punch-cards my dad showed me back in 1985. I can tell you how I got my first computer when I was seven. And I can tell you that I’ve loved programming since 1989. I can tell you a great many things about all that, but I’m not sure how interesting they’d be.

Instead, let me tell you about my quest for an answer of sorts. There’s been one issue about our industry that has continued to puzzle me over the years: why is it that no software project is ever as simple as it seems? Why is it that no project ever comes out on time and on budget? Why are there always bugs? Why doesn’t it ever quite do what was intended? And why is it always so hard to make changes to the software? No matter how clean the slate is when a project starts, why does it always become a big ball of mud?

Almost everyone acknowledges the problem, and they seem to accept the status quo. Most of our industry deals with it by adding buffers to schedules and budgets, and by accepting mediocre software. Isn’t there a better way?

This book is not the answer, not by a long shot. But it is part of my exploration of the way forward. It is my notion that better tools can help us create better software.

This raises the obvious question: what is a better tool? Better at what? I believe the answer is that a better tool is one that helps manage complexity better. After all, complexity is one of the root causes for the state of things in our world. Indeed, Fred Brooks wrote about complexity in a paper as early as 1986. He drew a distinction between essential complexity and accidental complexity. Essential complexity is inherent in the problem domain, whereas accidental complexity is introduced by things external to the problem domain. For example, in a software project that deals with filing taxes, complexity that arises from convoluted tax-codes is part of the domain, and hence essential. Any complexity that arises from, say, employing the rather intricate visitor pattern, is accidental.

So let me rephrase my statement: a better tool helps us minimize accidental complexity. It lets us do our job as best as we can, while getting out of the way. And great tools go beyond that; they give us leverage. They let us amplify our effectiveness as designers and programmers, without introducing problems of their own. The Lisp programming language was designed to be just such a tool. And Clojure is an amazingly well designed Lisp.

Every programmer who stumbles onto Lisp has a story, and mine is similar to many. I started my professional career with Java, and eventually ran into a wall with what I could create with it. I started exploring dynamic languages and they felt more expressive and malleable. Mostly, I enjoyed using Python and Ruby, and wrote several nontrivial applications with them. I was working at a company called ThoughtWorks at the time, and I had a lot of like-minded colleagues to work with. Eventually, one of them turned me onto Common Lisp. The more I read about the language, the more I began to realize how primitive other languages were. I used Common Lisp on a few personal projects, but never did anything major with it; it did however have a profound effect on my code in all the other languages I was using, and I kept looking for an opportunity to use a Lisp on a real-world project.

I finally got my chance in 2008. I had moved to the Bay Area in California, and ended up joining the founding team of a startup named Runa. In true Silicon Valley tradition, our first office was in the founder’s garage. We wanted to disrupt the world of eCommerce with Runa. The idea was to collect lots of data, use machine-learning techniques to make sense of it all, and then present personal deals to select shoppers in real-time. And in order to do all that, we had to overcome serious technological challenges. The system needed to handle thousands of requests a second. It needed to handle several terabytes of data a day. It needed to be scriptable via a set of high-level, declarative DSLs. It needed to support hot code-swaps so it could be updated on the fly. It needed to run on the cloud, and it needed to be entirely API-driven. And we had to build it without much in the way of resources; we were an engineering team of three.

With these kinds of constraints, we needed a language that gave us leverage. So we turned to this new language called Clojure. It was a modern, functional language that ran on the JVM. It also promised to solve the problems inherent in concurrent, multithreaded code. And it was a Lisp!

I was the architect at this startup, and am now the VP of Engineering. I staked the success of our future on this new (pre-release at the time) programming language created by someone who I had never heard of before. But everything I read about it resonated with me; all the pieces fit. We’ve been using Clojure ever since with incredible success. Our team has grown over the past three years, but it’s still about an order of magnitude smaller than other teams at similar companies. I suspect they’re using plain old Java. If nothing else, the past three years have upheld my belief that tools matter, and that some are far superior to others.

When we started out, I used to think of our usage of Clojure as our secret weapon—but the Clojure community is so strong and supportive that making it an open secret seemed like a much better idea. I started the Bay Area Clojure User Group, and we’ve now got hundreds of members. I like to think there are dozens of people who have come to our meetings, liked what they heard, and decided to use Clojure on their own projects.

In that same spirit, I wrote this book to share my experience with Clojure with you. It’s my hope that I can convince some of you to look beyond the parentheses to what is possible with a Lisp in general, and with Clojure specifically. I hope you find this book useful and enjoyable.