Dreyfus at Work: Herding Racehorses and Racing Sheep

Now that we’ve looked at the Dreyfus model in detail, let’s see how to apply the Dreyfus lessons at work. In software development at least, it turns out that we tend to apply them pretty poorly.

Experts aren’t perfect. They can make mistakes just like anyone else, they are subject to the same cognitive and other biases that we’ll look at later (in Chapter 5, ​Debug Your Mind​), and they will also likely disagree with one another on topics within their field.

But worse than that, by misunderstanding the Dreyfus model, we can rob them of their expertise. It’s actually easy to derail an expert and ruin their performance. All you have to do is force them to follow the rules.

In one of the Dreyfus studies, the researchers did exactly that. They took seasoned airline pilots and had them draw up a set of rules for the novices, representing their best practices. They did, and the novices were able to improve their performance based on those rules.

Rules ruin experts.

But then they made the experts follow their own rules.

It degraded their measured performance significantly.^[11]

This has ramifications for teamwork as well. Consider any development methodology or corporate culture that dictates iron-clad rules. What impact will that have on the experts in the team? It will drag their performance down to the level of the novice. You lose all competitive advantage of their expertise.

But the software industry as a whole tries to “ruin” experts in this fashion all the time. You might say that we’re trying to herd racehorses. That’s not how you get a good return on investment in a racehorse; you need to let them run.^[12]

Intuition is the tool of the expert in all fields, but organizations tend to discount it because they mistakenly feel that intuition “isn’t scientific” or “isn’t repeatable.” So, we tend to throw out the baby with the bathwater and don’t listen to the experts to whom we pay so much.

Conversely, we also tend to take novices and throw them in the deep end of the development pool—far over their heads. You might say we’re trying to race sheep, in this case. Again, it’s not an effective way to use novices. They need to be “herded,” that is, given unambiguous direction, quick successes, and so on. Agile development is a very effective tool, but it won’t work on a team composed solely of novices and advanced beginners.

But forces in the industry conspire against us in both directions. A misguided sense of political correctness dictates that we treat all developers the same, regardless of ability. This does a disservice to both novices and experts (and ignores the reality that there is anywhere from a 20:1 to 40:1 difference in productivity among developers, depending on whose study you believe).^[13]

The journey from novice to expert involves more than just rules and intuition, of course. Many characteristics change as you move up the skill levels. But the three most important changes along the way are the following:^[14]

• Moving away from reliance on rules to intuition

• A change in perception, where a problem is no longer a collection of equally relevant bits but a complete and unique whole where only certain bits are relevant

• Finally, a change from being a detached observer of the problem to an involved part of the system itself

This is the progression from novice to expert, away from detached and absolute rules and into intuition and (remember systems thinking?) eventually part of the system itself (see the following figure).

---

Figure 3. Dreyfus model of skill acquisition

The Sad Fact of Skill Distribution

Now at this point you’re probably thinking that the great bulk of people fall smack in the middle—that the Dreyfus model follows a standard distribution, which is a typical bell curve.

It does not.

Sadly, studies seem to indicate that most people, for most skills, for most of their lives, never get any higher than the second stage, advanced beginner, “performing the tasks they need and learning new tasks as the need arises but never acquiring a more broad-based, conceptual understanding of the task environment.”^[15]  A more accurate distribution is shown in the figure here:

Most people are advanced beginners.

---

Figure 4. Skill distribution

Anecdotal evidence for the phenomenon abounds, from the rise of copy-and-paste coding (now using Google as part of the IDE) to the widespread misapplication of software design patterns.

Also, metacognitive abilities, or the ability of being self-aware, tends to be possible only at the higher skill levels. Unfortunately, this means practitioners at the lower skill levels have a marked tendency to overestimate their own abilities—by as much as 50 percent, as it turns out. According to a study in Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments [KD99], the only path to a more correct self-assessment is to improve the individual’s skill level, which in turn increases metacognitive ability.

You may see this referred to as second-order incompetence, not knowing just how much it is that you don’t know. The beginner is confident despite the odds; the expert will be far more cautious when the going gets weird. Experts will show much more self-doubt.

Unfortunately, we’ll always have more advanced beginners than experts. But even though it is weighted at the bottom, it’s still a distribution. If you’re lucky enough to have an expert on your team, you need to accommodate them. Similarly, you need to accommodate the few novices, the many advanced beginners, and the small but powerful number of competent and proficient practitioners.

The hallmark of the expert is their use of intuition and the ability to recognize patterns in context. That’s not to say that novices have zero intuition or that competents can’t recognize patterns at all but that the expert’s intuition and pattern recognition now take the place of explicit knowledge.

Intuition and pattern matching replace explicit knowledge.

This transition from the novice’s context-free rules to the expert’s context-dependent intuition is one of the most interesting parts of the Dreyfus model; so our goal, for most of the rest of this book, is to see how we might better harness intuition and get better at recognizing and applying patterns.^[16]