CHAPTER 2

Smell the Problem

An essential part of seeing clearly is finding the willingness to look closely and go beyond our own ideas.¹

—CHERI HUBER, THERE IS NOTHING WRONG WITH YOU

All too often, attempting to solve hard problems is done almost exclusively at a desk, in a conference room, or behind a computer. Or, if you’re in the field, you’re just taking action, trying out different solutions. For problems of any significant difficulty, you need to start with your hands in your pockets, and your senses open. You need to get out there and study the problem in detail.

As a problem-solver of any system—be it a machine, a circuit, a code base, your body, or your habits—your mission is to smell the problem. Get out there and thoroughly describe the problem in detail with all of your senses: Record your observations of when and where the problem is occurring. Get the information you need to understand the problem, but don’t just get reams of data that you’re hoping will help you guess the solution—getting to the root cause of the problem comes later. Go back to smelling the problem more as you need to.

GOOD PROBLEM-SOLVERS ARE PROBLEM-SMELLERS

People have been solving problems since the early days of human civilization. Let’s use an example everyone is familiar with: going to the doctor. In the days of Hippocrates in ancient Greece, doctors had little technology at hand to diagnose illnesses. Back then before lab testing, doctors would poke and prod, bend an arm, listen to your stomach or chest. They’d even smell wounds for gangrene and infection, smell your breath or your stool to learn more about gastrointestinal problems. They were very rigorously studying the problem and the patterns of failure.

In modern times, good doctors study the problem before coming up with potential diagnoses or even running more complex or expensive tests. They’ll also poke and prod you. They ask you to move, bend, twist, cough, and report how you feel. They listen with a stethoscope, and take your temperature, blood pressure, and pulse. They want to know what you ate or what your physical activity has been. If the stakes are high or the cause is hard to pin down, they will schedule an X-ray, lab work, or an MRI, but only after first using simpler methods.

Doctors will also ask you to keep a log of what you eat or your physical activity if you have a chronic or recurring problem that they can’t diagnose by looking at your body. You’ll write what you ate, what you did, and how you felt, with an eye to the specific problem. Great problem-solvers will make sure their data, too, is robust and problem-specific.

Psychologists and therapists do the same thing. They have a battery of questions. They dig into your past. Good ones may spend hours dissecting your mind before attempting to diagnose. It’s a long way from your friend telling you that you need a better attitude, after listening to you for a few minutes and jumping to a solution.

You can do this, too. A friend of mine has asked me whether he was drinking too much and if he has a problem with alcohol. Now I don’t think I am particularly qualified to make this judgment, but I can help him look at what is actually going on. Most people have simply told him either, “don’t worry about it,” or “yes, you should drink less.” How they have the information to draw that conclusion beats me. I have had him smell the problem by keeping a diary of how much he drinks and when, as well as noting his emotional state when doing so. Where it will lead I am not sure, but getting actual data will allow for progress on what could be a hard problem with serious consequences.

This approach, smelling the problem, can give you lots of insight about the nature of the problem you are experiencing. With simpler problems, establishing a strong pattern of failure can give you the solution on its own.

HOW TO SMELL WELL

Early on you’ll want to smell the problem to develop a pattern of failure. Where possible, understanding where the problem does and doesn’t happen, when the problem started, and how often the problem occurs will generate critical insights for the problem-solving effort. For easy or fairly moderate problems, this pattern of failure can lead to the key insight that solves the problem or puts you just one step away.

A friend of mine was helping his mother understand why her car’s push-to-talk (PTT) feature was malfunctioning. When she pushed the button and asked it to play Lady Gaga, it would sometimes work great, and she could rock out. But intermittently, it would bring up the car’s navigation feature. It was driving her up the wall (blessedly, only figuratively so). The car dealer and its shop couldn’t figure it out, and had actually written it off as a software bug, a short circuit, or entirely nonexistent and all in mom’s head. They wasted lots of time, nearly lost a customer, and even offered to replace the car, because they didn’t smell the problem.

My friend, while visiting his mother one weekend, smelled the problem by repeatedly mashing the button and using all of his senses. A subtle pattern emerged: there were different numbers and tones of beeps just before navigation and just before music. He noted this, read the owner’s manual with this knowledge, and learned that you needed to hold the PTT button for more than two seconds to access music. He instructed his mom and solved the problem permanently. Now of course others had read the manual before him, but just the section on the onboard computer was dozens of pages, and they didn’t know what they were looking for.

There are literally dozens of problems like this that we encounter every year that waste our money, try our patience, and consume our time. Simply smelling the problem well can help you resolve them and lead a better life.

DEVELOPING A PATTERN OF FAILURE

There are problem-solving methods that can help us add rigor and practice. They ask thought-provoking questions that guide you to collect information and look for very specific patterns, rather than shotgunning and looking at everything in the system. These vary in their level of rigor, detail, and prescription. In Chapter 10, “How to Choose Your Method,” we’ll discuss how to select some of these methods over others. For now, know that when you’re dealing with hard problems, having guidance is extremely helpful, but not exhaustive. Hard problems tend to be unique, so use these as a guide rather than a recipe. As you develop insight, you’ll come up with your own questions to ask.

I can provide some basic guidance on where to get started in developing a strong pattern of failure (see Table 2.1). Most generally, describe in detail the conditions in which a problem does and does not occur.

All of the questions in Table 2.1 are guides, not directions. Look at one occurrence and many together, when possible, and see what you discover. Again, you are not trying to guess the solution. You are simply trying to understand the facts of how the failure manifests.

A friend of mine renting an old house had been experiencing his computer shutting down occasionally because it temporarily lost power. This had of course been very frustrating as it meant he would lose work or at least be interrupted from whatever he was working on as he went to flip the breaker. He had been “living with it” for a while, but when he lost some important work he got fed up. As he went for the third time into the kitchen to complain of his computer failing, he realized that he was always complaining to the same roommate—and that the roommate was always heating up leftovers in the microwave. As it happened the house had a very large, powerful microwave, and his room was next to the kitchen.

Once my friend had noticed this pattern, he could quickly deduce the answer to this fairly simple problem: The very large microwave might be tripping the breaker when other appliances (and his desktop computer) were all running at the same time. He tested it and saw the trip happen again. He smiled and grabbed an extension cord to move his computer’s supply to his opposite wall. While he and his housemates waited for a less aggressive replacement microwave, he taped over that outlet to make sure he didn’t use it by mistake.

BREAKING DOWN BARRIERS

Great problem-solvers will overcome barriers to getting the specific information they need. If a machine is moving too quickly for a problem-solver to see a pattern with their own eyes, a great problem-solver will get a camera to record it, and then slow it down. If a manufacturing line doesn’t automatically count units produced, great problem-solvers get out there and do the counting. They work closely with those who best understand the system or process to find where to look to answer each question and find the clearest information.

In a chemical processing plant, some very large (10-ton) pumps were breaking down every 3 months, costing the company tens of millions of dollars per year and posing a safety risk, as containment was lost during failure. The plant had spent years and tens of millions of dollars repeatedly upgrading the pumps to be able to add bigger, harder seals into the pumps to prevent the failures—but they just kept coming.

After yet another failure, a new team had been formed to work on this problem, and it included some members of my team. After the next pump failure occurred, rather than work on even bigger seals back at the desk, we insisted on looking at the pump as it was disassembled in order to better understand what was happening. The team found once again that the seal had been eroded, and there was a smattering of black solid particles all over the seal, mixed into the lubricant. We decided to “smell it” chemically, running it through the lab: The techs there found that the particles were actually an oxidized (or literally “cooked”) version of the very chemical that the pumps were pumping. This was a huge insight for the team and set them up to quickly solve this long-thought impossible problem by digging into the fundamentals. You’ll see how this problem was solved in Chapter 5, “Dig Into the Fundamentals.”

USING PROBLEM-SMELLING TO BUILD ALIGNMENT

Getting a good pattern of failure can build organizational belief that a problem is solvable. Often hard problems require a commitment of resources and attention from an organization to solve them and implement a solution, and a good pattern can develop buy-in to muster these resources.

One of my favorite examples of this comes from working with a national drink brand that was selling out in stores. The demand for this product was expanding rapidly as the marketing and sales teams had done a phenomenal job. There was huge pressure to increase production to retain as much of the available market share as possible, because copycat products were starting to fill empty shelf space. The organization was planning to build new plants and lines to do this, but it would take 18 months. In the meantime, the existing facilities were working furiously to get everything out the door that they could, and I was brought in to help.

Our analysis turned up a number of opportunities to immediately increase production. The most interesting was to increase the speed of one of the lines. The idea was understandably met with some resistance: Everyone knows that if you run faster you will make more stuff, and the local team knew this, but they also “knew” that the equipment was not capable of going faster. They proved this to me by turning up the speed and creating a huge mess of half-filled bottles flying onto the floor that three of us had to clean up.

I realized that the problem I needed to solve first was to give the local team hope and belief that the speed could be successfully increased (I’ll talk more about figuring out what problem to work on in Chapter 4, “Know What Problem You’re Solving”). Luckily, a conversation with the VP got me permission to work with the operator to test the speed and smell the problem, and we learned a lot. First, we saw that the bottles were kicked off the line due to being underfilled. With careful study I saw that the underfilled bottles came from the same three (of 36) filling heads.

Once the operator and I showed this to the VP, it was clear the problem was solvable, and we got the team on board to move forward. After all, if 33 of the filling heads work then the other three could be made to work as well. Pattern of failure alone had not solved the hard problem of how to get more output from the filler, but it had helped me solve the political one of how to get the team on board.

HOW MUCH IS ENOUGH?

Developing a thorough problem description and pattern of failure is not a matter of gathering reams of data and burying yourself with them. Bad problem solvers in businesses will download long histories of data from points along the entire process or even measure lots of new stuff to try to find something useful. They’ll compare every single part between a “good machine” and a “bad” one. These efforts waste a lot of time and money at best, and lead to red herrings in hard problems that mean bad solutions, wasted money, new problems, and a loss of credibility.

Great problem solvers develop the questions that they want to answer before they go about collecting information and data, rather than depending on whatever data streams they see. They grab the signal, not the noise.

But when are you done with describing the problem or developing a pattern of failure? In short: never. As you develop insight and dig into the fundamentals, you’ll be coming back iteratively to keep smelling: You’ll have new understanding to ask new questions. This isn’t a step: It’s a behavior. Smell the problem to answer questions about it that arise.

If you are a less experienced problem-solver then you will want to practice this rigor on some easier and moderate problems in order to hone the skills you’ll need to make rapid progress on harder ones. Some hard problems may only occur once or be very hard to see; being a skilled problem-smeller helps you make decisions when you face these more challenging scenarios.

NOW: SMELL THE PROBLEM

Get yourself out of your chair and into the field with a notebook. Start gathering information about your problem, and ask the questions in Table 2.1. But most importantly, practice using your powers of observation, with all of your senses, to sift out what is actually happening at the problem. And remember to not distract yourself with solution-guessing!