1
The Perfect Storm of Complexity

When you want to know how things really work, study them when they’re coming apart.

—William Gibson, Zero History

For every complex problem there is an answer that is clear, simple, and wrong.

— H. L. Mencken

Humanity has always suffered plagues, famines, floods, and warfare. In modern times we have faced new horrors, such as nuclear weapons and AIDS. One common stance toward our current challenges is that we will adapt just as we have always adapted. The trouble with this stance is that our current challenges are profoundly different from those of the past. Our familiar modern responses no longer work because they’re based on a fundamental misunderstanding of what we are facing.

THE PERFECT CHALLENGE

Just how different our challenges are crystallized for me in the summer of 2008. It began with a mysterious call from two strangers. I met them in an empty cafe on Cowley Road in Oxford, not far from where I live. Both had been working in Yemen for a number of years. They wanted to know if we could help. I knew very little about Yemen and so asked them to explain the situation to me. The pair, Henry Thompson and Ginny Hill, spoke in hushed voices, occasionally looking around to make sure no one else was listening. I was bemused at their behavior and not quite sure what to make of them.

Yemen, they told me, was in serious trouble. It was collapsing. The facts were startling. Bordering Saudi Arabia and Oman, on the other side of the Red Sea from Somalia, Yemen occupied a geostrategic location due to the Suez Canal and its proximity to the oil fields of Saudi Arabia and the United Arab Emirates. One of the oldest civilizations in the Middle East, it also had the youngest and fastest growing population in the region, over twenty-three million people, 50 percent of whom were under fifteen.¹

First, Al Quaeda was using Yemen as a major base for operations. Second, the country was suffering from two incipient civil wars, which threatened to flare up at any moment. One was a secessionist movement in the south, and the other involved a religious minority in the north. In addition, Yemenis were running out of what meager resources they had: water, oil, food, and foreign exchange to buy food. Yet, Yemen had four times as many AK47s as people.²

Finally, they explained, the crux of the problem was that a cabal of criminals and quasi-criminals ran the country, a situation sometimes known as state-capture. This shadow elite lived behind anti-missile walls and in some cases held no official positions despite wielding great influence. When I asked about official channels, they looked at each other and shrugged. Could we help?

WHAT IS A COMPLEX SOCIAL CHALLENGE?

The situation in Yemen is a textbook example of a complex social challenge because of three characteristics: (1) the situation is emergent, (2), as a result, there is a constant flow of information to negotiate, and (3) this means actors are constantly adapting their behavior.³

Complex social challenges are emergent because their properties arise from the interaction of many parts. Imagine the difference between throwing a rock and throwing a live bird. The rock will follow a path that is predictable, that is, it can be predicted with a high degree of accuracy in advance. The path of the bird, on the other hand, is emergent, which means that path cannot be predicted in advance. It emerges from the interactions of many factors from the physiology of the bird to environmental factors. The system of the person (throwing the bird) and the bird is therefore said to be characterized by emergence.

In complex systems new information is constantly being generated.⁴ When we study a complex system, we are deluged by new information. If we tied a GPS to the bird and tracked its movements, we would be capturing a new stream of information about where the bird was going. (According to Nate Silver, “IBM estimates we are generating 2.5 quintillion bytes of data per day, more than 90 percent of which was created in the last two years.”⁵)

This new information gives rise to the third characteristic of a complex system, that of adaptive behavior. This means that actors in complex systems are constantly and autonomously adjusting their behaviors in response to new information. This feedback loop in turn gives rise to a whole new set of emergent characteristics. If our task is to re-capture the bird once it’s been thrown, then we use information to adapt our behaviors to ensure we succeed.

These three characteristics make complex challenges distinct by nature from technical challenges. Ronald Heifitz and his colleagues at Harvard’s Kennedy School of Government define a challenge as being technical when the problem and the solution are clearly defined.⁶ And they point out that confusing adaptive, or complex, challenges with technical challenges is a classic error.

An example of a technical challenge is sending a man to the moon. The problem is clearly defined and the solution unequivocal. Implementation may require solving many difficult problems, but the desired outcome is plainly understood and agreed upon. In contrast, multiple perceptions of both the problem and the solution are characteristic of complex systems.

Complex challenges are therefore dynamic and can change in unexpected ways over time, whereas technical challenges are relatively stable and static in comparison. The nature of gravity, for example, is not changing while we try to come up with solutions for putting a man on the moon. This is just one reason why it is hard to address complex social challenges.

In the past, everything was less connected. Today, interconnectivity is rapidly increasing, creating an age defined by its complexity. This connectivity has many dividends, but it also means that our landscape of challenges has changed dramatically in the last few decades. In the past, problems could be dealt with in isolation, while today, most of our most intractable social challenges are deeply interconnected. They don’t respect man-made boundaries, such as national borders. The nature of interconnectivity means that we are seeing challenges that are entirely new and fast changing.

These challenges are sometimes referred to as wicked problems, a phrase coined in the early 1970s.⁷ The trouble with the word wicked is that it makes us think that complex situations are somehow deviations from a non-wicked norm, that they are somehow temporary aberrations. And the problem, if you like, with the word problem is that it conveys the impression that everyone thinks of the situation as a problem (when some actors, typically those holding minority positions, might not).

One practitioner compares christening complex challenges as wicked to a story of a grandfather and the coming of cars.⁸ The grandfather couldn’t understand why cars didn’t behave like horses (resulting in many accidents) and considered them wicked. Much as we might love our grandfathers, calling complex social challenges wicked betrays a way of thinking that doesn’t make much sense today. Forty years ago we had just started to wrap our heads around the idea of complexity. Since then we have learned a lot, and many ideas from complexity science are in common use. Complexity is the norm for us—not an anomaly—and there is no returning to a simpler “non-wicked” time.

THE FUTILE OPTIMISM OF OPTIMIZATION

It’s 1959. The USSR is on the brink of Utopia. Comrades, let’s optimize!

—Francis Spufford, Red Plenty

Today it is common to address a wide range of complex social challenges using methods that are technical and planning based. Together they define a culturally dominant technocratic approach, which characterizes efforts at addressing challenges as diverse as public health care, environmental degradation, poverty, and inequality.

This dominant technocratic approach was born during the early twentieth century, a time when the belief that science would solve all human problems was widespread. The work of mathematicians such as Kurt Gödel and physicists such as Werner Heisenberg shattered this belief. By then, however, the technocratic paradigm had rooted itself deeply in an entire generation of problem solvers, who then passed it on.

Technocratic approaches typically seek to optimize, that is, to incrementally improve a situation through efficiency gains. For example, if ten thousand people are hungry, then a technocratic approach would seek to ensure that every day some of these people were fed, thus incrementally improving the situation. The end goal, of course, is to ensure that all ten thousand people are fed. This is a classic optimization strategy.

Optimization makes sense in some instances, such as when the number of hungry people is static and not increasing. Economists call this inelastic demand, as opposed to elastic demand.⁹ This means that if we manage to feed two hundred hungry people per day, in fifty days we would have fed all ten thousand people, therefore optimizing our way toward solving the problem of ten thousand hungry people.

This strategy is dramatically less effective in dynamic situations. Imagine that we feed ten thousand hungry people at a rate of two hundred per day. If, for whatever reason, the number of hungry people increases by 5 percent per day (compounded), then we’re in trouble. After five days of feeding two hundred people a day, we end up with just under 11,300 hungry people. After 10 days we end up with just over 13,300 hungry people, after 50 days we wind up with nearly 70,000, and so on. The dream of optimization, of course, is the other way round—that we increase the number of hungry people we feed every day by a percentage, which, when compounded over years, leads to a utopian society free of hunger.¹⁰ Of course, all of this assumes that gains will not be wiped out by unexpected events, such as a famine or some other natural disaster.

Another problem with technocratic approaches, including optimization, is that it addresses parts in isolation, rather than the whole. This could look like feeding a small number of hungry people by cutting down massive swathes of rainforest, which helps a small minority, while vast resources are spent with massive longterm negative impact. A side effect of optimization is that the underlying causal dynamics are frequently untouched.

This is what’s happening in Yemen with malnutrition. The system is generating more malnourished people every day than can be fed. Efforts to support them are helping small parts and are being outstripped by the dynamic nature of the challenge, where the problem as a whole is getting worse day by day.

The same logic applies to many issues, including climate change, deforestation, and poverty. Imagine that ten new light bulbs are turned on every second, each emitting a tiny puff of carbon dioxide. This pumps greenhouse gases into the atmosphere, which increases the risk of dangerous climate change.

An optimization response would be to turn off three light bulbs every second, striving in time to turn off four or five, and believe this is adequate. Unfortunately this leaves us with a net increase of emissions, despite our efforts. That is what’s happening with greenhouse gases dramatically increasing the probability of dangerous climate change.

All complex challenges have what could be thought of as an engine that produces the symptoms we are most concerned about, be that too many hungry people or too many greenhouse gas emissions. We see these symptoms as trends. For example, one of the trends governing the situation in Yemen is population growth, which, in itself, is not a problem. But when coupled with other trends, such as steadily declining agricultural productivity, we can see how it creates the complex social challenge of malnutrition.

This reflects a situation where demand for different forms of capital is increasing, including natural capital, such as fossil fuels and food. Simultaneously, there is a decline in our ability to meet this increasing demand sustainably. This is represented, for example, by declining forests, topsoil loss, less fresh water, and the shrinking envelope of carbon dioxide we can safely emit—which puts limits on how much fossil fuel we can safely burn. In other words, we are now hitting boundaries beyond which our actions seem to be causing irreparable damage to critical ecosystems.

It’s not simply that we’re running out of resources. The story is more complex. Ramez Naam demonstrates how we have used technological innovation to produce greater output from the same natural resources. For example, we have managed to dramatically increase yields from the same acre of land and convert greater percentages of solar energy into electricity. While the efficiencies are getting better and costs are dropping, they are not dropping fast enough to shift the underlying negative trends. Furthermore, market-based approaches have yet to figure out what to do with the environmental consequences of economic growth.¹¹

Technocratic approaches, therefore, represent a bet, a “grand wager” that our ability to optimize will be faster than the rate at which our problems grow.¹² If our problems are growing exponentially and our ability to optimize is growing linearly (or worse, declining), then we are staring at a mathematical certainty of collapse. This is what happened with the Soviet Union and what’s currently happening with many responses to complex social challenges across the world.

YEMEN AS A NATURAL EXPERIMENT

My first response to the request for helping in Yemen was “No, of course we can’t help.” The situation was too far along in its trajectory of collapse. Henry and Ginny wanted to bring the elite—including the shadow elite—into a room and run a scenario planning exercise on the future of Yemen.¹³ The elite would then see the implications of what they were doing to the country, and this insight would cause them to act proactively in the interests of the whole.

I pointed out that the shadow elite would not voluntarily step forward into such a conversation. Our usual approaches would not work with people who were loath to step up in any formal way, which is what defines a shadow elite.

Originally I assumed this situation was unique to Yemen. However, I later came across the work of Janine Wedel, a professor and author, who argues that the phenomenon of the shadow elite is widespread: “A new breed of players has arisen in the past several decades … whose manoeuvrings are beyond the traditional mechanisms of accountability. They, for example, play multiple, overlapping, and not fully disclosed roles.”¹⁴ And what she describes applies to Yemen as well as many other places, including the United States, Europe, and China.¹⁵

Even if by some miracle the shadow elite did agree to participate, I was dubious that such a top-down exercise would result in fundamental change. I offered advice relating to the nature of the problem but largely felt that I didn’t have anything useful to say. My two guests politely thanked me and left.

A few months later they invited me to a talk called “Crisis in Yemen: A Holistic Approach?” being given by a state department official, who had worked at the US Embassy in Yemen for many years. Out of curiosity, I put on a suit and tie and caught the train to London.

The talk was at the Royal Institute of International Affairs, known as Chatham House. Ginny worked there, helping organize a forum on Yemen. Officially a think tank, Chatham House serves as a global rallying point for those concerned with foreign affairs issues. This constellation, including both Yemenis and non-Yemenis, was out in full force that day.

As the talk ended, I turned to my neighbor and asked, “I might have missed something, but what’s the holistic approach?” He looked at me a little blankly and said, “Oh, he doesn’t really have a holistic solution, he’s just saying that we need one.”

Later, I quizzed organizers on the purpose of the talk. One person told me that the speaker was there to deliver a message to friends of Yemen. The startling message was that there was time to act in order to avert disaster in Yemen, but if this window was passed, the response would unfortunately shift to the Pentagon and the military planners.

Soon afterward I read a New York Times article with the headline “Is Yemen the Next Afghanistan?” which made me both intensely concerned and curious. In it Robert Worth writes, “I spoke to a number of American officials in Washington and to a variety of diplomats at the embassy in Sana. They all told me the same thing: no one has a real strategy for Yemen.”¹⁶

Over the next few years I went to Chatham House whenever there was a talk on Yemen.

My colleagues and I had worked on many challenges singularly: food and energy security, child malnutrition, water stress, and security issues. Almost all of these were happening in Yemen simultaneously, creating the perfect storm of complex challenges.

Yemen is what Jared Diamond calls a “natural experiment.”¹⁷ These are situations we could not recreate for reasons both practical and ethical. We are unable to cause a drought in order to study the effects of water instability on communities; nor would we do so in good conscience. Naturally occurring phenomena present us with options to study situations and learn from them.

Yemenis were facing down all the problems that other countries, regions, cities, and people were conceivably going to face in the future. A lot could be learned from examining not just the trajectory of challenges in Yemen but also the responses to these challenges. Yemen represents the future of a lot of places.

While the circumstances are unfortunate, Yemen is at the forefront of developing innovative strategies for how to address complex challenges. It is a little like the Dutch experience of building dikes. With the challenges of climate change, the Dutch are working all over the world helping communities build dikes to protect themselves from the rising oceans. It’s conceivable that the Yemenis will build a skill set to address a complex series of interlinked problems before anyone else.

The Yemen Forum gatherings I attended were consultations with civil society on what should be done. They were also often attended by Yemeni government representatives. But it seemed that few people had any faith in the ability of Yemen’s government to do anything.

At the end of a Chatham House event I chaired, I conducted an impromptu straw poll, asking the audience to raise their hands if they had faith in the government’s ability to come up with a centralized response to Yemen’s problems (as opposed to a decentralized one).¹⁸ The only people who voted in the affirmative were a handful of Yemen government officials sitting in the front row. To my astonishment, all other hands stayed down. I was astonished because these were the same people who told me that plan A for Yemen ran straight through its government.

When I asked one foreign office official (FCO) why they haven’t tried to catalyze a track-two effort, involving NGOs and other civil society actors as well as government, he gave me the official line: they had to deal directly with the government of Yemen because doing anything else would be seen as interference in the sovereignty of a nation. This was not even vaguely true in practice. When I asked the same question to the head of a UN agency struggling with a myriad of problems in Yemen, he responded, “Plan A has to be to work with the government; perhaps if that fails, we will examine a plan B.”

TOO BIG TO FAIL, TOO BIG TO JAIL

What seemed both obvious and crazy to me about Yemen was that everyone seemed to be saying that plan A had not only failed but had been failing for years. The unfortunate narrative in the international community was that the government of then President Saleh had little capacity to implement anything; it did not keep its promises and could not be trusted.

Every couple of years the government of Yemen would come up with a new plan and present a “Christmas list” of requests to the international community—asking for the plan to be funded. Each time this happened, the international community would demand assurances that the plan would be implemented, and of course very little actually happened.

The government representatives from Yemen who came to these meetings were repeatedly lectured on their failings by their Western counterparts. They sat with their arms crossed, listening mostly in silence, occasionally responding to questions or to say that they needed more resources and support. I asked one deputy minister how he managed to sit silently while being repeatedly patronized like that. He gave me a wry smile and shrugged.

I felt there was little cognizance in the international community of the nature of the challenges being faced by the Yemenis. From my work globally, I knew that it wasn’t as if someone else had figured out how to deal with these issues effectively, resource rich or not. It wasn’t simply that the Yemenis were doing a bad job, as was implied—they were also faced with a titanic set of challenges that no one anywhere really knew how to address.

Yemen was too big to fail. This idea, first popularized during the US 2008 financial crisis, applies to countries and development programs as well.¹⁹ In these situations, a system, be it a government or a program, is deemed as too politically sensitive to fail. So it is kept alive at massive cost, despite the fact that it may be failing in almost all dimensions beyond the political.

The situation in Yemen was fascinating because there was such widespread agreement about the failure of the Saleh regime, but the international community seemed to think it was powerless to do anything. Saleh was not simply too big to fail—he was too big to jail.²⁰ Indeed, during the Arab Spring, it was the Yemenis who forced Saleh to resign, but he managed to negotiate an immunity deal in which he would not be held accountable for anything that happened during his tenure.

What was I seeing? What were the stakeholders who came to Chatham House hoping would happen? What were they doing? It felt like people were operating on autopilot—they were all doing their jobs, and, almost regardless of what was actually happening, they would keep doing them. They were going through the motions of business as usual, or BAU.