12


Fault tolerance, randomness and pattern

In this chapter I will delve into what good, imperfect management means. It absolutely does not mean being satisfied too easily. It does mean striving hard to make the most of what you have, calibrating and reviewing and above all dealing credibly with the real world, a world in part always unknowable, ungraspable, unclear, unhelpful and chaotic.

Living with imperfection

One of the great objectives of 21st-century engineering science is to create machines and computers which are ‘fault tolerant’. This means that they are able to cope with mis-programming, defective instructions or circumstances that don’t exactly fit the preordained bill. The key, though, is that having recognised the fault, the failure, the problem, or simply the new circumstance or environment, the machines do something about it, they adjust, they modify their approach.

Organic life has developed and evolved by becoming increasingly and, in ever more sophisticated ways, fault tolerant. Our bodies repair injuries and cope with a very adverse range of unexpected circumstances, and our consciousness is partly about re-computing the circumstances around us in all their adversity and unexpectedness and modifying our behaviours to accommodate them.

As for people, so for businesses. Think of it not as a matter of crude occasional correction but as always and constantly at the very heart – or in the genes – of an organisation. You want it to be hard-wired in: a subtle and complex process, endlessly adjusting and re-computing what it faces, and each time acting there and then on what it knows.

It is essential to emphasise that this approach is sharply differentiated from, indeed the converse of, slackness and simply allowing mistakes to accumulate. That is a completely different sort of tolerance: the process of inattention, of ploughing on regardless, which elsewhere in this book I categorise as the road to ruin. The process I am describing here is one where you acknowledge, identify and respond to failures, recognising they signal a new or altered problem which can then be coped with. It is quite different from tolerating mistakes but not learning from them, not amending actions as a result of them. That is the worst of both worlds.

Fault tolerance is necessary because the world outside is constantly changing. It will always defy our expectations and challenge our insights. Much of the future is unknown and some of it is perhaps unknowable. As you travel on, your horizon changes. The language you use, though, in these circumstances, tells you a lot about what you are seeing and how you are behaving. Four terms pop up again and again here: randomness, chance, inevitability and luck. They need to be treated with care because they easily become the excuse, the get-out clause. If it really is random or chance then the outcome is down to luck or inevitable. But is it?

Managing the unknown

People have a tendency to characterise events that they can’t explain as random or down to bad luck. This can imbue them with a sense of fatalism. But a willingness to confront the unknown, I argue in this chapter, is critical to successful management.

In fact, many apparently unexplained events have an underlying pattern and are, at least in part, predictable. The real world is one of imperfect data, partial understandings and chance discoveries. Most of the time people operate within the area of the known, while trying to cope with what is on the edge, and sometimes with the totally unknown. Donald Rumsfeld’s originally much-derided but now universally quoted categorisation of situations into ‘known knowns’, ‘known unknowns’ and ‘unknown unknowns’ actually succinctly expresses this reality, although I would add a fourth one: ‘unknown knowns’ (what we think we know but actually don’t).

There are always unknowns. They occur again and again in everyone’s personal and working lives. Dealing with them is what life is all about and, for that reason, they are best seen not as barriers but as challenges and opportunities. Each area of unknown that is identified, understood and then managed moves across into the area of ‘knowns’.

By contrast, what people don’t realise they don’t know – unknown unknowns – is a huge limit to their ability to fix things. Some of the most poignant examples of this come from schemes and organisations taking a leap forward in information, often in the form of shiny new IT schemes. NPfIT, the comprehensive multi-billion NHS IT investment programme covering the last several years was the largest IT investment in the world. It was implemented principally as a technical scheme. In fact, its potential for success lay not in how technically advanced it was but in how effectively it took into account the ways hospitals work and how doctors, nurses and patients behave. By not taking these basic factors into account – because those who managed it didn’t realise they didn’t know this – it went spectacularly wrong.

The manager’s task is to find the pattern, whatever it is, and use it to reduce gradually the area of the unknown and the unmanageable. By breaking the problems down into small manageable bits, much of what appears to be random can be distilled into patterns based upon what people do, when, and what they need. There will always be a residue that is beyond anyone’s control, but by reducing the ‘unknown’ substantially, management can increase its capacity to act, and then have a fresh go at what remains.

London Transport

Not long ago, waits, difficulties and delays in accessing transport across London were perceived as inevitable because of variations in demand, necessary bottlenecks and the general unpredictability of problems. But London Transport showed that you could reduce queues and increase usage. It began with the underground system and straightforward things like simplifying fares to speed up payment, increasing capacity where it could and rationing (often by long ticket queues) where it couldn’t. Automated ticketing helped, but the key innovation was probably the introduction of the Oyster card, which meant that people had an automatic, easily renewable way of travelling which avoided entry queues. Travellers were financially incentivised to use the cards, which further reduced queuing, and the system was extended to London buses, which in turn increased their popularity and encouraged cross-usage between different forms of transport. As the usage increased, other measures were taken to extend this easy on–easy off system, which is highly user friendly. Means were found to offer it to casual visitors, particularly tourists, and to extend it to the rail network within and around London. Bit by bit, travel across London by public transport became easier and more pleasant, with a positive outcome for the providers as well as their customers.

The 2012 London Olympics showed how good and how interconnected a system it has become, despite the ill-informed predictions of many Jeremiahs and Cassandras in the media of inevitable breakdown, meltdown and chaos. It coped with huge volumes and peak flows without waits or inconvenience.

Turning randomness into patterns

Is something really random or does it just appear random to me because it hasn’t been properly analysed? Some form of pattern, of coherence, underlies practically everything, and what people are often saying when they say that something is random is that they don’t understand this pattern and because of that – or, possibly, anyway – it is outside their control. But until patterns are detected and made explicit, you cannot know what you can control and what you can’t. Nor is an understanding that you can’t control something the end of it. Once you know what you can’t control, then you can start to develop avoidance strategies – contingencies that enable you to manage nonetheless.

Reducing waiting times in emergency departments

Patients turning up at a hospital with an emergency problem have to be dealt with. That’s why emergency departments exist. And one thing that is not at all unusual in them is a wait or a queue to be seen. This is true in the USA, in the UK, in Canada, in Sweden and no doubt in many other places around the world. The universality of these waits prompts many people to say that, unless you put enormous resources into solving this problem, then people will just have to wait because we can’t predict who is coming in when. So when the UK government said that long waits in casualty were no longer acceptable and that all patients must be seen and treated within four hours, there was an outcry. It was impossible. I don’t make this comment from a position of superiority. I was one of the hospital CEOs who said it!

But it was not all down to chance. A third of the patients only needed simple, straightforward treatment. So by setting up special units within emergency departments staffed by nurse practitioners to deal with ‘minor injuries’, 33% of the demand could be siphoned off. Next, there turned out to be predictable patterns depending on the day of the week or the time of day. The reason why many patients came in when they did was that a couple of hours earlier their GP had told them to. The likely timings of such interactions were highly predictable. So it was possible to amend staffing to reflect when more or fewer patients would be turning up at the door.

And so on. The random was reduced from 100% to 70% to 40% to 25% and lower. Countrywide over 95% of patients were now being seen within the ‘impossible’ four-hour target time.

The random had been turned into pattern and process. Word seeped out to the separately and differently run NHS systems in Scotland, Wales and Northern Ireland. They copied and they improved. Across Canada, where they had similar problems, they used the UK learning, and they improved too. So did Sweden and other countries.

So dealing with apparent randomness is about finding the pattern and the process within it and taking precautionary or preventive action. For those who don’t, it’s a matter of ‘bad things always happen to me’. At the end of the famous film The Big Sleep, the beautiful blond turns to Bogart and says, ‘Wish me luck, I got a raw deal.’ In his unimpressed, laconic way, Bogart simply replies, ‘Your kind always does.’ Which kind are you?

Randomness transmutes into chance and chance transmutes into inevitability with surprising ease. Efforts to reduce infection from the superbug MRSA bring some of this out.

Is MRSA inevitable?

There are huge variations in the prevalence of MRSA in hospitals in different countries, and in the UK it was relatively high. In Holland it is tiny, and in Denmark it is very low. The Dutch and the Danes don’t think this is a matter of chance. They think it is a matter of their approach.

Even within English hospitals, there were huge variations in MRSA infections but the setting of a target reduction prompted cries of unfairness. It wasn’t their fault. It was down to geography and social conditions. And because they assumed it was inevitable, they didn’t make enormous efforts to avoid the things they could avoid. On the other hand, those who took the stance that things weren’t inevitable and they could do things, behaved proactively and in a precautionary way, and their ‘inevitable’ MRSA infections reduced.

And what about chance? The famous scientist Louis Pasteur noted that ‘Chance favours only the prepared mind.’ If you don’t know what is going on, you might well assume that no one does and no one can. You will almost certainly be wrong. Alternatively, you can start looking and keep looking, and you’ll probably find something.

Spikiness in MRSA infections

In visiting a whole range of hospitals that were trying to reduce their MRSA cases, I became used to looking at statistics based on small numbers. Typically the hospitals I visited would have had between zero and seven cases per month, so interpretation of variation required care. Very often people said, ‘This peak [interestingly, it’s usually a peak and not a trough]) was just random variation. There is nothing we can do about it and next month or the month after it’s bound to be lower.’ But – one of my recurring themes – this isn’t necessarily so.

A hospital team noted that over the previous two years they had had peaks or spikes in the same two months. At first sight this seemed chance, but then they asked themselves if anything unusual went on in these two months. After much sifting and questioning, one event was peculiar to these four months but not to the other 20: twice a year, large numbers of junior medical staff moved on from their training posts and were replaced by newer, less experienced and, at that point, less well-trained staff. It was in their first month that the spikes occurred.

Their inference was that this was likely to be due to the pressure on new and inexperienced staff, the fact that what they needed to do to prevent infections hadn’t been highlighted to them clearly enough and they hadn’t fully absorbed it, with the result that they were likely to be making more mistakes, and engaging in less good practice. So the hospital at the next intake upped its emphasis in induction training on all aspects of infection prevention and control. The next time the figures came out for that month there was no spike; nor was there for the next changeover month. Having heard about this, we then found another hospital that had come to the same conclusion and had done the same thing with the same positive result.

Conclusion: This improvement is ready and waiting for everyone. Ascribing it to random variation means it’s unlikely to be secured.

Talent or luck?

Those of us who are not great sportsmen or sportswomen can all too easily assume that ‘greatness’ is a matter of being born with exceptional talents. But while this might have some part to play, people know in their heart of hearts that it’s not entirely that way. Those who succeed may have natural talents but they have to try – and try very hard. Arnold Palmer, one of the greatest golfers of the twentieth century, gave his take on it: ‘It’s a funny thing. The more I practise, the luckier I get.’ You need to practise at everything you do, to be relentless, obsessive even, never to give up, to get into that virtuous cycle of continuous improvement, recognising your imperfection but being restless with it.

And what about that strange and perhaps elusive commodity, luck? One of my favourite accounts of this is in the science fiction TV series Red Dwarf. In a particularly difficult situation and on a distant planet where the task is to unpick the most appallingly complex series of choices and make the right one, the protagonist discovers a series of positive viruses among which is a luck virus. If you become infected by it then you keep making lucky choices.

We know it isn’t quite like that, but sometimes I think it isn’t quite unlike it either. Napoleon was once asked why his generals were so lucky. He replied that he only appointed lucky generals. It’s a funny, paradoxical comment and at first sight it seems only to say, ‘I have great judgement in choosing the right sort of people with the right sort of attitude to make things happen.’ And indeed that seems to be part of what he meant. But he went on to explain that when he was looking to choose a general, he went round the companies of his commanders, who were candidates to be new generals, and asked people what they thought of the candidates. When he was repeatedly told that a particular person was lucky, he appointed them. His reasoning was that people saw repeated success and, partly out of envy, partly out of disbelief, were reluctant to ascribe it to skills and judgement, although these were the real causes. Instead they ascribed it to luck.

Dealing with bad luck

An organisation I was managing was recovering well from its previous failures. We were visited by a national inspection team and assumed they would see the improvement. When we received the report, we were shocked. It gave us the lowest possible markings on practically everything and painted a picture of an organisation that was getting almost nothing right. Our view was that the inspection team had come with an unconscious prior view that things must be bad and during their visit sought and found confirmatory evidence.

As we went through the report, we saw how it kept emphasising the worst of our failings and omissions without crediting us for the multitude of things we had done or were doing to make things better.

We were told that we had ten days in which to respond and that the response would have to be restricted to questioning matters of factual accuracy. We noted that the ten days included the Easter holiday. On the day we received the report, I called a meeting of all the people whose areas of responsibility were affected by the report. The chorus of response was loud and universal. It went something like this. ‘This report simply isn’t fair. What can we do to get the truth across?’ I had also realised that, on the basis of the scores that we had been given, I would probably lose my job despite our other successes.

We went through the draft report point by point and when we felt a point was inaccurate or misleading we assigned individuals to marshal supporting evidence and rebut it. We created a small but highly dedicated team to coordinate this work and immediately start drafting our response. We met daily, assembling more and more detailed evidence that the statements and conclusions were not soundly based. We started to interlink themes and form a coherent narrative.

We completed our response within the deadline and sent it with a dozen boxes of supporting documents. I sent a cover letter pointing out our concern at the amount of time we had had to raise questions on the report, and to assemble our case. I asked for personal assurances that all the work and supporting material we provided would be examined and considered. I asked the Director to monitor what was done with our response.

We heard nothing for two weeks, then we received a letter saying that we had sent so much material that they were not yet able to respond. Two weeks later we received a second response saying that it would be three months before we got a definitive response. Some months later we were given a revised upgraded marking that took us out of the bottom zone.

To sum up …

Maybe luck isn’t that elusive. Maybe it’s a matter of choice and persuasion and approach:

  • You will get your share of good and bad luck. That isn’t an excuse for inaction. Inaction will lead to failure.
  • Disbelieve in the chance, the random, the unknowable, the unforeseeable.
  • Seek out your luck, seek patterns, seek variations, seek explanations.
  • When you get it, don’t pause to say you are doing too well. Run with it.
  • You will have earned it, you will need it and you can use it.
  • That way the imperfect manager becomes the lucky manager, who becomes the successful manager.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.135.213.212