Nestled in the gentle hills of California's Silicon Valley sits Stanford University: the breeding ground for the area's innovators since a horse-stud farm was converted into the original campus in 1891. Yet this university is renowned for more than being the intellectual hub of the most innovative community in the world's leading technological nation. In the 1960s and early 1970s Stanford University became known as a major centre for psychology – and particularly for a series of ground-breaking experiments on children that were to shape thinking on motivation, drive and success.

And while many of the experiments have been forgotten by all but a select group of professionals, one has entered the realms of folklore: the 1972 marshmallow tests on impulse control.

Brought into a room and given a single marshmallow, a succession of four-year-old children were then offered a choice: eat the marshmallow now, or resist for 15 minutes and receive a second marshmallow as a reward. Unbeknown to them, this simple choice – dividing the children into those that managed to wait for the additional marshmallow and those that didn't – revealed a fissure that would potentially run right through their lives, according to the psychologists at Stanford (led by Walter Mischel). This was between those able to defer gratification – and therefore develop productive, future-oriented organizational competence – and those preferring impulse-driven instant gratification, who were thus condemned to organizational incompetence and underachievement.

These were not stupid kids. They were mostly the offspring of campus professionals or graduates, so were likely to be destined for strong educational attainment. Yet, when tracked down and interviewed in adolescence, they again fell into two camps that corresponded closely with the results of the earlier experiments. Those capable of resisting the single marshmallow at four were more likely to be optimistic, competent, self-reliant and trustworthy. They were confident teenagers with strong initiative and clear goals. Yet those who'd been unable to resist the marshmallow were more troubled: revealing traits such as pessimism, impulsiveness, envy, mistrust, anger, resentment and indecision.

Basically, one group – the marshmallow resisters – expected and were organized for success; while the other group – the marshmallow eaters – were not.

‘There is perhaps no psychological skill more fundamental than resisting impulse’, writes Daniel Goleman in Emotional Intelligence (1995), one of the many books to cite these famous experiments. ‘It is the root of all self-control, since all emotions, by their very nature, lead to one or another impulse to act.’

Delayed or deferred gratification is, therefore, a key trait in productive competence. Those lacking willpower or self-control will seek instant gratification, states Goleman, whether through sought pleasure or avoided pain. While those with self-control will ignore short-term inconveniences and temptations in order to focus on future potential rewards.

Early-life conditioning

Of course, everyone who's ever read about the marshmallow test immediately wonders how they'd have reacted to such temptation as a four-year-old child. The truth is, we cannot know; although I had that uneasy feeling of recognition – suspecting I'd have been incapable of delayed gratification.

Yet my feelings of unease extended to the marshmallow test itself. As a four-year-old, my guess is I'd have misunderstood the terms of the offer. So focused would I have been on the treat in front of me, I'd have filtered out any other information, including the potential for reward if I waited for the adult to return. Did this, therefore, condemn me at four as innately incapable of deferring gratification? Or did it simply suggest it was something I'd yet to learn?

This left me wondering whether there was anything hardwired (or even genetic) being measured by the marshmallow test, as some of those commenting on the Stanford experiments suggest. Or could early-life conditioning be dictating the result – encouraging the obvious follow-on that, unless this conditioning is reversed or amended at some point on our route towards adulthood, we'll reveal the same traits throughout our lives?

So, while I was convinced I'd have failed the marshmallow test at four – and probably even at eight – I was far from convinced this meant I was innately disposed (potentially genetically) towards unproductive, impulse-driven behaviour. It was simply poor, yet reversible, conditioning.

Testing the marshmallow test

To test this I did my own, totally unscientific, experiment on four children I knew well: my own and those of a friend and neighbour. Left together (although the Stanford children were alone I wanted to observe the impact of influence on the children) – each with a single marshmallow – I secretly watched the reactions of boys aged three, four and six and a girl who'd turned seven that day.

Despite the distractions of the occasion, the seven-year-old immediately understood my promise to return with an additional marshmallow if the original remained uneaten. She held on the required 15 minutes, which was no more than I expected from this emotionally aware young girl.

And her good behaviour influenced the six-year-old boy. That said, he seemed able to wait only by creating a game that mimicked the movements of the girl: with them alternating between sitting on their hands and clasping their hands over their mouths. I suspect he'd have found it a lot harder without her good example – especially as he talked constantly about the reward, as well as how long my return was taking.

The younger two had a far tougher time, however. The three-year-old lasted less than a minute, although I became convinced he only understood the premise after realizing the cost of his action: as excitedly reminded by his elders. This distressed him to the point he had to be removed from the room to avoid disrupting the experiment entirely.

Meanwhile, the four-year-old hung on, although was constantly asking his sister for an explanation and was clearly troubled by the challenge. Only the verbal intervention of the older boy (a strong influencer of the younger boy's behaviour) prevented him gobbling the marshmallow at around the five-minute mark and then repeatedly from around minute eight. Again, I worried throughout that he'd misunderstood the proposition.

Delayed gratification is developmental

Of course, my own version of the experiment proves nothing, although I thought the exact matching of the children's age to their ability to resist was surely no coincidence – meaning that delayed gratification is as likely developmental as it is innate. It's something we learn. As for the Stanford marshmallow failures and their negative traits in young adulthood, could the same poor conditioning that prevented them developing strong productive behaviour at four last right into young adulthood?

In fact, it could last a lifetime. A 2011 follow-up study – conducted by Dr B.J. Casey of Weill Cornell Medical College in New York – noticed that those adept at delayed gratification in the original Stanford test revealed similarly enabling traits as they approached middle-age. As for the instant-gratifiers in 1972, they too revealed similarly disabling propensities in the most recent study. And worryingly (at least for those assuming conditioning the central issue), the 2011 experiments recorded brain-pattern correlations suggesting the existence of a ‘seat of self-control’ (or otherwise) in the prefrontal cortex of the brain.

Yet Casey's experiments sampled just 59 out of the original 600 tested at Stanford. And these were the most extreme cases (at either end of the spectrum) recorded by Mischel in 1972. So a correlation was always a likely outcome. Of more interest – at least for the vast-majority of middling types – would have been a study of those that fell between these two extremes. It is this group – the 540 not retested by Casey – where success or failure would have been most likely due to conditioning.

Indeed, my guess is that many of the 1972 failures would not only now be competent individuals, they'd be able to recall the events that motivated their change from impulsive instant-gratifiers into productive future-oriented professionals. Perhaps, at some point, they became motivated by strong desires or goals (see Part Two). Or maybe they were jolted into productivity via professional training or from starting a new job. Or maybe a new influencer – perhaps from beyond the family – gave them the direction they lacked.

Sure, some will have prospered while others struggled. And those to adopt such competences early will have an advantage. But – given the obvious benefits of delayed gratification and its related traits of productive competence – late adoption is still better than no adoption.

Freud's id, ego and superego

So we're not condemned to a life of impulse-related ineffectiveness at the age of four – an opinion supported by the godfather of psychoanalysis himself. Sigmund Freud's 1920 essay ‘Beyond the Pleasure Principle’ deals with exactly this issue – later (in a 1923 essay called the ‘The Ego and the Id’) elaborating his ideas of the ‘id’, ‘ego’ and ‘superego’ to explain the various stages in the development of the human psyche to encompass socialization, planning and organization.

Freud's id contains ‘the psychic apparatus at birth’ – the instincts to seek pleasure and avoid pain. The id is, according to Freud, ‘a cauldron full of seething excitations’ that deals with basic needs such as food, water and sex. It's amoral and selfish. It has no sense of time, is completely illogical – primarily sexual – and infantile in its emotional development.

The id is clearly incapable of deferred gratification and needs tempering, although this is something we have to learn (largely from external influences), which is where the ego comes in. The ego acts according to the ‘reality principle’, says Freud. In contrast to the id's ‘pleasure principle’, this comprises the organized part of our personality.

‘The ego is that part of the id which has been modified by the direct influence of the external world’, writes Freud, representing ‘what may be called reason and common sense, in contrast to the id, which contains the passions’.

While the ego is only partly conscious, it acts as a restraint on the id – perhaps overriding it with semi-conscious concerns for, say, safety or, importantly, organization. Judgement, tolerance, control, planning, intellect, memory: all are part of Freud's ego.

Guilt, meanwhile, is one of the central characteristics of the superego – the fully-organized part of Freud's personality structure, which acts according to the ‘morality principle’. The superego is our conscience – punishing misbehaviour with feelings of guilt. It strives for perfection and is determined to act in a socially appropriate manner – countering the id's need for instant gratification. The superego emerges from learning and brings a sense of personal progress, of future orientation and of integration with social norms.

Given the above descriptions, Freud would have surely viewed a four-year-old child winning a second marshmallow as the early influence of the moderating ego and the organizing superego, most likely due to external guidance. And, importantly, he would have viewed the continuation into early adulthood of traits suggesting an inability to defer gratification as the failure or absence of those same external influences.

For whatever reason, those unable to defer gratification continued to flounder in the id's ‘cauldron full of seething excitations’. A cauldron, moreover, that will only lose its appeal once its cost becomes apparent in adulthood: perhaps via wasteful hedonism, addictions, procrastination or an inability to organize for the future.

Maslow's hierarchy of needs

Yet this leaves one question unanswered. Why would one child develop strong awareness and acceptance of external influences, while another – potentially ignoring or rejecting those same influences – be concerned only with their immediate id-induced stimulation? Of course, this is where some of the marshmallow experimenters suggest an innate propensity for such behaviour, although another famous psychologist offers an alternative explanation.

Abraham Maslow is best known for his ‘hierarchy of needs’. First proposed in his 1943 paper ‘A Theory of Human Motivation’, Maslow's hierarchy is usually expressed as a pyramid – record­­ing a human's development towards what Maslow termed ‘self-actualization’.

At the base of the pyramid are basic needs such as air, water and food. Only once these are satisfied can we move to the next level and seek shelter and safety. With these needs met, we seek love and companionship, which – once won – allow us to develop our self-esteem, often via achievement or praise. And once we have self-esteem, we can seek self-actualization, which involves needs such as morality and creativity.

Figure 1 Maslow's Hierarchy of Needs

An important point here is that we cannot move to the next level until we have satisfied the needs below. We will not seek shelter without food, or love without safety. Self-esteem, therefore, is impossible without love, meaning we have no motivation towards achievement – and acquiring the productive skills for making progress – unless we first acquire the sense of belonging that comes with love, friendship and acceptance.

Could it be, therefore, that those incapable of deferred gratification at four were revealing early signs of low self-esteem? Lacking the security of love, their impulses were immediate – even basic. And, unless this was actively tackled, such a disablement could stay with them into adulthood.

This certainly had a resonance with me. Having developed low self-esteem as a child – mostly due to feelings of rejection from my father – I felt Maslow's hierarchy explained my erratic tendencies, which lasted well into adulthood.

Texts that ignore the psychological issues

Yet there's one more aspect to this for those trying to acquire organizational competence in adulthood. It's just possible that the specialist sub-sector of the self-help industry – focusing on time and task management – is of little or no help (at least initially). In fact, it could be viewed as just another distraction, as well as – in the failing – further confirmation of our poor self-beliefs.

Certainly, my response was usually to scan the first chapter of these texts fearful of the emotional and physical investment required to even finish the book, let alone commit to its programme: no wonder they seemed to exacerbate, rather than alleviate, my feelings of low self-worth.

Such books have become a popular genre in recent years. The Time Trap by Alec Mackenzie is perhaps the granddaddy. First published in 1972, it adopted the no-nonsense ‘take control’ approach that has become the accepted style for the majority since. These include Time Tactics of Very Successful People by B. Eugene Griessman (1994), Time Management for Busy People by Roberta Roesch (1998), Streetwise Time Management by Marshall Cook (1999), Getting Things Done by David Allen (2001), Organizing From The Inside Out by Julie Morgenstern (2004) and Eat That Frog by Brian Tracy (2004). Yet there are hundreds of such books.

Although useful – and ransacked for tips and methodologies for the pages ahead – they rarely address the fundamental psychological issues at play with respect to our poor productivity. Nearly all take our organizational inabilities as a starting point and march forward – offering admittedly strong ideas and actions for making the unproductive productive.

Yet this approach assumes we're willing and able to change, as well as capable of putting aside our personal histories in order to take responsibility for our poor productivity. Of course, eventually that's exactly what we have to do. But, first, we must fully understand what led us to this sorry point – something that may take professional psychological help (as it did for me). Only when we comprehend the psychology can we develop strong attributes that convert our destructive and incompetent habits into something more constructive.