How Social Science Can Hurt Those It Loves

In his fascinating book Science Fictions, Stuart Richie presents graphic details of how scientists have distorted the facts. Beyond actual fraud (much more prevalent than might be expected), distortions in data collection, analysis, and interpretation can lead to inaccurate and often downright incorrect conclusions. Awareness of these pitfalls became particularly apparent in what became known as the ‘replication problem, ‘ the difficulty of obtaining the same results in two separate studies of the same topic.

A charming example of this replication problem, explored delightfully in the recent bestselling novel by Nathan Hill called Wellness, is the well-known ‘Marshmallow Test.’ The 1990 study put a marshmallow in front of young children, giving them the choice of eating it now or delaying their gratification for 15 minutes and then being allowed to have many more. The claim was that children who delayed their gratification did much better in later life. This spurred on a whole raft of studies that claimed delayed gratification was the secret to success. In some accounts, it was even implied that people stayed working class because they did not have the habit of delaying their gratification.

However, a more recent replication of the marshmallow experiment that took account of whether the children’s mothers had a college degree, found virtually no support for the claim that this test of delayed gratification was a strong indicator of future success. Those children without degree-baring mothers showed a marginal relationship between delayed gratification and later success. Those without such mothers showed no such relationship.

You can read into this various explanations for the results, but what amazes me about the comments on the difference between the earlier and later experiments is that I can find no reference to the distortion and artificiality that is at the heart of these studies. They are daft experiments. They introduce limited, controlled conditions in fake settings. They are then extrapolated to lived experiences with little or no attempt to explore their ecological validity.

In order to get tidy, replicable results, social scientists create laboratory-based situations in which ‘controlling for unwanted variables’ consists of removing everything that might make their studies even marginally realistic. This is especially true of psychologists, with their fondness for comparing themselves to the natural (pure) sciences. They do this partly because they do not have the resources to set up the complex studies that take account of all the confounding variables. Physicists, for example, rely on huge funds and large teams to collect their data in a way that allows them to filter out the noise.   

I have become painfully aware of the destructiveness of many psychological experiments when preparing for the extensive revision of my 1977 book The Psychology of Place. The conundrum that emerges from reading so many studies in the area, generally known as Environmental Psychology, is that the ‘environment’ is so often treated as a bundle of stimuli that exist independently of any actual context. The pressure to fit studies into the shackles of a tidy experiment that will give clear statistically ‘significant’ results, leads for example, to people being shown photographs of landscapes and asked to rate them. They may be taken to a scenic view and their preferences assessed. In some cases, this is done after participants have been made to experience some other brief event, which may cause stress or discomfort. The resulting impact of ‘being in a natural environment’ is then heralded as revealing something profound about how humans are slaves to their evolutionary past.

The tragedy is that the ease of carrying out such ‘experiments’ means that slight tweaks of the setup, such as comparing older with younger participants, enables generations of PhD students to follow a well-trodden path. The senseless results ignore such things as time of day, season, huge varieties of what may be regarded as a ‘natural environment’ and, most importantly of all, why anybody would decide to be in that setting – their purposes for being there, do not matter because they accord with what hundreds of other people have published.

There is no doubt that, under particular circumstances, being outside in settings without the noise, air, and other pollutants, as well as pleasant views, has benefits. The physical opportunities that the great outdoors provide clearly can contribute to physical and mental well-being. Beyond the physiological benefits of moving around, though, contorted experiments do not explain why the experience of nature has such powerful potential. Poets, novelists, travel writers and others in the humanities, even composers, often provide much greater insights than the limited findings supported by the insignificance of ‘statistical significance.’