The Guardian’s report of a recently published scientific article was headlined 'Religious children are meaner than their secular counterparts, study finds'. The same conclusion has been published elsewhere, including the Daily Mail, Time magazine and other esteemed outlets.
But there are several reasons why the claim is hardly worth the paper its written on (and worthy in fact of The Guardian’s excellent former ‘Bad Science’ columnist Ben Goldacre). Here are seven reasons why:
1. Social science experiments such as this have limited application to the wider population, for a number of reasons. Psychological experiments often test issues in artificial ways that can’t be extrapolated to real life.
2. This particular study seems to have a serious design flaw: the use of the ‘dictator game’ as a measure of altruism, when that particular experiment has been critiqued to be more a measure of susceptibility to peer pressure. See this post from John Baskette for details. Even if it was a good measure of altruism, the other problems with such experiments still apply, and 'experimenter demand' effects are common confounding variables.
3. The fact that this study is in a biology journal, when it is a social science study, makes me wonder if it was rejected from journals where a rigorous and learned peer review would have taken place. Biology and psychology are very different disciplines, with very different experimental designs.
4. The children being tested were from the USA, Canada, China, Jordan, Turkey, and South Africa. Are these countries representative of people as a whole? How did they select the religious children? From which kind of religious communities were they? As anyone who has faith will know, even within one particular religion, there is wide diversity of practice and belief. There is very little information on the sampling – how the researchers chose the schools and children involved, which seems as though it would have a crucial effect. For example, if their USA school was one run by the Westboro Baptist Church, it’s likely that Christian children would not be representative of Christian children as a whole!
5. The researchers appear to have a particular axe to grind. They say at the end: 'More generally, they call into question whether religion is vital for moral development, supporting the idea that the secularization of moral discourse will not reduce human kindness—in fact, it will do just the opposite.' To make such a politically loaded statement in the conclusion of a research article indicates a degree of experimenter bias.
6. The fact that the experimenters claim their findings ‘robustly demonstrate’ less altruism in religious children, and omit all the usual qualifications about generalising from experiments that are included as standard in reputable social science papers, again suggests experimenter bias and poor interpretation.
7. There are other experiments that could suggest that religious people are more altruistic. These are barely mentioned in the paper. Of course, such experiments also have limitations in how they can be generalised and should have lots of qualifications too. But they should surely be given more weight and discussion in a report that finds the opposite?
There are other problems too. George Yancey points out the lack of control variables, along with other similar criticisms; and William M Briggs points out problems with the statistical analysis. Even atheists who have an understanding of the social sciences are pointing out the problems with this study.
This article is so poor, that I’m sure I could find more flaws if I spent more time on it, but I’ve just whizzed through. It’s a shame that it is being taken seriously by media and public alike.
Please add any other flaws you see in the experiment in the comments below.
This post originally appeared on heathersmag.wordpress.com
