Can bad science lead us to draw wrong conclusions about the world we live in? “Of course it can“, we are inclined to think. And if so, can this have real-life consequences? Investigating these meta-questions is not as easy as it might seem, for it would require an exact manner to distinguish the good from the bad science, and it would require a subject that has been thoroughly investigated in both the ‘good’ and the ‘bad’ ways to compare the outcomes.

One such subject would be the vast amount of research done on the psychological consequences of undergoing an induced abortion. This heavily researched (and heavily debated!) subject focusses primarily on the questions whether or not a women has a higher chance of suffering from anxiety, feeling of guilt, depression, or (other) mental disorders caused by undergoing an induced abortion. The conclusions drawn in the vast literature on this basic question vary form an abortion having no consequences, to an abortion having a negative impact on the psychological well-being of a woman.

Charles, Polis, Sridhara and Blum (2008) did a systematic review of this literature. Interestingly, they did not only classify the findings, but also derived a set of guidelines by which to evaluate the methodological quality of each research paper.

Analytical studies were rated on a scale from Excellent to Very Poor using five major criteria: (1) use of an appropriate comparison group; (2) use of valid mental health measures; (3) control for preexisting mental health status; (4) control for confounders; and
(5) whether there was comprehensive exploration of the research question. (p. 437)

Trained researchers would find it difficult to disagree with these ‘five guidelines for quality research’. However, hardly any of the studies they evaluated did comply with all of them. Try to imagine what the consequences of failing to comply with any of these guidelines might be. For instance, it is known that mental health status relates to the chance of becoming pregnant unintendedly, and subsequently it relates to the decision whether or not to keep the baby. Not taking that into account would attribute the difference in mental health between women who have had an abortion, and women who did not, to the abortion, whereas these differences in fact were already present before the abortion. A similar argument goes for taking into account differences regarding age, educational level, and religious conviction: these characteristics all relate to both mental health and the odds of having an unintended pregnancy and / or an abortion. As a last example, the groups of women that are compared should be as equivalent as possible, except for having had an abortion. However, studies compared women having had an abortion with, for example, the general population, or just with women having carried out their pregnancy. This practice does not allow for causal inference, for it cannot exclude the effects of pregnancy intentions.

In total, Charles et al. evaluated 21 studies, the methodological quality of which varied widely, as did their conclusions on the consequences of an abortion. Their most fascinating finding was that a relation exists between the methodological quality of a study and the conclusions drawn in it: the better studies hardly found any consequences from undergoing an abortion, whereas the poorer quality studies did find negative consequences of an abortion. In other words: bad science tends to overestimate the consequences of an induced abortion.
Generally, these differences are due to the fact that the studies with poor methodology did not take into account differences between women prior to the abortion, thereby attributing differences between women in mental health to having had an abortion, whereas in fact these differences were pre-existing.

There we have it: bad science distinguished from good science, and different outcomes. So, yes, bad science indeed leads to wrong outcomes and in real life, this can have consequences. In the policy-discussions on induced abortion, the possibility of negative consequences of an abortion plays an important role. It would depend on which research papers the policy makers turn to, what their stand on the issue would be.

The question that remains is now: can we trust policy makers to distinguish the good from the bad (science)?

V CHARLES, C POLIS, S SRIDHARA, R BLUM (2008). Abortion and long-term mental health outcomes: a systematic review of the evidence Contraception, 78 (6), 436-450 DOI: 10.1016/j.contraception.2008.07.005