In the Netherlands it is commonly believed that the newspapers have a left-wing political orientation. Perhaps, that would be the reason for the attention a research report that was published yesterday is receiving. Based on a content analysis of primarily newspaper articles, the conclusion is drawn that newspapers are more right-wing oriented that we thought. Dutch newspaper ‘Dag’ draws these conclusions on a report by André Krouwel.

Below, I will discuss some aspects of the report that I think are incorrect. I will treat this as a scientific report, the status it claims, focussing mainly on the reliability of the empirical basis of the conclusions. It do not argue that the newspapers are either left- or right-wing, I just argue that we better not answer that question based on this report. Subsequently, I will discuss the lack of compliance with scientific standards, how the research design is not capable of answering the research question, how percentages are calculated and compared, how keywords were categorized, and finally the absence of reliability tests.

Scientific standards

After reading the report, it is immediately clear that it does not comply to common scientific standards. For instance, while some references to scientific literature are made in the text, no bibliography is given. Nor are references to be found to literature on all the techniques of analysis used. Also, the presented tables are incomplete: no units of analysis are given, nor marginal totals or number of observations. Terminology is used very interchangeably, for instance the report is focused on newspapers, but often the word ‘media’ is used; similarly, the use of the dichotomies `left – right’ and ‘progressive – conservative’ to indicate political orientations seems interchangeably and arbitrary. Last, but certainly not least, a concluding paragraph is lacking. Basically, it is up to the reader to formulate an answer to the posed research question.

Research question and design

The stated research question focusses on a longitudinal question, but the data-collection is a single cross-section in a period much shorter than the time-periods implied by the research question. According to the report, it is focused on the question whether or not dutch newspapers have become more right-wing. That implies a change over time. However, only a five-month period of published newspapers is investigated and in the analysis even this short time-span is not taken into account explicitly.

Wrong Percentages

The percentages presented in the tables of the report are often miscalculated. For instance, in table 2, both absolute numbers and percentages are given. In this table, a marginal total is given indicating the percentage of the absolute number and the mean of the percentages. These two number should be identical, but they are not. No explanation of these differences is provided.

Besides this error, also the basis of the calculations is wrong. Let’s take for instance tables 4 and 5. In table 4 for certain news issues a number is presented, presumably indicating the number of articles on a specific subject. This is cross-tabulated for all the different newspapers and per topic a mean is computed. Subsequently, in table 5, for each combination of topic and newspaper the difference between prevalence and mean per topic is calculated, supposedly indicating under- or overrepresentation of topics in specific newspapers. This is clearly wrong, for it assumes that all newspaper have an identical chance to report on a topic to start with, so that all differences from the mean result from a editorial choice. But, not all Dutch-papers have the same layout, nor the same number of articles or even words on their front-pages (which were analyzed). So, these number cannot be compared in this way.

Keywords

The way the categories mentioned above were formed are a bit off as well. Based on word-frequencies, the authors categorized keywords under headings as ‘war’, ‘education’, ‘china’ and ‘healthcare’. The use of these key-words used to determining the prevalence of political issues in newspapers is wrong in two ways. First, per category a different number of keywords is used, ranging from 1 to as much as 17. Thereby, some issues are easier ‘found’ than others. This would not be that bad if only the issues were compared over newspapers only, but there are also comparisons between newspapers within categories of issues. This is clearly wrong.
Secondly, not all keywords are even applicable to the category. Clearly, ‘fitna’ is always about the movie / category ‘fitna’ However, the keyword ‘airport’ obviously is not always about Dutch main airport ‘Schiphol’. And interestingly, the category ‘China’ does not contain the keyword ‘China’, but only ‘Chinese’. Since keyword-based computer searches have been performed, this procedure has missed out on all articles mentioning China, without using the adjective ‘Chinese’.

Combined with the previous paragraph, this leads to the conclusion that the calculated percentages cannot be compared within a topic over newspapers, not within newspapers over topics. Basically, all the comparisons implied and made in the report do not have a firm, valid empirical basis.

No reliability analysis

Besides the strong validity issues described above, there is also a problem with the reliability of the coding of news-messages. Nothing is said on any reliability analysis. Although they refer to Krippendorf for a definition of content analysis, the authors apparently missed out on Krippendorf’s seminal work on reliability analysis. The procedure he described basically come to having multiple coders to code the same material. Subsequently, a reliability coefficient can be coded, which indicates whether or not the results of the coding procedure are independent of the individual coders. I cannot imagine such an article, without any report on the reliability analyses, being published in any renowned peer-reviewed journal.

Conclusion

Besides simply criticizing this research report, I would like to refer to an article in which a similar research question has been investigated in a much more reliable approach. This approach is to be found in the article by Lubbers, Scheepers, and Wester (1998). They investigated the representation of ethnic minorities in Dutch newspapers. Their sampling frame was all newspaper articles on ethnic minorities published in a time-span of five years (1990-1995). Clearly describing their procedure and narrowly defining keywords and categorization, the authors were able to make longitudinal inferences reliably. What makes their article quite sophisticated is that the authors compared their findings on the representation of ethnic minorities in the newspaper with additional information on attitudes towards ethnic minorities in Dutch population. Due to this comparison, the understanding to be gained by the investigation is greatly enlarged.

The results of the report on right-wing Dutch newspapers are however, I argue, not reliable at all and the conclusion of this report is not warranted. Although the report is about much more, the discussion on it seems to focus on the statement that newspapers are not as left-wing as generally thought. Basically, this statement is only based on the finding that Dutch newspapers have written a lot about a specific right-wing politician. Does writing about a right-wing politician make a newspaper inherently right-wing? I wouldn’t think so.

In science, research papers are normally send to a peer-reviewed journal and media-attention is best sought after the results have been published in such a journal. This procedure functions as an important measure of quality control. The apparent eagerness of publishing these results in a newspaper, before the peer-reviewing process has been concluded, has led to the publication of a content-analysis done completely wrong.

References

Krouwel, A.G. (2008). Links en rechts in het nieuws. Aandacht voor politieke partijen en politici in Nederlandse media.. Report by the \’Centre for Applied Political Science\’, 358(1), 143-186.

Lubbers, M., Scheepers, P., Wester, F. (1998). Ethnic Minorities in Dutch Newspapers 1990-5: Patterns of Criminalization and Problematization. International Communication Gazette, 60(5), 415-431. DOI: 10.1177/0016549298060005004