Guideline for improving the reliability of Google Ngram studies: Evidence from religious terms

PLoS One. 2019 Mar 22;14(3):e0213554. doi: 10.1371/journal.pone.0213554. eCollection 2019.

Abstract

The Google Books Ngram Viewer (Google Ngram) is a search engine that charts word frequencies from a large corpus of books and thereby allows for the examination of cultural change as it is reflected in books. While the tool's massive corpus of data (about 8 million books or 6% of all books ever published) has been used in various scientific studies, concerns about the accuracy of results have simultaneously emerged. This paper reviews the literature and serves as a guideline for improving Google Ngram studies by suggesting five methodological procedures suited to increase the reliability of results. In particular, we recommend the use of (I) different language corpora, (II) cross-checks on different corpora from the same language, (III) word inflections, (IV) synonyms, and (V) a standardization procedure that accounts for both the influx of data and unequal weights of word frequencies. Further, we outline how to combine these procedures and address the risk of potential biases arising from censorship and propaganda. As an example of the proposed procedures, we examine the cross-cultural expression of religion via religious terms for the years 1900 to 2000. Special emphasis is placed on the situation during World War II. In line with the strand of literature that emphasizes the decline of collectivistic values, our results suggest an overall decrease of religion's importance. However, religion re-gains importance during times of crisis such as World War II. By comparing the results obtained through the different methods, we illustrate that applying and particularly combining our suggested procedures increase the reliability of results and prevents authors from deriving wrong assumptions.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Books*
  • Data Mining*
  • Guidelines as Topic
  • Language*
  • Religion*
  • Search Engine*

Grants and funding

Support for this research was provided to U.-D. Reips by the Committee on Research at the University of Konstanz and the Excellence Clusters "Politics of Inequality" (EXC 2035/1) and "Collective Behaviour" (EXC 2117/1).