Surprisingly little – at least to my taste. Science is supposed to be cumulative: we should build knowledge on solid foundations. But how do we know what knowledge is solid? Through independent verification. If someone claims to have found a regularity, it is only through repeated testing of the same claim that we know that the result holds in other populations and contexts. It turns out that scientists rarely conduct such replications. Here are all the estimates of the prevalence of replications in various scientific disciplines that I could find in an extensive search on Google Scholar.
Some observations on the prevalence rates above.
The replication prevalence in psychology increased strongly after 2013. The detection of fraud by Diederik Stapel and the publication of a paper on ‘precognition’ by Daryl Bem may have contributed to this increase.
In marketing, in contrast, fraud cases such as those of Dirk Smeesters and Brian Wansink did not increase the prevalence of replications. In the period 2000-2011 the proportion of articles replicating previous research was lower than in the preceding period 1990-2004.
So called ‘top’ journals – publishing papers with a higher citation count on average – are more likely to publish replications in criminology. This also holds generally speaking in the social sciences – see the graph below.
There is no hierarchy of the sciences in the prevalence of replications. The “hard sciences” do not attempt replications more often than the social sciences – it’s the other way around. Publications in economics are far less likely to attempt replications than publications in marketing or in communication science. The prevalence of replications in ecology is devastatingly low at 0.03% (11 out of 38730 published papers).
If you know of a study I have missed that allows for an estimate of the prevalence of replications, please let me know. I could not find studies in public administration, political science, or sociology. The lack of studies documenting the prevalence of replication attempts in public administration and political science is surprising because in these disciplines several journals considered to be leading in the field have introduced data availability policies.
The table below and the ppt with the graphs above contains the data and references to the sources represented.
| Field | Rate | Reference | Remarks |
| Social Sciences | 1.3% (2/156) | Hardwicke T.E., Wallach, J.D., Kidwell, M.C., Bendixen, T., Crüwell, S. & Ioannidis, J.P.A. (2020). An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017). Royal Society Open Science, 7(2), 7190806190806. https://doi.org/10.1098/rsos.190806 | Random sample of 250 articles, 2014-2017 |
| Social sciences | 2.8% (44/1559) | McNeeley, S., & Warner, J. J. (2015). Replication in criminology: A necessary practice. European Journal of Criminology, 12(5), 581-597 https://doi.org/10.1177/1477370815578197 | 5 ‘top’ journals, 2006-2010 |
| Natural sciences | 1.4% (94/6637) | McNeeley, S., & Warner, J. J. (2015). Replication in criminology: A necessary practice. European Journal of Criminology, 12(5), 581-597 https://doi.org/10.1177/1477370815578197 | 5 ‘top’ journals, 2006-2010 |
| Discipline | Rate | Reference | Remarks |
| Psychology | 5.32% (10/188) | Hardwicke, T. E., Thibault, R. T., Kosie, J. E., Wallach, J. D., Kidwell, M. C., & Ioannidis, J. P. (2022). Estimating the prevalence of transparency and reproducibility-related research practices in psychology (2014–2017). Perspectives on Psychological Science, 17(1), 239-251. https://doi.org/10.1177/1745691620979806 | Random sample of 250 articles, 2014-2017 |
| Communication science | 3.7% (21/562) | Keating, D. M., & Totzkay, D. (2019). We do publish (conceptual) replications (sometimes): Publication trends in communication science, 2007–2016. Annals of the International Communication Association, 43(3), 225-239. https://doi.org/10.1080/23808985.2019.1632218 | 10 journals, 2007-2016 |
| Advertising | 2.87% (82/2856) interstudy replications | Park, J. H., Venger, O., Park, D. Y., & Reid, L. N. (2015). Replication in advertising research, 1980–2012: a longitudinal analysis of leading advertising journals. Journal of Current Issues & Research in Advertising, 36(2), 115-135. https://doi.org/10.1080/10641734.2015.1023874 | 4 ‘top’ journals, 1980-2012 |
| Criminology | 2.32% (16/691) | McNeeley, S., & Warner, J. J. (2015). Replication in criminology: A necessary practice. European Journal of Criminology, 12(5), 581-597 https://doi.org/10.1177/1477370815578197 | 5 ‘top’ journals, 2006-2010 |
| Experimental linguistics | 1.81% (153/8437) | Kobrock, K., & Roettger, T. B. (2023). Assessing the replication landscape in experimental linguistics. Glossa Psycholinguistics, 2(1). http://dx.doi.org/10.5070/G6011135 | 100 journals, 1945–2020 |
| Marketing | 1.01% (12/1185) | Kwon, E. S., Shan, Y., Lee, J. S., & Reid, L. N. (2017). Inter-study and intra-study replications in leading marketing journals: a longitudinal analysis. European Journal of Marketing, 51(1), 257-278. https://doi.org/10.1108/EJM-07-2015-0450 | 5 ‘top’ journals, 2000-2011 |
| Marketing | 1.70% (41/2409) | Evanschitzky, H., Baumgarth, C., Hubbard, R., Armstrong, J.S. (2007). Replication research’s disturbing trend. Journal of Business Research, 60, (4), 411-415. https://doi.org/10.1016/j.jbusres.2006.12.003 | 5 ‘top’ journals, 1990-2004 |
| Business and management | 1.48% (1235/83682) | Ryan, J. C., & Tipu, S. A. (2022). Business and management research: Low instances of replication studies and a lack of author independence in replications. Research Policy, 51(1), 104408. https://doi.org/10.1016/j.respol.2021.104408 | All 121 journals, 2008-2017 |
| Psychology | 1.08% (347/321411) | Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur?. Perspectives on Psychological Science, 7(6), 537-542. https://doi.org/10.1177/1745691612460688 | Random sample of 500 articles, 1900-2012 |
| Criminology | 0.45% (178/39275) | Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2018). Replication in criminology and the social sciences. Annual Review of Criminology, 1, 19-38. https://doi.org/10.1146/annurev-criminol-032317-091849 | All journals, until 2014 |
| Addiction Medicine | 0.41% (1/244) | Adewumi, M. T., Vo, N., Tritz, D., Beaman, J., & Vassar, M. (2021). An evaluation of the practice of transparency and reproducibility in addiction medicine literature. Addictive Behaviors, 112, 106560. https://doi.org/10.1016/j.addbeh.2020.106560 | All journals in PubMed, 2014-2018 |
| Management | 0.15% (240/159242) | Block et al. 2022 https://doi.org/10.1007/s11301-022-00269-6 | 56 top journals until 2020 |
| Economics | 0.10% (130/126505) | Müller-Langer et al. 2019 https://doi.org/10.1016/j.respol.2018.07.019 | Top 50 journals, 1974-2014 |
| Educational science | 0.13% (221/164589) | Makel et al. 2014 https://doi.org/10.3102/0013189X14545513 | Top 100 journals, until 2013 |
| Ecology | 0.02% (11/38730) | Kelly, C. D. (2019). Rate and success of study replication in ecology and evolution. PeerJ, 7, e7654. https://doi.org/10.7717/peerj.7654 | All 160 journals, until 2017; Python code at https://doi.org/10.17605/OSF.IO/WR286 |
Rene Bekkers 