How to find truth in an ocean of correlations – with breakers, still waters, tidal waves, and undercurrents? In the old age of responsible research and publication, we would collect estimates reported in previous research, and compute a correlation across correlations in a meta-analysis. Those days are long gone.
It is a frustrating experience for anyone who has conducted one: endless searches on the Web of Science and Google Scholar to collect all published research, input the estimates in a database, find that a lot of fields are blank, email authors for zero-order correlations and other statistics they had failed to report in their publications and get very little response.
Meta-analysis is not only a frustrating experience, it is also a bad idea when results that authors do not like do not get published. A host of techniques have been developed to find and correct publication bias, but the problem that we do not know results that do not get reported is not solved easily.
As we enter the age of open science, we do not have to rely any longer on the far from perfect cooperation from colleagues who have moved to a different university, left academia, died, or think you’re trying to prove them wrong and destroy their career – and frustrate yours in retribution. In the age of open science we can simply download all the raw data and analyze them.
Enter mega-analysis: include all the data points relevant for a certain hypothesis, cluster them by original publication, date, country, or any potentially relevant property of the research design, and add the substantial predictors you find documented in the literature. The results reveal not only the underlying correlations between substantial variables, but also the differences between studies, periods, countries and design properties that affect these correlations.
The method itself is not new. In epidemiology, Steinberg et al. (1997) labeled it ‘meta-analysis of individual patient data’. In human genetics, genome wide association studies (GWAS) by large international consortia are common examples of mega-analysis (e.g., Major Depressive Disorder Working Group of the Psychiatric GWAS Consortium, 2013). In personality research, mega-analysis has also been called ‘integrated data analysis’ (Curran & Hussong, 2009). Previous mega-analyses pooled fMRI data (Costafreda, 2009) and brain volume data (Boedho et al., 2016). A recent mega-analysis analyzed the association of personality characteristics with life outcomes (Beck & Jackson, 2022).
Mega-analysis includes not only the file-drawer of papers that never saw the light of day after they were put in. It also includes the universe of papers that have never been written because the results were unpublishable.
If meta-analysis gives you an estimate for the milky way of published research, mega-analysis can be used to detect just how unique that universe is in the universe. I predict that correlations in published research are mostly further from zero than the same correlation in a mega-analysis.
Mega-analysis bears great promise for the social sciences. Samples for population surveys are large, which enables optimal learning from variations in sampling procedures, data collection mode, and questionnaire design. It is time for a Global Social Science Consortium that pools all of its data. As an illustration, I have started a project on the Open Science Framework that mega-analyzes generalized social trust.
The idea behind mega-analysis originated from two different projects designed to test the effects of volunteer work on dispositions such as generalized trust and subjective well-being. In the first project, Erik van Ingen and I analyzed the effects of volunteering on generalized trust, to check whether results from an analysis of the Giving in the Netherlands Panel Survey (Bekkers, 2012) would replicate with data from other panel studies (Van Ingen & Bekkers, 2015). Confirming the original findings, we found essentially the same results in five panel studies, although subtle differences emerged in the quantative estimates.
In the second project, with Arjen de Wit and colleagues from the Center for Philanthropic Studies at VU Amsterdam, we analyzed the effects of volunteering on well-being conducted as part of the EC-FP7 funded ITSSOIN study. We collected 845.733 survey responses from 154.970 different respondents in six panel studies, spanning 30 years (De Wit, Bekkers, Karamat Ali & Verkaik, 2015). We found that volunteering is associated with a 1% increase in well-being.
In these projects, we initially analyzed the data from different studies separately. For a mega-analysis the data need to be harmonized. Then we can pooling them in one single analysis, and learn much more. In a project with Arjen de Wit and Ellie (Heng) Qu we’ve done that to further study the association between volunteering and health in a longitudinal design. We have added more observations to the analysis and applied quantile regression models (De Wit, Qu & Bekkers, 2022). The study has been accepted for publication in the European Journal of Aging. Read more about it in this post.
References
Beck, E. D., & Jackson, J. J. (2022). A mega-analysis of personality prediction: Robustness and boundary conditions. Journal of Personality and Social Psychology, 122(3), 523-553. https://doi.org/10.1037/pspp0000386
Bekkers, R. (2012). Trust and Volunteering: Selection or Causation? Evidence from a Four Year Panel Study. Political Behavior, 34 (2): 225-247. https://renebekkers.wordpress.com/wp-content/uploads/2011/08/bekkers_pobe_12.pdf
Boedhoe, P. S., Schmaal, L., Abe, Y., Ameis, S. H., Arnold, P. D., Batistuzzo, M. C., … & members of the ENIGMA OCD Working Group. (2017). Distinct subcortical volume alterations in pediatric and adult OCD: a worldwide meta-and mega-analysis. American Journal of Psychiatry, 174(1), 60-69. https://doi.org/10.1176/appi.ajp.2016.16020201
Costafreda, S. G. (2009). Pooling FMRI data: meta-analysis, mega-analysis and multi-center studies. Frontiers in Neuroinformatics, 3, 33. https://doi.org/10.3389/neuro.11.033.2009
Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14(2), 81–100. https://doi.org/10.1037/a0015914
De Wit, A., Qu, H. & Bekkers, R. (2022). The health advantage of volunteering is larger for older and less healthy volunteers in Europe: a mega-analysis. European Journal of Aging, https://doi.org/10.1007/s10433-022-00691-5
De Wit, A., Bekkers, R., Karamat Ali, D., & Verkaik, D. (2015). Welfare impacts of participation. Deliverable 3.3 of the project: “Impact of the Third Sector as Social Innovation” (ITSSOIN), European Commission – 7th Framework Programme, Brussels: European Commission, DG Research. http://itssoin.eu/site/wp-content/uploads/2015/09/ITSSOIN_D3_3_The-Impact-of-Participation.pdf
Major Depressive Disorder Working Group of the Psychiatric GWAS Consortium (2013). A mega-analysis of genome-wide association studies for major depressive disorder. Molecular Psychiatry, 18(4): 497-511. https://dx.doi.org/10.1038%2Fmp.2012.21
Van Ingen, E. & Bekkers, R. (2015). Trust Through Civic Engagement? Evidence From Five National Panel Studies. Political Psychology, 36 (3): 277-294. https://renebekkers.wordpress.com/wp-content/uploads/2015/05/vaningen_bekkers_15.pdf
Steinberg, K.K., Smith, S.J., Stroup, D.F., Olkin, I., Lee, N.C., Williamson, G.D. & Thacker, S.B. (1997). Comparison of Effect Estimates from a Meta-Analysis of Summary Data from Published Studies and from a Meta-Analysis Using Individual Patient Data for Ovarian Cancer Studies. American Journal of Epidemiology, 145: 917-925. https://doi.org/10.1093/oxfordjournals.aje.a009051
Rene Bekkers 
This has helped no end (previously very lost MSc student!). Thank you so much.
Pingback: Health benefits of volunteering | Rene Bekkers