Category Archives: methodology

Introducing Mega-analysis

How to find truth in an ocean of correlations – with breakers, still waters, tidal waves, and undercurrents? In the old age of responsible research and publication, we would collect estimates reported in previous research, and compute a correlation across correlations. Those days are long gone.

In the age of rat race research and publication it became increasingly difficult to do a meta-analysis. It is a frustrating experience for anyone who has conducted one: endless searches on the Web of Science and Google Scholar to collect all published research, input the estimates in a database, find that a lot of fields are blank, email authors for zero-order correlations and other statistics they had failed to report in their publications and get very little response.

Meta-analysis is not only a frustrating experience, it is also a bad idea when results that authors do not like do not get published. A host of techniques have been developed to find and correct publication bias, but the problem that we do not know the results that do not get reported is not solved easily.

As we enter the age of open science,  we do not have to rely any longer on the far from perfect cooperation from colleagues who have moved to a different university, left academia, died, or think you’re trying to prove them wrong and destroy your career. We can simply download all the raw data and analyze them.

Enter mega-analysis: include all the data points relevant for a certain hypothesis, cluster them by original publication, date, country, or any potentially relevant property of the research design, and add the substantial predictors you find documented in the literature. The results reveal not only the underlying correlations between substantial variables, but also the differences between studies, periods, countries and design properties that affect these correlations.

The method itself is not new. In epidemiology, and Steinberg et al. (1997) labeled it ‘meta-analysis of individual patient data’. In human genetics, genome wide association studies (GWAS) by large international consortia are common examples of mega-analysis.

Mega-analysis includes the file-drawer of papers that never saw the light of day after they were put in. It also includes the universe of papers that have never been written because the results were unpublishable.

If meta-analysis gives you an estimate for the universe of published research, mega-analysis can be used to detect just how unique that universe is in the milky way. My prediction would be that correlations in published research are mostly further from zero than the same correlation in a mega-analysis.

Mega-analysis bears great promise for the social sciences. Samples for population surveys are large, which enables optimal learning from variations in sampling procedures, data collection mode, and questionnaire design. It is time for a Global Social Science Consortium that pools all of its data. As an illustration, I have started a project on the Open Science Framework that mega-analyzes generalized social trust. It is a public project: anyone can contribute. We have reached mark of 1 million observations.

The idea behind mega-analysis originated from two different projects. In the first project, Erik van Ingen and I analyzed the effects of volunteering on trust, to check if results from an analysis of the Giving in the Netherlands Panel Survey (Van Ingen & Bekkers, 2015) would replicate with data from other panel studies. We found essentially the same results in five panel studies, although subtle differences emerged in the quantative estimates. In the second project, with Arjen de Wit and colleagues from the Center for Philanthropic Studies at VU Amsterdam, we analyzed the effects of volunteering on well-being conducted as part of the EC-FP7 funded ITSSOIN study. We collected 845.733 survey responses from 154.970 different respondents in six panel studies, spanning 30 years (De Wit, Bekkers, Karamat Ali & Verkaik, 2015). We found that volunteering is associated with a 1% increase in well-being.

In these projects, the data from different studies were analyzed separately. I realized that we could learn much more if the data are pooled in one single analysis: a mega-analysis.

References

De Wit, A., Bekkers, R., Karamat Ali, D., & Verkaik, D. (2015). Welfare impacts of participation. Deliverable 3.3 of the project: “Impact of the Third Sector as Social Innovation” (ITSSOIN), European Commission – 7th Framework Programme, Brussels: European Commission, DG Research.

Van Ingen, E. & Bekkers, R. (2015). Trust Through Civic Engagement? Evidence From Five National Panel StudiesPolitical Psychology, 36 (3): 277-294.

Steinberg, K.K., Smith, S.J., Stroup, D.F., Olkin, I., Lee, N.C., Williamson, G.D. & Thacker, S.B. (1997). Comparison of Effect Estimates from a Meta-Analysis of Summary Data from Published Studies and from a Meta-Analysis Using Individual Patient Data for Ovarian Cancer Studies. American Journal of Epidemiology, 145: 917-925.

Leave a comment

Filed under data, methodology, open science, regression analysis, survey research, trends, trust, volunteering

Four Reasons Why We Are Converting to Open Science

The Center for Philanthropic Studies I am leading at VU Amsterdam is converting to Open Science.

Open Science offers four advantages to the scientific community, nonprofit organizations, and the public at large:

  1. Access: we make our work more easily accessible for everyone. Our research serves public goods, which are served best by open access.
  2. Efficiency: we make it easier for others to build on our work, which saves time.
  3. Quality: we enable others to check our work, find flaws and improve it.
  4. Innovation: ultimately, open science facilitates the production of knowledge.

What does the change mean in practice?

First, the source of funding for contract research we conduct will always be disclosed.

Second, data collection – interviews, surveys, experiments – will follow a prespecified protocol. This includes the number of observations forseen, the questions to be asked, measures to be included, hypotheses to be tested, and analyses to be conducted. New studies will be preferably be preregistered.

Third, data collected and the code used to conduct the analyses will be made public, through the Open Science Framework for instance. Obviously, personal or sensitive data will not be made public.

Fourth, results of research will preferably be published in open access mode. This does not mean that we will publish only in Open Access journals. Research reports and papers for academic will be made available online in working paper archives, as a ‘preprint’ version, or in other ways.

 

December 16, 2015 update:

A fifth reason, following directly from #1 and #2, is that open science reduces the costs of science for society.

See this previous post for links to our Giving in the Netherlands Panel Survey data and questionnaires.

 

July 8, 2017 update:

A public use file of the Giving in the Netherlands Panel Survey and the user manual are posted at the Open Science Framework.

Leave a comment

Filed under academic misconduct, Center for Philanthropic Studies, contract research, data, fraud, incentives, methodology, open science, regulation, survey research

The Fishy Business of Philanthropy

Breaking news today: the essential amino acid L-Tryptophan (TRP) makes people generous! Three psychologists at the University of Leiden, Laura Steenbergen, Roberta Sellara, and Lorenza Colzato, secretly gave 16 participants in an experiment a dose of TRP, solved in a glass of orange juice. The 16 other participants in the study drank plain orange juice, without TRP. The psychologists did not write where the experiment was conducted, but describe the participants as 28 female and 4 male students in southern Europe – which is likely to be Italy, given the names of the second and third authors. Next, the participants were kept busy for 30 minutes with an ‘attentional blink task that requires the detection of two targets in a rapid visual on-screen presentation’. After they had completed a task, they were given a reward of €10. Then the participants were given an opportunity to donate to four charities: Unicef, Amnesty International, Greenpeace, and World Wildlife Fund. And behold the wonders of L-Tryptophan: the 0,8 grams of TRP more than doubled the amount donated from €0.47 (yes, that is less than five percent of the €10 earned) to €1.00. Even though the amount donated is small, the increase due to TRP is huge: +112%.

Why is this good to know? Why does tryptophan increase generosity? Steenbergen, Sellara and Colzato reasoned that TRP influences synthesis of the neurotransmitter serotonin (called 5-HT), which has been found to be associated with charitable giving in several economic experiments. The participants in the experiment were not tested for serotonin levels, but the results are consistent with these previous experiments. The new experiment takes us one step further into the biology of charity, by showing that the intake of food enriched by tryptohan is making female students in Italy more generous to charity.

Tryptophan is an essential amino acid, commonly found in protein-rich foods such as chocolate, eggs, milk, poultry, fish, and spinach. Rense Corten, a former colleague of mine, asked on Twitter: how much spinach the participants would have had to digest to obtain a TRP intake that would make them give an additional €1 to charity? Just for fun I computed this: it is about 438 grams of spinach. Less than the 1161 grams of chocolate it would take to generate the same dose of TRP as the participants got in their orange juice.

The fairly low level of giving in the experiment is somewhat surprising given the overall level of charitable giving in Italy. According to the Gallup World Poll some 62% of Italians made donations to charity in 2011, ranking the country 14th in the world. But wait – Italians eat quite some fish, don’t they? If there is a lot of tryptophan in fish, Italians should be more generous than inhabitants of other countries that consume less fish. Indeed the annual fish consumption per capita in Italy (some 25 kilograms, ranking the country 14th in the world) is much higher than in the Czech Republic (10 kilograms; rank: 50), and the Czech population is less likely to give to charity (31%, rank: 30).

Of course this comparison of just two countries in Europe is not representative of the any part of the world. And yes, it is cherry-picked: an initial comparison with Austria (14 kilograms of fish per year, much less than in Italy) did not yield a result in the same direction (69% gives, more than in Italy). But lining up all countries in the world for which there are data on fish consumption and engagement in charity does yield a positive correlation between the two. Here is the excel file including the data. The relationship is modest (r = .30), but still: we now know that inhabitants of countries that consume more fish per capita are somewhat more likely to give to charity.

fishconsumption_givingtocharities

Leave a comment

Filed under experiments, household giving, methodology, philanthropy

Why a high R Square is not necessarily better

Often I encounter academics thinking that a high proportion of explained variance is the ideal outcome of a statistical analysis. The idea is that in regression analyses a high R Square is better than a low R Square. In my view, the emphasis on a high R2 should be reduced. A high R2 should not be a goal in itself. The reason is that a higher R2 can easily be obtained by using procedures that actually lower the external validity of coefficients.

It is possible to increase the proportion of variance explained in regression analyses in several ways that do not in fact our ability to ‘understand’ the behavior we are seeking to ‘explain’ or ‘predict’. One way to increase the R2 is to remove anomalous observations, such as ‘outliers’ or people who say they ‘don’t know’ and treat them like the average respondent. Replacing missing data by mean scores or using multiple imputation procedures often increases the Rsquare. I have used this procedure in several papers myself, including some of my dissertation chapters.

But in fact outliers can be true values. I have seen quite a few of them that destroyed correlations and lowered R squares while being valid observations. E.g., a widower donating a large amount of money to a charity after the death of his wife. A rare case of exceptional behavior for very specific reasons that seldom occur. In larger samples these outliers may become more frequent, affecting the R2 less strongly.

Also ‘Don’t Know’ respondents are often systematically different from the average respondent. Treating them as average respondents eliminates some of the real variance that would otherwise be hard to predict.

Finally, it is often possible to increase the proportion of variance explained by including more variables. This is particularly problematic if variables that are the result of the dependent variable are included as predictors. For instance if network size is added to the prediction of volunteering the R Square will increase. But a larger network not only increases volunteering; it is also a result of volunteering. Especially if the network questions refer to the present (do you know…) while the volunteering questions refer to the past (in the past year, have you…) it is dubious to ‘predict’ volunteering in the past by a measure of current network size.

As a reviewer, I give authors reporting an R2 exceeding 40% a treatment of high-level scrutiny for dubious decisions in data handling and inclusion of variables.

As a rule, R Squares tend to be higher at higher levels of aggregation, e.g. when analyzing cross-situational tendencies in behavior rather than specific behaviors in specific contexts; or when analyzing time-series data or macro-level data about countries rather than individuals. Why people do the things they do is often just very hard to predict, especially if you try to predict behavior in a specific case.

1 Comment

Filed under academic misconduct, data, methodology, regression analysis, survey research

Update: Giving in the Netherlands Panel Survey User Manual

A new version of the User Manual for the Giving in the Netherlands Panel Survey is now available: version 2.2.

The GINPS12 questionnaire is here (in Dutch).

Leave a comment

Filed under data, empathy, experiments, helping, household giving, methodology, philanthropy, principle of care, survey research, trends, trust, volunteering, wealth

What works in crowdfunding? Field experiments will tell

Crowdfunding is a new model of financing public goods. In the Netherlands it has become highly visible by the cutbacks in public funding for the arts. Crowdfunding has grown exponentially in the past years, not just in projects for the arts, but also in financing commercial startups and recently also in funding for science. However, the success of projects advertised on crowdfunding platforms varies enormously. What are the characteristics of successful crowdfunding projects? How can crowdfunding be made more effective as a fundraising tool? How does crowdfunding change the way nonprofit organizations raise funds? These questions will be answered in new research funded by the Netherlands Organization for Scientific Research, NWO. Together with Marcel Veenswijk, Irma Borst (postdoc) and Barend Vernooij (PhD candidate) from VU University Amsterdam and with crowdfunding experts Douw&Koren and crowdfunding platforms Flintwave.com, Seeds.nl, and Voordekunst.nl I will conduct large scale field experiments to test what works in crowdfunding.

Here is a press release (in Dutch) describing the project.

Leave a comment

Filed under crowdfunding, experiments, household giving, methodology

Dag van de Filantropie en Boekpresentatie Geven in Nederland 2013 op 25 april

Op de Dag van de Filantropie 2013 – het jaarlijks terugkerend evenement op de laatste donderdag van april – is dit jaar het boek ‘Geven in Nederland 2013’ gepresenteerd. Dit jaar kreeg een bijzonder tintje door het aanvaarden van een bijzondere leerstoel met het uitspreken van de rede ‘De maatschappelijke betekenis van filantropie’ door René Bekkers.

Kiezen om te Delen: Filantropie in Tijden van Economische Tegenwind

Nu het economisch niet voor de wind gaat zien we allerlei verschuivingen in de filantropie in Nederland. We zien een  terugval in het geefgedrag en verschuivingen in bestedingen van bedrijven en huishoudens. Zij moeten bewustere keuzes maken; onderscheid maken tussen wat écht belangrijk is en wat niet. De dynamiek binnen de bronnen van filantropische bijdragen en maatschappelijke doelen vormden het hoofdthema van het symposium. De presentatie van het onderzoek naar geefgedrag door huishoudens en vermogende Nederlanders vindt u hier. De resultaten van het onderzoek naar bedrijven, sociale normen rond filantropie en de trends in de cijfers van de bijdragen van huishoudens, bedrijven, en loterijen vindt u later op de Geven in Nederland website.

De Maatschappelijke Betekenis van Filantropie

De groeiende aandacht voor filantropie wordt meestal verklaard uit het feit dat de overheid moet bezuinigingen. Men vergeet echter dat de sector filantropie zich vanaf begin jaren ‘90 in rap tempo heeft ontwikkeld. Het “Geven in Nederland”onderzoek maakt deel uit van deze ontwikkeling. Van bezuinigingen was in die periode geen sprake, eerder het tegendeel. Particulier initiatief liet weer van zich horen. Met het sluiten van het Convenant “Ruimte voor Geven” in juni 2011 tussen het kabinet en de sector filantropie is een nieuwe situatie ontstaan, waarin filantropie de ruimte krijgt om meer maatschappelijke betekenis te krijgen.

Wat is de maatschappelijke betekenis van filantropie? Die vraag beantwoordt René Bekkers in zijn oratie. Bekkers is per 1 januari 2013 aan de Faculteit Sociale Wetenschappen van de Vrije Universiteit Amsterdam aangesteld als bijzonder hoogleraar Sociale aspecten van prosociaal gedrag. De leerstoel is mede mogelijk gemaakt door de Van der Gaag Stichting van de Koninklijke Nederlandse Akademie van Wetenschappen (KNAW) voor een periode van vijf jaar. Bekkers gaat in op de herkomst en bestemming van filantropie in de samenleving. Waarom zien we meer filantropie in sommige sociale groepen, landen en perioden dan in andere? In welke sociale omstandigheden doen mensen vrijwilligerswerk en geven ze geld aan goededoelenorganisaties? In welke mate en in welke omstandigheden zullen Nederlanders overheidsbezuinigingen op kunst en cultuur, internationale hulp en andere doelen compenseren?

De volledige tekst van de oratie vindt u hier.

Leave a comment

Filed under altruism, charitable organizations, corporate social responsibility, empathy, foundations, helping, household giving, law, methodology, philanthropy, principle of care, taxes, trust