Big adventure: how do you get a €1.5 ($1.7) million grant for five years of research at the Netherlands Organization for Scientific Research (NWO)? Here are my experiences with Global Giving, a research proposal for a ‘vici’-grant. After a rollercoaster extending a period of five years, I did not get the grant.
Update: December 3, 2020. The Journal of Trial and Error published the rejected grant application, along with the reviews. JOTE aims to publish answers to the question “what went wrong?”. Here, I want to share four reflections with you about the things that went well after things went wrong.
First, the rejection made me realize that when resources are scarce, team science is a solution. I’ve shifted my research agenda, away from projects that can only work with personal grants. So I am now focusing on building the field with a meta-research project Revolutionizing Philanthropy Research, in computational social science on the organizational level of nonprofit organizations, and in longitudinal research on the individual level causes of blood donor behavior and the effects of volunteering, using registry data.
Second, after I calculated the collective costs of competitive research grant competitions I’ve reduced my ambitions to win competitive research grants and the willingness to invest in writing research proposals that are evaluated by a committee with little chance of getting funded. I also discourage others to spend time on research proposals that are unlikely to get funded.
Third, I’ve radicalized my commitment to openness and transparency in research. Perhaps sharing these materials help others. If I’m not getting funds to do the research, others may be able to acquire them. I will share everything in new research.
Fourth, I am giving lower priority to research, and more priority to matters of research quality – research ethics, integrity and data transparency – and to teaching.
May 27, 2020. I’ve decided not to file an appeal. It is abundantly clear what a waste of time the grant writing circus is. Collectively, the cost of the competition was almost €20 million before any grant was awarded – see the calculations here. 87% of all applicants received no grant. If I would force an appeal committee to reconsider the decision of the Domain Board Social Sciences and Humanities of NWO that waste of time would only become larger.
On February 19, 2020, I received a letter with the motivation for the rejection. The award selection committee invited me for an interview on December 12, 2019. The 5 minutes pitch went well. The 15 minutes of interview questions focused on theory aspects of the proposal. The committee started out with questions on the conceptual innovation in the proposal and the relevance of research on giving for debates in specific disciplines. Next came a question on the relationship between informal and formal giving. Then the committee asked about knowledge utilization and current policy debates that may be informed by the research. The final two questions were on the possibility to include experiments with volunteering and study giving behavior of children. Here is a full report of the interview. All in all, the interview went well, I think. I was cautiously optimistic.
The proposal ‘Global Giving’ is here. Also I wrote a response to reviews of the pre-proposal by committee members, posted here. Anonymous reviews of the proposal by experts were unanimously positive, thoughtful, and constructive. Here are the reviews. They could hardly be better: 1 A and 4 times A+. Thanks to all who have contributed to responding to the reviews. Here is my ‘rebuttal’. The reviews are similar to those I received previously. Here are the reviews from the first attempt (2016), and here are the reviews of the second attempt (2018).
The Back Story
I originally wrote the grant proposal in 2016. I invested a lot of time in it. My first two attempts failed. In both cases the committee did not like the proposal, although the expert reviewers gave very favourable ratings. The first time I applied for funding I received five reviews that were all ‘excellent’, or A+. The marks could not have been higher (the reviews are posted here). The second time, I received 3 times an A+, one A and one B. I will tell you about the B later, but it was in fact an A+. Still, the committee did not recommend the proposal for funding. Why not?
The first time, the rejection letter (posted here) stated that my responses during the interview were ‘not convincing enough and lacked depth’. In retrospect, I now understand why the committee wrote this. During the interview, I had kept my responses brief and did not engage in speculations, in keeping with an advice I had received during the preparations of the interview. I now see that this was a mistake – I should have engaged more with the committee.
For the second attempt I submitted virtually the same proposal, because the reviews had been so good the first time. Meanwhile, fieldwork in Africa had become cheaper, and I was able to include several countries on the continent in the proposal. But at the second attempt, I did not even get an invitation for an interview. Again, the committee had recommended that I should not submit a full proposal. This time, the committee found confirmation in the less than perfect evaluations by the reviewers (posted here). I received 3x an A+, 1 A, and 1 B. The reviewer who gave a B (R1) wrote: “my major concern is about the realization of the multi-national data collection. If the project is really about collecting >1,000 responses in 145 countries, it will have a major impact. But for this aim, I assume the funds planned for international researchers (80,000) are insufficient.” The reviewer misread the budget. In my rebuttal (posted here), I explained to the selection committee that the 80k budgeted for international collaboration is not for the survey data collection in WP3, which is budgeted separately, and that the plan really is to analyze microdata from all 145 countries in the Gallup World Poll. The committee did not even respond to this. I’m hoping the third attempt will be more satisfactory.
- Pre-application – March 26, 2019; read the pre-proposal here.
- Non-binding advice from committee: please do not submit a full proposal – June 17, 2019; read the reviews from NWO here.
- Full proposal – August 27, 2019; read the full proposal here – response to committee members
- Expert reviews – October 29, 2019; read the reviews here.
- Rebuttal to criticism in expert reviews – 5 November 2019; read the rebuttal here.
- Selection for interview – November 2019
- Interview – December 12, 2019
- No grant – February 20, 2020
Previously, on “Global Giving”
Season 1, Episode 1: “First Contact” – Rene has a good idea: study generosity across the world using the best possible methods. When the evaluation committee recommends not to submit a full proposal for a grant competition, he ignores the advice.
Season 1, Episode 2: “Second Best” – Expert reviews come in: the external advice is unanimously positive. The reviews are excellent: five times A+. Even the most experienced grants advisors say this is unheard of.
Season 1, Episode 3: “Such a Shame” – Rene messes up the interview with the evaluation committee. Or vice versa, that’s hard to tell.
Season 2, Episode 1: “I remember now” – Strengthened by re-reading the external reviews, Rene resubmits the proposal.
Season 3, Episode 1: “Open the door!” – Rene publishes the pre-proposal.
Season 3, Episode 3: “Now what?” – You are currently watching this episode.
Frequently Asked Questions
Here are responses to some of most frequently asked questions from colleagues and competitors about the open grant review initiative.
Why are you participating in a research grant competition that you think is a waste of time?
As I’ve explained in previous blog posts, I think that research grant competitions like the vici are inefficient. Still I participate in them, because they are the only way university researchers can get sufficient funds to do research at a meaningful scale. Collectively speaking, however, the competition is a huge waste of time. The research we all do does not get much better because of the review reports on the proposals we get rejected. My case is an example: I did not get funding for a research plan that according to the review reports cannot be improved. And I am not the only one. There is simply too much excellent research out there that does not get funded. We would be much better off with a lottery than with an extensive peer review procedure (see some of the evidence here).
Is this legal? Can you publish a research proposal before it is funded?
Yes. The Netherlands Organization for Scientific Research (NWO) does not forbid publication of research proposals. In fact, article #56 of the standards of good practice in the code of conduct for researchers in the Netherlands even explicitly requires researchers to do the following: “As a supervisor, principal investigator, research director or manager, provide for an open and inclusive culture in all phases of research.” All phases includes the stage at which researchers produce ideas and design the research, before it receives funding. The proposal has been admitted to the competition, without further conditions. On Twitter, NWO even liked this post, and the public relations officer wrote an email to say the organization supports the initiative.
Are you not afraid that your research ideas will be stolen?
No. Open publication of research ideas promote their execution. I have never encountered a researcher whose ideas were stolen after publication. In the unlikely event that someone else obtains funding for exactly the same research plans, the publication of this research proposal will be important evidence about the chronology of events. In the more likely scenario that someone else obtains funding for a similar research project, I will be happy that the research is funded.
How will publishing the proposal affect its chances of getting funded?
I have no idea, but it should not matter. The review process is not double blind, so for them it does not make a difference. I hope that the committee likes it. In previous attempts, the committee did not like the pre-application, and discouraged me to submit a full proposal. These committees did not yet have expert reviews though.
How much time did you invest in this proposal?
This is the third time I am trying to get a vici-grant. The first time I wrote the proposal, I spent about three months (full time) thinking, reading previous research, designing the study, and writing up the plans. Responding to reviews took another two weeks. Preparations for the interview took about one week. The second time I submitted the proposal, I spent about three weeks revising it. This third time, I spent about three days.
Why is a multidisciplinary approach required to understand philanthropy?
Societies do not observe the artificial boundaries between academic disciplines, and neither do people. Giving can be a political act; it can be religiously motivated; it can be a form of conspicuous consumption by the wealthy; and it may spring from humanitarian concerns. Different disciplines have focused on explanations at different levels, and with particular clusters of determinants. In reality, they are interwoven in a complex web. To explain philanthropy, we need to understand how these motivations and circumstances interact.
Why do you use a western definition of giving? You are ignoring informal giving.
All forms of giving, both formal and informal, will be measured in Global Giving. Some societies may have very levels of informal giving, for instance when people are dependent upon kin and extended family networks. The low levels of philanthropy do not imply that people in these countries are not generous. I will test the hypothesis that formal and informal giving are substitutes. As societies advance economically and politically, citizens are less dependent upon each other. Practices of formal philanthropy are likely to emerge in societies in which citizens trust strangers and institutions. At the individual level, however, formal and informal giving go hand in hand, because the same prosocial citizens engage in both forms of giving.
Why are you using registered reports in WP4?
The scientific record is littered with false positives. This is a particularly large problem in research relying on experiments with small numbers of observations. Registered reports are the best way to avoid false positives.