Cut the crap, fund the research

We all spend way too much time preparing applications for research grants. This is a collective waste of time. For the 2019 vici grant scheme of the Netherlands Organization for Scientific Research (NWO) in which I recently participated, 87% of all applicants received no grant. Based on my own experiences, I made a conservative calculation (here is the excel file so you can check it yourself) of the total costs for all people involved. The costs total €18.7 million. Imagine how much research time that is worth!

Cost

Applicants account for the bulk of the costs. Taken together, all applicants invested €15.8 million euro in the grant competition. As an applicant, I read the call for proposals, first considered whether or not I would apply, decided yes, I read the guidelines for applications, discussed ideas with colleagues, read the literature, wrote a short draft of the proposal to invite research partners, then wrote the proposal text, formatted the application according to the guidelines, prepared a budget for approval, collected some new data and analyzed it, considered whether ethics review was necessary, created a data management plan, corresponded with: grants advisors, a budget controller, HR advisors, internal reviewers, my head of department, the dean, a coach, and with societal partners. I revised the application, revised the budget, and submitted the preproposal. I waited. And waited. Then I read the preproposal evaluation by the committee members, and wrote responses to the preproposal evaluation. I revised my draft application again, and submitted the full application. I waited. And waited. I read the external reviews, wrote responses to their comments, and submitted a rebuttal. I waited. And waited. Then I prepared a 5 minutes pitch for the interview by the committee, responded to questions, and waited. Imagine I would have spent all that time on actual research. Each applicant could have spent 971 hours on research instead.

Also the university support system spends a lot of resources preparing budgets, internal reviews, and training of candidates. I involved research partners and societal partners to support the proposal. I feel bad for wasting their time as well.

The procedure also puts a burden on external reviewers. At a conference I attended, one of the reviewers of my application identified herself and asked me what had happened with the review she had provided. She had not heard back from the grant agency. I told her that she was not the only one who had given an A+ evaluation, but that NWO had overruled it in its procedures.

For the entire vici competition, an amount of €46.5 million was available, for 32 grants to be awarded. The amount wasted is 40% of that amount! That is unacceptable.

It is time to stop wasting our time.

 

Note: In a previous version of this post, I assumed that the number of applicants was 100. This estimate was much too low. The grant competition website says that across all domains 242 proposals were submitted. I revised the cost calculation (v2) to reflect the actual number of applicants. Note that this calculation leaves out hours spent by researchers who eventually decided not to submit a (pre-)proposal. The calculation further assumes that 180 full proposals were submitted and 105 candidates were interviewed.

Update, February 26: In the previous the cost of the procedure for NWO was severely underestimated. According to the annual report of NWO, the total salary costs for its staff that handles grant applications is €72 million per year. In the revised cost calculation, I’m assuming staff spend 218 hours for the entire vici competition. This amount consists of €198k variable costs (checking applications, inviting reviewers, composing decision letters, informing applicants, informing reviewers, handling appeals by 10% of full proposals, and handling ‘WOB verzoeken’ = Freedom Of Information Act requests) and €20k fixed costs: preparing the call for proposals, organizing committee meetings to discuss applications and their evaluations, attending committee meetings, reporting on committee meetings, evaluating the procedure).

Leave a comment

Filed under academic misconduct, economics, incentives, policy evaluation, taxes, time use

Revolutionizing Philanthropy Research Webinar

January 30, 11am-12pm (EST) / 5-6pm (CET) / 9-10pm (IST)

Why do people give to the benefit of others – or keep their resources to themselves? What is the core evidence on giving that holds across cultures? How does giving vary between cultures? How has the field of research on giving changed in the past decades?

10 years after the publication of “A Literature Review of Empirical Studies of Philanthropy: Eight Mechanisms that Drive Charitable Giving” in Nonprofit and Voluntary Sector Quarterly, it is time for an even more comprehensive effort to review the evidence base on giving. We envision an ambitious approach, using the most innovative tools and data science algorithms available to visualize the structure of research networks, identify theoretical foundations and provide a critical assessment of previous research.

We are inviting you to join this exciting endeavor in an open, global, cross-disciplinary collaboration. All expertise is very much welcome – from any discipline, country, or methodology. The webinar consists of four parts:

  1. Welcome: by moderator Pamala Wiepking, Lilly Family School of Philanthropy and VU Amsterdam;
  2. The strategy for collecting research evidence on giving from publications: by Ji Ma, University of Texas;
  3. Tools we plan to use for the analyses: by René Bekkers, Vrije Universiteit Amsterdam;
  4. The project structure, and opportunities to participate: by Pamala Wiepking.

The webinar is interactive. You can provide comments and feedback during each presentation. After each presentation, the moderator selects key questions for discussion.

We ask you to please register for the webinar here: https://iu.zoom.us/webinar/register/WN_faEQe2UtQAq3JldcokFU3g.

Registration is free. After you register, you will receive an automated message that includes a URL for the webinar, as well as international calling numbers. In addition, a recording of the webinar will be available soon after on the Open Science Framework Project page: https://osf.io/46e8x/

Please feel free to share with everyone who may be interested, and do let us know if you have any questions or suggestions at this stage.

We look forward to hopefully seeing you on January 30!

You can register at https://iu.zoom.us/webinar/register/WN_faEQe2UtQAq3JldcokFU3g

René Bekkers, Ji Ma, Pamala Wiepking, Arjen de Wit, and Sasha Zarins

1 Comment

Filed under altruism, bequests, charitable organizations, crowdfunding, economics, experiments, fundraising, helping, household giving, informal giving, open science, philanthropy, psychology, remittances, sociology, survey research, taxes, volunteering

The Magic of Science

Dinosaurs are like magic. They capture the attention because of their size and sharp teeth. The fact they are no longer among us may also contribute to their popularity. In science, we still have dinosaurs. They do date back to the prehistoric age, when scientists could build careers on undisclosed data and procedures. But we have entered the new age of open science, with comets and earthquakes causing dark clouds in the sky and blocking our view of the sun.

dino_bummer

In the prehistoric age, a lot of science was like magic. The wizard waved his wand, and…. poof: there was the result that only the wizard could reproduce. If nobody can repeat your trick, it’s not science. When you dig up old research, you are stuck with a lot of ‘magic’. Make sure you can detect it.

magic

Unlike real magic, the tricks of illusionists are highly reproducible. It may take some time to learn tricks and you will need the appropriate equipment, but if you know the secret recipe, you can dress up like a magician, and perform the very same act you could not figure out when you were in the audience.

Needless to say, it is our collective responsibility to disclose all the tricks and equipment we use in our research. Here’s a list of things we can do to make this happen.

Leave a comment

Filed under open science

A Conversation About Data Transparency

The integrity of the research process serves as the foundation for excellence in research on nonprofit and voluntary action. While transparency does not guarantee credibility, it guarantees you will get the credibility you deserve. Therefore we are developing criteria for transparency standards with regards to the reporting of methods and data.

We started this important conversation at the 48th ARNOVA Conference in San Diego, on Friday, November 22, 2019. In the session, we held a workshop to survey which characteristics of data and methods transparency that help review research and utilize past work as building blocks for future research.

This session was well attended and very interactive. After a short introduction by the editors of NVSQ, the leading journal in the field, we split up in three groups of researchers that work with the same type of data. One group for data from interviews, one for survey data, and one for administrative data such as 990s. In each group we first took 10 minutes for ourselves, formulating criteria for transparency that allow readers to assess the quality of research. All participants received colored sticky notes, and wrote down one idea per note: laudable indicators on green notes, and bad signals on red notes.

IMG-2421

Next, we put the notes on the wall and grouped them. Each cluster received a name on a yellow note. Finally, we shared the results of the small group sessions with the larger group.

IMG-2424

Though the different types of data to some extent have their own quality indicators, there were striking parallels in the match between theory and research design, ethics, sampling, measures, analysis, coding, interpretation, and write-up of results. After the workshop, we collected the notes. I’ve summarized the results in a report about the workshop. In a nutshell, all groups distinguished five clusters of criteria:

  • A. Meta-criteria: transparency about the research process and the data collection in particular;
  • B. Before data collection: research design and sampling;
  • C. Characteristics of the data as presented: response, reliability, validity;
  • D. Decisions about data collected: analysis and causal inference;
  • E. Write-up: interpretation of and confidence in results presented.

bty

Here is the full report about the workshop. Do you have suggestions about the report? Let me know!

Leave a comment

Filed under data, experiments, methodology, open science, survey research

Gevonden: student-assistent Geven in Nederland 2020

De werkgroep Filantropische Studies van de Faculteit Sociale Wetenschappen aan de Vrije Universiteit Amsterdam is het expertisecentrum op het gebied van onderzoek naar filantropie in Nederland. De werkgroep houdt zich bezig met vragen zoals: Waarom geven mensen vrijwillig geld aan goede doelen? Waarom verrichten mensen vrijwilligerswerk? Hoeveel geld gaat er om in de filantropische sector? Voor het onderzoek Geven in Nederland heeft de werkgroep een student-assistent gevonden: Florian van Heijningen. Welkom!

Leave a comment

Filed under bequests, Center for Philanthropic Studies, charitable organizations, corporate social responsibility, data, foundations, household giving, Netherlands, philanthropy, statistical analysis, survey research

Research on giving in the Netherlands continues, funding secured

We are pleased to announce that the Center for Philanthropic Studies has been able to secure funding for continued research on giving in the Netherlands. The funding enables data collection for the Giving in the Netherlands Panel Survey among households, as well as data collection on corporations, foundations, charity lotteries, and bequests.

In the past 20 years, Giving in the Netherlands has been the prime source of data on trends in the size and composition of philanthropy in the Netherlands. Continuation of the research was uncertain for more than a year because the ministry of Justice and Security withdrew 50% of its funding, calling upon the philanthropic sector to co-fund the research. In an ongoing dialogue with the philanthropic sector, the VU-Center sought stronger alignment of the research with the need for research in practice. The Center has organized round table discussions and an advisory group of experts from the sector has been composed. The Center will use the insights from this dialogue in the research.

Meanwhile the fieldwork has started. Preliminary estimates of giving in the Netherlands will be discussed at a symposium for members of branch organizations in the philanthropic sector in the Fall of 2019. Full publication of the results is scheduled mid-April 2020, at the National Day of Philanthropy.

5 Comments

Filed under Center for Philanthropic Studies, charitable organizations, corporate social responsibility, data, foundations, fundraising, household giving, informal giving, Netherlands, survey research, trends

Grootste goededoelenorganisaties ontvangen minder uit giften

Het ‘sectoronderzoek’ van Goede Doelen Nederland onder de 24 grootste goede doelen organisaties is weer verschenen. https://www.goededoelennederland.nl/system/files/public/Sector/190726%20Overzicht%20cijfers%20grote%20goede%20doelen%20met%20meer%20dan%2020%20miljoen.pdf

In het persbericht is onder de optimistische kop “Maatschappelijke betrokkenheid bij goede doelen onveranderd groot” te lezen dat “dat de resultaten in 2018 nagenoeg hetzelfde zijn als in 2017”. Ik zie iets anders. Drie tekenen aan de wand:

  1. De vrijgevigheid in Nederland neemt af. In 2018 ontvingen de grootste goede doelen in Nederland €571,6 miljoen, uit giften van particulieren en bedrijven. In 2017 was dit nog €583,6 miljoen. Een afname van 2%. De afname van giften van particulieren was bijna 4%. Tegelijk was de inflatie in Nederland 1,7%. De ontvangen euro’s zijn dus ook nog minder waard geworden.
  2. Een toename in inkomsten uit overheidssubsidie (goed voor 35% van de inkomsten van deze goede doelen), nalatenschappen (goed voor 12% van de inkomsten) en de goede doelen loterijen (nu 10% van de inkomsten) hebben de daling van de vrijgevigheid van particulieren en bedrijven opgevangen, maar niet goed gemaakt.
  3. De grootste goede doelen in Nederland besteden exact nul euro (= €0) aan opleidingen.

Update, 26 september 2019: ook de Volkskrant heeft de jaarverslagen van de grootste goededoelenorganisaties geanalyseerd en komt tot vergelijkbare conclusies. Zie hier het hoofdartikel dat op de voorpagina stond en hier het achtergrondartikel. Navraag bij het CBF leert dat de nul euro besteed aan opleidingen geen betrekking heeft op opleidingen van medewerkers, maar opleidingen van doelgroepen; de grootste goededoelenorganisaties zijn dus niet actief op het terrein van onderwijs.

Leave a comment

Filed under bequests, charitable organizations, data, economics, fundraising, household giving, Netherlands, trends