The Work & Worries of a Webinar

Can everyone hear me? Does my hair look OK? What does the audience think about what I just said? Did I answer the most important questions? Some of these worries are the same now in the Webinar Age as for an old style Pre-COVID-19 in-person conference presentation, but many are new. In a webinar setting it is very difficult to get cues from the audience. Solution: organize an honest feedback channel, separate from your audience.

This is just one of the things we have learned at the Center for Philanthropic Studies at the Vrije Universiteit Amsterdam from transforming an in-person conference to an online webinar. The day before yesterday we organized our Giving in the Netherlands conference entirely online. We had planned this conference to be an in-person event for 260 participants – the maximum capacity of the room that a sponsor kindly offered to us. We were fully booked. Registration was free, with a €50 late cancellations penalty.

Then the ‘intelligent lockdown’ and physical distancing measures imposed by the government in the Netherlands made it impossible to do the conference as planned. After some checks of various presentation platforms, we decided to move the conference online, using Zoom. We reworked the program, and made it shorter. We removed the opening reception, break, and drinks afterward. We first did 3 plenary presentations, and then a panel discussion. Total length of the program was 90 minutes.

We pre-recorded two of the three the presentations (using Loom) so we could broadcast them in a zoom-session. This worked well, though it was a lot of work to create good quality sound and a ‘talking head’ image in the presentations. We have learned a lot about audio feedback loops, natural light effects, and the importance of a neutral background for presentations.

In the preparations for the symposium, I also benefited from the experience moderating the opening plenary at the ARNOVA conference last year. In our online format, instead of having volunteers going around the room, I gave the audience the opportunity to pose questions through a separate online channel, www.menti.com. The online format even had an advantage compared to the hotel ballroom stage setting. During the interview I was able to keep an eye on the questions channel, and I could secretly look at my phone as colleagues sent me texts and emails identifying the questions as they came in. As a result, the discussion went smoothly, and the audience was engaged. After the unilateral research presentations, the panel discussion was a lively change of scene. I interviewed three sector leaders in the Netherlands about COVID-19 effects, and again presented questions from the audience.

Overall, this was a good experience for us, proving that it is possible to do a traditional symposium in an online setting. We also learned that it was a lot of work. You need new audiovisual skills that you don’t learn in graduate school.

You need a team of people working behind the scenes to make it work. We had a moderator, Barry Hoolwerf, introducing the house rules, broadcasting the pre-recorded presentations, and giving the floor to the live speakers – unmuting their microphones and allowing their video to be visible on screen. We had two people, Arjen de Wit and Claire van Teunenbroek, monitoring the questions channel, selecting the most important ones.

Finally, we learned how important it is to test, learn and adapt. We tested the presentations for a smaller audience that we gave a ‘sneak preview’ and learned about technical issues. The test was additional work, but worth it because it took away most of our worries.

You can watch the presentations (in Dutch) here: https://www.geveninnederland.nl/presentatie-geven-in-nederland-2020/. If you’re interested in the book you can download it here: https://www.geveninnederland.nl/publicatie-geven-in-nederland-2020/. A visual summary of the book in English is here: https://renebekkers.files.wordpress.com/2020/04/giving-in-the-netherlands-2020-summary.pdf

Cut the crap, fund the research

We all spend way too much time preparing applications for research grants. This is a collective waste of time. For the 2019 vici grant scheme of the Netherlands Organization for Scientific Research (NWO) in which I recently participated, 87% of all applicants received no grant. Based on my own experiences, I made a conservative calculation (here is the excel file so you can check it yourself) of the total costs for all people involved. The costs total €18.7 million. Imagine how much research time that is worth!

Applicants account for the bulk of the costs. Taken together, all applicants invested €15.8 million euro in the grant competition. As an applicant, I read the call for proposals, first considered whether or not I would apply, decided yes, I read the guidelines for applications, discussed ideas with colleagues, read the literature, wrote a short draft of the proposal to invite research partners, then wrote the proposal text, formatted the application according to the guidelines, prepared a budget for approval, collected some new data and analyzed it, considered whether ethics review was necessary, created a data management plan, corresponded with: grants advisors, a budget controller, HR advisors, internal reviewers, my head of department, the dean, a coach, and with societal partners. I revised the application, revised the budget, and submitted the preproposal. I waited. And waited. Then I read the preproposal evaluation by the committee members, and wrote responses to the preproposal evaluation. I revised my draft application again, and submitted the full application. I waited. And waited. I read the external reviews, wrote responses to their comments, and submitted a rebuttal. I waited. And waited. Then I prepared a 5 minutes pitch for the interview by the committee, responded to questions, and waited. Imagine I would have spent all that time on actual research. Each applicant could have spent 971 hours on research instead.

Also the university support system spends a lot of resources preparing budgets, internal reviews, and training of candidates. I involved research partners and societal partners to support the proposal. I feel bad for wasting their time as well.

The procedure also puts a burden on external reviewers. At a conference I attended, one of the reviewers of my application identified herself and asked me what had happened with the review she had provided. She had not heard back from the grant agency. I told her that she was not the only one who had given an A+ evaluation, but that NWO had overruled it in its procedures.

For the entire vici competition, an amount of €46.5 million was available, for 32 grants to be awarded. The amount wasted is 40% of that amount! That is unacceptable.

It is time to stop wasting our time.

 

Note: In a previous version of this post, I assumed that the number of applicants was 100. This estimate was much too low. The grant competition website says that across all domains 242 proposals were submitted. I revised the cost calculation (v2) to reflect the actual number of applicants. Note that this calculation leaves out hours spent by researchers who eventually decided not to submit a (pre-)proposal. The calculation further assumes that 180 full proposals were submitted and 105 candidates were interviewed.

Update, February 26: In the previous the cost of the procedure for NWO was severely underestimated. According to the annual report of NWO, the total salary costs for its staff that handles grant applications is €72 million per year. In the revised cost calculation, I’m assuming staff spend 218 hours for the entire vici competition. This amount consists of €198k variable costs (checking applications, inviting reviewers, composing decision letters, informing applicants, informing reviewers, handling appeals by 10% of full proposals, and handling ‘WOB verzoeken’ = Freedom Of Information Act requests) and €20k fixed costs: preparing the call for proposals, organizing committee meetings to discuss applications and their evaluations, attending committee meetings, reporting on committee meetings, evaluating the procedure).

Revolutionizing Philanthropy Research Webinar

January 30, 11am-12pm (EST) / 5-6pm (CET) / 9-10pm (IST)

Why do people give to the benefit of others – or keep their resources to themselves? What is the core evidence on giving that holds across cultures? How does giving vary between cultures? How has the field of research on giving changed in the past decades?

10 years after the publication of “A Literature Review of Empirical Studies of Philanthropy: Eight Mechanisms that Drive Charitable Giving” in Nonprofit and Voluntary Sector Quarterly, it is time for an even more comprehensive effort to review the evidence base on giving. We envision an ambitious approach, using the most innovative tools and data science algorithms available to visualize the structure of research networks, identify theoretical foundations and provide a critical assessment of previous research.

We are inviting you to join this exciting endeavor in an open, global, cross-disciplinary collaboration. All expertise is very much welcome – from any discipline, country, or methodology. The webinar consists of four parts:

1. Welcome: by moderator Pamala Wiepking, Lilly Family School of Philanthropy and VU Amsterdam;
2. The strategy for collecting research evidence on giving from publications: by Ji Ma, University of Texas;
3. Tools we plan to use for the analyses: by René Bekkers, Vrije Universiteit Amsterdam;
4. The project structure, and opportunities to participate: by Pamala Wiepking.

The webinar is interactive. You can provide comments and feedback during each presentation. After each presentation, the moderator selects key questions for discussion.

We ask you to please register for the webinar here: https://iu.zoom.us/webinar/register/WN_faEQe2UtQAq3JldcokFU3g.

Registration is free. After you register, you will receive an automated message that includes a URL for the webinar, as well as international calling numbers. In addition, a recording of the webinar will be available soon after on the Open Science Framework Project page: https://osf.io/46e8x/

Please feel free to share with everyone who may be interested, and do let us know if you have any questions or suggestions at this stage.

We look forward to hopefully seeing you on January 30!

You can register at https://iu.zoom.us/webinar/register/WN_faEQe2UtQAq3JldcokFU3g

René Bekkers, Ji Ma, Pamala Wiepking, Arjen de Wit, and Sasha Zarins

The Magic of Science

Dinosaurs are like magic. They capture the attention because of their size and sharp teeth. The fact they are no longer among us may also contribute to their popularity. In science, we still have dinosaurs. They do date back to the prehistoric age, when scientists could build careers on undisclosed data and procedures. But we have entered the new age of open science, with comets and earthquakes causing dark clouds in the sky and blocking our view of the sun.

In the prehistoric age, a lot of science was like magic. The wizard waved his wand, and…. poof: there was the result that only the wizard could reproduce. If nobody can repeat your trick, it’s not science. When you dig up old research, you are stuck with a lot of ‘magic’. Make sure you can detect it.

Unlike real magic, the tricks of illusionists are highly reproducible. It may take some time to learn tricks and you will need the appropriate equipment, but if you know the secret recipe, you can dress up like a magician, and perform the very same act you could not figure out when you were in the audience.

Needless to say, it is our collective responsibility to disclose all the tricks and equipment we use in our research. Here’s a list of things we can do to make this happen.

A Conversation About Data Transparency

The integrity of the research process serves as the foundation for excellence in research on nonprofit and voluntary action. While transparency does not guarantee credibility, it guarantees you will get the credibility you deserve. Therefore we are developing criteria for transparency standards with regards to the reporting of methods and data.

We started this important conversation at the 48^th ARNOVA Conference in San Diego, on Friday, November 22, 2019. In the session, we held a workshop to survey which characteristics of data and methods transparency that help review research and utilize past work as building blocks for future research.

This session was well attended and very interactive. After a short introduction by the editors of NVSQ, the leading journal in the field, we split up in three groups of researchers that work with the same type of data. One group for data from interviews, one for survey data, and one for administrative data such as 990s. In each group we first took 10 minutes for ourselves, formulating criteria for transparency that allow readers to assess the quality of research. All participants received colored sticky notes, and wrote down one idea per note: laudable indicators on green notes, and bad signals on red notes.

Next, we put the notes on the wall and grouped them. Each cluster received a name on a yellow note. Finally, we shared the results of the small group sessions with the larger group.

Though the different types of data to some extent have their own quality indicators, there were striking parallels in the match between theory and research design, ethics, sampling, measures, analysis, coding, interpretation, and write-up of results. After the workshop, we collected the notes. I’ve summarized the results in a report about the workshop. In a nutshell, all groups distinguished five clusters of criteria:

• A. Meta-criteria: transparency about the research process and the data collection in particular;
• B. Before data collection: research design and sampling;
• C. Characteristics of the data as presented: response, reliability, validity;
• D. Decisions about data collected: analysis and causal inference;
• E. Write-up: interpretation of and confidence in results presented.

Here is the full report about the workshop. Do you have suggestions about the report? Let me know!

Gevonden: student-assistent Geven in Nederland 2020

De werkgroep Filantropische Studies van de Faculteit Sociale Wetenschappen aan de Vrije Universiteit Amsterdam is het expertisecentrum op het gebied van onderzoek naar filantropie in Nederland. De werkgroep houdt zich bezig met vragen zoals: Waarom geven mensen vrijwillig geld aan goede doelen? Waarom verrichten mensen vrijwilligerswerk? Hoeveel geld gaat er om in de filantropische sector? Voor het onderzoek Geven in Nederland heeft de werkgroep een student-assistent gevonden: Florian van Heijningen. Welkom!

Research on giving in the Netherlands continues, funding secured

We are pleased to announce that the Center for Philanthropic Studies has been able to secure funding for continued research on giving in the Netherlands. The funding enables data collection for the Giving in the Netherlands Panel Survey among households, as well as data collection on corporations, foundations, charity lotteries, and bequests.

In the past 20 years, Giving in the Netherlands has been the prime source of data on trends in the size and composition of philanthropy in the Netherlands. Continuation of the research was uncertain for more than a year because the ministry of Justice and Security withdrew 50% of its funding, calling upon the philanthropic sector to co-fund the research. In an ongoing dialogue with the philanthropic sector, the VU-Center sought stronger alignment of the research with the need for research in practice. The Center has organized round table discussions and an advisory group of experts from the sector has been composed. The Center will use the insights from this dialogue in the research.

Meanwhile the fieldwork has started. Preliminary estimates of giving in the Netherlands will be discussed at a symposium for members of branch organizations in the philanthropic sector in the Fall of 2019. Full publication of the results is scheduled mid-April 2020, at the National Day of Philanthropy.