Open call for contributors: Transparency in Nonprofit Research

How do publications in nonprofit and philanthropic studies report on the type of data they report on? What are the characteristics of samples and measurement instruments used in research? Which research designs and methods are used in the analysis of data? To what extent do publications describe the generalizability and validity of observations? Which criteria do publications use to support the methodological quality of research in nonprofit and philanthropic studies?

Research in nonprofit and philanthropic studies uses a variety of data and methods. In order to evaluate their quality, it is imperative that the data and methods are clearly described. In this meta science project, we provide an assessment of the transparency in research products sampled randomly from the Knowledge Infrastructure for Nonprofit and Philanthropic Studies (KINPS).

The assessment starts with the current state of research. Retrospective assessments include research from the past 50 years, working backwards from 2022, at ten year intervals such that the state of research of today can be compared with representative samples from 2012, 2002, 1992, 1982 and 1972. The goal of the project is to identify weaknesses in the quality of research in nonprofit and philanthropic studies, and provide suggestions for how to improve research quality.

Ideas for the current meta-science project are outlined here: https://osf.io/hvw73. This project is supported by the Revolutionizing Philanthropy Research (RPR) initiative. See https://osf.io/46e8x/ for the foundational ideas. Previous projects of the RPR consortium include the Knowledge Infrastructure for Nonprofit and Philanthropic Studies (KINPS), https://osf.io/g9d8u/. You can access the database through https://public.tableau.com/app/profile/ji.ma/viz/KINPS/Story1.

Would you like to contribute to this project? If you care about the quality of data and methods in our field, are interested in its development over time, or if you want to be part of it in the future we need your help. We could use assistance to go through a sample of publications, build a database, and analyze it. First results will be published at the upcoming ARNOVA Conference in November 2022. We strongly believe in an open science approach. Therefore this will be an open project, to which all members of the nonprofit research community can contribute. We are open to any suggestions you may have. Anyone with basic skills can participate, and become a co-author on the paper we will write together. If you have advanced skills in programming, data analysis, visualization, or writing, you can assume more responsibility. If you’re interested, please describe your interest and potential contribution in an email message to René Bekkers, r.bekkers@vu.nl.

René Bekkers will give further details on the project and answer questions in an online webinar, on Thursday September 29, at 3 PM Amsterdam time | 9 AM EDT/EST | 9 PM CST | 11 PM AEST. Please write an email to obtain access details for the webinar.

Leave a comment

Filed under data, experiments, history, household giving, methodology, open science, philanthropy, publications, research, science, statistical analysis, survey research, trends

Webinar: Longitudinal Data on Philanthropy

The Researchers, Academics, and Data Enthusiasts (RADE) working group of WINGS is organizing a session “Longitudinal Studies: Contributions to Understanding Giving Behaviour and the Practice of Philanthropy” on September 19th, from 12:30 pm to 2:00 pm (UTC). 

I am delighted to participate in a conversation with Verma and Swati Shresth from the Centre for Social Impact and Philanthropy at the Ashoka University (India), and Una Osili from Lilly Family School of Philanthropy at the Indiana University (USA). 

You can register here; feel free to share this link with colleagues who might also be interested in this conversation.

The session will address the issue of how research and data can influence practice in the field of philanthropy and public policies that affect giving and civil society. The speakers will be referring to the following studies: Lilly Family School of Philanthropy’s Philanthropy Panel Study (PPS), Giving in the Netherlands Panel Survey (GINPS), and How India Gives, which just completed its first survey. These are longitudinal studies, where researchers track the philanthropic engagement of participants for years or even decades, offering important insight into how such behaviors evolve over time. We will also discuss the challenge of funding such research.

Leave a comment

Filed under data, experiments, fundraising, health, helping, household giving, law, methodology, Netherlands, open science, philanthropy, policy evaluation, remittances, research, science, statistical analysis, survey research, trends, volunteering

Nederlandse miljonairs geven €1274 per jaar aan goede doelen

Het was *het* nieuws van de Dag van de Filantropie: Nederlandse miljonairs geven per jaar €1274 aan goede doelen. Het nieuws komt uit het hoofdstuk ‘Geven door huishoudens‘ van Geven in Nederland 2022, dat we op 16 juni 2022 presenteerden op een symposium aan de VU.

Na afloop van de presentatie sprak ik drie heel verschillende mensen. Zij vonden het om uiteenlopende redenen een laag bedrag. Voor de een was het een laag bedrag omdat zij mensen kende die niet vermogend zijn, maar toch vergelijkbare bedragen doneren aan goede doelen. Voor de ander was het een laag bedrag omdat de miljonairs die hij kende aanzienlijk hogere bedragen gaven. De derde kende geen miljonairs maar leek het een laag bedrag omdat zij had verwacht dat miljonairs meer zouden geven.

Het bedrag is in ieder geval flink minder dan eerder onderzoek had gesuggereerd. Een eerste onderzoek uit 2013 onder 741 vermogende klanten van ABN/AMRO-MeesPierson die de bank helpt met advies over giften aan goede doelen zeiden miljonairs gemiddeld €11.100 te geven. Dat was een grove overschatting. Die groep was niet representatief voor alle miljonairs in Nederland: hun vermogen was hoger dan dat van alle miljonairs in Nederland, en bijna de helft (46%) zei gebruik te maken van de giftenaftrek, terwijl in werkelijkheid dat percentage toen maar 17% was. Het is niet gek dat klanten van een bank die hen advies geeft over het opzetten van een eigen stichting en over andere vormen van filantropie meer geven dan de gemiddelde miljonair. Het bedrag is ook hoger dan een vervolgonderzoek uit 2016 onder 690 miljonairs, die gemiddeld zeiden €7900 per jaar te geven. Ook dat onderzoek was niet representatief; 41% zei de giftenaftrek te gebruiken terwijl dat toen ook maar 17% was.

Het bedrag was ook lager dan het gemiddelde van €5200 dat we in 2013 hadden gevonden onder 1300 vermogende Nederlanders die hadden gereageerd op een uitnodiging om een vragenlijst in te vullen. Ook die groep was niet representatief voor alle miljonairs in Nederland: ongeveer 40% zei de giftenaftrek te gebruiken. Sindsdien wegen we de bedragen die we aantreffen in de respons op vragenlijsten onder vermogende Nederlanders naar het gebruik van de giftenaftrek. Zo kwamen we voor de jaren 2013-2015 uit op bedragen van iets boven de €2000 per jaar (zie tabel 2.9 op pagina 21 van dit rapport).

Maar ook deze bedragen waren dus nog een overschatting, nu we door samenwerking met het CBS een schatting hebben gemaakt op basis van een aselecte steekproef. En daarin is het totaalbedrag aan giften niet hoger dan €1274. Dat is 0,98% van het gemiddelde jaarinkomen van €129.370 van miljonairs. Daarmee geven miljonairs meer dan het gemiddelde huishouden, dat gemiddeld 0,38% van de consumptie besteedt aan giften.

Maar de 0,98% van het inkomen van miljonairs is weer minder dan de 1,16% van het inkomen die de 10% armste huishoudens in Nederland per jaar geven.


Het gebruik van de giftenaftrek neemt af

Er was meer schokkend nieuws: de cijfers over het gebruik van de giftenaftrek laten een duidelijke daling zien, met name onder de meest vermogende Nederlanders.

Wat is hier aan de hand? Misschien zijn miljonairs bezig met andere dingen dan met goede doelen. Of misschien geven miljonairs wel degelijk, maar steeds vaker via een eigen bedrijf, via een familiefonds of een fonds op naam. Het is onwaarschijnlijk dat die trends al sinds 2011 de afname van het gebruik van de giftenaftrek kunnen verklaren. Ook giften aan fondsen op naam en aan familiefondsen zijn aftrekbaar. Het zou vreemd zijn als miljonairs van die mogelijkheden geen gebruik maken. We hebben vorig jaar geconstateerd (zie pagina’s 42-43 van dit rapport) dat een steeds groter deel van de huishoudens in Nederland de giftenaftrek niet gebruikt omdat zij de drempel niet halen. Van de vermogende huishoudens die Geven in Nederland-vragenlijsten hebben ingevuld en de giftenaftrek niet gebruiken zegt twee derde de drempel niet te halen. Het lijkt er dus op dat het geefgedrag van miljonairs de laatste jaren toch echt is gedaald.

Leave a comment

Filed under charitable organizations, data, household giving, methodology, Netherlands, publications, research, taxes, wealth

Better Academic Research Writing

If you’re struggling with writing your dissertation, bachelor or master thesis, or if you know students who are, then my blog Better Academic Research Writing may help: betteracademicwriting.wordpress.com/. You’ll find answers to questions like:

The blog is a guide for academic writing. It provides suggestions for students writing their bachelor, master or PhD thesis. It’s a free and open educational resource. It’s cc by 4.0 – free to download and share. A pdf is here: https://betteracademicwriting.wordpress.com/2022/04/08/guide-in-pdf/

Need help on other issues? See the FAQ here: https://betteracademicwriting.wordpress.com/2019/03/28/the-journey-begins/

Leave a comment

Filed under open science, research, science, teaching, writing

Vrijwilligers blijven langer gezond

Het lijkt te mooi om waar te zijn: help anderen met vrijwilligerswerk, en je wordt er zelf ook beter van. Het houdt je gezond en je leeft uiteindelijk langer. Zou het zo eenvoudig zijn? Er zijn steeds meer onderzoeken die suggereren dat er een kern van waarheid schuilt in het idee dat vrijwilligerswerk mensen langer gezond houdt. Veel van deze studies gebruiken alleen kleine steekproeven. Ook hebben veel eerdere onderzoeken het effect van vrijwilligerswerk op de gezondheid overschat omdat ze gebruik maken van gegevens op één moment in de tijd. Daardoor is het om moeilijk rekening te houden met het feit dat een minder goede gezondheid een barriere vormt om vrijwilligerswerk te kunnen doen.

Mega-analyse. Een nieuwe studie heeft deze problemen opgelost met een mega-analyse van gegevens van bijna een miljoen waarnemingen in zes longitudinale panelstudies in Europa, die 22 landen bestrijken over een periode van maximaal 33 jaar. Arjen de Wit van het Center for Philanthropic Studies van de afdeling Sociologie van de VU Amsterdam is de hoofdauteur van deze studie. Heng Qu van de Bush School of Governance van Texas A&M University en René Bekkers van de VU Amsterdam zijn co-auteurs.

Vrijwilligers zijn gezonder dan niet-vrijwilligers. De resultaten van de studie wijzen op een bescheiden maar duidelijk gezondheidsvoordeel voor vrijwilligers. Vrijwilligers in bijna alle landen van Europa rapporteren een betere gezondheid dan niet-vrijwilligers. Gemiddeld scoren vrijwilligers 8 punten hoger dan niet-vrijwilligers op een schaal van subjectieve gezondheid van 0 tot 100. Uit de longitudinale analyses blijkt dat het voordeel vooral te danken is aan zelfselectie: mensen met een betere gezondheid zijn eerder geneigd om vrijwilligerswerk te gaan doen en daarmee door te gaan.

Vrijwilligerswerk draagt bij aan de gezondheid. Bovendien is er een klein maar consistent positief longitudinaal verband tussen veranderingen in vrijwilligerswerk en de gezondheid. Europeanen die met vrijwilligerswerk beginnen, blijven langer gezond dan degenen die niet actief worden. Het verschil bedraagt ongeveer 0,5 punt. Onder vrijwilligers die doorgaan met vrijwilligerswerk, blijft de gezondheid beter dan onder degenen die stoppen. Het verschil is ongeveer 0,6 punten.

Het gezondheidsvoordeel voor vrijwilligers is het grootst voor mensen met een slechtere gezondheid, zo blijkt uit geavanceerde analyses. Het verschil onder degenen met de laagste gezondheidsscores is 0,88 punten; onder degenen met de hoogste gezondheidsscores is het 0,45.

Het gezondheidsvoordeel voor vrijwilligers neemt toe met de leeftijd. Het longitudinale verband tussen vrijwilligerswerk en gezondheid is veel sterker voor ouderen. Voor respondenten in hun tiener-, twintiger-, dertiger-, veertiger- en vijftiger-jaren vindt het onderzoek geen gezondheidsvoordeel voor vrijwilligerswerk. Voor respondenten van 60 jaar en ouder zijn veranderingen in vrijwilligerswerk wel significant gerelateerd aan veranderingen in de zelf geschatte gezondheid. Voor respondenten van 80 jaar en ouder is het gezondheidsvoordeel van vrijwilligers zelfs 1,9 punten.

Gezond ouder worden. Hoewel het onderzoek geen causaal effect van vrijwilligerswerk op de gezondheid aantoont, bieden de resultaten wel ondersteuning voor overheidsinterventies die vrijwilligerswerk bevorderen als een manier om gezond oud te worden. Zelfs met kleine gezondheidsvoordelen van jaar tot jaar kan vrijwilligerswerk kwetsbare groepen beschermen tegen gezondheidsachteruitgang op oudere leeftijd.

Het onderzoek is gepubliceerd in European Journal of Aging en is gratis toegankelijk.

De Wit, A., Qu, H. & Bekkers, R. (2022). The health advantage of volunteering is larger for older and less healthy volunteers in Europe: a mega-analysis. European Journal of Aging. https://doi.org/10.1007/s10433-022-00691-5

Leave a comment

Filed under health, regression analysis, survey research, volunteering

Health benefits of volunteering

The proposal seems too good to be true: give time to help others, and benefit yourself too. Be healthier and eventually live longer. Would it be so simple? Studies actually suggest there may be some truth in the idea that helping others through volunteer work helps people stay healthy. Yet many of these studies use small samples. Also most previous research is likely to have overestimated the effect of volunteering on health by using cross-sectional designs. They make it difficult to take into account that health is also a factor that enables people to volunteer in the first place.

Mega-analysis. A new study has taken care of these problems in a mega-analysis of data from almost one million observations in six longitudinal panel studies in Europe, covering 22 countries over a maximum period of 33 years. Arjen de Wit from the Center for Philanthropic Studies at the Department of Sociology of VU Amsterdam is the lead-author of this study. Heng Qu from the Bush School of Governance at Texas A&M University and René Bekkers from VU Amsterdam are co-authors.

Volunteers are healthier than non-volunteers. The results of the study indicate a modest but clear health advantage for volunteers. Volunteers in almost all countries in Europe report better health than non-volunteers. On average, volunteers score 8 points higher than non-volunteers on a scale of subjective health from 0 to 100. The longitudinal analyses show that the advantage is due mostly to self-selection: individuals in better health are more likely to start and continue volunteering.

Over time, volunteering contributes to health. In addition, there is a small but consistently positive longitudinal association between changes in volunteering and self-reported health within individuals. Europeans who start volunteering stay in better health than those who remain in active. The difference is about 0.5 points. Among volunteers who continue volunteering, health remains better than among those who quit. The difference is about 0.6 points.

start_quit

The health advantage of volunteers is largest for those in worse health. Quantile regressions show that the longitudinal association is much larger for respondents in worse health. The difference among those with the lowest health scores is 0.88 points; among those with the highest health scores it is 0.45.

quantile

The health advantage of volunteers increases with age. The longitudinal association between volunteering and health is much stronger for the elderly. For respondents in their tens, twenties, thirties, forties, and fifties the study finds no health advantage for volunteers. Only for respondents 60 years and older, within-person changes in volunteering are significantly related to changes in self-rated health. For respondents 80 years and older, the health advantage of volunteers is 1.9 points.

byage

Healthy aging. Though the study does not establish a causal effect of volunteering on health, the results do provide support for government interventions that promote volunteering as a route to healthy ageing in the long run. Even with small health advantages from year to year, volunteering may protect vulnerable groups from health decline in old age.

Publication history. The study has been in the making for more than ten years. The authors and their research assistants invested greatly in the harmonization of data from different surveys, developing methods for analysis of harmonized data, and in extensive analyses. The analysis plan for the current study was pre-registered. The pre-analysis plan, the code for the analyses, the paper and supplementary materials are publicly available at the OSF page. The study was published under an open access (CC BY 4.0) license in the European Journal of Ageing.

We invite you to use the code to conduct new analyses on volunteering and health. Also we’d appreciate your thoughts and suggestions for future research!

De Wit, A., Qu, H. & Bekkers, R. (2022). The health advantage of volunteering is larger for older and less healthy volunteers in Europe: a mega-analysis. European Journal of Aging. https://doi.org/10.1007/s10433-022-00691-5

Leave a comment

Filed under data, Netherlands, open science, regression analysis, statistical analysis, survey research, volunteering

Insights from a journey in assessment: From a written exam on campus to a digital exam at home

Here’s a set of insights on remote assessment in higher education, from another COVID-19 year filled with remote teaching, hybrid online / in person classes and online proctoring. I’m focusing on remote assessment – exams students can take from home. My suggestions: first, assess students through tasks that engage them in theory and skills application, analysis of evidence, and creation of knowledge; second, give students sufficient time to complete these tasks. Ask them to apply, analyze, evaluate, and create. These are activities from the higher ranks of Bloom’s taxonomy of teaching, learning and assessment.

The insights come from two classes I taught in the late Fall of 2020 and 2021 (November – December), one in the two year research master Societal Resilience (~15 students) and one in the master program Sociology (~50 students). I’ve learned a lot from teaching these classes about assessment.
In the design of courses in higher education, assessment is the third element of constructive alignment. Assessment forms the triangle with course objectives and learning activities, which should all be aligned. The pandemic produced an opportunity to transform the assessment in the exams, and take them to a higher level.


From a written exam on campus to a digital exam at home

Before the pandemic, the courses assessed student performance on campus through written exams. The written exams on campus asked students to reproduce some of the theories discussed in the course. Students were not allowed to take textbooks and readings with them to the exam. These were easy questions to grade, and they could be assessed with multiple choice questions.

Because campus rooms could not be used for exams due to COVID-19 restrictions, students had to take the exams from home. Because there was no easy way to prevent students using the readings and other materials to improve performance on this type of questions I allowed them to use these materials. Also after graduation there will be few situations in which students cannot consult sources and have to know things by heart. In their future jobs, being able to apply and analyze ideas and evaluate evidence is more important for graduates than simply remembering facts and figures. Assessing students on these higher order skills is a better practice in higher education.


Exam questions engaging students in higher learning activities

In both courses, assessment focused on the application of theories to new cases and data. The exam questions test students’ performance in understanding ideas and concepts, applying them to new materials, analyzing connections between ideas and materials, and evaluating arguments based on theories and results. The questions do not test recall of facts and concepts. To answer the questions, students are allowed to consult the readings, slides, and other materials available on the web. The questions presuppose that students understand the theories and hypotheses discussed in the readings and in course meetings.

The prototypical question starts with a piece of new material: a quote, a cartoon, a news item, a table or a figure from an article or a piece of output not discussed in class. Students interpret the new material and explain it from theories and concepts covered in readings and class meetings.

Here’s an example of this type of question.

Income and wealth inequality are not the same. Explain the difference between the two concepts. Describe the degree of income and wealth inequality in the Netherlands, using statistics from the World Inequality Database https://wid.world/country/netherlands/

Questions of a second type work in the reverse order, and ask students to first draw connections between theories and hypotheses, and then invite students to present examples from new materials collect themselves.

Here’s an example of this type of question.

  1. State two hypotheses about the difference in the level of civic voluntarism between countries with a relatively high and a relatively low proportion of university graduates. Base one hypothesis on the absolute education model, and the other on the relative education model.
  2. Compare the level of civic voluntarism for three countries. Which hypothesis that you stated in question a is supported by the differences in civic voluntarism between the three countries?

Questions of a third type ask students to apply, analyze, and evaluate existing theoretical explanations, or produce a new explanation. These questions presuppose that students understand the logic of theory building and are able to find flaws in explanations and repair them.

Select two perspectives on resilience that we have discussed in the course, and contrast them in the case of Dutch approach to COVID-19. What kind of research questions would be asked in the two perspectives? Which aspects emerge as important from the two perspectives? Are they complementary or mutually exclusive? Explain which perspective is most fruitful in your view.


Investment required

The creation and grading of exams with the type of questions above takes time – certainly more time than multiple choice exams. For each exam, I’ve spent about one 8 hour work day to create them, and about 20 minutes per candidate grading. Taken together, that is a lot of time, and you may not be able or willing to invest that much. It will not be feasible for courses with larger numbers of students. It’s hard to automate grading exams with the type of questions that require higher order learning. On the other hand, I’ve found that these exams are more fun to create, and that they’re also more fun to grade. Students found the questions more challenging, but also more interesting.


Aligning assessment with learning objectives

These types of exam questions do not test the ability of students to memorize and reproduce concepts and theories. Instead, they test students’ understanding, and their ability to apply and analyze ideas, to evaluate ideas and evidence, and create new predictions. The higher order exam questions were aligned with the objectives of both classes.

Foundations of Societal Resilience is a pure theory class, in which students encounter different disciplinary perspectives on resilience. The course manual is here: https://renebekkers.files.wordpress.com/2022/02/2021-course-manual-foundations-of-societal-resilience.pdf. Students read a lot in the course, learn concepts, distinguish disciplinary approaches, and reason from theories. The course does not have an empirical component. I’ve posted the exam for this course here: https://renebekkers.files.wordpress.com/2022/02/2021-fsr-exam-assignments.pdf.

Inequality and Conflict in Societal Participation is a combined theory and data analysis class, in which students learn about societal value change over time, and learn to analyze survey data from the World Values Survey. The course manual is here: https://renebekkers.files.wordpress.com/2022/02/course-manual-social-inequality-2021-2022.pdf. Students read quite a bit, and learn social science theories. In addition, students write a short paper about an empirical data analysis. I’ve posted the exam for this course here: https://renebekkers.files.wordpress.com/2022/02/2021-icsp-exam-assignments.pdf.


Giving students enough time to answer the questions

The challenging questions created dissatisfaction among students under time pressure. To make it infeasible for students to collaborate with each other while not under surveillance I created time pressure by allowing a limited amount of time for each question. The time available was scarcely enough to answer all questions. There was no opportunity to meaningfully exchange ideas or insights without losing time writing. I’ve tried this twice, with very clear results: don’t do this. Students dislike it. You’re testing a set of skills that are very different from the skills in the learning objectives of the course. The ability to quickly read abstracts, assess and classify ideas embedded in new texts in English is related to working memory and processing speed, and gives native speakers of English an advantage. Also students who are more able to handle stress, keep calm, and set priorities do better under time pressure, while improving these abilities are not learning objectives of the course. With time pressure, students cannot demonstrate what they have learned.

In course evaluation surveys, students suggested giving an entire day to complete the exam. This would allow students to give more detailed and thought-through answers, that allow students to better demonstrate their abilities to engage in higher learning. On the other hand, it would create more opportunities to share ideas and texts before submitting them, not only with each other, but also with other helpers. Also it would make the exam more similar to one of the other assignments in the course, in which students wrote an essay about a topic of their own choice. Therefore in the resit of the exams I reduced the number of questions, so students had more time to answer each question. The time limit of a regular exam (2 hours and 15 minutes) served to reduce the opportunity for collaboration.


Context and technicalities

These insights come from teaching courses at the master’s level. I understand the argument that bachelor (BA) students should learn elementary concepts and facts first before they can apply theories, analyze cases and create new insights. For master (MA) level students the higher level tasks are certainly more appropriate. They are more challenging and more fun for students to complete for BA and MA students alike.

At VU Amsterdam we have a digital platform for exams (TestVision). It’s a bit tedious in creating exams and test questions, but once you know your way around the system, it works quite efficiently. The platform is not connected with the electronic learning environment we use at VU (Canvas). The university contracted online proctoring services (Proctorio) to facilitate students taking exams off-campus.

I first tried exams through the online learning environment because of two advantages: students know their way around it because they use it on a daily basis, and the platform allows for a plagiarism check on the materials students submit. The disadvantage of the online learning environment is that it cannot handle online proctoring services that the university uses (Proctorio).

To help students prepare the exam, I provided the questions of last year’s exam, which had a similar design. In a meeting with students before one of the exams I explained the form of the exam, and asked them for suggestions to prevent plagiarism and collaboration. The student group appreciated being consulted, and suggested a code of honor. All students completed it. Indeed the plagiarism check flagged no one’s answers. The plagiarism check did produce a large number of false positives. Students who had copied the exam questions in the materials they submitted were flagged.

On Canvas, it is a good idea to set the grading policy to ‘manual’, so that students do not get updates for each question you grade. This will stress them out and can generate a flurry of emails before you’re even done grading.


Strategies I’ve used to let students work creatively with different versions of the same question

  1. Let students use a random number generator to select a single case from a set of cases.

Example: in a question about political parties, students described characteristics of the winning political party in one out of four different recent elections.

Another example: based on data from the World Values Survey and the European Values Survey, I created a table with scores for 129 different countries in three groups of 43 countries each. The random number generator determined which three countries the students compared.

2. Ask students to discuss their own context in the answer to a question.

Example: in a question about political socialization, students described the difference between the level of education of their parents and their own level of education.

Another example: on the website of Statistics Netherlands, students looked up statistics of the neighborhood in which they are currently living, and compared them with another (fixed) neighborhood.

Leave a comment

Filed under teaching

New PhD project: Learning to Donate

In the new academic year, we’re starting a PhD project at the department of Donor Studies at Sanquin Research and the Center for Philanthropic Studies at Vrije Universiteit (VU). Meanwhile, we’ve completed interviews and selected Alexandra Ciauşescu for the position.

Social relations (e.g., with family members, teachers or friends) are critical to the onset and maintenance of different types of prosocial behaviour, e.g., charitable giving or volunteer work. Solicitations by others, observation of prosocial behaviour in others, awareness of need, and norms about prosocial behaviour all travel through social relations and may result in social ‘contagion’. Parental role-modeling for example and conversations about giving behaviour are strongly related to adolescents’ giving and volunteering. Evidence about whether and how social relations shape blood donation behavior, however, is scarce. Previous research shows that one of the most effective recruitment strategies for blood donors is the donor-recruits-donor strategy. But we do not know which specific relationships are producing this effect: romantic relations, parent-child relations, friends and/or colleagues? In addition, the specific mechanisms at work, such as social learning, awareness of need, and value transmission, remain elusive. Knowledge about such mechanisms increase possibilities for more effective recruitment of new donors.

With this project we address three objectives:
•    Examine learning and inter/intragenerational transmission in blood donor behavior 
•    Develop and test educational material about donation for children and adolescents (e.g., Science exhibition)
•    Develop and test interventions for blood donor recruitment by using social information (i.e., information about donor status and donation behaviour of others)

The project duration is four years and the PhD candidate will participate in the VU Graduate School for Social Sciences as well as in the PhD network at Sanquin. Supervisors are dr. Eva-Maria Merz (VU/Sanquin) and prof. dr. René Bekkers (VU). The research proposal is here.

Organizations
This project is a collaboration between the department of Donor Studies at Sanquin Research and the Center for Philanthropic Studies at VU, both located in Amsterdam. The department of Donor Studies is an internationally recognized center for high-quality (blood) donor research. Staff at the department consists of about 15 researchers, including 5 PhD students. The Center for Philanthropic Studies at VU conducts research and educates professionals in all areas of the Dutch philanthropic sector. Since 1995, the Center has been the leading unit for research on philanthropy in the Netherlands and in Europe. 

Leave a comment

Filed under blood donation, Center for Philanthropic Studies, data, experiments, research, science

The New Frontier: Research Quality

In the ideal research publication infrastructure, the value or a piece of research is not about the prestige of the authors, but it is determined by the validity of theories, data and methods. Research quality should be the only criterion that determines the prestige of research. To get there, we need to reverse course. We switch seats at the table: journals will bid for the best articles that researchers produce. We must flip academic publishing.

Scientific knowledge is a public good. It is not about the advancement of careers of individuals. Science is about getting it right, not about being right. The point of doing research is the development of knowledge. We should seek to discover new phenomena that throw a new light on old intuitions. We should investigate anomalies that call into question established ideas. We should strive to obtain findings that discredit the current consensus. The pursuit of black swans is difficult when their territory is off limits, policed by authorities in the field.

In the current research publication infrastructure, authors compete for space in the most prestigious journals. Journals reward novelty – coining an attractive label for common intuitions, inventing a new method, or claiming a controversial finding. Authors support their predictions by appeals to authority, defined by prestige. Authors selectively cite previous research that supports their predictions, omitting proper attention for previous research that indicates otherwise. Authors taut the uniqueness of their findings, ignoring and withholding relevant previous research and crucial limitations of their own research design. The fate of research products is in the hands of anonymous reviewers, who impose their arbitrary and idiosyncratic preferences in a secret exchange mediated by journal editors.

The current incentives for research publications are not aligned with the public good character of science because they reward the prestige of authors instead of the quality of their work. Research careers in academia depend on previous success, defined as a high number of publications in journals that previously published successful people, and acquisition of large grants in the past.


The Complete Publication Model

In the ideal research infrastructure, we follow a Complete Publication Model. All research gets published – nothing is withheld, every piece of research is publicly available.

The Complete Publication Model

1. Authors post their work on preprint servers that charge no fees for access and impose no barriers to publication. No research goes to waste; publication bias no longer exists.

2. In the flipped publication industry, the most prestigious journals publish the best research. Bots crawl the complete body of all publicly available research, and direct research work that fits journal criteria to algorithms that disregard author rank and affiliation and automatically assign quality assessments, based on standardized indicators. This also happens out in the open. No longer will journals burden researchers with the effort to advertise their research to editors and reviewers in secret communication. Instead, the manuscripts assessed are public, the criteria used for the assessment are public, the algorithms that produce the assessment are public, and the assessments are public as well.

3. Journals compete with each other to obtain the right to publish the best research. Researchers receive offers from journals, which authors can choose to ignore if they are satisfied already with the quality assessment of their work.

4. Once researchers accept the invitation to submit their work to a journal, the peer review procedure begins. Authors, editors and reviewers communicate openly, and reviews are published, so that everyone can check their quality.

5. Researchers improve their work in response to the reviews.

6. After each revision, the paper goes through the automatic quality assessment again, and receives a new rating with a time stamp.

7. Once the manuscript under review achieves an acceptable rating, it is published in the journal.


Incentivizing research quality

The automatic quality assessment system incentivizes the journals to actually provide a service: the improvement of research through peer review. After all, the research already has a quality certificate. The prestige of researchers is visible already from the stars assigned to their research products, also when it is not invited for review by a journal. This system shows the added value of peer review and the journals that organize it. Some journals may do a very good job improving the quality of research; others may not improve it at all, or even reduce it. If researchers are satisfied with the quality assessment, they can choose to ignore invitations for peer review at journals. The extent to which the quality assessment improves is the added value of the journal. Because all quality assessments are time stamped, journals can be ranked both in terms of the eventual quality  of the research they publish as well as in terms of the improvement. These rankings provide researchers with a choice of journals. The improvement achieved through peer review will become the hallmark of quality for a journal.

It is now up to the scientific community to formulate quality standards for publications and publish them.

6 Comments

Filed under academic journals, open science, publications, regulation, research, science

Real research, or just looking things up?

“I’ve done some research….” You’re planning a city trip. Where to stay? Which places to visit? Opening times? How to reach the places you want to go? The answers to these questions are not the result of real research. You just looked things up.

“Research shows that…” You’re wondering what the latest insights are about a topic that you’re interested in. What does the current research tell you? The answer to this question is not the result of your research. You just looked things up.

“The rules are…” You’re planning a study and need to go through an ethics review. What rules and guidelines do you need to follow? What forms do you need to complete? The answers to these questions are not the result of your research. You just looked things up.

“The research models show….” You’re considering to prolong the lockdown, and need evidence for your decision. How will ending the lockdown reduce the number of COVID-19 infections? How will the number of ICU patients grow when the lockdown ends? You ask a mathematician to produce a set of predictions and compare the results of the models with preferred outcomes. That comparison is not research. You just looked things up.

Real research is the collection and analysis of data, to evaluate a claim.


“Research into the case showed…” You’re responsible for the punishment of offenders, criminal prosecution, investigating suggestions of misconduct, or you’re a journalist looking for news. What evidence is there that somebody did something terrible? You compare reports by victims to declarations of alleged or possible offenders. That’s real research – but it does not produce new knowledge; you produce an assessment of legality or newsworthiness.

“And the winner is…” You’re responsible for the payment of prizes, recognition of awards, permission to enter a country, or the allocation of rights. You need to know whether the person in front of you is truly the lottery ticket holder, prize winner, national citizen or a person entitled to drive a car. You ascertain the identity of the person or the authenticity of documents through comparison of some biometric or other physical quality of the person or document with entries from a database. That’s real research – but it does not produce new knowledge; you produce an assessment of authenticity.

Real research is the collection and analysis of data, to evaluate a claim to truth, and generate new knowledge.


Studies of how people plan city trips, a meta-analysis of previous research, a comparison of ethics review practices across countries or research disciplines, mapping social contacts and networks through which COVID-19 infections spread, or an evaluation of predictive models for the prevalence of COVID-19, those are examples of real research.

Studies of how judges make judgments, how people infer certain traits of perpetrators from reports misconduct, how journalists produce news and how people consume it, those are examples of real research. Studies of why people participate in lotteries, what design of award schemes is optimal to generate interpersonal jealousy and competition, how immigration officers try to detect false documents, or what types of checks and authenticity marks work best to reduce fraud are also examples of real research.

This very text is not real research – it is a set of descriptions of analytical categories. It could become research on research if I’d present a table with counts of documents – say, papers published in academic journals – that present themselves as research in each of the categories distinguished above, and next we analyze the properties of these documents: do they deduce claims from theories, contain mathematical formulas, conduct statistical tests, state limitations on generality?

I hope the categories are useful the next time you hear the word “research” and wonder: “Is it real research, or just looking things up?”

Leave a comment

Filed under research, science