Category Archives: statistical analysis

Uncertain Future for Giving in the Netherlands Panel Survey

By Barbara Gouwenberg and René Bekkers

At the Center for Philanthropic Studies we have been working hard to secure funding for three rounds of funding for the Giving in the Netherlands Study, including the Giving in the Netherlands Panel Survey for the years 2020-2026. During the previous round of the research, the ministry of Justice and Security has said that it would no longer fund the study on its own, because the research is important not only for the government but also for the philanthropic sector. The national government no longer sees itself as the sole funder of the research.

The ministry does think the research is important and is prepared to commit funding for the research in the form of a 1:1 matching subsidy to contributions received by VU Amsterdam from other funders. To strengthen the societal relevance and commitment for the Giving in the Netherlands study the Center has engaged in a dialogue with relevant stakeholders, including the council of foundations, the association of fundraising organizations, and several endowed foundations and fundraising charities in the Netherlands. The goal of these talks was to get science and practice closer together. From these talks we have gained three important general insights:

  • The Giving in the Netherlands study contributes to the visibility of philanthropy in the Netherlands. This is important for the legitimacy of an autonomous and growing sector.
  • It is important to engage in a conversation with relevant stakeholders before the fieldwork for a next round starts, in order to align the research more strongly with practice.
  • After the analyses have been completed, communication with relevant stakeholders about the results should be improved. Stakeholders desire more conversations about the application of insights from the research in practice.

The center includes these issues in the plans for the upcoming three editions. VU Amsterdam has been engaged in conversations with branch organizations and individual foundations in the philanthropic sector for a long time, in order to build a sustainable financial model for the future of the research. However, at the moment we do not have the funds together to continue the research. That is why we did not collect data for the 2018 wave of the Giving in the Netherlands Panel Survey. As a result, we will not publish estimates for the size and composition of philanthropy in the Netherlands in spring 2019. We do hope that after this gap year we can restart the research next year, with a publication of new estimates in 2020.

Your ideas and support are very welcome at r.bekkers@vu.nl.

Advertisements

Leave a comment

Filed under Center for Philanthropic Studies, charitable organizations, contract research, data, experiments, foundations, fundraising, household giving, methodology, Netherlands, philanthropy, policy evaluation, statistical analysis, survey research

Onderzoek Geven in Nederland in gevaar

Door Barbara Gouwenberg – uit de nieuwsbrief van de werkgroep Filantropische Studies aan de VU (december 2018)

Het Centrum voor Filantropische Studies werkt momenteel met man en macht om de financiering voor het onderzoek Geven in Nederland voor de komende 6 jaar (3 edities) veilig te stellen. Het Ministerie van Justitie en Veiligheid (J&V) heeft bij de opzet van Geven in Nederland 2017 medio 2015 te kennen gegeven dat het onderzoek niet langer alleen door de overheid zal worden gefinancierd, met als belangrijkste argumentatie dat het onderzoek van belang is voor overheid én sector filantropie. De overheid ziet zichzelf niet langer als enige verantwoordelijke voor de financiering van het onderzoek.

Het Ministerie van J&V wil zich wel voor een langere tijd structureel verbinden aan Geven in Nederland en geeft 1:1 matching voor financiële bijdragen die de VU vanuit de sector ontvangt.

Om de maatschappelijke relevantie van – en commitment voor – het onderzoek Geven in Nederland te versterken heeft de VU de afgelopen maanden de dialoog opgezocht met diverse relevante doelgroepen. Doel: wetenschap en praktijk dichter bij elkaar brengen.

Deze rondgang heeft ons – naast specifieke inzichten – drie belangrijke algemene inzichten opgeleverd; te weten:

  • ‘Geven in Nederland’ draagt bij aan de zichtbaarheid van maatschappelijk initiatief in Nederland. Belangrijk ter legitimatie van een zelfstandige en snel groeiende sector.
  • Communicatie met relevante doelgroepen vóór de start van het onderzoek dient verbeterd te worden met als doel om inhoudelijk beter aansluiting te vinden bij praktijk en beleid.
  • Communicatie over onderzoeksresultaten naar relevante doelgroepen dient verbeterd te worden. Het gaat dan om de praktische toepasbaarheid van het onderzoek, de vertaling van de onderzoeksresultaten naar de praktijk.

De onderzoekers nemen deze verbeterpunten mee in hun plan van aanpak voor de komende drie edities. De VU is al enige tijd in gesprek met de brancheorganisaties en individuele fondsen om tot een duurzaam financieringsmodel voor de toekomst te komen. Op dit moment is de continuering van het onderzoek echter nog niet gegarandeerd. Dat betekent dat er helaas geen Geven in Nederland 2019 komt en dus ook geen presentatie van de nieuwe onderzoeksresultaten zoals u van ons gewend bent op de Dag van de Filantropie. We spreken echter onze hoop uit dat we zeer binnenkort met een Geven in Nederland 2020 kunnen starten!

Leave a comment

Filed under Center for Philanthropic Studies, charitable organizations, contract research, data, foundations, fundraising, household giving, methodology, Netherlands, open science, philanthropy, statistical analysis, survey research, trends, VU University

Closing the Age of Competitive Science

In the prehistoric era of competitive science, researchers were like magicians: they earned a reputation for tricks that nobody could repeat and shared their secrets only with trusted disciples. In the new age of open science, researchers share by default, not only with peer reviewers and fellow researchers, but with the public at large. The transparency of open science reduces the temptation of private profit maximization and the collective inefficiency in information asymmetries inherent in competitive markets. In a seminar organized by the University Library at Vrije Universiteit Amsterdam on November 1, 2018, I discussed recent developments in open science and its implications for research careers and progress in knowledge discovery. The slides are posted here. The podcast is here.

2 Comments

Filed under academic misconduct, data, experiments, fraud, incentives, law, Netherlands, open science, statistical analysis, survey research, VU University

Multiple comparisons in a regression framework

Gordon Feld posted a comparison of results from a repeated measures ANOVA with paired samples t-tests.

Using Stata, I wondered how these results would look in a regression framework. For those of you who want to replicate this: I used the data provided by Gordon. The do-file is here. Because wordpress does not accept .do files you will have to rename the file from .docx to .do to make it work. The Stata commands are below, all in block quotes. The output is given in images. In the explanatory notes, commands are italicized, and variables are underlined.

A pdf of this post is here.

First let’s examine the data. You will have to insert your local path at which you have stored the data.

. import delimited “ANOVA_blog_data.csv”, clear

. pwcorr before_treatment after_treatment before_placebo after_placebo

These commands get us the following table of correlations:

There are some differences in mean values, from 98.8 before treatment to 105.0 after treatment. Mean values for the placebo measures are 100.8 before and 100.2 after. Across all measures, the average is 101.2035.

Let’s replicate the t-test for the treatment effect.

The increase in IQ after the treatment is 6.13144 (SE = 2.134277), which is significant in this one-sample paired t-test (p = .006). Now let’s do the t-test for the placebo conditions.

The decrease in IQ after the placebo is -.6398003 (SE = 1.978064), which is not significant (p = .7477).

The question is whether we have taken sufficient account of the nesting of the data.

We have four measures per participant: one before the treatment, one after, one before the placebo, and one after.

In other words, we have 50 participants and 200 measures.

To get the data into the nested structure, we have to reshape them.

The data are now in a wide format: one row per participant, IQ measures in different columns.

But we want a long format: 4 rows per participant, IQ in just one column.

To get this done we first assign a number to each participant.

. gen id = _n

We now have a variable id with a unique number for each of the 50 participants.
The Stata command for reshaping data requires the data to be set up in such a way that variables measuring the same construct have the same name.
We have 4 measures of IQ, so the new variables will be called iq1, iq2, iq3 and iq4.

. rename (before_treatment after_treatment before_placebo after_placebo) (iq1 iq2 iq3 iq4).

Now we can reshape the data. The command below assigns a new variable ‘mIQ’ to identify the 4 consecutive measures of IQ.

. reshape long iq, i(id) j(mIQ)

Here’s the result.

We now have 200 lines of data, each one is an observation of IQ, numbered 1 to 4 on the new variable mIQ for each participant. The variable mIQ indicates the order of the IQ measurements.

Now we identify the structure of the two experiments. The first two measures in the data are for the treatment pre- and post-measures.

. replace treatment = 1 if mIQ < 3 (100 real changes made) . replace treatment = 0 if mIQ > 2
(100 real changes made)

Observations 3 and 4 are for the placebo pre- and post-measures.

. replace placebo = 0 if mIQ < 3 (100 real changes made) . replace placebo = 1 if mIQ > 2
(100 real changes made)

. tab treatment placebo

We have 100 observations in each of the experiments.

OK, we’re ready for the regressions now. Let’s first conduct an OLS to quantify the changes within participants in the treatment and placebo conditions.

The regression shows that the treatment increased IQ by 6.13144 points, but with an SE of 3.863229 the change is not significant (p = .116). The effect estimate is correct, but the SE is too large and hence the p-value is too high as well.

. reg iq mIQ if placebo == 1


The placebo regression shows the familiar decline of .6398003, but with an SE of 3.6291, which is too high (p = .860). The SE and p-values are incorrect because OLS does not take the nested structure of the data into account.

With the xtset command we identify the nesting of the data: measures of IQ (mIQ) are nested within participants (id).

. xtset id mIQ

First we run an ’empty model’ – no predictors are included.

. xtreg iq

Here’s the result:

Two variables in the output are worth commenting on.

  1. The constant (_cons) is the average across all measures, 101.2033. This is very close to the average we have seen before.
  2. The rho is the intraclass correlation – the average correlation of the 4 IQ measures within individuals. It is .7213, which seems right.

Now let’s replicate the t-test results in a regression framework.

. xtreg iq mIQ if treatment == 1

In the output below we see the 100 observations in 50 groups (individuals). We obtain the same effect estimate of the treatment as before (6.13144) and the correct SE of 2.134277, but the p-value is too small (p = .004).

Let’s fix this. We put fixed effects on the participants by adding , fe at the end of the xtreg command:

. xtreg iq mIQ if treatment == 1, fe

We now get the accurate p-value (0.006):

Let’s run the same regression for the placebo conditions.

. xtreg iq mIQ if placebo == 1, fe


The placebo effect is the familiar -.6398003, SE = 1.978064, now with the accurate p-value of .748.

Leave a comment

Filed under data, experiments, methodology, regression, regression analysis, statistical analysis, survey research