Category Archives: Uncategorized

Hunting Game: Targeting the Big Five

Do not use the personality items included in the World Values Survey. That is the recommendation of Steven Ludeke and Erik Gahner Larsen in a recent paper published in the journal Personality and Individual Differences. The journal is owned by Elsevier so the official publication is paywalled. Still I am writing about it because the message of the paper is extremely important. Ludeke and Gahner Larsen formulate their recommendation a little more subtle: “we suggest it is thus hard to justify the use of this data in future research.”

What went wrong here? Join me in a hunting game, targeting the Big Five.

The World Values Survey (WVS) is the largest, non-commercial survey in the world. It is frequently used in social science research. The most recent edition contained a short, 10 item measure of personality characteristics (BFI-10), validated in a well-cited paper by Rammstedt and John in the Journal of Research in Personality. The inclusion of the BFI-10 enables researchers to study how the Big Five personality traits is related to political participation, happiness, education, and health, among many other things.

So what is wrong with the personality data in the WVS? Ludeke and Gahner Larsen found that the pairs of adjectives designed to measure the five personality traits Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism are not correlated as expected. To measure openness, for instance, the survey asked participants to indicate agreement with the statement “I see myself as someone who: has few artistic interests” and “I see myself as someone who: has an active imagination”. One would expect a negative relation between the responses to the two statements. However, the correlation between the two items across all countries is positive, r = .164. This correlation is not strong, but in the wrong direction. Similar discrepancies were found between items designed to measure the four other dimensions of personality.

The BFI-10 included in the WVS is this set of statements (an r indicates a reverse-scored item):

I see myself as someone who:

  • is reserved (E1r)
  • is generally trusting (A1)
  • tends to be lazy (C1r)
  • is relaxed, handles stress well (N1r)
  • has few artistic interests (O1r)
  • is outgoing, sociable (E2)
  • tends to find fault with others (A2r)
  • does a thorough job (C2)
  • gets nervous easily (N2)
  • has an active imagination (O2)

In a factor analysis of the 1o items, we would expect to find the five dimensions. However, that is not the result of an exploratory factor analysis applying the conventional criterion of an Eigen value > 1. In this analysis and all following analyses negative items are reverse scored. Including all countries, a three factor solution emerges that is very difficult to interpret. Multiple items show high loadings on multiple factors. Removing these one by one, as is usually done in inventories with large numbers of items, we are left with a two-factor solution. If a five-factor solution is forced, we obtain the following component matrix. This is a mess.

Component

1

2 3 4

5

O1 not artistic (r)

-.116

-.054 .105 -.049

.961

O2 active imagination

.687

.162 -.031 .197

-.140

C1 lazy (r)

.249

-.004 .836 -.045

.159

C2 thorough

.640

.425 .231 .078

.071

E1 reserved (r)

-.110

-.825 -.022 -.183

-.047

E2 outgoing

.781

.097 -.004 -.105 -.068
A1 trusting

.210

.722 .003 -.160 -.137
A2 fault with others (r)

-.430

.079 .614 -.259

-.051

N1 relaxed (r)

-.461 -.377 .235 .534

.144

N2 nervous

.188 .133 -.291 .770

-.112

So what is wrong with these data?

Upon closer inspection, Ludeke and Gahner Larsen found that the correlations were markedly different across countries. Bahrain is a clear outlier. The weakly positive correlation between O1 and O2r is due in part to the inclusion of data from Bahrain. Without this country, the correlation is only .135. Still positive, but not as strongly. The data for Bahrain are not only strange for openness, but also for other factors. In the table below I have computed the correlations among recoded items for the five dimensions.

Without Bahrain, the correlations are still strange, but a little less strange.

O

C E A N
With Bahrain

-.164

.238 -.207 -.036

.008

Without Bahrain

-.135

.275 -.181 -.009

.044

What is wrong with the data for Bahrain? The patterns of responses for cases from Bahrain, it turns out, are surprisingly often a series of ten exactly the same values, such as 1111111111 or 555555555555. I routinely check data from surveys for such patterns. While it is impossible to prove this, serial response patterns suggest fabrication of data. Participants and/or interviewers skipping questions may follow such patterns. Almost half of all the cases from Bahrain follow such a pattern. Other countries with a relatively high proportion of serial pattern responses are South Africa, Singapore, and China. The two countries for which the BFI-10 behaves close to what previous research has reported, the Netherlands and Germany, have a very low occurrence of serial pattern responses.

Number of serial pattern responses

%
Bahrain

598

49.83%

South Africa

250

7.08%

Singapore

108

5.48%

China

52

2.26%

Netherlands

8

0.42%

Germany

2

0.10%

Even without the data for Bahrain and the serial responses from all other countries, however, the factor structure is err…not what one would expect. Still a mess.

Component

1

2 3 4

5

O1 not artistic (r)

-.094 -.040 .086 -.031 .968
O2 active imagination

.691

.150 -.046 .158 -.130
C1 lazy (r)

.297

.023 .815 -.017 .146
C2 thorough

.637

.410 .241 .050 .088
E1 reserved (r)

-.098

-.828 -.033 -.158 -.058

E2 outgoing

.771 .070 -.001 -.140

-.052

A1 trusting

.192 .710 .022 -.190

-.133

A2 fault with others (r)

-.405 .080 .628 -.230

-.048

N1 relaxed (r)

-.421 -.352 .218 .592

.123

N2 nervous

.192 .133 -.315 .750

-.104

Only for Germany and the Netherlands the factor structure is somewhat in line with previous research. Here is the solution for the two countries combined. In both countries, the two statements for agreeableness do not correlate as expected. Also the second statement for conscientiousness (thorough) has a cross-loading with one of the agreeableness items (trusting).

Component

1

2 3 4

5

O1 not artistic (r)

-.047

-.056 .842 .120

-.089

O2 active imagination

.208

.050 .729 -.140 .173
C1 lazy (r)

.061

-.083 -.040 .865 -.087
C2 thorough

-.064

.053 .057 .627 .440
E1 reserved (r)

.715

-.113 .130 .032 -.219

E2 outgoing

.732 -.166 .126 .166

.210

A1 trusting

-.008 -.100 .042 .049

.853

A2 fault with others (r)

-.657 -.272 .090 .177

-.001

N1 relaxed (r)

.012 .804 -.002 .116

-.259

N2 nervous

-.052 .835 -.006 -.160

.117

This leaves us with three possibilities.

One possibility was raised by Christopher Soto on Twitter: acquiescence bias could be driving the results. In a study using data from another multi-country survey in the International Social Survey Program (ISSP), Rammstedt, Kemper & Borg subtracted each respondent’s mean response across all BFI-10 items from his or her score on each single item. Doing this, however, does not clear the sky. Looking again at the correlations for the pairs of items measuring the same constructs, we see that they are not ‘better’ in the second row. In contrast, they are less positive.

O

C E A

N

Unadjusted

-.122

.286 -.166 .001

.053

Attenuated

-.310

.078 -.235 -.107 .049

Also the factor structure of the attenuated scores is not anything like the ‘regular’ five-factor structure. Still a mess.

Component

1

2 3 4

5

O1a

-.192

-.025 .096 -.117

-.957

O2a

.509

.190 -.319 .034

.269

C1a

-.133

.469 .617 -.460

.174

C2a

.351

.681 -.005 .050

.071

E1a

-.043

-.846 .080 -.250

.017

E2a

.823

.029 .034 .045

.114

A1a

.086

.285 .026 .821

.148

A2a

-.497

-.246 .555 .274

.067

N1a

-.598

-.345 -.223 -.345

-.047

N2a

-.123

.043 -.854 -.031

.178

The second possibility is that things went wrong in the translation of the questionnaire. The same adjectives or statements may mean different things in different countries or languages, which makes them useless as operationalizations of the same underlying construct. It will require a detailed study of the translations to see if anything went wrong. The questionnaires are available at the World Values Survey website. The Dutch questionnaire is good. I looked at a few other languages. The Spanish questionnaire for Ecuador also seems right. “Me veo como alguien que…… es confiable” is quite close to “I see myself as someone who is… generally trusting”. My Spanish is not very good though. Rene Gempp wrote on Twitter that the BFI-10 is a Likert-type scale, but the Spanish translation asks about the frequency, and one of the options, “para nada frecuentemente” is *very* confusing in Spanish.

I am not sure about your fluency in Kinyarwanda, the language spoken in Rwanda, but the backtranslation of the questionnaire in English does not give me much confidence. Apparently, “…wizera muri rusange” is the translation of “is generally trusting”. The backtranslation is “…believe in congregation”.

rwanda_back

The third possibility is that personality structure may indeed be different in different countries. This would be the most problematic one.

Data from the 2010 AmericasBarometer Study, conducted by the Latin American Public Opinion Project (LAPOP) support this interpretation. The survey included a different short form of the Big Five, the TIPI, developed by Gosling, Rentfrow, and SwannA recent study by Weinschenk published in Social Science Quarterly shows that personality scores based on the TIPI are hardly related to turnout in elections in the Americas. This result may be logical in countries where voting is mandatory, such as Brazil. But the more disconcerting methodological problem is that the Big Five are not reliably measured with pairs of statements in most of the countries included in the survey. Here are the correlations between the pairs of items for each of the five dimensions, taken from the supplementary online materials of the Weinschenk paper.

Big5_rel_LAPOP

The graphs show that the TIPI items only work well in the US and Canada – the two ‘WEIRD’ countries in the study. In Brazil, to take one example, the correlations are <.10 for extraversion, agreeableness and conscientiousness, and lower than .25 for emotional stability and openness.

Back to the WVS case, which raises important questions about the peer review process. Two journal articles based on the WVS (here and here) were able to pass peer review because neither the reviewers nor the editors asked questions about the reliability of the items being used. Neither did the authors check, apparently. Obviously, researchers should check the reliability of measures they use in an analysis. In case authors fail to check this, reviewers and editors should ask. Weinschenk reported the low correlations in the online supplementary materials, but did not report reliability coefficients in the paper.

The good thing is that because the WVS is in the public domain, these problems came to light relatively quickly. Of course, they could have been avoided if the WVS had scrutinized the reliability of the measure before putting the data online, if the authors of the papers using the data had checked the reliability of the items or if the reviewers and editors had asked the right questions. Another good thing is that the people at the WVS (volunteers?) at the WVS twitter account have been frank in tweeting about the problems found in the data.

Summing up:

  1. We still do not know why the BFI-10 measure of the Big Five personality does not perform as in previous research.
  2. It is probably not due to acquiescence bias. Translations may be problematic for some countries.
  3. Do not use the WVS BFI-10 data from countries other than Germany and the Netherlands.
  4. Treat the WVS data from Bahrain and with great caution, and to be on the safe side, just exclude it from your analyses.
  5. The reliability of short Big Five measures is very low in non-WEIRD countries.

The code for the analyses reported in this blog is posted at the Open Science Framework.

Update 22 March 2017. The factor loadings in the table with the results of the analysis of attenuated scores has been updated. The table displayed previously was based on a division of the original scores by the total agreement scores. Rammstedt et al. subtracted the original scores from the total agreement scores. The results of the new analysis are close to the previous one and still confusing. The code on the OSF has been updated. Also a clarification was added that the negative items used in the factor analyses were all recoded such that they scored positively (HT to Christopher Soto).

2 Comments

Filed under Uncategorized

Five Reasons Why Social Science is So Hard 

1. No Laws 

All we have is probabilities. 

2. All Experts 

The knowledge we have is continuously contested. The objects of study think they know why they do what they do. 

3. Zillions of Variables 

Everything is connected, and potentially a cause – like a bowl of well-tossed spaghetti. 

4. Many Levels of Action 

Nations, organizations, networks, individuals, time all have different dynamics. 

5. Imprecise Measures 

Few instruments have near perfect validity and reliability. 

Conclusion
 

Social science is not as easy as rocket science. It is way more complicated.

Leave a comment

Filed under Uncategorized

Gevonden: Student assistent voor het onderzoek Geven in Nederland

De werkgroep Filantropische Studies van de Faculteit Sociale Wetenschappen aan de Vrije Universiteit Amsterdam is het expertisecentrum op het gebied van onderzoek naar filantropie in Nederland. De werkgroep houdt zich bezig met vragen zoals: Waarom geven mensen vrijwillig geld aan goede doelen? Waarom verrichten mensen vrijwilligerswerk? Hoeveel geld gaat er om in de filantropische sector? Voor het onderzoek Geven in Nederland heeft de werkgroep Suzanne Felix aangenomen als onderzoeksassistent.

 

Werkzaamheden

Geven in Nederland is een van de belangrijkste onderzoeksprojecten van de werkgroep. Sinds 1995 wordt het geefgedrag van huishoudens, individuen, fondsen, bedrijven en goededoelenloterijen elke twee jaar in kaart gebracht en samengevoegd tot een macro-economisch overzicht. De werkgroep Filantropische Studies brengt de resultaten van het onderzoek tweejaarlijks uit in het boek ‘Geven in Nederland’. Felix werkt mee aan het onderzoek naar nalatenschappen en giften door vermogensfondsen en huishoudens.
update: 3 september 2016

Leave a comment

Filed under Uncategorized

Brief guide to understand fMRI studies

RQ: Which regions of the brain are active when task X is performed?

Results: Activity in some regions Y is higher than in others.

Leave a comment

Filed under Uncategorized

Heeft de culturele sector de cultuuromslag naar ondernemerschap gemaakt?

Presentatie rapport Culturele instellingen in Nederland’

Werkgroep Filantropische Studies Vrije Universiteit Amsterdam

 

Vrijdag 10 juni 2016, Theater Griffioen, Uilenstede 106, 1183 AM, Amstelveen

 

In 2012 werd de Geefwet ingevoerd met een multiplier die de aftrekbaarheid van giften aan culturele instellingen verhoogde. Bovendien kregen culturele instellingen meer mogelijkheden eigen inkomsten te genereren uit commerciële activiteiten. Tegelijk kregen veel instellingen te maken met bezuinigingen en de vraag om meer ondernemerschap. Hoe hebben Nederlandse particulieren en bedrijven met een hart voor cultuur gereageerd op de verhoogde aftrekbaarheid van giften aan cultuur? Zijn zij ook inderdaad meer gaan geven? En hoe hebben de culturele instellingen gereageerd op de bezuinigingen enerzijds en de multiplier anderzijds? Wat voor instellingen hebben de omslag naar ondernemerschap wel kunnen maken en wat voor instellingen niet?

Deze vragen stonden centraal in een onderzoek dat de werkgroep Filantropische Studies heeft uitgevoerd op verzoek van het ministerie van OCW naar de effecten van de Geefwet op het genereren van inkomsten door culturele instellingen. Het onderzoek verschaft inzicht in de stand van zaken van de culturele sector op dit gebied en de mate waarin de Geefwet bijdraagt aan de versterking van de culturele sector door stimulering van giften aan cultuur.

U bent van harte welkom op een symposium waarop de onderzoekers de resultaten presenteren aan de culturele sector. U kunt zich hier aanmelden.

 


Programma

15.30    Aanmelden

16.00    Presentatie onderzoek door prof. dr. René Bekkers

16.30    Annabelle Birnie, Drents Museum

16.45    Marielle Hendriks, Boekmanstichting

17.00    Drankje

 

 

Locatie

Theater Griffioen, Uilenstede 106, 1183 AM, Amstelveen

Routebeschrijving – klik hier

 

 

Meer informatie

Meer informatie over het onderzoek vindt u op www.cultuursector.nl

Leave a comment

Filed under Uncategorized

Conditional Review Acceptance Policy (R1)

“Thank you for your invitation to review. Did the authors provide the data and the code they have used to produce the paper? Will the paper be published in open access mode? If twice yes, I will consider reviewing the paper.”

This is my new automatic reply to requests for review journal articles that I receive from editors and their assistants. In june 2014, I introduced a conditional review acceptance policy (CRAP). The policy was to review only those articles that the journal agrees to publish in a Free Open Access mode – making the article publicly available, without charging any fees for it from universities, authors, or readers. The revised policy now also includes the question whether the data and code will be publicly available, as proposed by the Peer Reviewers’ Openness (PRO) initiative. The revision rewards open science.

pro_lock

 

Leave a comment

Filed under Uncategorized

The Future of Foundation Support for Research and Innovation

Recently the EUFORI Study was published, in which a network of experts coordinated from our Center for Philanthropic Studies at the Vrije Universiteit Amsterdam mapped the support from foundations for research and innovation in Europe. Read the synthesis report here. The research was based on an extensive survey of 1 591 foundations supporting R&I in Europe and a qualitative analysis of 29 different country reports. We concluded that foundations contribute a significant amount of money to R&I: annually at least €5 billion, out of an asset base worth at least €127 billion. These are lower bound estimations, because it was impossible to estimate contributions by foundations that did not participate in the study.

The final chapter of the synthesis report presents six recommendations. The main objective of the recommendations made in this final chapter is to increase the potential of R&I foundations in Europe. Considering the underlying potential, actions towards greater support by foundations for research and innovation should and must involve engaging all actors: national governments, EU institutions, the foundations themselves, the corporate sector, universities and other research institutes, and the public at large.

Recommendation 1: Increase the visibility of R&I foundations

This recommendation is addressed to foundations, national governments, the EC and EU administration, businesses and the public at large. It relates to the current fragmented landscape of R&I foundations in Europe. The landscape of foundations in Europe is characterised by a few well-established foundations and many smaller foundations with modest resources mainly operating in the background. Growing visibility will enhance the impact of existing funding. If foundations become more aware of each other’s activities, the effects and impact of their contributions can be increased. Moreover, the other stakeholders involved such as the business community and research policy-makers will become knowledgeable about the foundations’ activities. From the perspective of the beneficiaries, research institutes, universities and researchers will more easily find their way to foundations. Visibility will lower the transaction costs for all the parties involved. For foundations, governments and businesses it will increase their knowledge about ongoing research/new research funded and vice versa. For grantmaking foundations it will facilitate the review process of research proposals and submissions; it is to be expected that more visibility will reduce the amount of incorrect applications. For the beneficiaries of the foundations’ support (research institutes, universities and researchers) – the grantseekers – it will increase their funding opportunities, they will more easily find their way to foundations, and it will facilitate submission processes. For potential (major) donors it will offer visible causes to benefit. Increasing the visibility of R&I foundations could have a positive effect on potential (major) donors as it could encourage them to support a research foundation. Increasing the visibility of and information about R&I foundations was already addressed by an expert group in 2005. They argued: ‘.. foundations and their donors would be more aware of the foundation landscape (increasing collaborative working and, possibly, giving), foundations’ contribution to various sectors could be properly assessed and the information could inform policy-making in this area. It is in fact a prerequisite to other actions’. The present EUFORI study is a step forward. A lot of information is now available. Next to this synthesis report, 29 country-reports, new data, an active network of researchers and the EUFORI website can contribute to the profiling of the R&I foundation sector in Europe.

With the exception of some large and well-established foundations in Europe, there is a lack of a common research identity among the foundations supporting R&I in most countries. Research and innovation are often not seen as a purpose/field in itself but are instead used as an instrument for other purposes and areas in which foundations specialise (such as health, technology, society). This is reflected by a lack of dialogue between the foundations supporting R&I (occasionally between foundations that deal with similar topics, e.g. foundations supporting cancer research). Bringing foundations together at a European level and following the recommendations of the expert group from 2005, the European Foundation Center (EFC) created the European Forum of Research Foundations. This forum provides a platform for a group of large and well-known R&I foundations in Europe. In order to increase the visibility of foundations supporting R&I at a national level, the encouragement of the creation of national forums of research foundations is recommended as the next step. The opportunities and mutual benefits for foundations supporting R&I at a national level should be explored. The next step: Explore the opportunities and mutual benefits of the creation of national forums of research foundations.

Recommendation 2: Explore synergies through collaboration

Unity in diversity is one of the main challenges for all the players involved in the R&I domain. These players can be distinguished in the domain of research (governments, business, foundations and research institutes/researchers), each with their own distinctive role. Together these groups can make a difference in increasing the potential for R&I. They can create synergy through collaboration, which should be interpreted in the broadest sense, varying from information sharing, networking, co-funding and partnerships. Mutual advantages can be derived from pooling expertise, sharing infrastructure, expanding activities, pooling money due to a lack of necessary funds, avoiding the duplication of efforts and creating economies of scale.

Get to know each other, meet and see where to reinforce each other’s efforts

Based on the conclusions of the EUFORI Study there is an indication for the need for improved dialogue, information exchange, networking and cooperation between the foundations supporting R&I, as well as between foundations, governments, business and research institutes (researchers). The needs, opportunities, mutual benefits and barriers for collaboration should be further explored, including mutual responsibilities when cooperating. The creation of national forums or networks of foundations supporting research and innovation, regular meetings between the foundations and other stakeholders involved (national government, EU government, research institutes and business) could bring these groups together.

An EU-wide study is recommended on the needs, opportunities, mutual benefits and barriers for collaboration between foundations, national governments, the European Commission, the business sector and research institutes. A network of national experts (mostly members from ERNOP) built for the EUFORI study can be of added value for this study and can facilitate the collaborative relations between the EC/ RTD, the R&I foundation sector and other stakeholders in Europe. It would be well-advised to set up an independent expert group before the start of this study with selected experts and stakeholder representatives in the field of foundations, the business sector, research institutes and public authorities at a national and European level. The expert group should provide input for the design of the study and could adopt an advisory role. Subsequently, it is recommended that the study will be finished by a follow-up conference for all the players involved aimed to discuss the implementation of the outcomes of the Collaboration Infrastructure Study. In this call for collaboration we have to consider two possible, interrelated pitfalls; namely the danger of ‘substitution’ and the danger of threatening the independence of foundations. Foundations, and civil initiatives in general, make their own choices and preferences and are based on social democracy. Governments, on the other hand, have their own responsibility based on political democracy. Businesses have their own market-driven values. Sometimes they reinforce each other, sometimes they may act as opponents. It concerns different worlds, differing in terms of constitution, values, legitimacy and organisation style. The independence of private R&I foundations should be respected. Foundations derive their legitimacy from many contacts with the ‘capillaries’ in society, thus offering them the opportunity to function ‘as the eyes and ears’ for innovation. This grass-roots connection represents the philanthropic tradition in Europe: ‘voluntary action to serve the public good’. The next step: Launch a Collaboration Infrastructure Study.

Continue reading the recommendations from the EUFORI study here.

 

 

 

Leave a comment

Filed under Uncategorized