Abstract

During the last four decades, Loek Halman became from a master assistant to Ruud de Moor a driving force of the European Values Study. This chapter is a tribute from the ‘Tilburg team’ that together with the team from the GESIS Data Archive facilitated all the steps from the questionnaire to the available datasets. We present the innovations made in the fifth and last wave with respect to translation, sampling, harmonization of cross-national measurement, and mixed mode and web surveys. Also, the collaborations of EVS with other European and global research infrastructures that will lead us into the future are highlighted.

5.1 Introduction

The history of the European Values Study (EVS) project and how this history is connected with the academic life of Loek Halman was already presented in the Introduction to this Liber Amicorum. In this chapter, we will concentrate on the different aspects of the EVS data, how they were gathered in the five waves since 1981 with an emphasis on the last wave, and the innovations made during these four decades.

Social survey methodology strengthened a lot in the last decades and this, together with the available datasets and the different ways the EVS had impact, will be presented in this chapter. Attention will be paid to the synergetic cooperation with other research infrastructures, such as the World Values Survey (WVS), that developed from EVS in the early 1990s and with whom EVS collaborated ever since. The fifth wave was a close collaboration where EVS took the lead in Europe and WVS in the rest of the world.

In the coming years, we will (hopefully) see a further consolidation of close cooperation between the different survey programs and a development towards a solid infrastructure to keep the social sciences at the core of a developing World (and Europe).

5.2 The Five EVS Waves

The European Values Study was initiated at the end of the 1970s to address political and social issues that were pressing at that time, e.g., the centrality of Christian values for Europeans and its implication for the cultural unity of Europe itself. In terms of organization, the EVS is managed by the Council of Program Directors, which includes representatives of all countries/regions participating in EVS. The program is steered by an Executive Committee, for a long period led by Loek Halman, a Methodology Group, and a Theory Group. The central operations are mostly carried out at Tilburg University and at the GESIS - Leibniz Institute for the Social Sciences, in constant communication with the national EVS teams. The EVS is primarily funded by its participating members’ institutions, such as universities, research institutes, national science foundations, and private sponsors. The European Values Foundation, established in the late 1970s and of which Loek Halman was a member for a long time, played an important role in the gathering of funds, especially to make the surveys possible in Eastern Europe.

The EVS questionnaire, which has undergone improvements and changes throughout the years, generally revolves around the measurement of values and attitudes in several domains of life: family, work, religion and morale, politics and society, environment, as well as national identity, tolerance, and social solidarity. EVS data has been used to investigate important societal trends and changes, such as secularization (see, e.g., Halman & van Ingen, 2015), modernization (see, e.g., Inglehart & Baker, 2000), social inequalities (see, e.g., Hertel & Groh-Samberg, 2019), demographic behaviours (see, e.g., Arpino et al., 2015) – and the list could continue.

The first wave of the EVS was conducted in 1981 and collected data from almost 20,000 individuals spread over 16 countries. Since then, the EVS was carried out every 9 years (1990, 1999, 2008, 2017) and expanded tremendously, reaching up to 47 countries/regions in 2008 (see the geographical coverage in Figure 5.1), and interviewing over 225,000 individuals throughout the waves (see Table 5.1).

Table 5.1 Countries that participated in the different waves with their sample sizes.

* Mixed-mode data collection; ** only web survey

The EVS relies on probability-based sampling, and there is a strong commitment to ensure comparability over time and across countries. Similarly to other large-scale population surveys, the EVS had to adapt its methodology to a changing survey research landscape. While the EVS is traditionally carried out as an interviewer-administered face-to-face (F2F) survey, in the latest wave, a mixed-mode design was implemented in six countries (see Luijkx et al., 2021) so as to better respond to the decreasing response rates and increasing survey costs (Wolf et al., 2021). The main features and improvements in the EVS methodology are outlined in the next section.

5.3 Strengthening Methodology in the EVS 2017

In this section, we will discuss the sampling, the construction of weights, and the mixed mode surveys with the matrix design, In addition, the translation process and the harmonization of cross-national measurements are presented, as well as the improvements of the project management and the level and transparency of the documentation.

Sampling

Sampling is a crucial aspect in the design of a general population survey, with strong implications on the quality of the collected data. The EVS has relied on probability-based samples since the start, although different sampling methods and sampling frames have been used (see, e.g., Scherpenzeel et al., 2017). Also thanks to the work undertaken in SERISS (the Synergies for Europe’s Research Infrastructures in the Social Sciences, Horizon 2020-project), in the EVS 2017 more attention has been paid to the use of population-registers as sampling frames, and clearer guidelines have been written to aid national teams in their sampling procedures (EVS, 2020a). Sampling design forms have been adopted to identify a suitable sample size in each country, conditional on the sampling method available. This allowed for more control over the sampling design phase, ensuring higher quality overall.

Weights

The EVS also includes survey weights. In the EVS 2017, three sets of weights are provided. First, population size weights that correct for differences in the ratio sample/population in each country and should be used when producing aggregate statistics. Second, calibration weights were computed by the EVS central team in consultancy with a team of statisticians at GESIS, to adjust the characteristics of the samples to the characteristics of the population (age, gender, educational level, and region). Finally, design weights – which are only available for a selected number of countries – allow to correct for the unequal inclusion probabilities of individuals in the samples. More information on the weighting procedure can be found in EVS (2020b).

Mixed Mode – Matrix design

While the EVS has traditionally been conducted face-to-face, changes in the survey climate are pushing towards mixed-modes, and the EVS has taken up that challenge. In this respect, the EVS was the first large-scale cross-national survey program to officially approve mixed-mode data collection, hence providing pioneering insights. Six countries (Denmark, Finland, Germany, Iceland, the Netherlands, and Switzerland) complemented the traditional face-to-face survey with self-administered surveys, either web surveys or postal surveys. The questionnaire has been adapted accordingly, with item batteries split into multiple single items and some changes in wording. Due to the length of the questionnaire, four of these countries (Germany, Iceland, the Netherlands, and Switzerland) decided to adopt a matrix design, slicing the questionnaire into smaller modules and administering only some of the modules to a larger pool of respondents. In Iceland, the Netherlands, and Switzerland, respondents were also contacted again to administer the modules that were skipped in the first round.

The article by Luijkx et al. (2021) explains the mixed-mode strategy in larger detail, and attempts to assess the outcomes. Overall, the strategy is considered successful, with significant cost reduction and good outcomes, even in the long, one-hour version. In Iceland and Germany, the self-administered mode yielded better outcome rates than the traditional face-to-face. The paper version, as a complement of the web survey, was particularly important for some segments of the population. The main drawbacks of the strategy are the complexity of the resulting data file structure – especially with the matrix design, and the stronger representation bias in the self-administered mode. All in all, however, the data quality results are acceptable, and recent studies looking at the comparability of the measurements across modes yield promising results (Cernat, 2021; Lomazzi, 2022)

Translation

Translating the questionnaire preserving comparability across countries but also over time is a key aspect of a cross-national longitudinal survey project like the EVS, and also in this respect the EVS has improved its standard. The questionnaire, designed by the Theory Group and approved by the Council of Program directors, is written in English, and later adapted by each national team to their own language(s) and contexts. In principle, languages spoken by 5% or more of the population in a country are included. In the EVS 2017, a thorough review of existing translations has been conducted, whereas new questions have been translated using state-of-the-art standards, and most notably the Translation, Review, Adjudication, Pretest and Documentation (TRAPD) procedure (see Mohler et al., 2016).

The Translation Management Tool (TMT), developed by Centerdata under the SERISS funding, has been adopted by EVS to assist its national teams in the translation process. Among the many functionalities of the TMT, the re-use of the translation of repeated elements has improved the efficiency during the translation process, whereas the possibility to document changes and doubts in notes attached to survey items has smoothly flowed into translation notes in the survey’s variable report (EVS, 2020c). TMT is now available as TranslationCTRL – see http://tmt.centerdata.nl/.

Harmonization of cross-national measurements

Alongside translation, another cornerstone of comparative research is the harmonization of measurements, such as national classifications (e.g. educational attainment, political parties, income categories), which allows to make meaningful comparisons across contexts. On the one hand, the EVS adopts the international standards, e.g., ISCED for education, ISCO for occupation, ISO3166 for countries and regions, which are widely used in social surveys and increase the interoperability of diverse data sources. On the other hand, the EVS is also committed to aid the development of new classification schemas. In the EVS 2017, thanks to SERISS, the EVS has adopted the ES-ISCED classification (Schneider, 2009), enhancing the links to ESS. In the framework of enriching SurveyCodings (https://www.surveycodings.org/), an online tool to foster the (re)use of multilingual classifications developed under SERISS and SSHOC (Martens & Tijdens, 2021), the EVS has expanded a coding classification to harmonize religious denominations building on the ONBound project (https:// www.gesis.org/en/services/processing-and-analyzing-data/data-harmonization/onbound). Moving beyond the harmonization of national classifications, in the ESS-SUSTAIN-2 project, the EVS and the ESS have started a comparison of substantive items to establish whether they can be harmonized, potentially allowing to pool the two data sources and unlocking new research opportunities for comparative researchers and survey methodologists.

Project management

Organizing and monitoring the work of dozens of different teams is a cumbersome task, and when it comes to a survey, poor project management can negatively affect the data quality. At the onset of EVS, in the late 1970s and early 1980s, a lot of the alignment and coordination work had to happen face-toface, with members of the central teams travelling across Europe to meet and work with the national teams. Face-to-face meetings continue to be important in such a large cross-national program; for instance, the yearly General Assembly still takes place preferably in person. However, for the daily tasks, the process has improved through the years, mainly thanks to technological developments which make it more efficient to organize the work, and coordinate with partners spread all over the continent. In 2017, the EVS adopted myEVS, a new online Survey Project Management Platform – SMAP, developed under SERISS and tailored to the needs of a large-scale survey project (Brislinger et al., 2019). MyEVS facilitated access to guidelines and templates, and enabled a smoother communication between national and central teams.

More documentation and transparency

In addition to the methodological improvements hereby outlined, over the years the EVS has taken significant steps into improving transparency and providing more survey documentation. A large array of documents is provided alongside the latest data release, including standard survey reports (e.g., the codebook, the method report) but also an in-depth explanation of the matrix design data set, the script to compute the calibration weights, and the full set of methodological guidelines as defined prior to the data collection.

EVS data and documentation are stored in the GESIS data archive, a Core Trust Seal Repository. Several steps are undertaken by the archive to maximize compliance with the FAIR data principles (Wilkinson et al., 2016), including assigning globally unique persistent identifiers to the datasets (doi) and adopting internally-recognized metadata standards (e.g. DataCite, DDI).

5.4 Overview of Datasets Available

Official EVS data and documentation is available free of charge for research purposes from the GESIS Data archive upon registration. An overview of the available datasets and relative persistent identifier is included in Table 5.2. 

Integrated datasets are available for each wave, constituting the preferred source for cross-sectional cross-national analyses. For time-series analyses, the EVS Trend File includes data collected over almost 40 years. Comparisons can also be expanded globally thanks to the Joint EVS-WVS dataset and the Integrated Values Surveys dataset – for which scripts and info are provided and not the compiled dataset –, which covers 115 countries/territories globally over 40 years.

Table 5.2 Overview of available EVS datasets.

^a Single-country datasets also available; ^b Sensitive data version also available, under stricter conditions; ^c Not a dataset, but steps to construct it can be found on https://europeanvaluesstudy.eu/methodology-data-documentation/integrated- values-surveys-ivs-1981-2021/data-and-documentation-ivs-1981-2021/

5.5 Cooperation with Research Infrastructures

The EVS has thrived also thanks to its cooperation with other survey programs and initiatives.

Cooperation with WVS

In the early 1990s, the World Values Survey (WVS, see www.worldvaluessurvey.org) originated from the EVS and expanded the investigations on a global scale. The two projects have been cooperating ever since, releasing joint files to enable global analyses of values and value change.

European projects

Through the participation in European projects funded via Horizon 2020, the EVS collaborated with other large-scale cross-national survey programs to strengthen the methodology of social surveys. The Synergies for Europe’s Research Infrastructures in the Social Sciences (SERISS) project, operational from 2015 to 2019, allowed establishing a cooperation with the Consortium of European Social Science Data Archives (CESSDA ERIC, https://www.cessda.eu/), the Survey of Health Aging and Retirement in Europe (SHARE ERIC, http://www.share-project.org/), the European Social Survey (ESS ERIC, https:// www.europeansocialsurvey.org/), WageIndicator (https://wageindicator.org/) and the Gender and Generation Programme (GGP, https://www.ggp-i.org/). Tools developed under SERISS and piloted by EVS include the project management platform myEVS and the translation management tool (TMT, developed by Centerdata).

Collaboration between the survey programs has continued and expanded to other domains through another project, the Social Sciences & Humanities Open Cloud (SSHOC, https://sshopencloud.eu/), in which EVS has contributed to build a European Question Bank and to further develop SurveyCodings, a tool to foster the documentation and reuse of socio-demographic classifications.

The cooperation with the ESS has continued also via the ESS-SUSTAIN-2 project, in which the two surveys are exploring their similarities (and differences) and outlining potential scenarios of future collaboration. 

5.6 Impact

Such a vast amount of data has inspired high quality and impactful research. Since the early 1980s, over 2,800 scientific publications have appeared which are based on the EVS data sets (https://europeanvaluesstudy.eu), including journal articles, theses, books, and sourcebooks. A significant number of these publications is in languages different from English, showing engagement also with the local scholar communities.

Beyond the scientific impact, EVS data is also used for dissemination and training purposes. The exhibition ‘United in Diversity’ at the Visitors’ Centre of the European Parliament, in Brussels, made use of EVS data. Three versions of the Atlas of European Values are now published (Halman et al., 2005, 2011, 2022), each time updated with new data and another focus, including informative graphs and tables which are more accessible for a non-scientific audience, therefore widening the dissemination opportunities. The new digital version of the Atlas of European Values (https://www.atlasofeuropeanvalues.eu/), developed in the EVALUE project, includes interactive tools and educational materials for teachers and pupils of secondary schools.

5.7 The Future

As a conclusion, let’s look ahead towards 2026, the year when the sixth wave of EVS is due. What are our challenges?

The covid-19 pandemic made us once more aware that in case of emergencies you need a flexible, ready to go into the field infrastructure. A well-functioning web panel would be a great good in these cases. For the Netherlands, the LISS panel is such a panel, and it could be used to reinterview the 2017 respondents of EVS and observe their value changes (Reeskens et al., 2021). A Europe wide survey as follow-up on the last wave of EVS would have been ideal in this situation. In general, face-to-face surveys are becoming very costly and several surveys already took or are ready to take the decision to move to self-completion in web and mail surveys. This is a great challenge ahead, and also EVS will have to take this up. If this can lead to a common infrastructure with other social surveys that would be a great gain. Ideally this would happen on a European level with possible global outreach, but it is important to notice that good examples on the national level are already being developed and visible, e.g., ODISSEI (https://odissei-data.nl/en/) in the Netherlands.

In the field of survey tools, a lot of progress has been made in the last decades. Question banks have been developed that aid the translation process and have more control on comparability over time and space within surveys but also between surveys. An example is MCSQ (the multilingual corpus of survey questionnaires, https://www.upf.edu/web/mcsq) in which EVS takes part. A further consolidation of the coding of core demographic variables also is an important challenge ahead. With SurveyCodings, we are on a good path and we should continue and elaborate the cooperation with other social science surveys here.

Concluding, exciting times are ahead, to continue the great work started by EVS-founding fathers Ruud de Moor and Jan Kerkhofs in the late 1970s. Forty-five years later, the world of survey research changed dramatically. In the last years, Loek Halman was pivotal in keeping the EVS-train going. We are grateful for that and will continue that work in an ever-rapidly changing context.

List of References

Arpino, B., Esping-Andersen, G., & Pessin, L. (2015). How Do Changes in Gender Role Attitudes Towards Female Employment Influence Fertility? A Macro-Level Analysis. European Sociological Review, 31(3), 370–382. 

Brislinger, E., Kurti, D., Davari, M., & Quandt, M. (2019). Survey Project Management Portal: Quick guide to the demo version of SMaP (Deliverable 4.11 of the SERISS project funded under the European Union’s Horizon 2020 research and innovation programme GA No: 654221).

Cernat, A. (2021). Transition to multimode data collection: Understanding the measurement dimension.  https://www.youtube.com/watch?v=F54TV5wU174

EVS (2020a) European Values Study (EVS) 2017: Methodological Guidelines (Vol. 2020/13).  European Values Study (EVS). https://doi.org/10.21241/ssoar.70110

EVS (2020b) European Values Study (EVS) 2017: Weighting Data (Vol. 2020/15). European Values Study (EVS). https://doi.org/10.21241/ssoar.70113

EVS (2020c) European Values Study (EVS) 2017: Variable Report (Vol. 2020/16). GESIS – Leibniz Institute for the Social Sciences.

Halman, L., Luijkx, R., & Van Zundert, M. (2005). Atlas of European Values. Leiden: Brill.

Halman, L., Sieben, I., & Van Zundert, M. (2011). Atlas of European Values: Trends and Traditions at the turn of the Century. Leiden: Brill.

Halman, L., Reeskens, T. Sieben, I., & Van Zundert, M. (2022). Atlas of European Values: Change and  Continuity in Turbulent Times. Tilburg: Open Press TiU.

Halman, L., & van Ingen, E. (2015). Secularization and Changing Moral Views: European Trends in

Church Attendance and Views on Homosexuality, Divorce, Abortion, and Euthanasia. European  Sociological Review, 31(5), 616–627.

Hertel, F. R., & Groh-Samberg, O. (2019). The Relation between Inequality and Intergenerational Class  Mobility in 39 Countries. American Sociological Review, 84(6), 1099-1133.

Inglehart, R., & Baker, W. E. (2000). Modernization, Cultural change, and the persistence of traditional values. American Sociological Review, 65(1), pp. 19-51.

Lomazzi, V. (2022). Improving measurements by survey infrastructures synergies: Insights from the revised gender role attitudes scale in the European values study 2017. Quality & Quantity.  https://doi.org/10.1007/s11135-021-01312-6

Luijkx, R., Jónsdóttir, G. A., Gummer, T., Ernst Stähli, M., Frederiksen, M., Ketola, K., Reeskens, T., Brislinger, E., Christmann, P., Gunnarsson, S. Þ., Hjaltason, Á. B., Joye, D., Lomazzi, V., Maineri,  A. M., Milbert, P., Ochsner, M., Pollien, A., Sapin, M., Solanes, I., Verhoeven, S.,Wolf, C. (2021). The European Values Study 2017: On the Way to the Future Using Mixed-Modes. European Sociological Review, 37(2), 330-346.

Martens, M., & Tijdens, K. (2021). D3.4 Multilingual ontologies for Occupation, Industry, Regions and cities, Food items, and Religion, with use case. https://doi.org/10.5281/zenodo.4724837

Mohler, P., Dorer, B., de Jong, J., & Hu, M. (2016). Translation. In Guidelines for Best Practice in Cross-Cultural Surveys. Survey Research Center, Institute for Social Research, University of Michigan. 

http://ccsg.isr.umich.edu/.

Reeskens, T., Muis, Q., Sieben, I., Vandecasteele, L., Luijkx, R., & Halman, L (2021). Stability or Change of Public Opinion and Values During the Coronavirus Crisis? Exploring Dutch Longitudinal Panel Data. European Societies, 23(sup 1), S153-S171.

Scherpenzeel, A., Maineri, A., Bristle, J., Pflüger, S.-M., Mindarova, I., Butt, S., Zins, S., Emery, T., & Luijkx, R. (2017). Report on the use of sampling frames in European studies. SERISS.

Schneider, S. L. (2009). Confusing credentials: The cross-nationally comparable measurement of educational attainment [Http://purl.org/dc/dcmitype/Text, Oxford University, UK]. https://ora.ox.ac.uk/objects/ uuid:15c39d54-f896-425b-aaa8-93ba5bf03529

Wilkinson, M. D., Dumontier, M., Aalbersberg, Ij. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas,  M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., … Mons, B. (2016). The FAIR Guiding  Principles for scientific data management and stewardship. Scientific Data, 3(1), 160018.

Wolf, C., Christmann, P., Gummer, T., Schnaudt, C., & Verhoeven, S. (2021). Conducting General Social Surveys as Self-Administered Mixed-Mode Surveys. Public Opinion Quarterly, 85(2), 623-648.