Skip to main content
SearchLoginLogin or Signup

Beware! Surveys Are Not Universal Tools

Published onMay 09, 2022
Beware! Surveys Are Not Universal Tools


Comparative surveys, such as the European Value Survey, are sources for relevant and important information about the formation and distribution of opinions and values in our contemporary societies. However, the data should not be regarded as given entities which are just to be collected and fed to the computers. They are products of a wellplanned, comprehensive, and meticulous process, which involves reflections about how respondents interpret the items. This contribution aims to stress the importance of the preliminary work behind the surveys, and the reflections behind it. It is the conscientious work of coordinators, like Loek Halman, which elevates survey data to sociologically meaningful indicators. Furthermore, this contribution also points out that the challenges of survey studies cannot be answered by more sophisticated statistical models. Supplementary qualitative analyses are also called for. Thus, the quantitative data analyses may become more valid and meaningful by reflections based qualitative studies of the sociohistorical context and the practical language of the respondents.

4.1 Introduction

International survey data are a product of a long-term collaboration within a scattered team of researchers. Users of these data sets hardly recognize the challenges of harmonizing the task among contributors from different academic cultures. However, Loek Halman has proven to be a master conductor of the EVS team. When the EVS was launched, it was met by a wide-spread scepticism, especially among humanistic intellectuals and theologians. There are some issues to beware of in survey studies. The points presented in this chapter are common-sense among serious values researchers. Empirical articles tend to focus on the findings resulting from quantitative analysis rather than its underlying assumptions. Textbooks tend to present an idealized procedure rather than the actual working process. Surveys on social values are valuable tools provided you recognize what you must beware of when you read such analyses.


4.2 The Survey Sample as a Social Formation

The sample is supposed to be a kind of micro version of society. If the sampling is performed correctly, and if there is no systematic loss of sampled informants, then the composition of the sample will probably correspond with that of the population. However, the obtained sample does not form an interacting group. Whereas social groups are characterized by interaction, communication and cooperation, the sample only interacts indirectly, by the media of the questionnaire and the interviewers. In a normal, social group, individuals who are subject to questions about social values, will take up discussions with others, and their eventual answers will be influenced by special persons, who are assumed to know about the issues, so-called ‘opinion leaders’. For instance, when laypersons are asked about their religious beliefs, they may refer to people, they ascribe with authority on the matter. Grace Davie (2000) describes such a religiosity as ‘vicarious’. This is, however, a special case of a well-known general pattern of opinion forming (Lazarsfeld 1957). Respondents will often answer with their opinion leadership in mind. The values survey is situated in media res by prompting an immediate answer to abstract questions. These questions may be addressed in societal discourses, and the debate may end with a referendum. However, the result of the referendum will probably differ from the findings of the survey, due to the social discourses which follow from the initial questioning to the final and collective answer. This reservation does not mean that the survey results are invalid provided we acknowledge that they just depict the initial individual position, before the issue has been subject to a collective process of deciding.


4.3 Interviewing as Measurement

By asking a sample of respondents the same questions and register their responses by standardized categories, comparisons are made possible and observer-induced bias is minimized.

The influence of the interviewer is only erroneous if the situation is regarded as an exact measurement of a fixed position. We may instead interpret the answers as expressions of how they would like to present themselves to others. For instance, when respondents over-report their church attendance, they tell us about what they would like to do and how they would like to be seen. The answers are truthful, if we understand what they refer to. The measurement problem only arises from researchers’ interpretation of the response. In order to clarify whether this is the case, we ideally need to supplement the survey with open interviews, asking some types of respondents how they understand the question and what they meant by their answers.

One major study of how questionnaires are perceived is by Henning Olsen (1998). He approached the issue by involving a large group of people in first responding to a questionnaire and then to reconsider their answers in a laboratory. He observed that a fifth of the participants changed their answers on second thought. The change was mostly moderate and was also directed towards moderation. He further noticed that measurement issues were more noticeable at questions which used nouns with a wide semantic range. Surveys on social values often use such nouns in the questionnaire, such as religion, independence, tolerance or democracy.

Olsen’s study (1998) was not based on the EVS questionnaire, but it is relevant for interpreting responses to it. Social values are referred to by abstract concepts with a wide semantic range. An abstract question about values may invoke a complex set of associations. Therefore, values are often operationalized by referring to specific issues or acts. This calls for reflections about whether the selected issues are typical and central for the meaning of the concept. For instance, Inglehart’s (1977) well-known index of values operationalizes economic values by referring to fighting rising prices. This operationalization is only relevant in societies which have experienced inflation recently. Other economic issues may be more salient in some contexts, such as unemployment.

Validation of a measurement calls for a meticulous reflection on the meaning of the theoretical concept, its focus, scope, and range, and how it relates to people’s practical life-world. As the theoretical concept typically is quite abstract, it is needed to consider the adequacy of efforts to make it more concrete. For instance, attendance at ritual services does not always point to church affiliation; e.g., members of a choir attend services regularly.

An evaluation of a measurement calls for theoretical and methodological reflections. For instance, when we ask people about their ‘religion’, some associate it with the established ‘public’ institution while others associate it with their ‘private’ beliefs. The methodological dilemma cannot be solved by putting a set of data into the computer, since no algorithm can identify what people thought as they answered the questions. A scientific argument must be based on connection between theoretical reflections on the one hand and information about how the respondents associate the questions with their lifeworld on the other. Knowledge about this calls for in-depth interviews. When a sociological analysis points to different responses between social groups, we need to consider the possibility that the question evoked different associations between the groups.

A methodological distinction can be made between four basic types of responses to survey items. The first type includes ‘conformist’ responses; these responses adhere to ideas which are historically rooted and expressed by wellknown concepts, and which are supposed to be shared by a majority in society.

To confirm a conformist position does not call for a self-critical reflection. It affirms embeddedness in the local history and belonging to the community. To express support for a position which is novel or alien in the context calls for a critical reflection. We may label such a position as ‘innovative’. Such a position must be stated in novel terms, and they are therefore difficult to communicate to people who take the common language for granted. A third type comprises attitudes which doubt or reject the traditional position, but remain undecided in the search for a firm answer. We may label such a position as ‘seekers’. They may be offered to respond by ‘don’t know’; but that answer does not represent their position. They positively know that they reject the traditional position and they have also rejected some alternatives. They may even know in which direction their search goes, but they have not found a firm answer, and hence they are not able to identify their present position. A further type consists of ‘sceptics’, who do not seek, since they believe that it is futile. Philosophers and psychologists may argue that scepticism is a precarious position. However, this does not preclude scepticism to be empirically present in a population. A wide-spread state of scepticism may indicate a state of anomie which forms an overture for an emotionally charged rebellion. This makes it even more important for sociologists to identify the prevalence of a general scepticism.

These four types can be traced in the question about beliefs in the European Values Study. The option of ‘a personal God’ conforms with the historical tenets of the Christian churches. Belief in ‘a spiritual force’ typically represents an innovative type of religiosity in a European context. ‘I don’t believe in any kind of spiritual force or personal God’ represents scepticism. However, the option ‘I don’t know what I believe in’ does not necessarily indicate seekership. The questionnaire for the survey on “Religious and Moral Pluralism” (Dobbelaere & Riis 2002) added a question about spirituality: “I believe that God is something within each person rather than something out there”. This formulation also points towards an innovative tendency. It caught a rather wide affirmation. The decline of the personal image of God proclaimed by the churches has not led to a corresponding increase in scepticism, but rather to a spiritually oriented transformation.

Surveys focused on ideas and terms which people recognize. Therefore, they are better able to identify the status of conformist attitudes and their decline. In order to present innovative attitudes, the questions need to be comprehensible to the respondents. By using fixed questions, which most respondents can understand and relate to, surveys are thereby inherently better able to indicate how established ideas and values recede, than to trace the growth of new ideas and values. Thus, surveys are better at providing indications of a decline of established type of practices and beliefs, such as service attendance or belief in the Christian dogma proclaimed by the churches, while they are less able to illuminate innovative ideas and practices. This may explain why research based on surveys tend to focus on secularization, in the sense of decline in established practices and belief, rather than to produce indicators for religious innovations, such as the emergence of new types of spirituality. It is, of course, interesting to indicate whether the membership foundation of the established churches erodes, and surveys studies are adequate tools for this. However, they ought to be supplemented by other research designs in order to complete the picture of religious and spiritual transformations in our time.


4.4 Surveys as Measurements

The exact natural sciences are empirically based on measurements, referring to objective standards. In the era of Positivistic predominance, the social sciences attempted to construct similar measurements for human attitudes and social values. Obviously, it is not possible to produce such measurements in an objective sense. Such reflections led to a broad criticism of survey methods from defenders of a qualitative research design. Thus, Aaron Cicourel (1964, p. 244) therefore argued: “Conventional measurement systems may have a moderate correspondence with the institutional features of everyday life.” This critique kicks in an open door. It is well-known to empirical researchers that the interpretation of the data depends on a hermeneutic reflection. This is acknowledged when we construct the questionnaires and try to make sense of the answers. However, in the final reports, this hermeneutical reflection is mostly under-reported.

Surveys on social values do not produce measurements in an objective sense. Their operation of assigning numbers to certain expressions should provide a structurally true depiction of the phenomenon. This basic preliminary reflection is, however, often skipped, in the drive to get into the practical analysis.

The statistical measurement models developed by Georg Rasch (1968) is based on an attempt to specify a ‘specific objectivity’ which allows comparisons of the items (response patterns) in a manner which is statistically independent from the composition of the respondents. The probability that a participant can solve a task depends on both that person’s ability and the difficulty of the task. Rasch’s solution assumes a ‘local statistical independence’, which implies that all variation among responses to an item is accounted for by the variable x; so for a given x, there is no further relationship among responses.

This approach allows to indicate whether a set of responses to a questionnaire can be regarded as fulfilling a measurement or not in Rasch’s sense. This type of insight is of great interest since it warns against a naïve assumption of measurability. When a series of proposed value scales were tested at University of Copenhagen, they did not identify the same scaling patterns across socio-economic groups, and thus did not fulfil Rasch’s criterion (Gundelach 2002 p. 39).


4.5 Causality Problems in Surveys

Survey studies are inherently fixed in time, while causal explanations necessitate analyses of the conditions and mechanisms which influence a process. Cross-sectional analyses of survey studies are based on causal hypotheses which may be refuted but cannot be proven by them. Regression analyses depend on a set of assumptions about boundary conditions, such as about the relations of dependence or independence between the variables. The resulting coefficients are only meaningful provided that these assumptions are correct. Many researchers recognize this and therefore study supplementary sources about the processes relating to their problem. Nevertheless, many reports from survey studies skip the preliminary reflections.

It is necessary to consider the logic of the analyses. They do not prove the causal mechanisms, but rests on causal hypotheses. If these assumptions are correct, then the regression coefficient indicates how much the relative degree of change of the dependent factor as the independent factor changes with one unit. The analysis does not observe this change. The causal status of the independent variables must be clarified. An independent variable may indeed identify an assumed cause, as a mechanism or hindrance. Thus, distance to the nearest church may influence church attendance in a causal sense. However, independent variables are often used in regression models without considerations about their causal status. Thus, surveys of patterns of religiosity have identified three determining indicators: gender, generation, and geography (Halman & Riis 2003). The causal assumptions behind these indicators are seldom made explicit. Sociologists do not assume that religiosity is stimulated by chromosomes or hormones. Gender as an indicator points back to patterns of socialization and social roles. In a similar manner, generation is affiliated with changing patterns of socialization and socio-economic and cultural fluctuations, rather than with age. Geography indicates located clusters of religious values, for instance in rural districts. The analysis may point out relevant indicators, but it does not reveal the causal mechanisms and hindrances. In order to identify how social values emerge and how they influence social life, we need to include both operative and latent causal mechanisms.


4.6 Including the Actor in Survey Data

Schutz (1953, p. 32) criticized social scientists of reducing social actors to puppets, created and directed by the scientists themselves: “Yet these models of actors are not human beings living within their biographical situation in their social world of everyday life.” This critique could either be read as confronting all analytical reductions made by social science. It could also be read as an appeal to include considerations about the life-world of the social agents.

Social surveys may, in fact, provide an opening to this perspective. Surveys result in a data-matrix. It is normally read from a structural perspective, seen from the variable angle. However, the data may also be read from a respondent perspective. We may focus on a particular respondent, identify the background data about gender, age, location, education, et cetera and then trace the answers to the posed questions. Of course, it is not possible to perform this kind of reading for all the thousand respondents and try to find some significant patterns. However, the survey aids us to provide an overview of the population, which enables to identify some types of respondents which have a theoretical significance. Our theory may identify types of social agents, such as firm believers, atheists, spiritual seekers and so forth. We may then pick out respondents which conform to the type and read through their responses to the other questions. This may provide a supplement to the structural analyses which make the covariations comprehensible in human terms.


4.7 Surveys as a Tool for Selecting Cases

According to Andrew Sayer (1992), surveys represent an ‘extensive’ research design; it analyses formal relations of similarity, based on ‘taxonomic groups’. He contrasts it with an ‘intensive’ research design, which analyses substantial relations of connection based on ‘causal groups’. This distinction is not identical with the common contrasting of quantitative and qualitative methods, or with broad versus deep studies. Extensive designs are descriptively oriented while intensive designs are causally oriented (Sayer, 1992, p. 243).

The proper function of surveys is to provide an overview of a population. A survey may point out typical or special cases which call for further analyses. The intensive case studies enable us to investigate whether the statistical correlations between attitudinal variable also corresponds with associations in people’s minds. The intensive case studies could further inform about specific processes, their conditions, time sequence, operative mechanisms and hindrances. Such information may qualify regression analyses by indicating how the independent variables made the dependent variable change. Mixed designs have been criticized epistemologically for disregarding the philosophical opposition between positivism versus hermeneutics, but such designs are permissible according to both critical realism and pragmatism (Riis, 2001).

4.8 To Conclude: From Positivism to Pragmatism?

Whereas the early survey studies were based on positivism, they follow a more pragmatic line today. However, this opens for a new challenge; namely to identify the pragmatic criteria of research. When can we affirm that a procedure is fruitful and useable? We risk being caught into a tautological web, where our use of a procedure affirms its usefulness. With a pragmatic perspective in mind, the social function of surveys on social values is two-sided: These studies could either support commodification, manipulation of opinions, or a technocratisation, or become tools for democratization, social mobilization and emancipation (Habermas 1977). We, as social researchers, have an influence on the outcome by our dissemination of the studies and our presentation of the findings. The dissemination of our studies can inspire a public discourse about whether the societal structures enable or restrict the realization of people’s values.


List of References

Cicourel, A. (1964): Method and Measurement in Sociology. The Free Press.

Gundelach P. (ed.) (2002): Danskernes Værdier 1981-1999. Hans Reitzels Forlag: Copenhagen.

Davie,G. (2000): Religion in Modern Europe. A Memory Mutates. Oxford University Press: Oxford.

Dobbelaere, K. & Riis, O. (2002). Religious and moral pluralism: theories, research, questions and  design. Research in the Social Scientific Study of Religion, 13, 159-171.

Habermas, J. (1977): Knowledge and Human Interest. Heineman: London.

Halman.L. & and Riis, O. (eds.)(2003): Religion in Secularizing Society: The European’s Religion at the end of the 20th Century. Brill: Tilburg.

Inglehart, R. (1977): The Silent Revolution. Princeton University Press: Princeton.

Inglehart, R. (1990): Culture Shift in Advanced Industrial Society. Princeton University Press: Princeton.

Lazarsfeld, P. (1957) in E. Katz (ed.): Public Opinion. Free Press: New York.

Kreiner,. S (1996): An informal introduction to graphic modelling. In: H.C. Knudsen and G. Thorncroft (eds.) Mental Health Service Evaluation. Cambridge University Press: Cambridge.

Olsen, H. (1998): Tallenes Talende Tavshed. Måleproblemer i surveyundersøgelser. Akademisk Forlag: Copenhagen.

Rasch, G. (1968): “A Mathematical Theory of Objectivity and its Consequences of Model Construction” Statistisk Institut, Copenhagen.

Riis. O. (2001): Metoder på Tværs. DJØF Forlag: Copenhagen.

Sayer, A. (1992): Method in Social Science, a Realist Approach. Routledge: London.

Schutz, A. (1953): Common-sense and Scientific Interpretation of human action. Philosophy and  phenomenological research, vol. 14.

No comments here
Why not start the discussion?