Response sets in self-report data and their associations with personality traits

 

Key words: Self-report, Questionnaire, Bias, Confounding, Neuroticism.

ABSTRACT ­ Objective: With a strong reliance on the use of self-report questionnaires in psychiatric research, appropriate attention should be given to the existence and correlates of response sets. We studied the presence of response tendencies by using an old research paradigm from psychology.
Methods: We administered response options without questions to a sample of 91 second-year medical students, together with two personality questionnaires.
Results: The scores of the respondents demonstrated the willingness of respondents to fill in a content-free questionnaire and revealed the presence of a general tendency towards positive responses. In psychometric theory, this would be referred to as a 'constant error'. More importantly, positive answers were significantly related to low neuroticism, high extraversion and high well-being. In psychometric theory, this would imply 'response bias' in which the self-reports are confounded by personality features.
Conclusions: In the development and evaluation of questionnaires, researchers should be aware of the potential for this bias, particularly when asking questions about subjective health status.

Introduction

Researchers in psychiatry are heavily dependent on the use of self-report questionnaires. Although the advantages over the use of other forms of data collection, such as medical records or clinical ratings by specialists, are obvious in terms of costs, repeatedly the limited accuracy of self-report data has been shown (Goldman et al. 2003, Bush et al. 1989). Response errors can be due to poor memory, poor comprehension, personal norms and values but also to what has been referred to as general response sets. For example, an assessment of health status based on self-report may be confounded by a person's current mood or personality.

In the psychological literature, extensive research has been directed to describing forms of bias associated with self-report data, and to solutions for these phenomena. Examples include the use of both positively and negatively formulated items to prevent bias that is referred to as acquiescence, and the inclusion of a 'lie-scale' in order to detect subjects with the tendency to produce socially desirable answers (Smith 2004). Less is known about the potential effects of response sets on the validity of self-reports. A paradigm to study their presence was presented in 1954 by Berg and Rapaport (Berg & Rapaport 1954). They administered a questionnaire with a maximally ambiguous structure ­a questionnaire without questions, but only response options­ and found that respondents had a tendency for positive responses. Studies in The Netherlands (Van Heerden & Hoogstraten 1979) and India (Verma et al. 1980) replicated their findings despite cultural and temporal differences. If a questionnaire with response options only evokes non-random responses from respondents, the same may occur to questions that are beyond the scope of the subjects' interest or comprehension. We therefore studied whether (a) responses to an empty questionnaire are generally biased to the positive of negative and (b) whether responses are related to neuroticism and extraversion ­two personality traits associated with acquiescence­ and to social desirability. The presence of response tendencies per se would suggest the existence of what psychometrists refer to as 'constant error' (Nunnally 1976), which would be of interest mainly for descriptive studies. If these tendencies are also related to personality traits however, this suggests a 'response bias' (Nunnally 1976) (i.e. in a study on health status, the level of self-reported health status would depend on the true level of health status and the respondents' personality).

Methods

At the end of their college year, ninety-one second-year medical students of the University of Groningen were asked to participate in the study. The following questionnaires were administered:

1. The Dutch validated NEO-FFI (Costa & McCrae 1988) is a personality questionnaire consisting of 60 items to which the respondent should mark "strongly disagree", "disagree", "neutral", "agree", or "strongly agree". Of the NEO-FFI, we used the neuroticism and extraversion scale. Both traits were assessed with 12 items, both with 4 negatively formulated items.

2. A self-administered questionnaire with response options only was devised according to the description by Van Heerden and Hoogstraten (Van Heerden & Hoogstraten 1979), with a few additions. It consisted of 7 questions labeled "Question 1." to "Question 7" but where the formulation of a question was expected, no text was given. The questions had the following answering categories: 1). yes-uncertain-no, 2). true-false, 3). agree-indifferent-disagree, 4). very satisfied-satisfied-dissatisfied-very dissatisfied, 5). Always-sometimes-never, 6). 0-1-2-3-4-5-6-7-8-9-10, 7). a visual analogue scale (VAS) of 13 centimeters as follows: "I--------------I". We distinguished between questions that tap on an acquiescent response style (questions 1-3), positive affect (questions 4-6), or other (question 7). All questions were recoded so that a high score means a "positive" response.

The questionnaire was introduced as follows: "In the following questionnaire you will find a series of questions which lack content, followed by possible answers. The place where you would expect a question was deliberately left blank. We want you to mark the answer that fits the answer you would like to give to this "empty question". You don't have to think very long for each question. Please try not to skip questions, and mark only one answer per question".

3. The third questionnaire consisted of the social desirability scale of the EPQ (Eysenck et al. 1985). The questionnaire consisted of 12 items with yes and no as answering categories, with 3 negatively formulated items.

4. In addition, we asked respondents how well they were feeling today, on a scale 0-1-2-3-4-5-6-7-8-9-10, as an indicator of current well-being.

Data analysis

The pattern of answers on the questionnaire without questions was analyzed by χ² tests, in order to test whether answers were equally divided among the categories. Since Questions 6 and 7 can be considered as continuous variables, we analyzed the responses to these questions by means of a T-test with hypothesis 0 that the mean is in the middle of the answering categories.

The results of the present study were compared to the previous studies (Berg & Rapaport 1954, Verma et al. 1980). Spearman rank correlations were calculated to assess the associations between the empty questions and personality traits. In a sub sample of 28 students, a second assessment of the empty questions was made, three weeks after the original assessment.

Results

Of the ninety-one respondents, 57 were female (62.6%) and 34 were male (37.4%), and mean age was 21.6 years (minimum: 18; maximum 31). Virtually all questionnaires were completely filled in. For one respondent, one item on the questionnaire without questions was missing and was substituted by the mean score. Also, for one respondent one item on the social desirability was extrapolated. In Table 1, the responses on the questionnaire without questions is shown and, where possible, related to previous studies. For all items, significantly unequal distributions across the answering categories were found. Question 6 (0-1-2-3-4-5-6-7-8-9-10) had a mean score of 7.1 (sd. 1.8) and question 7 (visual analogue scale of 13 cm) had a mean of 9.2 centimeters (s.d. 2.4). The mean scores on question 6 and 7 were significantly higher than the middle of the scale (Question 6: T=10.9; df 90; p<0.001; Question 7: T=10.8; df 90; p<0.001 ).

The significant Spearman correlations between responses and personality traits revealed a pattern of significant negative associations with neuroticism (4 out of 7), positive associations with extraversion (2 out of 7), no associations with social desirability, and positive associations (2 out of 7) with well-being. As multiple significance testing was done (4x7=28 times), one would expect some significant correlations by chance (28x0.05=1.4). Since we found 8 significant correlations, and the directions of these correlations is in line with expectations, we consider them as meaningful and did not correct significance levels. Some differences were seen when comparing male and female respondents, but the general picture of neuroticism being associated with negative responses, extraversion and well-being with positive responses, and social desirability not being related to the responses to the empty questions, was confirmed for both sexes (Table II).

Significant correlations between the two assessments in time of the empty questionnaires were observed for 3 of the 7 questions: yes-uncertain-no (0.41; p<0.05); 0-1-2-3-4-5-6-7-8-9-10 (0.53; p<0.05); never-sometimes-always (0.40; p<0.05).

Discussion

Respondents were willing to complete a questionnaire consisting of very ambiguous items, and their responses were not random. In this situation of maximum ambiguity, respondents tended to produce positive answers, almost identical to studies performed 25 and 50 year earlier and in different countries. What we also found was that these response tendencies were related to specific personality traits, suggesting both a 'constant error' and a 'response bias'.

Interestingly, the tendency to produce positive answers was positively correlated to extraversion and negatively to neuroticism. Our findings suggest that acquiescence and positive affect, both of which may be low in neurotic subjects and high in extraverted subjects (Roccas et al. 2002), plays a role in producing the answers to the empty questions. Surprisingly, no correlations were found with respect to social desirability. Some of the responses were related to the subjects' sense of well-being. Understandably, this effect was most clearly present in the "question" very satisfied- satisfied-dissatisfied-very dissatisfied. Finally, for three of the questions a substantial test-retest correlation was found, giving support to the notion that the responses are not random but reflect a personal preference for some categories.

The results of the study should be viewed in the light of the following limitations. First, the task of providing answers to questions without content may appear artificial. Despite the fact that we introduced the questions in a neutral fashion, we do not know how respondents have experienced this task. However, the fact that all respondents except for one did give answers seems to suggest that the task was not too artificial. Second, since we used a sample of medical students, we may not have captured variables that would have been relevant for the outcome of the study, such as low socio-economic status or low education. However, the finding that even in this restricted sample associations with the response tendencies were present, suggests that this may be an even more relevant phenomenon in more heterogeneous samples. Finally, we mention the test-retest reliability of the response tendencies, which according to psychometric standards was relatively low (i.e. significant only for 3 out of 7, ranging from 0.40 to 0.53). This finding suggests that the response sets may be relatively unstable and depend also on more variable factors such as mood.

Our study provides further support that certain response sets exist that may threaten the validity of self report data, such as personality assessments or self-reported health. We found that response tendencies were correlated in a meaningful way with personality and self-reported well-being ­the chance of using negative response sets seemed to increase with increasing neuroticism and decreasing extraversion and well-being­. Although it may be argued that our assessment of self-reported well-being was limited, the fact that a similar picture in comparison to personality was observed, suggests that some bias might be present both in personality and health status assessment.

The chance of using negative response sets seems to increase with increasing neuroticism and decreasing extraversion and well-being. Remarkably, no significant correlations were found with social desirability. Together with the finding that correlations between response tendencies and personality were also found in the frequency question (never-sometimes-always), it remains diffcult to maintain that social desirability plays an important role. Instead, it seems more likely that a personal preference for some categories exists, that is independent of the content of the question. Since we found that not only our respondent's had a general tendency to give positive and agreeing answers but also that this tendency was related to personality traits, the conclusion should be that either our assessment of self-reported personality is confounded by response tendencies or that we can actually predict responses to empty questions based on a respondent's personality. We also found some stability in time for the answers to some of the empty questions, which supports the argument that some underlying tendency is present. We do not know to what extent response tendencies occur in existing questionnaires, and therefore the message is not that self-report data by themselves are unreliable. We would expect, however, bias introduced by personality differences among respondents to become larger when questions are more general, less relevant to the respondent, or vaguely stated. In the development and evaluation of questionnaires, researchers should be aware of the potential of this bias and think of ways to control for it by means of personality data.

References

Berg IA, Rapaport GM. Response bias in an unstructured questionnaire. J Psychol 1954; 38: 475-481.         [ Links ]

Bush T, Miller SR, Golden AL, Hale WE. Self-report and medical record agreement of selected medical conditions in the elderly. Am J Public Health 1989; 79(11):1554-1556.         [ Links ]

Costa PT, McCrae RR. Personality in adulthood: a six-year longitudinal study of self-reports and spouse ratings on the NEO Personality Inventory. J Pers Soc Psychol 1988; 54(5): 853-863.         [ Links ]

Eysenck SBG, Eysenck HJ, Barrett P. A revised version of the psychoticism scale. Pers Ind Diff 1985; 6: 21-29.         [ Links ]

Goldman N, Lin I, Weinstein M, Lin Y. Evaluating the quality of self-reports of hypertension and diabetes. J Clin Epidemiol 2003; 56: 148-154.         [ Links ]

Nunnally J. Psychometric Theory. New York: John Wiley & Sons; 1976.         [ Links ]

Roccas S, Sagiv L, Schwartz SH, Knafo A. The big five personality factors and personal values. Pers Soc Psychol Bull 2002; 28: 789-801.         [ Links ]

Smith PB. Acquiescent response bias as an aspect of cultural communication style. J Cross-Cult Psychol 2004; 35: 50-61.         [ Links ]

Van Heerden J, Hoogstraten J. Response tendency in a questionnaire without questions. Appl Psychol Meas 1979; 3: 117-121.         [ Links ]

Verma SK, Menon DK, Malhotra A. Response tendencies in a questionnaire without questions. Ind J Clin Psychol 1980; 7: 151-155.         [ Links ]

Adress of correspondence:
Peter de Jonge, Ph.D, methodologist
Department of Internal Medicine and Department
of Psychiatry, Hanzeplein 1, Gebouw 32
P.O.Box 30.001, 9700 RB Groningen, The Netherlands
Telephone: +3150 3619005, Fax: +3150 3619722
E-mail: p.de.jonge@med.rug.nl
THE NETHERLANDS