Brief ReportA methodological note on ordered Q-Sort ratings
Introduction
The Q-Sort method is widely used in psychological research. This research includes comparing patients in a clinical setting (Block, 1978), evaluating distinctive and normative similarity of personality (Furr, 2008), and evaluating the relationships between personality and perception (Sherman, Nave, & Funder, 2013) to name only a few.1 This paper reports the presence of substantial item ordering effects occurring in the Q-Sort method. We begin by briefly describing the Q-Sort method and several of its advantages. Then, we examine several large datasets using Q-Sorts for item order effects. After that, relationships between Q-Sort measures and dependent variables are examined for possible attenuation of relationships due to these item order effects. Lastly, implications for future research using the Q-Sort method are discussed.
Q-Sorts afford several advantages over traditional Likert-type measurements. For one, Q-Sorts are not susceptible to many response biases that can occur in Likert-type rating such as nay saying, acquiescence, extremes responding, and midpoint responding (Block, 1978). Using the Q-Sort method it is not possible for people to simply agree with every question (acquiescence bias) or pick moderate responses for every item (midpoint responding). Rather, characteristics are sorted into a fixed distribution. The Q-Sort method is also considered to be a more taxing procedure, taking longer to complete than a Likert-type rating. Presumably this leads to more valid characterizations of the target being rated. In addition, the Q-Sort method allows for comparisons of two or more targets on a large number of characteristics using a simple profile correlation (Block, 1978).
Using Block’s (1978) procedure, a Q-Sort works as follows: The rater is given a set of cards (items), describing possible characteristics of a target. The rater then sorts each item into one of three initial categories (e.g., uncharacteristic, neutral, or characteristic). After the items are sorted into these three categories, the participant further sorts the items into a pre-specified fixed distribution for the scale that the researcher is using. Often these ratings form a quasi-normal distribution whereby only a limited number of items fit into the most extreme categories (e.g., 1 and 9) while a larger portion of the items fit into the middle categories (e.g., 5). Traditionally, the Q-Sort method was conducted by hand (manually) using index-sized cards (for more detail on the Q-Sort method see Block, 1978). In more recent research, computer programs assist with the administration of this task (e.g., Sherman et al., 2010, Sherman et al., 2012, Sherman et al., 2013). In the computer-facilitated Q-Sort data reported here, the Q-Sorter program (Riverside Accuracy Project, 2013) is used, although a web-based version is now available (Funder, Guillaume, Kumagai, Kawamoto, & Sato, 2012).
In the course of conducting a preliminary analysis on some personality data collected using the aforementioned computer program, we noticed specific item order effects in participants’ responses to a Q-Sort measure of personality. In particular, we noticed that the item variances were substantially lower for items occurring later in the Q-Sort (i.e., item order was strongly and negatively correlated with item variance). We further noticed that item order was associated with an increased likelihood of having a score near the scale midpoint, such that items coming later in the sort were more likely to be placed in the middle categories. We wondered if these patterns were simply an artifact of the dataset we collected, or if something more was happening. Was this pattern of responding unique to the content in this measure or would the same pattern be found using other measures as well? Would these patterns be found in Likert-type ratings, or was the Q-Sort method the issue? Would these patterns also be found in Q-Sorts conducted by hand or was it only an issue with the computer-facilitated Q-Sorts? The result of our inquiry is an in-depth analysis of response patterns from several thousand Q-Sort and Likert-type ratings using numerous Q-Sets2 gathered from many studies over the course of almost two decades.
This study examines the presence (or absence) of such item order effects in Q-Sort and Likert-type rating measures of personality, behaviors, and situations. We contrast the procedures used to collect each dataset in an effort to isolate the potential cause of item order effects. Then, this study examines relationships between Q-Sort measures and dependent variables, evaluating possible attenuation of these relationships due to item order effects.
Section snippets
Methods
The data presented here come from seven different studies. These include the Riverside Accuracy Project (RAP I; see Funder, 1995), the Riverside Accuracy Project – II (RAP II; see Letzring, Wells, & Funder, 2006), the Riverside Situation Project (RSP; see Sherman et al., 2010, Sherman et al., 2012, Sherman et al., 2013), and the Perceptions of the Thematic Apperception Test (TAT; see Serfass & Sherman, 2013). Data are also included from three, as of yet, unpublished studies: Amazon’s M-Turk
Results
First, item order effects on Variance were calculated. For each set of Q-Set ratings, the item standard deviations were correlated with the item numbers (e.g. 1,2,3 … 100 for the CAQ).5 Second, item order effects on item Placement were calculated. For each set of Q-Set ratings, the average absolute distance from the scale midpoint (5 for all measures used) for each item was
Discussion
This study examined data from a variety of samples using thousands of Q-Sort and Likert-type ratings of Q-Set items collected over nearly two decades. The conclusion from these data is quite clear. When items are sorted, as opposed to Likert ratings, item order is associated with item variance and the propensity to be placed near the middle categories of the sort distribution. In other words, items presented near the end of a Q-Sort rating have lower variance and are more likely to be placed in
Acknowledgments
We thank David Funder for his feedback on a previous draft of this article. The collection of some data analyzed here was supported by NIMH grant MH-42427 to David C. Funder, Principal Investigator and National Science Foundation grant BNS BCS-0642243 to David C. Funder, Principal Investigator. Any opinions, findings, conclusions, or recommendations expressed in this article are those of the individual researchers and do not necessarily reflect the views of the National Institute of Mental
References (22)
- et al.
Personality and perceptions of situations from the thematic apperception test
Journal of Research in Personality
(2013) - et al.
Properties of persons and situations related to overall and distinctive personality-behavior congruence
Journal of Research in Personality
(2012) - et al.
Situational construal is related to personality and gender
Journal of Research in Personality
(2013) - et al.
Predicting more of the people more of the time: Assessing the personality of situations
Psychological Review
(1978) The Q-sort method in personality assessment and psychiatric research
(1978)A primer on Q methodology
Operant Subjectivity
(1993)- et al.
Applied multiple regression/correlation analysis for the behavioral sciences
(2003) - et al.
The NEO personality manual
(1985) Exploring attitudes: The case for Q methodology
Health Education Research
(2005)On the accuracy of personality judgment: A realistic approach
Psychological Review
(1995)
The riverside behavioral Q-sort: A tool for the description of social behavior
Journal of Personality
Cited by (25)
Is it just about me? A comparison between individual and cultural strategies of learning from failure
2022, International Journal of Educational Research OpenCitation Excerpt :Due to the requirement to think and decide in an authentic way, students are allowed to dive deep in the subject on hand and also reflect on their true preferences, whereas with application of Likert-type surveys they may stop at the level of liking or disliking certain topics or situations. Likert-type surveys are often completed without much reflection (Serfass & Sherman, 2013) and thus provide responses of less variance, discriminatory power, or meaning (Rieber, 2020). Such problems can be avoided when Q methodology is applied, as one of its strengths is the forced sorting of statements into the sorting grid, requiring participants to rank statements relative to all other statements.
Structuring educational decisions using the multiple sorting task: An example focusing on international placements in nursing
2017, Nurse Education in PracticeCitation Excerpt :Card sorting is an interesting, cost-effective and fun way to learn about how people think. Its origins can be found in personal construct theory and repertory grid technique (Winter 2013), and the Q-sort (Ellingsen et al., 2010; Serfass and Sherman, 2013). Sixsmith (1986) noted how the MST could also be used within a phenomenological framework to study individual accounts and personal meanings.
A situational construal approach to healthcare experiences
2015, Social Science and MedicineCitation Excerpt :In the present studies, participants instead rated each RSQ item on a 9-point, Likert–type scale (1 = extremely uncharacteristic, 9 = extremely characteristic). Block (1957) compared normative Likert ratings to ipsative Q-sort ratings and described the results of the two as “fully equivalent” (p. 52), although others have found that the Q-sort approach reduces item variance for items appearing towards the end of the scale (Serfass and Sherman, 2013), and Likert ratings may produce a positivity bias on RSQ items (Frascona, 2014). Concerns over the time and specialized programming required to complete a Q-sort online outweighed concerns over a broad positivity bias.
Predicting interpersonal behavior using the Inventory of Individual Differences in the Lexicon (IIDL)
2014, Journal of Research in PersonalityCitation Excerpt :Research assistants used a Q-sort computer program to rate the degree in which each RBQ item was characteristic of the behavior exhibited in that situation using nine categories (1 = extremely uncharacteristic, 9 = extremely characteristic) forming a forced-choice, quasi-normal distribution. Because data collected using Q-sorts may be susceptible to item order effects (Serfass & Sherman, 2013a), we calculated two statistics for the 24 RBQ items: the order effects on variance (i.e., the correlation between the item number and the standard deviation for each item) and item placement (the average absolute distance from the midpoint which reflects the tendency for a coder to place an item near the midpoint). Both effects were medium, r = −.39 (variance) and r = −.40 (placement).
To trust or to restrict?–mapping professional perspectives on intelligence powers and oversight in the Netherlands using Q-methodology
2024, Intelligence and National Security