As a general measurement method in psychology, self-report scales have long been used. In such scales, the respondent rates an item by checking a numerical response on a single item or multiple items. While self-report scales have the advantage of being easy to use and providing large amounts of data, they also have several problems. One of these is response validity. As they are self-reported, the answers do not necessarily correspond to objective facts but only to experiences, behaviors, attitudes, and knowledge as perceived by the respondent. Attitudes deserve particular attention. Attitudes are initially unobservable and must be inferred through questionnaires or verbalized opinions. However, depending on the question, opinions might not accurately reflect attitudes. For example, when we measure attitudes such as intention to help, compassion, and gratitude using a Likert scale, we may be unsure whether the answer reflects the true attitude because the respondent simply chooses one of the options presented (e.g., [1]). Various factors, such as social desirability and the presence of others, can affect the choice. In addition, as self-reporting is a method that relies on introspection, it may be difficult to measure attitudes of which respondents themselves are unaware. The disadvantages of the response format, which is merely a choice, must be overcome while retaining the advantages of self-report scales. Experimental and observational studies of behavior, however, require a great deal of time and effort. The purpose of the present study was to develop a new response format for a self-report scale that can substantially measure respondents’ willingness to incur a cost, making it easier to obtain results than previously possible only through laboratory experiments.
Oda & Hiraishi [2] developed the ‘checkbox method’ to address the above problems. In this method, participants check boxes in a grid of 10 × 10 checkboxes on a web page by pointing the mouse on each box and clicking. Each box is numbered from 1 to 100, from left to right, and it is only possible to check them in order, starting with the smallest number. For example, to answer 7, the participant must click seven times in the correct order. Compared to simply selecting a number or moving a slider to a desired position, checking the boxes is more tedious and time-consuming because participants have to keep clicking until they get to the number they want to answer. As it does not require a material cost such as money, the checkbox method is a convenient method of measuring willingness to incur a cost. Oda & Hiraishi [2] replicated Ohtsubo and Yagi [3], who tested the prediction that the more one values a relationship, the greater the cost one will pay to make an apology when reconciliation is needed. Ohtsubo and Yagi [3] asked participants to imagine that they had committed an interpersonal transgression against one of their real friends. The cost of apology was measured by the strength of their willingness to suffer some inconvenience (e.g., cancellation of an important meeting to make an apology as soon as possible). The results supported the prediction; participants reported that they were willing to make a more costly apology to more valuable partners than to less valuable ones. Ohtsubo and Yagi [3] employed hypothetical situations but could not directly measure the costs of making an apology. Oda & Hiraishi [2] measured willingness to apologize for an imagined interpersonal transgression to a participant’s real friend by counting the number of boxes participants checked. The primary factor (among others) affecting the number of boxes checked was the instrumentality of the friend, replicating the findings of Ohtsubo and Yagi [3]. In addition, the cost of the apology was higher in a stained book scenario than in a no-show scenario, which was also consistent with Ohtsubo and Yagi [3]. The results indicated the usefulness of the checkbox method as a measure of willingness to incur costs.
The checkbox method, which is costly to complete, is particularly useful for quantitatively measuring willingness to help or to express gratitude. Even in contexts not directly related to costs, such as knowledge, facts, and evaluations, we expect the checkbox method to yield more sincere responses than a simple multiple choice. Satisficing, a behavior in which participants do not devote an appropriate amount of attentional resources when answering questions, is a serious problem, especially in online surveys, and methods to prevent it have been proposed (e.g., [4]). The checkbox method could provide a solution. Before that, it needs to be tested whether the checkbox method of measurement shows similar results to traditional methods such as the Likert scale in contexts not directly related to cost. However, no study has compared Likert scale responses with a checkbox scale in a context unrelated to cost.
In this study, we asked participants to answer both a Likert scale and a checkbox scale about the Big Five personality traits and explored relationships between the results. We employed the Big Five traits because it is a well-established basic framework in personality psychology, on which there have been many studies. As rating one’s own personality traits is not a statement of attitude, the cost to participants when responding (i.e., the number of boxes they check) is not related to the rating level. By comparing the Likert scale scores with those of the checkbox scale for such questions, it is possible to examine the extent to which traditional measurement results can be replicated by the checkbox method.