Elsevier

Thinking Skills and Creativity

Volume 19, March 2016, Pages 279-289
Thinking Skills and Creativity

Epistemological orientations and evidence evaluation in undergraduates

https://doi.org/10.1016/j.tsc.2016.01.002Get rights and content

Highlights

  • New measure of epistemological orientations (Justifying Conclusions Inventory, JCI).

  • JCI provides scores for absolutism, multiplism, and evaluatism.

  • Evidence that about one third of undergraduates may be in transition.

  • Justifying Conclusions Inventory can be used globally or specifically.

Abstract

Epistemological orientations and evidence evaluation abilities influence processes related to critical thinking and conclusion justification across various reasoning domains. Participants (N = 500) were presented with the Justifying Conclusions Inventory (JCI) enabling the identification of epistemological orientation groups. Cluster analysis identified four groups: Absolutists, Multiplists, Evaluativists, and Low Evaluativists. Participants also read research vignettes and responded to Research Evaluation Inventory (REI) questions addressing evidence evaluation processes related to skepticism. REI responses were significantly affected by epistemological orientation group, with Evaluativists demonstrating the most skepticism. Participants with the most education and those who had taken a methodology course also demonstrated greater skepticism. These results suggest the JCI is a defensible assay of global and domain-specific epistemic cognition. In addition, the findings herein elucidate characteristics of a transitional epistemic cognitive state which could be common in undergraduates.

Introduction

Cognitive scientists studying how people approach knowledge and knowing use the terms epistemic cognition, epistemological understanding, personal epistemology, epistemological beliefs, and epistemological orientations to describe these processes Hofer and Pintrich, 1997, King and Kitchener, 1994, King and Kitchener, 2002, King and Kitchener, 2004; Kuhn, Cheney, & Weinstock, 2000). Epistemic cognition and epistemological understanding refer to thinking processes related to knowledge acquisition in a general sense, whereas personal epistemology, epistemological beliefs, and epistemological orientations refer to the varieties of epistemological beliefs and processes in individuals (Hofer and Pintrich, 1997). Included in all of these theoretical frameworks are taxonomies of beliefs or orientations related to the nature of knowledge (e.g., certainty) and how knowledge is justified (e.g., use of evidence; the role of authorities) (King and Kitchener, 1994). In short, an individual’s epistemological orientation reflects his or her belief system about the nature of and acquisition of knowledge (Hofer and Pintrich, 2002, King and Kitchener, 2004). For example, an individual may believe there is one correct answer (a truth) and that this answer is known by certain individuals (e.g., parents, church leaders, and/or teachers).

King and Kitchener (1994) obtained evidence that epistemological orientation (EO) typically reflects one of three levels: pre-reflective, quasi-reflective, or reflective, and refer to their framework as “Reflective Judgment.” Kuhn et al. (2000) later referred to these orientations as absolutists, multiplists, and evaluativists, respectively (Kuhn et al., 2000, Kuhn and Weinstock, 2002)1. Individuals with an absolutist orientation focus on one conclusion and authority privilege to that conclusion (e.g., parents, teachers). Multiplists believe that there can be more than one conclusion; viewing various opinions as equally correct. Influencing the tendency to view all opinions as equally valid is the tendency to view different types of compelling evidence as equally valid. Of the three levels, evaluativists engage in the most sophisticated approach to evidence evaluation and logical analysis. Evaluativists have reasonably well-developed abilities pertaining to evidence evaluation (e.g., empirical evidence provides greater support for a conclusion than personal opinion). In addition, evaluativists embrace dispositions related to a more rigorous analysis of evidence and/or conclusions, including the willingness to revisit evidence and analytical processes at a later time even if that results in a revision of the conclusion. Cognitive scientists in this area have found empirical evidence suggesting that it is possible to identify the epistemological levels of individuals reliably, particularly within problem contexts (Hofer, 2004, Kuhn et al., 2000).

According to many theorists and scientists studying critical thinking, epistemic orientations and processes are crucial (Bromme, Pieschle, & Stahl, 2010; Dawson, 2008; Dwyer, Hogan & Stewart, 2014; King and Kitchener, 1994, Kuhn et al., 2000). Dwyer et al. (2014) proposed a framework integrating epistemological orientations, metacognition, and critical thinking, in which the inherent metacognitive nature of various processes related to critical thinking is clarified (also see Hogan, Dwyer, Harney, Noone, & Conway, 2015). In other words, attempts to produce a reasoned conclusion involves thinking about the nature of knowledge, evaluating potential justifications, and self-regulating one’s thinking processes – all of which are metacognitive processes (also see: Bromme et al., 2010, Dawson, 2008, King and Kitchener, 1994, Kuhn et al., 2000). The metacognitive nature of epistemological processes related to critical thinking suggests that individuals may be aware of thinking propensities, thereby enabling researchers to use a variety of approaches to better understand what thinkers are doing mentally (Hofer, 2004).

The majority of studies addressing individual differences in epistemic cognition have used qualitative approaches, necessitating content analyses of responses to ill-defined problems (Baxter Magolda, 1992, King and Kitchener, 1994). Qualitative methodologies in this literature have met rigorous methodological standards resulting in empirical outcomes that can be highly regarded (Hofer and Pintrich, 1997). The development of inventories assessing epistemic cognition may contribute to empirical and theoretical progress that has already been demonstrated. DeBacker, Crowson, Beesley, Thoma and Hestevold (2008) examined the psychometric and content validity of three existing measures of epistemic beliefs and found problems with internal consistency and conceptual operationalization, suggesting that there is a need for instruments that are conceptually and psychometrically sound (also see Pintrich, 2002).

Greene, Torney-Purta and Azvedo (2010) developed a 13-item quantitative instrument that addresses epistemic cognition in two domains, math and history. Their instrument asks participants (N = 740; middle school to graduate school) to indicate their level of agreement with statements reflecting various forms of justification (authority, personal opinion, and facts that are certain and unchangeable), producing scores reflecting justification propensities. Greene et al. (2010) obtained evidence for the psychometric integrity of their questionnaire, demonstrating the utility of a quantitative approach when used in a specific content domain (also see Sosu, 2013).

The present study addresses the need for a quantitative instrument assessing global epistemological orientations. The inventory developed and tested herein, the Justifying Conclusions Inventory (JCI), was designed to address a range of epistemological tendencies characteristic of absolutism, multiplism, and evaluativism, such as the use (or non-use) of evidence, the relevance of evidence, the process of reasoning with evidence, and justification propensities. Scores on JCI subscales were used to categorize undergraduates as to epistemological orientation. It was hypothesized that third-year and fourth-year undergraduates would be more likely than first-year or second-year students to be classified into one of the more sophisticated epistemological orientation groups, although there should be undergraduates who are more or less sophisticated regardless of their academic standing. Using the JCI to group individuals into epistemological orientations facilitates the identification of individual differences in students at similar educational levels as well as across educational levels.

Post-secondary educational experiences, broadly speaking, may promote epistemological development, continuing cognitive development that begins in childhood and adolescence (Hofer, 2004, King and Kitchener, 1994, Kuhn, 1991, Perry, 1970, Valanides and Angeli, 2005). Post-secondary course content often showcases discipline-specific epistemologies, with an emphasis on why particular justifications are valued over others, potentially contributing to epistemological development. Furthermore, students who enroll in methodology courses are explicitly exposed to knowledge acquisition strategies that produce the most valued knowledge in a particular discipline which may promote development more specifically (Hofer, 2004, Angeli and Valanides, 2009). Dampening these conclusions somewhat is the evidence that epistemological development may be gradual or uneven for some proportion of students, suggesting that the overall linear relationship between epistemological orientations and education may be noisy (Greene et al., 2010, Hammer and Elby, 2002, Hofer, 2004, King and Kitchener, 2004, Kuhn et al., 2000, Rodriguez and Cano, 2006).

To evaluate the influence of research methodology exposure on epistemological orientations, the goals of the current study include comparing students exposed to a research methodology course to those who had not been exposed. It was hypothesized that students exposed to research methodology course content would be more likely to be classified into one of the two most sophisticated EO groups (Evaluativists or Multiplists). Utilizing two samples of undergraduates (Exposure, No-Exposure) enabled the examination of the influence of specific training in research methodology on epistemological development.

To examine epistemological orientations in a scientific context, participants were asked to evaluate a research vignette and to answer questions presented in a Research Evaluation Inventory (REI). The vignette presented a description of a study on extrasensory perception: a topic chosen because it would be more familiar than other research topics, and because such a topic is known to be questionable, and therefore it should elicit reactions related to epistemological orientations. The Research Evaluation Inventory (REI) included questions addressing the validity of the source, the quality of experimental design, and whether one should just believe these claims or use evaluative processes,. It was hypothesized that participants in more sophisticated EO subgroups and participants in the exposure group would evaluate the research vignette most critically. Including evidence evaluation in a scientific context enabled the exploration of domain-specific epistemic cognition relevant to the JCI (Muis, Bendixen, & Haerle, 2006; Sosu, 2013). If the JCI results in EO groupings with participants differing in critical thinking skills relevant to a specific content domain (in this case within a research evaluation domain), then REI subscale scores should differ across EO groups. For example, individuals who lean toward absolutism may consider print material from a range of publications as authoritative, resulting in Absolutists having higher REI subscale scores for overall credibility and source credibility. Conversely, Evaluativists should have lower REI subscale scores for overall credibility and source credibility, reflecting skepticism of unknown sources and skepticism of questionable conclusions. Hence, it is hypothesized that REI means will differ systematically across EO groups, providing evidence that the JCI is a defensible assay of epistemic cognition in a specific sense, as well as in a general manner.

To summarize, the goals of the present study were (1) to examine the utility of the Justifying Conclusions Inventory as a means to obtain epistemological subscale scores and to classify undergraduates into epistemological orientation groups; (2) to explore the influence of two types of educational variables (educational level and research methodology course participation) on epistemological orientation as reflected by JCI scores and epistemological orientation groups; and (3) to explore the influence of epistemological orientation groups on evidence evaluation processes specific to a research vignette (REI subscale scores).

Section snippets

Participants

A total of 500 young adults participated: 158 first-year students (31.6%); 94 second-year (18.8%), 140 third-year (28%), 104 fourth-year (20.8%), and 4 who had graduated (0.8%). There were 368 women (73.6%) and 132 men (26.4%). Participants received partial course credit in one of several Psychology courses. Nearly half of the students were enrolled in an Introductory Psychology Course (N = 230, 46.1%), all of whom reported that they had not completed a research methodology course (“no exposure”

Principle component analysis of JCI and REI scores

Significance levels were set at .05 for all analyses. There were six missing responses out of 11,022 responses (or .05%) for the JCI, and 20 out of 9,018 (or 0.22%) for the REI: reflecting low rates of missing data. Each missing datum was replaced with the overall mean for the non-missing responses. Principal Component Analysis (PCA) was used to identify a smaller set of conceptually-meaningful factors for each inventory.

Justifying conclusions inventory (JCI)

Correlational analysis using Bartlett’s test of sphericity was

Discussion

Exploring the utility of the Justifying Conclusions Inventory (JCI) as a self-report tool for identifying epistemological orientations in undergraduates was among the goals for the present study. Factor analysis of the JCI suggested a three-factor solution, resulting in three subscales corresponding to absolutist, multiplist, and evaluativist factors. The use of cluster analysis and discriminant analysis yielded three participant subgroups corresponding to high absolutism, high multiplism, and

Conclusions

The results of the present study substantiate the utility of the JCI as a tool to assess Absolutism, Multiplism, and Evaluativism in undergraduates. Cross-validation with cognitive processes associated with the evaluation of a scientific vignette supports this claim. In addition, the findings reported herein suggest that undergraduates may not transition quickly and unambiguously to more advanced epistemic cognitive levels, but that about a third may be questioning previous perspectives while

Acknowledgements

Debra McGinnis, Ph.D., Department of Psychology, Oakland University, Rochester, MI 48309. I acknowledge Oakland University students Jason Adams, Elona Braho, Mindee Juve and Melissa Wojnaroski, who assisted with data collection. The author was supported by an Oakland University Summer Fellowship for the final data collection.

References (25)

  • J.A. Greene et al.

    Empirical evidence regarding relations among a model of epistemic and ontological cognition, academic performance, and educational level

    Journal of Educational Psychology

    (2010)
  • J.F. Hair et al.

    Cluster analysis

  • Cited by (4)

    • Cross-cultural variation in the development of folk ecological reasoning

      2018, Evolution and Human Behavior
      Citation Excerpt :

      Menominee children were more likely to reason ecologically about biological kinds than were non-Native U.S. children (Unsworth et al., 2012). This variation in folk ecological knowledge between indigenous and non-native U.S. populations has primarily been attributed to cultural differences in epistemological orientations, or beliefs about the nature and acquisition of knowledge (McGinnis, 2016; Medin & Atran, 2004; Medin, Waxman, Woodring, & Washinawatok, 2010; Unsworth et al., 2012). From a Western scientific perspective, the world operates on a linear basis of cause and effect, whereas indigenous knowledge is more likely to construe the world as consisting of a complex web of interactions (Freeman, 1992).

    View full text