Accounting for variation during experimental design.

In the design phase of a biological investigation, a researcher identifies sources of variation and attempts to control for sources of unwanted variation. In the laboratory, researchers often try to use model organisms with very similar genetic backgrounds to minimize the influence of genetic and/or phenotypic variation on the experimental outcome. If a researcher aims to leverage the genetic and/or phenotypic variation of a population in their study, they may implement a large sample size to adequately capture this variation. To minimize unwanted environmental variation, one might place control and treatment samples side-by-side in a greenhouse or animal housing facility, with the intent of making environmental conditions (e.g., exposure to sunlight) as similar as possible between the two samples. When experimental variation (sometimes termed measurement error) is a concern, researchers may increase the number of technical replicates performed and average the measurement values or use a more precise instrument to minimize the impact of unwanted experimental variation. The variation in the measured outcome detected in a biological investigation is the cumulative effect of all of these types of variation (i.e. genetic variation, environmental variation, experimental variation) and any additional unexplained variation [19,20]. For each investigation they conduct, biological researchers must consider the sources of variation that are likely to impact the experimental outcome and which sources of variation are important to model or control so the results can be generalizable, yet precise.

Undergraduate students’ understanding of variation in the context of biological experimental design and data analysis is incompletely characterized, despite variation being inherent to biological experimentation. Studies that have investigated students’ ability to design experiments to answer biological questions suggest that students at many levels, from primary to undergraduate, fail to consider variation when designing an experiment [10,21]. Identifying that organisms within a population vary on a genetic or phenotypic level is a reported area of difficulty for students, evidenced by claims that a certain sample of experimental subjects will eliminate naturally occurring variability between subjects [6,10,21]. Additionally, designing experiments that account for sources of variability can be challenging [10,21]. Many students do not consider the amount of phenotypic variation in a population. When dealing with a population with a large amount of phenotypic variation, many students do not design experiments to address this variation by sampling a large number of individuals and by replicating the treatment conditions [10]. Measurement inaccuracy is another potential source of variation in a biological investigation that students can more easily identify [22].

Generating and interpreting representations of variation.

Variation is equally central to the analysis of data after it has been collected. A first step in data analysis is often the representation of variation in the measured outcome. This may entail visual or graphical representations of data to examine the distribution and spread of the data, or the calculation of summary statistics that mathematically represent variation in the data (e.g. standard deviation, variance, or standard error of the mean) [23,24]. Some graphical representations (e.g. histograms) typically allow for easier visualization of variation in a data set than others (e.g. bar plots). Similarly, different summary statistics provide different information about the variation in a population. For example, the standard deviation provides information about the amount of variation in the measured outcome, whereas the standard error of the mean provides information about the precision of the estimate. An investigator will choose the most appropriate representation of variation for the research question under investigation.

Students who are instructed on algorithmic methods for determining measures of central tendency and amount of variation in the data may prioritize algorithmic ‘plug and chug’ style thinking over the conceptual ideas that summary statistics signify (e.g. the concept of representativeness) [25,26]. When pre-instruction college biology students were asked to construct a graph of biological data to support a claim, many students did not calculate and plot standard summary statistics (e.g. group means) on their graphs and instead chose to plot individual replicate values or the sum of the replicate values [27]. In this case, it would be more appropriate to draw generalizable conclusions from data in aggregate form by calculating and comparing means to account for variation between individual data points. After instruction featuring quantitative thinking, nearly all students constructed graphs that plotted calculated quantities (e.g. the mean) [27]. Though constructing appropriate representations of variation can be challenging for some students, these concepts are not inherently esoteric. Primary school students are able to reason about these summary statistics without explicit statistical instruction on how these values are calculated [9,26].

Considering variation in the statistical analysis of biological data.

The statistical analysis of data, an essential and informative component of most biological investigations, requires consideration of variation [20]. Statistics are used to measure, model, and control for variation in an investigation to make predictions and explanations about the phenomenon being studied [19,20]. An understanding of the sources of variation present in a biological experiment and the methods that have been implemented to account for said variation is essential so that statistical techniques can be applied appropriately to model and explain the data.

Using statistics to test hypotheses is a nearly ubiquitous practice of biologists (and other scientists), albeit one in which errors are often made. Statistical tests and resulting p-values are often misused and misinterpreted by both students and experts [18,28–32]. The majority of these studies characterizing students’ ideas about the statistical analysis of data have done so in a non-biological context [28,31,33]. Comparatively, fewer studies have described students’ ideas about applying and interpreting statistical tests on biological data. Specific characterization of students’ understanding of statistical analysis of biological data is important because students may not transfer knowledge across disciplinary contexts [34].

The transfer of statistical knowledge to biological contexts requires students to recall mathematical concepts and manipulations and use these ideas in a novel setting to make sense of biological data. This process of transfer requires students to practice interdisciplinary thinking and translate between types of representations (mathematical/numerical to verbal/conceptual) to draw conclusions from data [34,35]. BioVEDA assessment items leverage multiple types of representations (pictorial, mathematical, graphical, and verbal) to query students’ understanding of variation as it relates to the statistical interpretation of data.

Small-scale pilot administration to assess quality of distractors.

The initial draft of the BioVEDA assessment was administered to post-second-semester students in Spring 2018. Student responses (N = 152) were analyzed to determine the proportion of students who correctly answered each item and very difficult items (fewer than 10% of students answered correctly) were discarded. Other items were eliminated on the basis of the quality of distractor options, such that all of the remaining questions had distractor options that were chosen by at least 5% of students. Items that we wished to include on the next version of the BioVEDA assessment that did not perform well (e.g. items that targeted prioritized concepts but were written in such a way that certain distractor answers were not chosen by any students) were revised.

Large-scale assessment administration to assess item difficulty and quality of distractors.

Sequential versions of the BioVEDA assessment items that remained after expert review, the student think-aloud interviews, and the pilot study item analysis were administered to pre-first-semester students in the Fall 2018 (N_FA18 = 138) and Spring 2019 (N_SP19 = 292). Student responses on the assessment were examined to determine 1) the distribution of student responses amongst the answer options, 2) the proportion of students who answered the item correctly, and 3) how well the item distinguished between the top 25% and the bottom 25% of students [43,44]. Items that did not perform well (e.g., had distractor options that were not selected, were answered incorrectly by more than 90% of students, or did not discriminate well between high- and low-performing students) were dropped or revised for subsequent administrations. Two items underwent revision after the Fall 2018 semester, and one of those items underwent an additional revision after the Spring 2019 semester. The final version of the BioVEDA assessment contains 16 items. The assessment questions are included in S1 File.

Expert review.

To provide evidence that the BioVEDA assessment appropriately represents the specified knowledge domain, the BioVEDA questions were submitted to four faculty members for expert review (one with a Ph.D. in learning sciences and a Ph.D. in biology, one with a Ph.D. in biology with expertise in biostatistics, and two with Ph.Ds. in biology with expertise in teaching science process skills in biology CURE laboratories). These faculty evaluated the questions on the basis of 1) the importance of the concept being assessed, 2) the scientific and statistical accuracy of the question stem and answer options, and 3) the clarity of the phrasing in the question stem and answer options. Questions were revised for accuracy and clarity following expert review and used in subsequent assessment administrations.

Student think-aloud interviews.

Student think-aloud interviews were used to determine that students are correctly interpreting the BioVEDA questions and selecting their answers based on the intended thought processes (as opposed to answering based on unintended non-content clues such as choice of phrasing). Think-aloud interviews were conducted with twenty students from a first-semester introductory biology lecture course in the Fall 2018 semester. Students who are enrolled in this course are typically in their freshman or sophomore year and are at a similar level to the students in the first-semester laboratory course. Students were selected on a volunteer basis, and they did not receive compensation for their participation. Each student was asked to complete five of the multiple-choice questions and to describe their thought process and reasoning out loud. Each question was answered by at least 5 students. Students were prompted to explain why the answer they selected was the best option, and why the other options were not optimal. Students were asked if there were any phrases or features of the included diagrams or graphs that were unfamiliar or unclear. Any phrases that were not familiar or unclear to multiple students were revised before the next stage of validation. At least 80% of the students interpreted the question stem and answer options as the authors intended for all of questions used for interviews. One area of difficulty that was reported and subsequently revised was the inclusion of the mathematical expression for a t-statistic. The initial version included standard mathematical symbols that some students reported to be unfamiliar (e.g. to indicate the sample mean), and the revised version included words or phrases instead of symbolic representations (e.g. mean instead of ).

Confirmatory factor analysis.

The BioVEDA assessment was designed to measure a single dominant ability: students’ understanding of variation as it applies to a biological investigation. To investigate the dimensional nature of the BioVEDA assessment, we assessed the fit of confirmatory factor analysis models to the data. This analysis was conducted on post-second-semester students’ dichotomously scored data collected during the Spring 2019 and Fall 2019 semesters (N_SP19 = 109, N_FA19 = 254, N_Total = 354). Given the low number of factors being tested (one or two) and the number of items loading on each factor (16 for the one-factor model, 7 and 9 for the two-factor model), the sample size has been generally recognized by others to be sufficient to perform factor analysis [47]. Post-second-semester student responses were used for this analysis because students’ answer selections were presumed to be less influenced by guessing than on the pre-first-semester administration (when students are less likely to have an accurate understanding of the concepts assessed by the BioVEDA test). Models were fit using the ‘lavaan’ package version 0.6–5 in R version 3.6.3 [48,49]. Weighted Least Squares (WLS) was used to estimate the model parameters. Latent factors were standardized, allowing for free estimation of all factor loadings. Model fit was assessed using three model-fit statistics (the Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), and the Root Mean Square Error of Approximation (RMSEA)). We compared the fit of a unidimensional model and a two-dimensional model using a Chi-square test with the null hypothesis that the models fit equally well, and the outcome of the Chi-square test was used to select the best fitting model.

Item response theory analysis.

Item Response Theory (IRT) was used to examine the degree to which the BioVEDA assessment appropriately measures the knowledge domain of ‘variation in biological investigations’ that we aim to measure. A Rasch analysis was performed on the post-second-semester student dichotomously scored responses (N = 354) to determine person ability and item difficulty scores. Models fitted to the data under IRT assume that test performance is due to person abilities and various characteristics of the individual item, including how difficult the item is, how well an item discriminates between respondents of high and low ability, and the extent to which responses are due to guessing [43,50–52]. Rasch models comprise a category of IRT models that assume the observed performance on an assessment is a product of person ability and only one item characteristic: the difficulty of the items. The person ability score takes into account a respondent’s performance on items of different difficulty, and all items are not presumed to be equal in difficulty (as is true when comparing raw percentage or sum scores from a multiple-choice assessment).

A unidimensional Rasch model was fitted to the data using the ‘mirt’ package version 1.32.1 in R version 3.6.3 [48,53]. Model fit was assessed by examining item and person infit and outfit statistics. The Rasch model was then used to compute ability scores for each respondent and difficulty estimates for each item to determine if the items can adequately capture the ability range of the respondent population being measured.

Assessing the suitability of using raw BioVEDA scores to indicate ability.

We recognize that researchers and instructors may be interested in using the BioVEDA assessment to evaluate students’ understanding of variation in biological investigations but may not have the time or the ability to conduct a Rasch analysis to generate person ability measures. To determine whether percentage scores could be used as an alternative to Rasch ability measures, a Pearson’s correlation was computed between respondents’ percent correct scores and their person ability scores calculated using the Rasch model. High correlation between the two values would indicate that the percentage scores could be used to report on students’ performance on the assessment [13,54].

Known-groups comparison.

A known-groups comparison was used to determine whether groups that should logically differ in their performance do indeed show differential performance on the assessment [55]. Graduate TAs (N_TA = 32) have more experience and instruction on experimental design and statistical analysis of data than the undergraduate students they teach, so they can be expected to perform better on the assessment. Similarly, students who have completed two semesters of biology laboratory courses (N_Sem2 = 354, data from Spring 2019 and Fall 2019) featuring instruction on variation in biological investigations can be expected to perform better than students who are just beginning their first semester of a biology laboratory course (N_Sem1 = 384, data from Fall 2019 and Spring 2020). Therefore, BioVEDA assessment scores were compared across these three groups of respondents (TAs, post-second-semester students, and pre-first-semester students). Samples of students were selected such that each assessment response was from one unique student, satisfying the assumption of sample independence. Raw percentage scores were compared across groups using a one-way ANOVA. Post-hoc comparisons of individual groups were done using Tukey’s HSD. We used an alpha level of .05 for all statistical tests.

Analysis of learning gains.

The BioVEDA assessment is intended to be used to detect changes in learning over time, perhaps in response to instruction. The curriculum for both the first- and second-semester lab courses contain several laboratory exercises that require students to consider and account for variation. Participation in these courses is thus expected to elevate students’ understanding of variation in biological investigations. A paired sample t-test on students’ (N = 282) pre-first-semester and post-second-semester assessment scores was used to determine whether the BioVEDA assessment can be used to detect learning gains (e.g. those assumed to occur after participating in this two-semester lab course series).

Rasch test information, standard error, and reliability.

One advantage of using Item Response Theory (e.g. the Rasch model described above) is that reliability is not measured as a uniform index for all possible respondents. Instead, reliability/precision is measured as a function of person ability, with the assumption that measurement precision is not constant across all ability levels. To describe the reliability/precision of BioVEDA assessment scores, we calculate the test information function and conditional standard error of measurement function using the Rasch model specified above. The Rasch model can also be used to calculate a Rasch reliability index (also termed the Person reliability). The Rasch person separation reliability index can be interpreted similarly to the Cronbach’s alpha or KR-20 indices of internal consistency: values range from 0–1, with higher values indicating higher internal consistency [56,57]. The Rasch person reliability index for the BioVEDA assessment is .68, which is reasonable for an assessment measuring a broad conceptual domain [57,58].

Cronbach’s alpha and item-total correlations.

Cronbach’s alpha is commonly used, often in incorrect ways, as a measure of assessment reliability [58]. Cronbach’s alpha is best interpreted as a measure of internal consistency; a measure of the degree to which all items on an assessment elicit a similar response pattern [58,59]. However, the BioVEDA assessment is intended to test a range of knowledge related to variation as it applies to experimental design. This is a broad topic and by necessity covers a broad range of experimental design and data analysis practices (Table 2). Therefore, such assessments would not be expected to yield very high alpha values which might be expected of attitudinal constructs [58]. Despite these considerations, the Cronbach’s alpha for the post-second-semester student responses on the BioVEDA assessment is .69 (95% CI = .64, .74) which has been reported as falling in the moderate to acceptable range and is on par with other multiple-choice concept assessments [58].

Item-whole correlations are reported in S2 File. Both the raw correlation (correlation of the focal item with the entire scale, including itself), and the correlation with the remaining items in the scale (excluding the focal item) are reported.

Consistency across semesters.

The BioVEDA assessment should enable instructors and researchers to consistently measure students’ understanding regardless of when the test is given. Test/retest strategies are often used to demonstrate reliability of test scores over time. However, this method cannot be employed in the context of this study as students begin receiving instruction on the topics assessed by the BioVEDA test immediately after the semester begins. The instruction students receive is assumed to affect their understanding of variation in biological investigations, which is assumed to affect their score on the BioVEDA assessment. Therefore, we implemented an alternative strategy to investigate the stability of BioVEDA assessment scores across administrations. The proportion of students who answer each item correctly was compared between the Spring 2019 and Fall 2019 second-semester course administrations by computing a Pearson’s correlation. The mean ability score of students in each semester was also compared using a two-tailed t-test.

The BioVEDA assessment appears unidimensional.

The BioVEDA assessment was designed to measure one dominant ability: students’ understanding of variation in a biological investigation. Confirmatory factor analysis models were fitted to the data to explore the dimensional structure of the assessment. The unidimensional model (including the final versions of the 16 multiple-choice questions) specified that all questions load on a single factor, hereafter termed ‘Understanding of Variation.’ The model fit is acceptable, with a CFI of .93, a TLI of .92, and an RMSEA of .04 (90% CI = .02, .05). All indicators have significant loadings, and the standardized factor loadings are positive, ranging from .21 to .80 (Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Factor loadings for the unidimensional confirmatory factor analysis.

https://doi.org/10.1371/journal.pone.0236098.t003

Although the BioVEDA assessment is designed to measure one dominant ability, students’ understanding of variation in a biological investigation can be organized into two subdomains: variation in the design phase and variation in the data analysis phase. The assessment is designed to include questions that interrogate respondents’ understanding of variation in both of these phases of a biological investigation (Table 1). Therefore, it is plausible that the assessment could be measuring two dimensions, namely, ‘Design’ and ‘Data Analysis.’ Items 1 through 7 relate to experimental design, and items 8 through 16 relate to data analysis, so a two-dimensional model was specified to reflect this structure. The fit statistics for the two-dimensional model are similar to those of the unidimensional model (CFI = .93, TLI = .92, RMSEA = .04 (90% CI = .02, .05). A scaled Chi-squared difference test indicates that the models fit the data similarly well (χ²(1) = 2.6, p = .11). Since the model fit indices are similar between the unidimensional and two-dimensional models, we default to the more parsimonious (unidimensional) model to describe the structure of the assessment. These results support the theoretical design of the assessment and indicate that the 16 items measure one underlying dominant ability that represents ‘understanding of variation in experimental design and data analysis.’

The BioVEDA assessment adequately measures student ability.

The results of the confirmatory factor analysis indicate that the 16 BioVEDA items collectively measure a single dominant ability or dimension, so a unidimensional Rasch model was fitted to the data (N = 354 post-second-semester student responses) to estimate item difficulty and person ability level. Item difficulty was estimated for each item (Table 4 shows difficulty estimates and standard error). Model fit was assessed by examining item mean-square infit and outfit statistics (Table 4), and person mean-square infit and outfit statistics. Mean-square infit/outfit values less than 0.5 indicate model overfit and those greater than 1.5 indicate model underfit; ideal values are close to 1.0 [60]. The average mean-square infit is 0.93, and the average mean-square outfit is 0.89. All of the 16 items meet criteria for acceptable fit [60], and therefore do not misfit the model. The average person mean-square infit is 0.91 (with 98% of respondents falling in the acceptable range), and the average person mean-square outfit is 0.89 (with 92% of respondents falling in the acceptable range) [60]. No respondents had infit or outfit statistics larger than 2.0 (above which may distort or degrade the model) [60].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Item difficulty and fit statistics for the Rasch model.

https://doi.org/10.1371/journal.pone.0236098.t004

Ability estimates for all respondents were computed using the Rasch model, and displayed beside the item difficulty values in a Wright map [57] (Fig 1). The mean score (for both person ability and item difficulty) is scaled to 0 and indicated by a dashed line. Respondents are displayed on the left, and items are plotted on the right. Items that are above the dashed line are more difficult than the average item, and items below the line are less difficult than average. Similarly, respondents with higher than average ability are above the dashed line, while respondents with lower than average ability are below the dashed line. Respondents and items are plotted on the same scale such that the position of an item on the scale indicates that a person of that ability has a 50% chance of answering that item correctly. A person of average ability (ability = 0) has a greater than 50% chance of answering items below the dashed line correctly, and a less than 50% chance of answering items above the dashed line correctly.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Wright map of respondent ability and item difficulty calculated using the Rasch model.

N = 354 respondents. Respondents are plotted on the left in a binned histogram, and items are plotted on the right. The axis for both respondent ability and item difficulty (in logits) is noted on the right. Dashed line indicates the mean respondent ability.

https://doi.org/10.1371/journal.pone.0236098.g001

Person ability and item difficulty measures calculated using the Rasch model can be compared to evaluate the test targeting of the assessment. A well-targeted assessment will have items that range in difficulty such that they can adequately capture the ability of the intended respondent population. A rule of thumb for comparing the mean item difficulty (M = -0.48) and the mean person ability (M = 0.00) suggests there should be less than a 1.00 logit difference between the mean item difficulty and mean person ability, which is the case for this data set [61]. Rasch measures can also be used to examine potential floor or ceiling effects by assessing the percentage of respondents at the highest or lowest measure of the instrument [62]. There are no respondents in this sample that score below the lowest measure of the instrument (minimum item difficulty = -1.70), suggesting that there is no floor effect with this assessment in this sample of respondents. A small proportion of respondents (9%, N = 32) score above the highest measure of the instrument (maximum item difficulty = 0.95), indicating that there may be a ceiling effect for this sample of respondents.

Raw BioVEDA scores are a reasonable indicator of person ability.

A Pearson’s correlation was computed between the respondents’ percent correct score and the person ability score calculated using the Rasch model. The correlation is very high (r = .99, p < .001), indicating that the raw percentage scores can provide a satisfactory indicator of students’ understanding of variation in a biological investigation.

The BioVEDA assessment can distinguish between naïve and advanced respondents.

The BioVEDA assessment is intended to be used to detect differences in content knowledge between individuals. To investigate the extent to which BioVEDA assessment scores can discriminate between respondents who should vary in their ability, BioVEDA scores were compared between first-semester students (pre-instruction), second-semester students (post-instruction), and graduate teaching assistants that served as the primary instructors for the first-semester lab course. Compared to undergraduate students, graduate students are assumed to have more advanced understanding of variation as it relates to a biological investigation, since they have completed more coursework in biology and have direct experience with conducting biological investigations. Similarly, undergraduate students who have completed two semesters of introductory biology lab courses are expected to have more advanced understanding of variation in biological investigations than students who have not yet completed any semesters of biology lab courses. As expected, BioVEDA scores differ significantly across the three groups of respondents (M_Sem1 = 51%, M_Sem2 = 59%, M_TA = 76%; one-way ANOVA, F(2, 767) = 43.72, p < .0001) (Fig 2). Post-hoc comparisons via Tukey’s HSD indicate that the differences between each group are also significant (Fig 2). The magnitude of the difference between first- and second-semester students (Cohen’s d = 0.5) is smaller than the difference between second-semester students and graduate TAs (Cohen’s d = 0.9). This aligns with the differences in experience and knowledge that one would expect for these groups. These data indicate that the BioVEDA assessment can be used to detect differences in respondents’ understanding of variation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The BioVEDA assessment can distinguish between groups with different ability levels.

Violin plots with inlaid box plots of BioVEDA scores (% correct). Pre-first-semester students are shown in light gray, post-second-semester students are shown in medium gray, and TAs are shown in dark gray Horizontal bar in the middle of the box indicates the median score for each group. Large black dot in center of boxplot indicates the mean score for each group. *** = p < .0001. Effect size shown as Cohen’s d. The table below the graph shows summary statistics of each group’s assessment performance (SD = Standard Deviation, SEM = Standard Error of the Mean).

https://doi.org/10.1371/journal.pone.0236098.g002

We envision that the BioVEDA assessment can be used to measure individual changes in learning over time. To determine if BioVEDA assessment scores can detect changes in learning that occur after instruction on the topics assessed in the test, we examined students’ pre-first-semester and post-second-semester scores using a paired sample t-test (N = 282). We detect a significant increase in students’ BioVEDA scores after two semesters of instruction (M_PreSem1 = 51%, M_PostSem2 = 61%; t(281) = 8.88, p < .0001). The difference between the pre-first-semester and post-second-semester scores represents a medium sized effect (Cohen’s d = 0.53). These data indicate that BioVEDA scores can be used to detect individual changes in learning when administered as a pre- and post- test.

Reliability/Precision from the Rasch model.

Item Response Theory models (e.g. Rasch models) can be used to describe the reliability/precision of measurement of a test. Measurement precision/reliability that is determined using Item Response Theory is expressed as a function of ability. It is not assumed that the test will be equally reliable/precise for persons at different ability measures. Test precision can be interpreted by examining both the test information function (Fig 3, solid black line) and the standard error of measurement function (Fig 3, dashed gray line). Test measurements are most precise/reliable when test information is high and standard error of measurement is low. The BioVEDA assessment provides the most information for people of near-average ability levels (ability of approximately 0) (Fig 3). The standard error of the estimate is lowest for individuals of average ability, indicating that estimates of these persons’ ability level are the most precise (Fig 3). The BioVEDA assessment provides less precise information for people of very high or very low ability measures.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Test information and standard error calculated using the Rasch model.

The black solid line indicates test information for persons at the indicated ability measure, and the gray dashed line indicates the standard error of the estimate for persons at the indicated ability measure. The gray shaded area under the information curve indicates the ability range of respondents in our sample.

https://doi.org/10.1371/journal.pone.0236098.g003

The BioVEDA assessment is consistent across semesters.

In order for an assessment to be used to measure student content knowledge in different contexts, the assessment must produce consistent results. We evaluated the consistency of BioVEDA scores across test administrations at the question and whole-assessment level. Student performance on the BioVEDA items was compared between the Spring 2019 and Fall 2019 post-second-semester test administrations. All test administrations were done in the same laboratory course, and students from different semesters have similar preparation and thus ability on the BioVEDA assessment. Indeed, the correlation between the proportion of correct responses on each item in each semester is high (Pearson’s r = 0.98, p < .0001). This indicates that the BioVEDA assessment produces consistent results across semesters. Additionally, comparing the mean percent correct score across semesters, there were no significant differences across undergraduate student total scores (M_SP19 = 61%, SEM_SP19 = 2%, M_FA19 = 58%, SEM_FA19 = 1%; two-tailed t-test, t(201) = 0.94, p = .34, Cohen’s d = 0.11), which provides additional evidence that BioVEDA assessment scores are consistent.