Validating an adapted questionnaire to determine perceptions of the clinical learning environment

The clinical learning environment is a key component of undergraduate medical students’ curriculum. At our institution there was a need for an electronic instrument to evaluate student perceptions of their clinical learning environment. Methodology The Uppsala Instrument, which had previously been used to improve student perceptions of their clinical learning environments, was adapted for the context of a South African Academic Hospital by a panel of experts and then completed by a group of students. The results were then used to determine the validity and reliability of the instrument using principal component analysis. The instrument proved to be highly reliable at measuring a single underlying component, ‘ perception of the clinical learning environment ‘, and due to this, further determination of validity was limited. The ﬁndings were similar to those reached by other authors’ studies of instruments evaluating the clinical learning environment in that most variance in responses were attributable to a single component. The fact that the instrument measures the clinical environment as a whole is useful, however it does not allow for deeper analysis of educational constructs within the clinical learning environment.


Introduction
The clinical learning environment (CLE) forms an important part of all health sciences curricula. Learning in the clinical learning environment differs from the formal curriculum: there is no clear timetable, with learning often happening in an unstructured way when as the opportunity presents (Swanwick, 2014).
Through obtaining student evaluations of the clinical learning environment, clinical educators can enhance the curricula of institutions (AlHaqwi, Kuntze, & van der Molen, 2014;Snell et al., 2000). Prior to this study, our institution did not have an evaluation instrument aimed at evaluating student perceptions of the CLE.
Various measuring instruments have been developed for the evaluation of clinical learning environments, as well as clinical teaching (AlHaqwi et al., 2014;Stalmeijer, Dolmans, Wolfhagen, Muijtjens, & Scherpbier, 2008), however, there is no evidence that any of these are objectively superior to others (Beckman, Ghosh, Cook, Erwin, & Mandrekar, 2004). In terms of selecting a particular evaluation instrument, previous researchers have suggested that the instrument should be "consistent with the culture of the organization, acceptable to teachers, easy to administer, and applicable to all levels of teachers" (Nation, Carmichael, Fidler, & Violato, 2011).
It is important to acknowledge that the terms "clinical teaching" and "clinical learning environment" are used somewhat interchangeably in the literature (AlHaqwi et al., 2014), and clear, widely accepted definitions for either are not forthcoming. For the purpose of this study, we refer to the clinical learning environment as a concept that encompasses clinical teaching, but also the "quality of supervision, the quality of teachers and facilities and atmosphere" (Boor et al., 2007, p. 27).
Some of the difficulties that have been observed when evaluating clinical teaching is that many clinical teachers do not supervise enough students at one time or do not supervise those same students over a sufficiently long period of time in order to be evaluated accurately (Kihlberg, Perzon, Gedeborg, Blomqvist, & Johansson, 2011). This is certainly the case at our institution where students rotate through various departments and wards, and have numerous teachers, each for a relatively short time periods. Due to these difficulties, some authors have elected to evaluate clinical rotations (or departments) using clinical teaching evaluation instruments (Kihlberg et al., 2011) in a context similar to where clinical learning environment evaluation instruments have been used (Boor et al., 2007) When using an instrument to gather information, the length of a questionnaire has been found to be inversely correlated with the response rate, therefore this was taken into consideration when selecting an evaluation instrument suitable (Bogen, 1994). For example, the CLEQ consists of 40 items, whereas the instrument developed for use at Kotze K, Archer E MedEdPublish https://doi.org/10.15694/mep.2017.000089 Page | 3 Uppsala University, Sweden, consists of only 10 items (Kihlberg et al., 2011).
Ultimately, the decision was made to use the latter instrument mainly because of its focus, length, and the success which the authors reported they had with improved student ratings of various departments, after providing the results of the instrument to the relevant departments and faculty leaders (Lewisson, Hellgren, & Johansson, 2013).
The Uppsala Instrument is based on the Stanford Faculty Development Program Clinical Teaching Framework (SFDPCTF), which has been extensively researched with large sample sizes at different sites, and found to be both valid, and reliable (Nation et al., 2011). One limitation of the Uppsala Instrument, however, is that, to our knowledge, it has not been tested for validity and reliability since its adaptation (Kihlberg et al., 2011).
Each of the 10 items that forms part of the Uppsala Instrument aims to assess one or more domains (interchangeably referred to by the authors as "behavioural anchors") (Kihlberg et al., 2011). It is worth noting here that the meaningfulness of the underlying domains for instruments on the clinical learning environment and clinical teaching has been questioned before (AlHaqwi et al., 2014), and that the principal component analysis of at least one other instrument has failed to support this notion entirely. Instead it was found that the instrument in question had only one underlying dimension (Boor et al., 2007).
The 10 questions originally used in the Uppsala evaluation instrument (translated from the original version in Swedish), as well as the proposed "domain" or "behavioural anchor" of each original question is displayed in Table  1.
Our study set out to determine the validity and reliability of the adapted questionnaire, aimed at measuring student perceptions of the clinical learning environment, after being adapted to suit the context of a South African Academic Hospital.

Methods
The Uppsala Instrument was reviewed and commented on by five experts in medical education from within our faculty for adaptation to our setting.
In the second phase of review, final year medical students were invited via email and social media (on a closed class Facebook group) to complete the newly adapted questionnaire anonymously, via Google Forms. The respondents could use any device capable of accessing the web page. Respondents selected the clinical rotation they wished to evaluate and answered each question on a Likert scale from 1 to 6. Table 1 presents the original items, as well as the expanded questionnaire which was finally used.
Next the principal component analysis was performed and results were interpreted in light of the steps proposed by Schönrock-Adema et al (2009). A scree plot of eigenvalues was generated, and considered alongside an eigenvalue cut-off of >1.5 and "the proportion of variance accounted for" minimum was set at 5%. Interpretability criteria as proposed by the above authors were also applied in identifying the underlying component, including requiring a component to load (show correlation with the given component) with a value of at least 0.4 on the component matrix for at least three questions. Convergent and divergent validity could further have been determined if multiple components were identified.
Following this the Cronbach's alpha was determined for all Likert scale questions with the intention to determine Cronbach's alpha for the identified component(s).

Participants
Participants were final year medical students who were invited to participate in the study via email, social media and class visits. Responses were collected during 2015. Students were asked to complete the electronic questionnaire after completion of every clinical rotation between January and July.

Ethical Approval
This study was approved by the Stellenbosch University Health Research Ethics Committee and was conducted in accordance with the international declaration of Helsinki (S14/09/185) and also received institutional permission. Participants provided informed consent for the study. Participants were informed that the purpose of the study was to validate and determine the reliability of a questionnaire aimed at evaluating the clinical learning environment. No identifying information was gathered from student respondents. Furthermore, only the researcher had access to the password protected data sets which were generated. These data sets were sent to the biostatistician using password protected email accounts.

Face Validity
Expert feedback on the Uppsala Instrument's face validity expanded the questionnaire from 10 items to 20, separating some of the original questions into more specific questions, several alterations to phrasings, as well as proposing a few new questions. (Table 1, columns 3 and 4)

Principal Component Analysis
The adapted questionnaire received 168 complete responses that could be used for analysis (n=168). The principal component analysis revealed only one component which satisfied the set criteria (Schönrock-Adema, Heijne-Penninga, Van Hell, & Cohen-Schotanus, 2009). The scree plot below supports this notion, as the slope after the first item is gradual. The point of transition does not count as a component (Brown, 2009b).  (Table 3) After varimax rotation with Kaiser extraction, instead of attaining simple structure, as is the intention with rotation (Brown, 2009a), multiple questionnaire items now loaded on both the first (accepted) and second (rejected) component (table 4). From this it could be concluded that the second component did not represent an interpretable component.
Therefore, the PCA failed to support any of the proposed underlying components (domains or "behavioural anchors") for the original Uppsala Instrument, but instead showed that most variance resulted from a single underlying component (component 1).

Reliability
The Crohnbach's alpha coefficient for Likert-scale type questions was 0.943. Because all Likert-scale questions belonged to the same component, the alpha for this component was 0.943, indicating very high reliability, and possible redundancy of certain questions.
This further supported the finding that a single component explained most of the responses, and that the original proposed components (domains), were not demonstrable.

Discussion
The results described showed that the items addressed in the Uppsala Instrument was acceptable to clinical teaching experts although ten additional questions (including demographic information) and the expansion of certain original items into clearer more directed questions were added. The problem, however, was that some of the added questions could not easily be answered on a Likert scale.
The principal component analysis failed to support the presupposition that this questionnaire meaningfully assessed several underlying components or domains, as was suggested in previous studies utilizing the Uppsala Instrument (Kihlberg et al., 2011;Lewisson et al., 2013). Rather, it seems that the questionnaire only addressed a single component; namely perception of the clinical learning environment. While this component is vague the evidence presented above is sufficiently rigorous to suggest that this is what the instrument actually measures.
These results found are somewhat similar to that published by Boor et al.(2007) while investigating the Postgraduate Hospital Educational Environment Measure (PHEEM), where it was also found that the variance in answers to the 40 items was due to a single component. The authors of the Clinical Learning Evaluation Questionnaire (CLEQ) made a similar finding after they did their principal component analysis. They found that all the identified components (referred to as "factors" in their study), still related to each other (as there was no divergent validity), raising the question of whether it is "meaningful to differentiate between these".(AlHaqwi et al., 2014, p. 6) It could also mean that each of the proposed domains are very closely related in the clinical learning environment, and that high ratings on one necessarily mean high ratings in all others. To use the "behavioural anchors" originally proposed by Lewisson et al.(2013) as an example, a rotation which spends time on "evaluation and feedback", would probably have "good control of session" and "communication of goals", which in turn will inspire "self-directed learning" in students (to quote the proposed domains from the original questionnaire). There is unfortunately no clear way of proving this hypothesis with the current data set. instrument which was used effectively to improve student perceptions of the CLE (Lewisson et al., 2013). Therefore it can still be of use, but will unfortunately not allow for deeper analysis of educational constructs within the clinical learning environment.
Furthermore the instrument's reliability compares well to that of instruments used to evaluate clinical teaching and clinical learning environments and the values determined are in line with those reported on other instruments based on the SFDPCTF instrument (with Crohnbach's alpha coefficient ranging from 0.82-0.98) (Nation et al., 2011). In light of this, the high value obtained here is not surprising.

Conclusion
There is a need for evidence based instruments to evaluate learning in the clinical environment and the various factors which influence it. This study found that variance in responses to an instrument aimed at evaluating perceptions of the clinical learning environment (CLE) amongst students was due to a single underlying component. The findings were similar to the outcomes of other studies evaluating instruments aimed at evaluating the CLE. This calls into question whether meaningful study of various domains is possible using this or similar instruments.

Limitations of the study
Due to limitations in time, certain portions of the research were conducted in an expedited fashion. It would have been preferable to have used a formalized method such as the Delphi method for the face validity (or "expert validation") step in the research.
The response rate was not as high what had initially been envisaged. Furthermore, given the software used (Google Forms), it was not possible to maintain participant anonymity at the same time as preventing multiple entries by an individual for the same rotation, so this created a possibility for students to answer more than once. This risk was taken due to the ease of distribution offered by this platform (over other platforms, which allow only one response per link to prevent multiple entries), and therefore the ability to distribute the questionnaire on social media. This medium allowed easy communication and distribution.

Take Home Messages
This study found that variance in responses to an instrument aimed at evaluating perceptions of the clinical learning environment amongst students was due to a single underlying component. This calls into question whether meaningful study of various domains is possible using this or similar instruments. Table 1: A comparison of the original questionnaire, as well as the original proposed "behavioural anchors", with the adapted questionnaire which was circulated to the participants (column 3).

Proposed Domain/Construct/"Behavioural Anchor" for Original Questions
Original Question (from the Uppsala Questionnaire)

Adapted Question Notes
New questions. No assigned domain.
N/A Which rotation are you currently on or regarding which rotation would you like to provide feedback?
Demographic information was gathered to facilitate analysis. What is/was the location of this rotation? Eg. Tygerberg, Worcester, Paarl, etc.
What year did your rotation take place?
During which month(s) did your rotation take place?

Learning Climate
What were the introduction and the welcoming at the department like?
What was the introduction to the rotation like?
The original was split into two questions to better suit the setting What was the reception at the ward like?

Declaration of Interest
The author has declared that there are no conflicts of interest.