Concept maps as a novel assessment tool in medical education [version 1; peer review: awaiting peer review]

Background: We conducted a pilot study to investigate the use of Concept Maps (CMs) in a Problem-Based Learning (PBL) setting as a complementary tool to current educational techniques for enhancing medical student knowledge and critical thinking. We also introduced a measure from the field of graph theory as an objective means of CM quality assessment. Methods: Participants were first-year medical students with no or minor prior CM experience. All participants completed questionnaires (demographic information and assessment of learning style) to establish a baseline measure against which the change in clinical and critical thinking was assessed. They were asked to prepare CMs for three PBL cases, and following the submission of the CMs they completed semi-structured critical and clinical thinking questionnaires. A clinical expert also created corresponding “benchmark” CMs for comparison. Qualitative (Wordclouds) and quantitative (graph theory) analysis provided a summary of the key concepts and quantified the CM quality respectively, compared to the “benchmark” CMs. Results: Questionnaires revealed that CMs helped students recall information, organize material in a concise manner, prepare better for their PBL session and provided a good revision tool. It was also found that graph-theoretical measures (graph density, modularity) were suitable for objectively distinguishing between CMs that captured more in-depth knowledge, compared to CMs that contained simpler associations. Conclusions: We have shown that it is possible to quantify CM quality using graph-theoretical measures, such as graph density and modularity. Open Peer Review Approval Status AWAITING PEER REVIEW Any reports and responses or comments on the article can be found at the end of the article. MedEdPublish Page 1 of 13 MedEdPublish 2022, 12:21 Last updated: 18 MAR 2022


Introduction
As the medical profession evolves, so do the educational methods by which medical students are taught and assessed. Over the last few years there has been a turn in the pedagogical system, from a teacher-centred to a student-centred approach. The traditional lecture-based methods are largely being evolved to more effective methods of teaching that enable medical students to learn, amalgamate new concepts with existing knowledge, integrate critical thinking skills, solve a range of complex clinical problems and lastly, retain and apply this vast knowledge in their medical learning and future careers.
Problem-Based Learning (PBL) is one of the more prominent student-centred approaches introduced in medical education to enrich the traditional lectures that haven't always succeeded in preparing medical students for problem-solving in clinical settings (Hung et al., 2008;Qin et al., 2016). PBL is an active, student-centred approach to learning, which enables students to gain competence in self-directed learning, collaboration, problem solving and critical thinking (Kong et al., 2014). Research has shown that PBL-based Medical Curricula promotes deep learning, critical and clinical thinking of future clinicians, and shifts the focus from a teacher-centred to a knowledge-centred system (Colliver, 2000;Hagi & Al-Shawwa, 2011;Rovers et al., 2018;Urrutia Aguilar et al., 2011;Yew & Goh, 2016).
Another effective educational method, which has gained ground in the last few years, is Concept Maps (CMs). CMs are visual hierarchical representations of conceptual knowledge and relationships between the various concepts (Croasdell et al., 2003;Eachempati et al., 2020). They were created by Novak and Gowin based on Ausubel's Assimilation Theory that suggests we learn by linking new knowledge to our already existing knowledge (Ausubel, 1968;Novak & Gowin, 1984). A CM consists of nodes (key concepts) connected via lines/arcs indicating relationships between concepts and how these are structured and organized hierarchically. When used for educational purposes, the instructor gets a snapshot of student knowledge and understanding based on the concepts and relationships present on the map, providing an insight as to how ideas were (or were not) absorbed by the students (Daley et al., 2016a). Thus, it can be used as an assessment tool of what material was understood and how it relates to the overall course goals. The number of interconnections and cross-linkages is an indicator of greater complexity and sophistication of understanding. Figure 1 provides an example of a CM illustrating the key features and structure of CMs (Novak & Cañas, 2006).
CMs have already been used successfully in allied health fields, such as nursing education, where they have been found to improve critical thinking and reasoning in nursing students and have been utilised as an alternative to traditional patient care plans (Daley et al., 2016b). Due to their success in other fields, their use in medical education is now of significant interest.
They function by promoting meaningful and deep learning, as it has been shown that students are able to integrate basic and clinical science information, move from linear to integrated holistic thinking, and demonstrate critical thinking through CM construction (Veronese et al., 2013). In a PBL setting there are significant learning gains when CMs are used in a group setting with very different knowledge structures, as CMs can function well across groups of learners with multiple and varied learning styles. It is thought that concept mapping complements PBL as it involves active learning rather than more superficial learning or rote memorisation and promotes problem solving and critical thinking in medical students (Hung & Lin, 2015;Slieman & Camarata, 2019). There are also indications that CMs may measure different cognitive domains compared with more standard tests, and are associated with the most impact regarding learning improvement in students who come into the PBL study with the lowest cognitive competence (thus could be used as a tool to help such students). Moreover, CMs provide an additional resource for learning, as CMs pre-prepared by faculty and used in teaching may offer alternative and innovative learning and teaching opportunities, developing PBL curricula or blueprinting for cross-departmental curricular learning objectives. For medical education in particular, CMs may assist students in promoting critical thinking, clinical reasoning and decision making (Daley & Torre, 2010;Rendas et al., 2006). Student resistance to using CMs is sometimes seen due to the time involved in creating a CM or inability to understand how CMs will subsequently help to increase performance scores. From a faculty point of view there needs to be an understanding that CMs are an implementation of meaningful learning and the fact that they change over time as the student learns is not a reliability issue, but an indication of how the students' learning has grown over time; this is analogous to the process of attaining expertise.
CMs could also become part of assessment, through assessing the CM creator's intentional effort to link, differentiate and relate various concepts in a way that traditional tests cannot (Gomez et al., 2014). Currently, formal analysis of a CM is done at a qualitative level, e.g. counting the number of concepts and/or links and, thus, indirectly measuring the map complexity, comparing with the CM created by an expert, and conducting longitudinal comparison of CMs throughout the course (Srinivasan et al., 2008). Informal analysis can also be performed, where the maps are assessed for accuracy, which is subjective as there is no "right or wrong". Currently, no objective measure of assessing a CM exists, which complicates the potential use of CMs as part of formal assessment. However, a novel solution to the lack of objective measures for CM assessment could be attained from the field of graph theory (Wilson, 1996). The visual representation of a CM is very similar to the visual representation of a network in graph theory applications, whereby each concept and its links with other concepts could be represented by "nodes" and "edges" respectively (Hevey, 2018;Watts & Strogatz, 1998). By treating each CM as a network and using graph-theoretical measures to objectively characterise the "network" (CM) complexity may allow CMs to be used for summative assessment purposes.
In this project, we first provide a table (Table 1) with a summary of reviewed studies on concept maps. Secondly, we fuse together expertise from the fields of pedagogy and dynamical network analysis in an effort to introduce tools that complement current educational techniques in medical schools through the conduction of a short-scale pilot study that introduces CMs in a PBL setting to investigate its applicability in further enhancing medical student knowledge and improving medical thinking; and to develop an objective measure from the field of graph theory to quantitatively assess a CM as part of assessment.

Methods
The benefits from the use of CMs as educational tools have been emphasized in the previous section. Based on this, we conducted a small pilot study to assess the benefits of introducing CMs in the PBL curriculum of first year medical students with respect to clinical reasoning. Additionally, we investigated whether the novel use of measures based on graph theory could provide a quantitative measure of CM characteristics and, thus, provide a means of objective CM assessment.

Ethics and consent
Following ethical approval from Cyprus National Bioethics Committee, we recruited volunteer participants from the first-year cohort of a 4-year graduate entry medical programme (MBBS) delivered at the University of Nicosia, Cyprus. Recruitment was random and was conducted through leaflet advertisements at the School premises, as well as through email list distributions. Participants were informed about the study through introductory tutorials on CM construction and gave written informed consent for their participation.
The volunteers were also informed that they were to attend their PBL sessions as per normal, but they had to prepare CMs for 3 of those PBL cases (Hepatitis, Breast Cancer and Tuberculosis). Participants completed a questionnaire on their learning style (VARK Learning Styles Self-Assessment Questionnaire) prior the beginning of the study and a questionnaire on their experience after the completion of the construction of all 3 CMs. The collected data measured the level of clinical and critical thinking of each participant at a qualitative scale.
All study participants attended the PBL sessions and completed the study questionnaires at two time points (shortly after recruitment but before the delivery of the PBL sessions, and after the end of the PBL sessions/construction of the CMs). The VARK questionnaire was printed and hard copies were given to the research participants during the delivery of the introductory session, while the post-intervention questionnaire was completed online (through SurveyMonkey) following the completion of the construction of the CMs. All participants developed a CM for each PBL case included in the study (3 Table 1. Summary of reviewed studies on CMs showing their benefits and limitations. (CMs: concept maps; PBL: problem-based learning; CBL: case-based learning.)

References
CMs as a leaning tool in small group teaching -CMs were incorporated into PBL and provided structure to the tutorials.
-CMs assisted students with critical thinking, problem solving, understanding causality and acknowledging relationships between concepts.
-CMs were helpful to identify misconceptions in student understanding.
-By comparing academic achievement between the CM intervention group and the non-CM control group, results showed moderately better performances by students in the CM groups compared to non-CM groups.
-Students also reported that they were 'unnecessarily time-consuming', overwhelming and intimidating to design and use.
-CMs are complementary to PBL in information gathering, hypothesis generation and identification of learning issues.
-CMs allowed 'previously misunderstood concepts' to be corrected, promoted eagerness to learn and was fun.
-CMs is a good strategy to integrate basic science knowledge with clinical reasoning, recalling prior basic science principles and improve understanding. Group discussion allowed collaboration and transformation of student knowledge by building upon each other's ideas.
-Use of CMs as a tool for hypothesis generation and for stimulating critical thinking.
-Student CMs reflected the 'knowledge organisation structure'.
-Groups consisting of students with different knowledge structures had greater improvements in their learning since input from other members was valuable to their learning.
-CMs were useful in understanding the 'meaning of relationships' among concepts.
-Some students that were required to construct a CM individually, reported that it is time consuming, overwhelming, complicated, and that at this level they already have their learning style so they may only be useful to some students. Thus, their integration into a group setting may be beneficial to reduce the burden of creating a CM alone and provide opportunities for students to learn from each other.
-Introducing CMs early on in a course might improve student perceptions of CMs because they may not have developed alternative or preferred study methods yet. CMs as a way of identifying gaps in student knowledge and providing feedback -CMs helped tutors to identify gaps in student thinking and understanding, which allowed them to provide more accurate feedback.
-Students also found the CMs helpful in 'identifying areas of weakness' prompting them to seek clarification.
-Introduction of CMs as a formative online assessment tool, which provided automatic feedback, gave students direction on where to focus their study for the rest of the semester.
-Students reported that the most helpful aspect of concept mapping was presenting their CMs to their peers and justifying connections between concepts as it allowed them to recognise any ' distortions' in their understanding.
-CBL Tutors were able to provide feedback that stimulated critical thinking in the students and in a qualitative study on student opinion of concept mapping students expressed that tutor feedback was an essential part of concept mapping.
-Incorporation of both tutor and peer feedback improved group concept mapping performance.
-Feedback is crucial to learning as students who received feedback on their CMs in a cell biology course displayed improved performance. On the contrary, there was no significant effect on the performance of students who received no feedback. CMs as a learning tool to understand relationships between concepts and to facilitate meaningful learning -CMs were more effective for retaining knowledge compared to passive learning, such as lectures, reading text passages and even participating in class discussions.
-CMs do promote meaningful learning in students learning physiology and increase their problem-solving ability but may be most beneficial for less high achieving students.
-Students found having an active role in their learning through the creation of CMs to be motivating despite initial resistance and concerns over the time required to complete the maps.
-CMs promote meaningful learning and problem-solving skills in students.
-Only a few studies found positive results in regards to meaningful learning in students and because the CMs were often used with other instructional methods, such as collaboration and feedback, it was difficult to conclude that any outcomes were purely attributable to CM activities. , 2006Gonzalez et al., 2008Colliver, 2000Gonzalez et al., 2008;Veronese et al., 2013Daley & Torre 2010Pudelko et al., 2012 PBL cases, a total of 3 CMs per participant). They were free to create their CMs in image or word format, or directly in Gephi, which is a freely available network visualisation tool (we also used Gephi for our analysis). For the purposes of the small pilot study (proof-of-concept) we recruited 10 volunteers. Following a review of the submitted CMs, we were able to use CMs from 8 participants, 6 of which were examples of CMs for which the students appeared to have grasped more complex concepts of the PBL cases, while the other 2 were examples of more simplistic CMs. 2 participants' data were excluded from analysis as they did not create usable CMs but rather short summaries of the information they collected. We specifically aimed at having this diversity in the participant CMs as a means of comparing the feasibility of the graph-theoretical measures for quantitative CM assessment.

Nesbit & Adesope
The data collection was conducted in 2 stages: (1) Preintervention: questionnaires for gathering demographic information and assessment of learning style were administered to all study participants to establish a baseline measure against which the change in clinical and critical thinking was assessed. During this stage, a clinical expert was also provided with the specific PBL cases and created the "benchmark" concept maps against which the participant concept maps would be compared.
The CMs developed by the students were assessed with respect to (i) format, compared to the "benchmark" CMs developed by the expert; and (ii) structure, using current methods of assessing concept maps (i.e. counting the number of nodes and connections).
Qualitative analysis investigated the key concepts that characterize the CMs developed by the participants, compared to those included in the "benchmark" CMs. NVivo (Release 1.4 (4)), which is specialised software for qualitative and mixed-methods research, was used for all qualitative analysis. To help in identifying key concepts that appear more frequently in the CMs, a wordcloud of the concepts was obtained (NVivo). A wordcloud is a visual representation of text data that can be used to identify keywords or visualize free form text. It is achieved through font size, whereby words that appear more frequently in the text are displayed in larger font. The font size by which each word is represented in the wordcloud is proportional to its frequency of occurrence in the text. The word clouds obtained from the CMs indicated which words (concepts) the participants deemed more important. Similarities and differences with the key concepts indicated by the experts over the various PBL cases can provide qualitative information as to whether the students' ability to grasp key concepts shows any changes longitudinally. The richness of the concepts identified through the wordclouds was also correlated with changes in the participant clinical/critical thinking from the questionnaires that were administered at various stages of the study.
While wordclouds provide a summary of the key concepts in a simple visual form, the richness of the concepts, as well as the density of connections between them, can be quantified via the novel application of graph theory to CMs. To estimate the graph-theoretical measures we used the freely available network visualisation tool Gephi. The expert and student CMs were imported into Gephi and the following measures were estimated: • Graph density, which reflects how connected a graph is. It is estimated as the actual number of connections (edges) in the graph as a fraction of the total number of possible edges. It tells us how connected a network is compared to how connected it could be. Higher density indicates more connections between the nodes. This could indicate more efficient transfer of information across the network, as information can diffuse rapidly from one node to all others rather than going from one node to another. But in the application of CMs, a densely connected network could also indicate a network where all concepts are connected to each other without any specific structure or purpose.
• Modularity, which is the degree to which the network may be subdivided into clearly delineated and nonoverlapping groups (number of communities). It provides a measure of partition of larger networks into basic "building blocks," i.e., internally densely connected clusters that are more weakly interconnected amongst each other. Networks with high modularity have a pronounced community structure, and the higher the modularity, the more distinct those communities of densely connected nodes are. A modularity of 0.4 or above implies that the network has quite a pronounced community structure; if it is less than 0.4, there are no big differences between the different clusters and most of the nodes are equally densely connected to each other across the whole network. In terms of the CMs, higher modularity would imply that the main concepts are also clustered into smaller groups that represent individual and more specific properties of the theme.

Results
We provide some example CMs for PBL case 1 (hepatitis). Figure 2 shows the expert CM, for PBL case 1. Figure 3 shows an example of a student-created CM that resembles the expert CM in terms of complexity. In contrast, a less complex student CM for the same PBL case is shown in Figure 4. Zoomable versions are also provided (see underlying data, (Loizou et al., 2022)).

Questionnaire analysis
Based on students' responses from the first questionnaire (that was completed prior to the beginning of the study/ pre-intervention stage), all eight students had medical/biological sciences background, six out of eight had no prior experience with PBL and only two students had some experience. Four out of eight students had no previous experience with CMs while the other four students mentioned they had some experience. Following completion of the VARK questionnaire (assessment of learning style -pre-intervention stage), results revealed that six  students belonged to the kinaesthetic type of learners, while one student belonged to the aural/auditory type and another to the visual type.
The most consistent finding from VARK questionnaire results is that people are very diverse and we cannot assume that others learn as we do. By definition, kinaesthetic refers to the "perceptual preference related to the use of experience and practice (simulated or real)" (Fleming & Mills, 1992, p. 140-141). It includes demonstrations, simulations, videos and movies of "real" things, as well as case studies, practice and applications. Aural/Auditory style describes a preference for information that is "heard or spoken." Learners with this modality report that they learn best from lectures, tutorials, tapes, group discussion, speaking, web chat and talking things through. Visual learning style describes a preference for information depicted in maps, diagrams, charts, graphs, flow charts etc. in place of using words.
Following the construction of the three CMs students were requested to complete a second questionnaire to evaluate their clinical/critical reasoning skills (post-intervention stage). Almost all students agreed that CMs helped their learning across the board. They commended that the connection of ideas helped them recall the information, organize the material in a concise manner and prepare better for their PBL session. Additionally, it provided a good revision at the end of the learning week. When the students were asked in what way they used the CMs in relation to their studying, they all commented that they helped them gather and revise the material that was covered in the PBL case of each specific learning week.
When students were asked how likely they were to use CMs in the future, the majority of them were hesitant to do so and only three students would definitely use them again reporting that 'It is a way of learning that I never used before and I was quite sceptical initially but it has improved my recall and I find myself wanting to use it more and more'. The ones who were indecisive commented that: 'While it helped me organize the material I felt it didn't help me understand the material any better, so while I may do it for a case with a lot of factors involved, it is not something I see myself using weekly' or 'I

Figure 4. A student-created concept map (CM) that is less complex in terms of key concepts and connections.
This CM is also very textual. A zoomable version is provided as underlying data.
think it could be helpful, but not for extremely detailed topics. Therefore, I might use it to study for some but not all topics.' Students would be positive on recommending the use of CMs to a fellow student mostly as a revision tool. More specifically, we received comments like 'This way of learning can help people to connect concepts that are difficult in a way that makes it easy to remember but they may not work for every person and every concept easily', or 'Although I find it very useful as a revision tool, I'm not sure many of my colleagues will find it quite so useful'.

Wordclouds
We provide some example wordclouds for PBL case 3 (tuberculosis). Figure 5 shows the wordcloud created from the expert CM. Figure 6 shows an example wordcloud from a student-created CM that resembles the expert CM in terms of complexity. In contrast, a wordcloud from a less complex but very textual, student CM for the same PBL case is shown in Figure 7.
The wordclouds can provide an initial qualitative assessment of whether a student has identified key concepts in the CM, as they provide a visual summary of the main concepts that can be found in a CM. However, they do not provide any information about the complexity of the connections between these concepts and they can help us identify whether there is high repetition of these key concepts throughout the CM (Figure 6). Such repetition could be an indication of a less in-depth understanding of how these are interconnected.
It is also difficult to use wordclouds to differentiate between less complex and not as well-structured CMs, such as those that are very textual (Figure 4). Figure 8 shows the estimated graph density for the student and expert CMs for the 3 PBL cases. CMs from students 1-5 and 8 have graph density that is closer to that of the expert CM. In   This CM is less complex in terms of key concepts and connections, but is very textual. This is not easily discernible from a wordcloud.

Graph theoretical measures
contrast, CMs 6 and 7 have very high graph density, indicating more densely connected networks, which most likely reflects CMs where concepts are connected to each other without specific structure. Figure 9 shows the corresponding modularity estimated for the student and expert CMs for the 3 PBL cases. The expert CM has a modularity index of above 0.4, which indicates a more pronounced community structure, i.e. the different concepts form distinct communities, but there is also some structure as to how each community is connected across the network. CMs of students 1-5 and 8 also have modularity that is above 0.4, reflecting CMs where the main concepts can be clustered into smaller groups that represent individual and more specific properties of the theme. The CMs of students 6 and 7, however, have lower modularity that in the majority of cases is below or just above 0.4. This implies that modularity can provide a means of quantitative assessment of CM complexity compared to the expert CM.
Based on the VARK questionnaire responses, students who produced CMs 1-5 had a kinaesthetic learning style, while the student who produced CM 8 had a visual learning style. CM 6 was produced by a student with an auditory learning style, which may explain why the CM was more simplistic as this may not be the appropriate learning tool for this particular student.
It is interesting, however, that the student who produced CM 6 also had a kinaesthetic learning style, despite the simpler structure of the CM.

Discussion
According to a number of studies, CMs can be integrated into PBL or tutorial-based curricula (Addae et al., 2012;Penũela-Epalza & De la Hoz, 2019;Rendas et al., 2006;Torre et al., 2017;Veronese et al., 2013). Most studies found group concept mapping in a PBL or case-based learning (CBL) curriculum was feasible and acceptable to both tutors and students. Students felt that they learnt from their peers, received constructive feedback and became more confident in sharing ideas and hypotheses. When confronted with the task of constructing a CM on their own, students often complain that they are time consuming and overwhelming and there are mixed opinions on whether they would utilise them in their own study in the future. Thus, group mapping is beneficial in that the students could learn from each other, and they are not faced with the overwhelming task of constructing a map on their own. Also, by incorporating the concept mapping activity into the time restricted PBL sessions, students are limited in the amount of time they are expected to spend on them.
Results from a number of studies that investigated students' perception of group CMs show that students feel they benefit greatly from both the sharing of knowledge with peers in a group and that the group CM environment allows for feedback from peers and tutors. The use of CMs to identify gaps in student knowledge allowing for specific feedback and more directed study is perceived as one of the most beneficial aspects of concept mapping to both students and tutors. When comprehensive, CMs allow students to see the whole picture of a topic thus allowing them to identify areas where their knowledge is less organised and cohesive. Furthermore, students' active involvement in creating maps and justifying connections between concepts indicates their understanding of the links between different concepts. As such, CMs provide opportunity for tutors and instructors to see where students are making incorrect connections and provide directed advice.
Student opinion on the use of CMs vary between studies. Common criticisms have been the time-consuming nature of concept mapping, that students already have their learning style and that CMs may not be useful to all students while other students have expressed that CMs facilitate a more active role in their learning and found this motivating. Furthermore, studies notably report more on the positive opinions of students with one study even offering an open-ended question regarding the positive aspects of the task but no space for criticism. These points raise concerns for the compulsory integration of CMs into curriculums. Student perception of CMs may not indicate their utility as a learning tool, however, there is some evidence to suggest that perception of the learning environment does impact academic performance (Wayne et al., 2013) and this should be taken into consideration by medical educators.
A limitation of a number of studies is that concept mapping activities often also involve other learning or instructional tools, such as feedback and collaboration, making it difficult to attribute all tested outcomes to the use of concept mapping alone. However, this may also be a point in support of CMs as a tool that allows the integration of different instructional methods that may complement each other and achieve better outcomes.
Studies into the use of CMs in medical education have largely been limited to individual courses or a single year of medical school. In the future, longitudinal studies are required to measure the long-term effects of CMs on student learning and whether implementation of CMs into medical curriculums benefits medical students and knowledge organisation as they transition into their future careers.

Limitations
The study is a pilot, proof-of-concept geared towards showing that graph theoretical measures can provide an objective measure of CM structure and complexity that can be further investigated for summative CM assessment purposes. Pilot studies inherently include a small number of participants and, therefore, we are cautious when interpreting our findings. However, the findings from the current study are encouraging for further investigations as part of a larger-scale study. Some graph theory measures also depend on the network structure itself, for example the maximum density of a graph is dependent on the number of nodes of the graph. Providing the students with the main concepts (i.e. number of nodes) to be used in the construction of the CMs will help to limit the differences in network measures to those related only to the interconnection and density of connections between the nodes, thus providing a more objective measure of the CM. Finally, there are many graph theory measures that can be used to quantify different characteristics of a graph. In this work we have only investigated the use of two such measures, graph density and modularity. In future work, we will investigate more measures and combinations of them, to develop an objective measure that could be used as part of summative CM assessment.

Conclusions
Concept maps are favourable to both students and tutors as a tool that allows recognition of where there are gaps in understanding and provide an example of student thinking so that tutors can provide meaningful feedback to direct study. They can be used in PBL or tutorial-based curriculums as a learning tool but it is necessary to consider that some students find them more useful than others who may consider them time consuming and overwhelming. Further research is required to determine the long-term benefits of using CMs in a tutorial style curriculum and how they help medical students make the transition into their future careers as doctors that can problem solve effectively and think critically. We have introduced here the novel notion of using measures from graph theory for CM assessment and have shown that it is possible to quantify CM quality through measures such as graph density and modularity.