Analyzing the Use of Concept Maps in Computer Science: A Systematic Mapping Study

. Context: concept Maps (CMs) enable the creation of a schematic representation of a domain knowledge. For this reason, CMs have been applied in different research areas, including Computer Science. Objective: the objective of this paper is to present the results of a systematic mapping study conducted to collect and evaluate existing research on CMs initiatives in Computer Science. Method: the mapping study was performed by searching five electronic databases. We also performed backward snowballing and manual search to find publications of researchers and research groups that accomplished these studies. Results: from the mapping study, we identified 108 studies addressing CMs initiatives in different subareas of Computer Science that were reviewed to extract relevant information to answer a set of research questions. The mapping shows an increasing interest in the topic in recent years and it has been extensively investigated due to support in teaching and learning. Conclusions: based on our results we conclude that the use of CMs as an educational tool has been widely accepted in Computer Science.


Introduction
Managing knowledge is a key challenge for organizations and research institutions, since once knowledge is well managed, it can become a consistent intellectual component in the decision making process (O' Leary, 1998;O'Leary and Studer, 2001;Zack and Serino, 2000).New challenges of how knowledge should be produced and transmitted have also been investigated (Correia, 2012).Among the strategies to work with knowledge, Concept Mappings (CMs) are considered an useful graphical technique for organizing, understanding and sharing knowledge (Correia, 2012;Novak and Cañas, 2008).
Originally developed by Novak and Cañas (2006) in 1972, CMs are a graphical representation of knowledge.CMs are visual representations of a particular topic and they provide support to organize and represent knowledge.CMs represent knowledge as graphs.These knowledge graphs are networks of concepts, where nodes represent concepts and links relations between them.These links contain associated words that describe the nature of the relationship between concepts (Novak and Cañas, 2008).
In summary, CMs provide a visual and holistic way to describe declarative knowledge relationships.They also provide an useful medium for sharing information in a concise and accessible form, making knowledge models based on CMs a promising vehicle for knowledge sharing (Novak and Cañas, 2008).Originally CMs are connected to the education domain (Novak et al., 1984).CMs originated from the need to find how best to represent the conceptual understanding of children about science (Novak and Cañas, 2006).It is possible to understand, from a simple search for related literature, that CMs are often applied as teaching and learning strategies (Gurupur et al., 2015;Jain et al., 2014;Katagall et al., 2015;Larraza-Mendiluze and Garay-Vitoria, 2013;Weinerth et al., 2014;Wu et al., 2012;Zvacek et al., 2012).CMs have been applied in pedagogical context in all disciplines including politics, philosophy of science, physics, chemistry, medicine, electrical engineering and mathematics (Katagall et al., 2015;Rueda et al., 2009a).CMs have been used in multiple settings in education, since they are able to clarify the relationships between the subcomponents of a particular topic and they are useful to organize one's own ideas (externalization) (Correia, 2012).
Given the potential benefits of CMs, this paper aims to identify CMs initiatives in Computer Science by means of a systematic mapping study.A systematic mapping study, or simply mapping study, provides a broad overview of an area of research, to determine whether there is research evidence on a particular topic.Results of a mapping study helps identifying gaps in order to suggest areas for future research and provides a map that appropriately allows the position of new research activities (Kitchenham and Charters, 2007;Petersen et al., 2015).
A mapping study presented in this paper investigates the following issues: Purposes of applying CMs in Computer Science.(i) Different subareas of Computer Science in which CMs have been applied.(ii) Resources used to support the development of CMs.(iii) Main guidelines used to evaluate CMs.(iv) Benefits and problems reported on use of CMs in Computer Science.

(v)
The rest of this paper is organized as follows.Section 2 presents related research.Section 3 discusses the research method applied to perform the mapping study.Results are presented in Section 4. Section 5 discusses the results, their implications, and limitations.Finally, Section 6 concludes the paper and presents directions for future work.

Related Work
In general, secondary studies are based on analyzing research papers (referred to as primary studies).Mapping studies provide an overview of a topic area and identify whether there are subtopics where more primary studies are needed (Petersen et al., 2015).In this paper we present a mapping study to identify and classify all research related to the use of CM in Computer Science.However, before accomplishing the secondary study presented in this paper, we performed a tertiary study looking for secondary studies that already investigated CMs in the same topic.Tertiary studies are considered as a review that focuses only on secondary studies, i.e., it is a review about other secondary studies (Kitchenham and Charters, 2007).In this tertiary study, we used the search string shown in Table 1.
The search string presented in Table 1 was applied in the following electronic databases: IEEE Xplore, ACM Digital Library, Scopus, Science Direct and Compendex.We found only one study that drew our attention (Weinerth et al., 2014).
In Weinerth et al. (2014), a literature review with a focus on usability was conducted.Authors conducted a research review to examine whether and how researchers have studied and considered usability when conducting computer-based CM assessments.From 119 journal studies that assessed computer-based CM only 24 discussed the usability issue in some way.According to Weinerth et al. (2014), the impact of usability on computer-based CM assessments is an issue that has received insufficient attention.
In addition, we decided to look for secondary studies investigating CMs separately (not in the context of Computer Science).We used the search string shown in Table 2: The same seven electronic databases previously mentioned were searched, returning 75 results.After eliminating duplications and applying the selection criteria, 19 secondary studies were returned, but, none was a secondary study on CMs.However, one study attracted our attention.Rosas and Kane (2012) did not present a secondary study, but they conducted a pooled analysis of 69 CM studies to describe characteristics across study phases, generated specific indicators of validity and reliability, and examined the relationship between selected study characteristics.A majority of CM studies analyzed were classified as directed to public health (59.4%).Others were in the fields of human services (20.3%), biomedical research (5.8%), social science research (2.9%), and business or human resources (2.9%).With respect to purpose, twenty-eight (40.6%) were used for strategic planning purposes.
Based on the results from this investigation by employing tertiary studies, we can say that there is only one recent extensive literature search that focused on computer-based assessment, CMs and usability.This motivated us to conduct this study.

Mapping Study
As previously mentioned mapping studies build a classification scheme and organize a research field of interest within the categories of the scheme.As a result, the coverage of the research field can be determined and the scheme can also be used to answer specific research questions (Petersen et al., 2015).Therefore, mapping study provides an overview of a research area by identifying and quantifying the related available research.
The research method for the mapping study presented in this paper was defined based on the guidelines given by Kitchenham and Charters (2007) and Petersen et al. (2015).This mapping involves three main phases: (i) Planning: refers to the pre-review activities, and aims at establishing a review protocol defining the research questions, inclusion and exclusion criteria, sources of studies, search string, and mapping procedures.(ii) Conducting: searches and selects the studies, in order to extract and synthesize data from them.The data extraction activity for a mapping study is broad and the analysis of a mapping does not include the use of in-depth analysis techniques, such as meta-analysis, but rather total amounts and summaries.Graphic representations can also be used to summarize the data.
Review "Systematic Literature Review", "Systematic Review", "Systematic Mapping", "Mapping Study", "Mapping Studies", "Systematic Literature Mapping", "Literature Review" Search String: ("Concept Map" OR "Concept Maps" OR "Conceptual Map" OR "Conceptual Maps" OR "Concept Mapping") AND ("Systematic Literature Review" OR "Systematic Review" OR "Systematic Mapping" OR "Mapping Study" OR "Mapping Studies" OR "Systematic Literature Mapping" OR "Literature Review") (iii) Reporting: final phase that aims at writing up the results and circulating them to potentially interested parties.In this phase the findings of the mapping study are used to answer the research questions.Kitchenham and Charters (2007) suggest, in addition to the searches in the databases, conducting a snowballing from reference lists of selected studies, in order to identify additional relevant studies through the reference lists of the studies found using the search strings (Jalali and Wohlin, 2012).Manual searches for works developed by important researchers and research groups can also be performed (Kitchenham et al., 2011).These researchers/ research groups are identified from previously selected studies (retrieved by searches in the databases, as well as from snowballing).We tried using these two approaches in our mapping to overcome the limitation of using a specific set of electronic databases.
The following are the main steps we performed for the mapping study.

Research Questions
This mapping study was conducted in order to identify principles of CMs initiatives in Computer Science.Based on this goal, seven research questions (RQs) were created.Table 3 presents the RQs that this mapping study aims to answer, as well as the rationale for considering them.The topic of this mapping study seems to be broad and new.This research question provides an understanding on whether there are specific publication sources for these studies, and when they have been published.
RQ2 What is the purpose of the application of CMs in Computer Science?
This question looks for the purposes declared in the studies for applying CMs.This is important to point out why such studies have been accomplished.

RQ3
In which different subareas the application of CMs have been presented?
Investigates the different subareas of Computer Science in which CMs have been applied.
RQ4 What are the main instruments used to develop the identified CMs?
Highlights the main instruments, as tools, techniques or methods, currently used to develop CMs.This is useful for researchers and practitioners that intend to accomplish new studies in this context, as well as to guide future research towards new technologies in order to fill the existing gaps.
RQ5 What are the guidelines or mechanisms used to evaluate the CMs identified?
This RQ highlights guidelines or mechanisms used to evaluate the CMs.Similarly to the previous one, this is useful for researchers and practitioners to guide future research.
RQ6 What are the benefits reported by authors related to the use of CMs?
Provides an overview of the main benefits reported by authors related to the main advantages in works with CMs.
RQ7 What are the problems reported by authors related to the use of CMs?
Provides an overview of the main limitations reported by authors related to the main difficulties in works with CMs.

Search Strategy and Selection
The purpose of a mapping study is to conduct a review of relevant studies to assess the quantity of evidence existing to address the RQs.The process needs to be rigorous and unbiased and it often involves a wide coverage of sources, such as online electronic databases, journals and conferences.In order to minimize bias and to maximize the number of sources examined, a pre-defined strategy to identify potential primary studies is required.Ours is described as follows.

Terms and Search String
First of all, we created our search string.The search string considers two areas -"Concept Map" and "Computer Science" (Table 4) -and it was applied in three metadata fields: title, abstract and keywords.The search string went through syntactic adaptations according to particularities of each source.
In order to elaborate the search string, the set of search terms were devised in an iterative way, i.e., we started with an initial set of terms and iteratively improved this set until all relevant pre-selected studies were found.The pre-selected studies are part of a control group.If the publications of the control group are not retrieved, the string needs to be calibrated (adjusted) or new databases need to be added in order to guarantee that at least these publications are retrieved.In this sense, we used a control group to calibrate the search string.

Sources
We chose the electronic databases based on predefined criteria, such as: (i) coverage (large number of conferences proceedings, journals, and indexed books in different knowledge areas); (ii) content update (publications are regularly updated); (iii) availability (full text of the study are available); (iv) quality of results (accuracy of the results obtained by the search); and (v) versatility export (a mechanism to export the results of the search is available).The control group, described before, also helped to define the databases.The most commonly used databases in Computer Science area that meet to the characteristics described above are: IEEE Xplore (http://ieeexplore.ieee.org)ACM Digital Library (http://dl.acm.org)ScienceDirect (http://www.sciencedirect.com/)Scopus (http://www.scopus.com/)Compedex (http://www.engineeringvillage2.org) As a result of searching from the search string defined and the selected sources, a total of 700 publications were returned.We considered the studies published until June 2016.

Selection Process
The selection criteria are organized in one Inclusion Criterion (IC) and seven Exclusion Criteria (EC).The following inclusion criteria were applied: (IC1) The study must be within the Computer Science and discuss the CMs initiative.
(EC1) The study does not have an abstract.

(EC2)
The study is just published as an abstract.
(EC3) The study is not written in English.

(EC4)
The study is an older version of other study already considered.
(EC5) The study is not a primary study, such as editorials, summaries of keynotes, workshops, and tutorials.(EC6) The study is not in the scope of Computer Science, although it uses CMs initiatives.(EC7) The study is in Computer Science context, however, does not use CMs initiatives.
The selection process was divided into five stages, as shown in Fig. 1.
In the 1 st stage duplicated studies were eliminated, resulting in 367 publications (reducing approximately 47.6%).
In the 2 nd stage, the selection criteria (inclusion and exclusion criteria) were applied for title, abstract and keywords, leading to 158 studies (reducing approximately 56.9%).
In the 3 rd stage, the selection criteria were applied considering the full text, resulting in a set of 93 studies (reducing approximately 41.1%).Over these 93 studies considered relevant, we performed backward snowballing, which resulted in 251 studies.From these 251 studies, the selection criteria were applied considering the title, abstract and keywords.Next, the selection criteria were applied considering the full text.A total of 37 studies were resulted from this stage.
Finally, from 130 studies selected until then (93 from sources and 37 from snowballing), we looked for publications authored by the researchers and research groups involved in these studies.We searched for their personal pages as well as other publications authored by them in electronic databases that we used as sources for this mapping.
A total of 52 studies were selected.After the selection criteria were applied, only 1 study remained.As a result, we got to 131 studies (93 from the sources, 37 from snowballing, and 1 from direct search to researchers and research groups).
In the 4 th stage, we applied Visual Text Mining (VTM) techniques in order to ensure that relevant studies were not eliminated (Felizardo et al., 2012).One VTM technique is to create a content map, i.e., a two-dimensional (2D) visual representation, where each document (i.e., a primary study) is graphically represented as an element on the plane, shown graphically as a circle.Positions of the studies in the map reflect the similarity relationships between their content (title and abstract).Therefore, if the studies are more similar (exhibiting content similarity) they are shown closer to one another; conversely, more dissimilar, further apart they are in the graph.
In summary, the strategy suggested by Felizardo et al. (2012) to review the studies' selection activity is to analyze the content map in terms of the included and excluded studies to find inconsistencies, i.e., studies with similar content (meaning that they are positioned close together on the map) with different classifications: included and excluded.These cases are hints to the reviewer, and the studies should be reviewed following the traditional method, i.e. reading the full text.Fig. 2 shows the clustering generated from the 131 studies.For example, the study highlighted with an arrow (see Cluster 7 in Fig. 2) was reviewed and its classification was changed from excluded to included.This same procedure was adopted for all clusters.In total, 6 studies initially excluded were included after the review, totaling 137 studies.In order to analyze the selected 137 studies, we conducted a quality assessment.This step allowed us to perform a 5 th stage in the selection process based on detailed quality criteria.In order to analyze the quality of the included primary studies we developed a checklist, as a quality instrument, containing seven questions (Table 5).The objectives of this list was to: identify the quality of the study regarding the main topic of this mapping and to identify the quality of the information that the study provides us, in order to know the relevance of the study within the SM.Our checklist was adapted from the generic quality criteria created by Kitchenham Kitchenham (2004).For each question in the checklist, the following scale-point was applied: Yes -1 point; No -0 points.The total quality score therefore fell into the range between: 0-1 (very poor); 2 (fair); 3-4 (good); 4-6 (very good) and 7 (excellent).The quality score was used to decide the inclusion or exclusion of a primary study.The cutoff scale-point, selected by us, to exclude studies from the list of selected studies was 2.0 (very poor or fair quality).A total of 29 studies were excluded from this stage.As a final result, we got to 108 studies to be analyzed.Appendix A presents the reference of the selected studies.

Data Extraction and Synthesis
We considered different facets for classifying the studies.The facets were defined according to the RQs and taking the selected studies into account.So, depending on the focus of each scheme, the study was classified as one or any combination of this.Following, the categories of these facets are presented.
Purposes (RQ2): in this facet, we wanted to learn the purposes, when applying CMs in Computer Science.We have identified five main categories of such purposes:

Learning Level Evaluation:
• it is related to the academic environment and quantifies how much knowledge was absorbed by a person from the use of CMs.

Assistance in Teaching and Learning:
• this classification is related to the initiatives that use CMs as an educational tool in Computer Science.

Knowledge Organization:
• from the structure of a CM, stakeholders can easily navigate through desired topics.This category is related to the initiatives that use CMs as a means to organize any knowledge on a particular subject.

Software Visual Representation:
• this classification is designed to incorporate the initiatives that use CMs as a conceptual model for software requirements that assist stakeholders in the development, evaluation and maintenance of software.

General:
• this category is used to classify those studies that discuss CMs in Computer Science in general.Evaluation (RQ5): this facet discusses the main guidelines or mechanism used to evaluate the CMs identified in the included studies.We adopted the following classification: (i) Support of a software for comparison; (ii) Structural with expert map; and (iii) Relational with expert map.

Subareas applied (RQ3
Reported benefits (RQ6): the categories for this facet are based on the main benefits related to use of CMs reported in the included studies.We have identified four main categories of benefits, namely: (i) Flexible structure and easy to understand; (ii) Knowledge representation; (iii) Knowledge Sharing; and (iv) Pedagogical contribution.
Reported Problems (RQ7): the categories for this facet are based on the main problems highlighted by the authors of the included studies related to use of CMs in Computing Science.We have identified four main categories of problems, namely: (i) Difficulty in creating the CMs; (ii) Cognitive overload; (iii) Increased workload; and (iv) Scalability.

Results of the Mapping
The results of our mapping study are presented.We used the facets of the classification schema aforementioned to answer the RQs.Appendix B presents the mapping of individual studies and its categories.

Frequency of Publication (RQ1)
In order to offer a general view of the efforts in the area of CMs in Computer Science, a distribution of the 108 selected studies over the years is shown in Fig. 3.As this figure suggests, although the research of CMs in Computer Science is not recent, the interest in CMs has been moderately increasing within 2006-2015, with a significant increase in 2011 and 2012.

Purposes of CMs (RQ2)
Fig. 4 shows the number of studies per purposes of application.We can notice that Assistance in Teaching and Learning (63 studies -58.33%) and Knowledge Organization (40 studies -37%) have the largest representativeness.Some purposes that were identified are strongly related.For instance, knowledge organization aims to support assistance in teaching or learning.Thus, many studies were considered in both categories.Furthermore, many studies assigned in those categories use CMs as a means of teaching undergraduate courses of Computer Science and Software Engineering.CMs are effective educational tools which consist of concepts (Atapattu et al., 2015a) and expertgenerated CMs can be used to guide students as in a content repository (Williams et al., 2012).Maps can improve learning when used as an advanced organizer in the classroom (Faily et al., 2012).
CMs can also be employed more effectively to determine the depth of knowledge gained by a student.A total of 19 studies (approximately 17,6%) have as a purpose Learning Level Evaluation.According to Gary et al. (2012), CMs offer an alternative assessment tool that can be used to show how much a student may have acquired knowledge.
Even with a smaller percentage of studies, CMs have also been used for Software Visual Representation (7 studies -6.5%) and it presented interesting results in software development.For instance, CMs have been used in object-oriented modeling (Sien, 2010), Software Requirements (Coffey et al., 2012;Faily et al., 2012;Kof et al., 2007), Software Security Assurance (Snider et al., 2014), process of developing software systems (Gurupur et al., 2014) and Software Assessment (Williams et al., 2012).

Subareas Applied (RQ3)
Fig. 5 shows the number of studies considering the different subareas in computing science.Note that most of the studies (76 studies -70.4%) did not clarify to which computing science sub-area the work is related.The studies discuss initiatives with CMs in computing science in a general form.For example, the studies (Álvarez Bermejo et al., 2015;Atapattu et al., 2015a;Huang et al., 2012) argue that the use of CM helps to conceptualize, schematize and present procedural information like software documentation, explaining laboratory reports, in classroom.The studies argue how the techniques for representing knowledge, as CM, can be effectively used in a classroom for the understanding of disciplines in computing science.
Looking at the studies that present initiatives in a specific sub-area of Computer Science, two categories have greater representation: Programming Languages (10 studies -9.6%).

•
Software Engineering (12 studies -11.1%).• In the Programming Languages category, CMs have been used mainly to present the Object-Oriented Programming (OOP) paradigm.For example, (Berges and Hubwieser, 2013;Berges et al., 2012;Dogan and Dikbiyik, 2016;Hubwieser and Mühling, 2011), presented results investigating how CMs have been used in OOP teaching and learning.Table 6 presents the other programming languages.
In relation to Software Engineering, CMs have been applied with the purpose of assisting in different activities that involve software development process (Coffey et al., 2012;Faily et al., 2012;Upchurch and Sobel, 2002;Williams et al., 2012), as well as how to organize concepts in Software Engineering domain (Gurupur et al., 2014;Koch and Landes, 2015;Leake and Wilson, 2001;Mohamed et al., 2013).In order to demonstrate this purpose, some studies are described below.In Coffey et al. (2012); Faily et al. (2012); Kof et al. (2007), for instance, CMs were used to make sense of and improve the quality of a software requirements specification and representation of traceability between requirements.In Gurupur et al. (2014) a process of developing software systems is described by capturing the conceptual domain knowledge of the problem domain using CMs.According to Gurupur et al. (2014), there is a gap between user's needs and software implementation efforts.This problem persists mainly because software systems often do not fully support user needs.Leake and Wilson (2001) describe a research on supporting aerospace design by integrating a case-based design support framework with interactive tools for capturing expert design knowledge through CMs.
Regardless of the sub-areas in which CMs are used, it is possible to note that the main motivation for integrating CM is the fact that it makes easy to represent and understand a specific domain.

Resources (RQ4)
In this RQ we investigated what were the resources used in the studies to support the CMs construction.Nine different resources are highlighted in this context.Fig. 6 shows the number of studies considering the different resources used to construct CMs.
The great majority of the studies use their own systems (58 studies -53.7%).The studies in this category propose the construction of a new system to assist in the creation of CMs in order to share and visualize knowledge.Many of the studies in this category suggest the construction of a tool for building CMs collaboratively (15 studies -13.9%).Recent approaches of teaching and learning put emphasis in activities that take place in a collaborative frame.Collaborative learning approaches seem to encourage knowledge construction and it can lead to better development of ideas and concepts through discussion and negotiation.For instance, Tomoto et al. (2011) presented a collaboration tool for knowledge integration and sharing in cross-cultural communication using a participant-constructed collaborative CM.MaGIA tool was presented in Blecic et al. (2007), an Internet-based multi-user system designed for a collective and collaborative construction of knowledge models represented as CMs.Martinez-Maldonado et al. (2015) described the Tabletop-Supported Collaborative Learning (TSCL), a conceptual model that provides foundations for building tabletopbased systems that can inform understanding of the collaborative learning process.Another resource used by many studies for the construction of CMs is the Concept Map Editor (Cmap Tools1 ) (34 studies -31.4%).Cmap is a traditional tool, commonly used to support the construction, navigation and sharing knowledge models represented through CMs.Besides providing basic operations to draw and label CMs, CmapTools includes extensive capabilities, for example, to link CMs with electronic resources, such as, images, diagrams, video clips, and other CMs, enabling the construction of richly connected concept-map-based knowledge models for particular domains.It also enables distributed storage and access to CMs on multiple servers, to support knowledge sharing across multiple sites (Cañas et al., 2004).
Besides the two most used resources, presented above, it is interesting to note the diversity of other resources used to support the construction or manipulation of CMs.For example, the creation of CM based on an existing ontology; the extraction of concepts and relationships from a text to automatically generate the CMs (called concept mining map); parser in XML-based document to extract concepts and their associated relations; and the traditional way of constructing CMs using paper and pencil.The studies in each one of these categories are presented in Appendix B, Table B3.

Evaluation of CMs (RQ5)
One of the main issues related to CMs is how to evaluate them.Several scoring schemes for human-based evaluation have been proposed.The analysis presented in Anohina and Grundspenkis (2009) shows that most of them are based on quantitative measures, that is, structural analysis (number of valid propositions, levels of hierarchy, etc.).Another method is to compare CMs created with one or more experts' maps.Currently, there are also proposals for automated evaluations of CM.
In this facet we analyze whether the CMs presented in the 108 studies that were left in this mapping were evaluated.Only 38 (35.1%) presented some type of evaluation.Then, we classified the main mechanism used to evaluate CMs identified in the included studies.Three types of evaluation were identified: (i) Relation with expert map; (ii) Structural analysis; and (iii) Support of a software to compare.Fig. 7 shows the number of studies considering the different types of evaluation used.
The first classification relates to evaluate the CM comparing with an expert's CMs.Normally, a CM which was developed by an expert is used as reference for evaluating other CMs created later.The great majority of the studies included in this mapping used this classification to evaluate the CMs generated (23 in 38 studies, corresponding to 84.2%).In Grundspenkis (2008), for instance, an approach is presented in which ontologies are transformed into CMs to be used for systematic creation of learner knowledge structure.In order to evaluate the CM created, a knowledge evaluation agent compares the learner's CM with a teacher's map and corresponding ontology, and recognizes synonyms and several patterns (correct or incorrect) of learner's solution.Rigby et al. (2009) examined means of increasing learners' expertise of Computer Information Technology course about the concepts through the use of varying types of representations, as CMs.The students created CMs of their understanding of threat analysis, which was then scored against an expert CM.Finally, Qasim et al. (2013) proposed an algorithm for automatic candidate term extraction from text documents to construct a CMs.CMs were verified by experts with doctoral degrees in the field of Information System.
The second classification refers to the structural analysis consisting of the comparison of CMs considering the number of valid propositions, levels of hierarchy, number of concepts, etc.In Gurupur et al. (2015) a tool is described that can be effectively used to evaluate student learning outcomes using CMs and Markov chain analysis.The CMs created are compared with each other from the calculation of number of concepts in hierarchy, number of support level concepts and number of detail level concepts.In Arruarte et al. (2014) a study was conducted using the concept mapping technique in computer engineering education for learning theoretical introductory database topics in Fig. 7. Evaluation of CMs. a collaborative way.Some sources of data have been analyzed to evaluate the results of the study, as individually created CMs and collaboratively created.CMs were evaluated in terms of the average number of nodes, average number of nodes relationships, completeness and thoroughness.
The third classification refers to evaluate CMs with software support.In Larraza-Mendiluze and Garay-Vitoria ( 2013), for instance, a process is presented to analyze CMs created by students about computer structure subject.All the CMs were merged to show the relationships that were missing.Another example of this category can be observed in Vaishnavi et al. (2016).The CMs elaborated by students are converted to a XML format and concepts and their relationships are identified.A probability distribution of concepts in the CM developed is calculated.The normal distribution is then represented as a form of learning curve.The evaluation of student's level of understanding content is assessed by analyzing the learning curve generated by this approach.

Reported Benefits (RQ6) and Problems Using CMs (RQ7)
As we can see from this mapping study, there are many benefits of implementing CMs in Computing Science: Flexible structure and easy to understand: • CMs are a simple graphical representation and it is easy to understand a link between different concepts.

Knowledge representation:
• knowledge represented in a visual format makes easier the recognition and the understanding of incoming information.In many disciplines various forms of CMs are already used as formal knowledge representation systems, for example: semantic networks in artificial intelligence, bond graphs in mechanical and electrical engineering, Petri nets in communications, requirements in the software development process.

Knowledge sharing:
• researchers have investigated CMs for collaboration through map sharing.Semantic integration has received a significant amount of attention in recent years, particularly from database researchers seeking to facilitate information sharing and integrate heterogeneous data sources (Kristensen et al., 2009).

Pedagogical contribution:
• in educational area CMs can be used for supporting learning and also for assessing learning (Arruarte et al., 2008).It is possible to note from our results (Fig. 8) that there are many studies that dealt with the effect evaluation of CMs on students' achievements in a wide diversity of areas, specially in Computer Science.According to Gurupur et al. (2015), CMs provide visual data to instructors on student misconceptions and their level of understanding.Hence CMs can be helpful to develop certain abilities such as, drawing reasonable inferences by identifying key concepts on a topic, synthesize and integrate the information and ideas, and learn concepts and theories in a study area.
Fig. 8 shows the main benefits in using CMs.The pedagogical contribution through CMs appears most representative.The fact that CMs can be used for different pedagogi-cal purposes, together with the nature of Computer Science, its many facets representing different abstraction levels and the interconnections among them, makes CMs relevant for Computer Science teaching and learning processes in general.Interestingly, despite the use of CM bringing many benefits, some authors, even though few, also highlight problems that may arise from the use of CM, such as:

Difficulty in creating the CMs:
• it is important to have the knowledge domain on the subject to develop a CM with quality.

Cognitive overload:
• in terms of the short-term memory, CM has a weakness; it might contain too many concepts in its structure causing learner's cognitive overload, i.e., becomes to understand the content wanted to be learned.

Increased workload: •
shortage of time is a potential risk to incorporate the use of CMs when working with deadlines, for example, in software development companies.The development or learning of another artifact can imply in increasing the employee workload and costs.However, automatic or semi-automatic creation of CMs from text documents is a worthwhile research very investigated nowadays (Chen et al., 2008;Qasim et al., 2013).

Scalability: •
CMs with large scale quickly become too complex to be processed by the learner.A representation of a CM can lose its utility when the amount of information increases.The high number of concepts and cross-links between concepts makes difficult to understand the represented knowledge.
Fig. 9 shows how the main problems aforementioned are perceived in the included studies.It is important to highlight that not all studies mention problems related to the use of CMs.Cognitive overload was the problem more cited in the studies.In fact, cognitive load can becomes a problem when the domain of application is large, and consequently, CMs become overburdened with too much information and complexity (Williams et al., 2012).According to Williams et al. (2012), this phenomenon (termed "map shock") results in student disengagement and reduction of expected learning gains.

Discussion
CMs were first proposed in education, to enable students to externalize their knowledge by constructing a based-representation of concepts and their relationships.This representation has been considered as a cognitive structures suitable for assessment or knowledge sharing.CMs are used worldwide to facilitate knowledge examination, construction, comparison, and reuse by users ranging from elementary graduated students to researchers.In the Computer Science area, this is no different.The main contribution of this research is to evidence how CMs have been used in Computer Science.This systematic mapping has shown that CMs have been heavily used with different purposes and in different subareas.
Within the Computer Science context, several studies reported that the CMs have been used, for example, to support artificial Intelligence, Human-Computer Interaction, Programming Languages and Software Engineering (Berges et al., 2012;Kof et al., 2007;León et al., 2008;Portmann et al., 2012).Many of the studies included in this mapping use CMs on general Computer Science subjects, that is, the subarea is not specified.On the other hand, Software Engineering and Programming Languages, although with few studies in the literature, were the subareas that stood out the most.It is possible to identify that CMs can provide important benefits in the representation of some programming languages paradigms (OOP, C), in the development of software product, in improving the requirements specifications quality or in software requirements traceability (Coffey et al., 2012;Gurupur et al., 2014).
Considering the two most outstanding subareas, Software Engineering and Programming Languages, we did an analysis of the year in which the studies were published.Software Engineering, for example, out of the 11 articles in this category, eight were published from 2009, and 3 in 2013.This shows that the interest in CM in software engineering can be considered recent.Similarly, Programming Language presents itself as a subarea that with interest in using CM recent, starting in 2003 to 2016.The subareas were also analyzed according to their purposes (RQ2).The selected studies of Software Engineering have as main purposes the Knowledge Organization and Software Visual Representation with 81.8% of representatively.For example, in Gurupur et al. (2014), semantic requirements sharing approach is shown to reduce the semantic gap involved in the process of developing software systems.According to Gurupur et al. (2014), sometimes a requirements engineer may not fully comprehend the problem domain.Therefore, a CM allow to build a schematic representation of the domain knowledge and its can be easily comprehended by every individual involved in the development of a software system.
The studies selected in the Programming Language subarea, six of the ten studies has the purpose of Learning Level Evaluation.This result makes sense, since CMs have been proposed as a potentially facilitative learning strategy.According to Hubwieser and Mühling (2011), in science education, there are many research activities that use CM in order to investigate cognitive structures.Consequently, CM can be used to measure the knowledge level and/or learning in programming disciplines.
We analyzed in the selected studies in order to identify which were the tools used to construct CM to represent programming language, for example.The same happens in other categories.The main tool used in the programming language subarea were property systems.CMs have been created mainly by systems or applications developed by the authors that wrote the papers.This happens because tools available in the literature do not always meet the different purposes of each study.This causes researchers to create a new tool or customize an existing tool.In addition, it is important to note the priority that studies have given to CM construction in a collaborative way from the tools created.According to Tomoto et al. (2011), our world becoming globalized needs no borders but must be opened to cross-cultural, collaborative communication in every social field, including education.International distance education could more broadly help participants experience concept construction in the distance communication environment.Technological contributions can contribute to achieving these goals, adding a new dimension to collaborative learning.
One result that attracted attention was in relation to the evaluation of CMs.Although there are in the literature numerous systems or methods of CM evaluation, only 35.1% of the CMs included in this mapping were evaluated.The minority of included studies in this mapping (38 of 108) reported some CM evaluation, being: Relation with expert map, structural analysis and support of software to compare are examples of evaluation types.
In addition to the benefits already mentioned about the use of CMs, we argue that they are also useful in secondary studies context, discussed as follows.Many researchers conducting secondary studies, as one of their first activities, need to read the studies abstracts to decide about their inclusion or exclusion.In some domains, such as clinical Medicine, Education, Psychology and Social Sciences, the review of abstracts is more than enough to determine its relevance (Kitchenham, 2004).However, in Computer Science, especially, in Software Engineering, there are unstructured abstracts in the studies.This becomes difficult to identify the relevance or not of the studies (Brereton et al., 2007).Recent literature has provided evidence that unstructured and poorly written ab-stracts may compromise the selection activity (Brereton et al., 2007;Dybå and Dingsøyr, 2008;Dybå et al., 2007;Kitchenham, 2008;Petersen and Nauman, 2011;Zhang andMuhammad, 2012, 2011a,b).
One potential solution to minimize such problem, experienced during a secondary study selection, is to promote the use of structured and graphical abstracts in Software Engineering (Hartley, 2004;Kitchenham, 2008).Since CMs can help in the understanding of a study, our research group have investigated the creation of graphical abstracts to support the selection activity in secondary studies using CMs (Felizardo et al., 2017).Graphical abstracts should enable reviewers to manipulate images to better understand the knowledge there represented (Oliveira and Levkowitz, 2003), since analyzing data in graphical format requires less cognitive effort from the reviewer to extract information.The main objective of the visualization is to help users explore and understand data (abstracts) (Keim, 2002).Visual representations allow users making use of their optical capabilities to strengthen the process of knowledge acquisition (Oliveira and Levkowitz, 2003).In this context, CMs can be useful tools to summarize a complex structure of textual information, contributing in identifying the most relevant information in an article.The results of this mapping will be used by us to support the construction of graphic abstracts based on CMs.

Limitations
The current review has some limitations.The study selection and data extraction steps were initially performed by just one of the authors, and thus some subjectivity could have been embedded.In order to reduce this subjectivity, the other authors performed these same steps over a sample of studies.The results of each reviewer were then compared in order to detect possible bias.
As with all reviews it was limited by the search terms used and the electronic databases included.However the studies discussed in this mapping provide a snapshot of empirical research on outcomes and impacts of existing research on CMs initiatives in Computer Science.We tried to overcome the limitations by using previous simulations in the selected databases.We just considered studies indexed by the selected electronic databases, and those obtained from by snowballing and direct search for works from researchers/research groups.The exclusion of other sources makes the review more repeatable, but possibly some valuable studies may have been left out of our analysis.

Conclusions
A CM study gives an idea, in the early phases, of shortcomings in existing evidence, which becomes a basis for future studies Kitchenham and Charters (2007).This study presents a review of published literature on CMs initiatives in Computer Science in order to identify the state of the art.Seven research questions were defined and addressed investigating the following facets: (i) distribution of the selected studies over the years; (ii) purposes to employ CM in Computer Science; (iii) different subareas that CMs have been applied; (iv) resources used to support the development of CMs; (v) guidelines or mechanism used to evaluate the CMs; (vi) main benefits reported on the use of CMs; and (vii) main problems reported on the use of CMs.
The most notable point about the current review was the diversity of research on positive impacts and outcomes associated with the use of CMs in Computer Science.Although it is a tool considered old in other areas, CMs in Computer Science have shown to be very promising research area, since CMs helps to provide visual representation of knowledge structures to assistance in teaching and learning.
The main contribution of this work is on making evident some aspects associated to the employment of CMs in Computer Science, in order to drive future research in this area.In this context, we highlight the following conclusions: (i) CMs in Computer Science is not a recent research, but has had a considerable increase in recent published studies; (ii) Assistance in Teaching and Learning and Knowledge Organization are the main purpose of applying CMs in Computer Science; (iii) there is a large diversity of the subareas of Computer Science in which CMs have been applied, such as Programming Language and Software Engineering; (iv) there seems to be no culture to evaluate CMs, which may impair the accuracy of CMs developed; (v) although CMs presents several benefits related to knowledge representation, sharing and pedagogical contribution, some problems also can be highlighted, such as cognitive overload, increased workload and scalability.

Appendix A. Included studies.
Table A1 presents the citation of the selected studies plus an identifier (ID) for each paper.In Appendix B, we use these identifiers to refer to the corresponding publication.Studies per Reported benefits .

Reported benefits Studies
Flexible structure and easy to understand
): this facet refers to the different subareas that CMs have been applied in Computer Science.We have identified seven main subareas: (i) Artificial Intelligence; (ii) Software Engineering; (iii) Database; (iv) Ontologies; (v) Human-Computer Interaction; (vi) Programming Languages; and (vii) Not Specified or General.Resources (RQ4): this facet discusses the different resources used to support the development of CMs in Computer Science, as tools, techniques or methods.Based on the selected studies, we considered the following categories: (i) XML Schema; (ii) Text Mining; (iii) Paper and Pen; (iv) Ontologies; (v) Own System; (vii) Intelligent Knowledge System; (vii) Data Ink; (viii) Collaborative Construction; (ix) Concept Map Editor; and (x) Not Mentioned.

Fig. 3 .
Fig. 3. Distribution of the selected studies over the years.

Fig. 4 .
Fig. 4. Purpose of applying CMs in Computer Science.

Fig. 5 .
Fig. 5. Subareas that CMs in Computer Science have been applied.