Guidelines for performing Systematic Research Projects Reviews

T number of scientific articles published, regardless of the academic discipline, has dramatically increased in the last decades. The publication in impact journals is considered one of the KPI (key performance indicators) in research centres and one of the measures to get funds. Moreover, in the current information society, most of the published works are available in online journals, repositories, databases, so researchers have access to them. One of the first tasks before conducting a research, regardless of the field of study, is to identify related works and previous studies as a way to support the need to conduct new research on a particular topic. Likewise, the review of available research provides answers to particular research questions and a knowledge base to learn from previous experiences and identify new research opportunities. Nevertheless, although the need to synthesise research evidence has been recognised for well over two centuries, it was not until the end of the last century that researchers began to develop explicit methods for this form of research. In particular, a literature review allows for achieving this objective. According to Grant and Booth [1], it involves some process for identifying materials for potential inclusion, for selecting included materials, for synthesizing them in textual, tabular or graphical form and for making some analysis of their contributions or value. There are different review types and associated methodologies. Specifically, before 1990, narrative reviews were typically used, but they have some limitations such as the subjectivity, coupled with the lack of transparency, and the early expiration because the synthetization process becomes complicated and eventually untenable as the number of studies increases [2]. The systematic review or systematic literature review method seeks to mitigate the limitations of narrative reviews. Systematic reviews have their origin in the field of Medicine and Health. Nevertheless, the logic of systematic methods for reviewing the literature can be applied to other areas of research such as Humanities, Social Sciences or Software Engineering; therefore there can be as much variation in systematic reviews as is found in primary research [3], [4]. A systematic review is a protocol-driven comprehensive review and synthesis of data focusing on a topic or related key questions. It is typically performed by experienced methodologists with the input of domain experts [5]. The systematic review methods are a way of bringing together what is known from the research literature using explicit and accountable methods [4]. According to Kitchenham [6][8], a systematic review is a means of evaluating and interpreting all available research relevant to a particular research question, topic area, or phenomenon of interest by using a trustworthy, rigorous, and auditable methodology. The analysis of related works and previous studies is not only associated with scientific literature. Another KPI in research centres is the number of projects funded in competitive calls. Project proposals, like other formal studies, have to justify the need to conduct them. Furthermore, most of the calls for funding projects require to justify the innovation of the proposal against other developed projects. Although it might be expected that the results of all funded projects are available in scientific publications, this is not always the norm. Determining the progress made through a research project requires the Guidelines for performing Systematic Research Projects Reviews


I. Introduction
T HE number of scientific articles published, regardless of the academic discipline, has dramatically increased in the last decades. The publication in impact journals is considered one of the KPI (key performance indicators) in research centres and one of the measures to get funds. Moreover, in the current information society, most of the published works are available in online journals, repositories, databases, so researchers have access to them.
One of the first tasks before conducting a research, regardless of the field of study, is to identify related works and previous studies as a way to support the need to conduct new research on a particular topic. Likewise, the review of available research provides answers to particular research questions and a knowledge base to learn from previous experiences and identify new research opportunities. Nevertheless, although the need to synthesise research evidence has been recognised for well over two centuries, it was not until the end of the last century that researchers began to develop explicit methods for this form of research.
In particular, a literature review allows for achieving this objective. According to Grant and Booth [1], it involves some process for identifying materials for potential inclusion, for selecting included materials, for synthesizing them in textual, tabular or graphical form and for making some analysis of their contributions or value. There are different review types and associated methodologies. Specifically, before 1990, narrative reviews were typically used, but they have some limitations such as the subjectivity, coupled with the lack of transparency, and the early expiration because the synthetization process becomes complicated and eventually untenable as the number of studies increases [2].
The systematic review or systematic literature review method seeks to mitigate the limitations of narrative reviews. Systematic reviews have their origin in the field of Medicine and Health. Nevertheless, the logic of systematic methods for reviewing the literature can be applied to other areas of research such as Humanities, Social Sciences or Software Engineering; therefore there can be as much variation in systematic reviews as is found in primary research [3], [4].
A systematic review is a protocol-driven comprehensive review and synthesis of data focusing on a topic or related key questions. It is typically performed by experienced methodologists with the input of domain experts [5]. The systematic review methods are a way of bringing together what is known from the research literature using explicit and accountable methods [4]. According to Kitchenham [6]- [8], a systematic review is a means of evaluating and interpreting all available research relevant to a particular research question, topic area, or phenomenon of interest by using a trustworthy, rigorous, and auditable methodology.
The analysis of related works and previous studies is not only associated with scientific literature. Another KPI in research centres is the number of projects funded in competitive calls. Project proposals, like other formal studies, have to justify the need to conduct them. Furthermore, most of the calls for funding projects require to justify the innovation of the proposal against other developed projects.
Although it might be expected that the results of all funded projects are available in scientific publications, this is not always the norm. Determining the progress made through a research project requires the analysis of the project itself, not only through its scientific publications but also through the information available (implementation details, results, etc.). Besides, as in any scientific study, it is important to ensure that the new project proposal does not repeat work previously done, or a research design found previously to be ineffective.
There is no established methodology that allows carrying out a systematic analysis of the studies and progress made through research projects in a specific area or topic. According to a systematic review conducted, even though there exists literature related to projects' review, the process is not fully systematised.
The research projects generate many types of results, and scientific publications are a small part of them. There are initiatives to facilitate open access to scientific data at national levels [9], and increased efforts to co-ordinate and support global data networks are necessary [10], but not all the projects are data-driven and data sets are not the only results in data-driven projects. The main problem to implement a systematised review process with research projects is their differences with the scientific literature. The research projects are a compilation of several types of documents, data sets, software, with different purposes. Even in projects inside the same funding call, that have to follow the same guidelines and meet the same requirements at the end of the funding period, the information available is entirely different. In fact, most of the information generated inside a research project is not publicly accessible, although it depends on the regulations of the funding call. Moreover, unlike the scientific literature, not all the research projects are available in accessible databases. Likewise, the project databases usually do not enhance projects findability with metadata, and the search tools provided are simple.
This work aims to present a set of guidelines to support systematic reviews of research projects as a way to summarise, synthesize, critique, and use that information to identify trends and lacks, justify the innovation of a new research project or collect valuable results that could be applied in other context beyond the project in which they were developed. The paper has been divided into five sections. The second section describes a review of other works focused on methods and guidelines to carry out systematic reviews, as a way to justify the added value of this work. The third section describes the set of guidelines to apply systematic reviews of research projects. The fourth section describes two examples in which the guidelines were applied. Finally, last section summarises the main conclusions of this work.

A. Identifying the Need of a Systematic Methodology
Literature reviews, and more specifically Systematic Literature Reviews (SLR) have reached a considerable level of adoption in many research fields. A literature review is a search and evaluation of the available literature in a given topic, providing state of the art about previous research and giving an overview of what are the strengths of the area of interest and which weaknesses need further improvement. Different methodologies have been proposed for performing literature reviews but most of them share the same base structure defined by the SALSA (Search, AppraisaL, Synthesis and Analysis) framework [11].
Within this framework, the aim of the Search phase is to gather a preliminary list of publications to analyse. During the Appraisal stage the papers collected during the previous stage are analysed, eliminating those that are irrelevant. Finally, the Synthesis and Analysis stages extract relevant data from the selected papers and draw conclusions from them in order to produce a final report.
In order to investigate the previous works carried out in the field of the systematic review of research projects, we have performed a systematic review of the related literature. The following subsections describe the process undertaken, which follows the recommendations of Kitchenham [7] and Petersen [12] regarding the methodologies for conducting systematic literature reviews and mapping studies.

Database Selection
In terms of the information sources of the papers to be included in the search process, we selected the Web of Science and Scopus electronic databases, as they fulfil the following requirements: • The database is available for us through our institution.
• The database can use logical expressions or a similar mechanism.
• The database allows full-length searches or searches only in specific fields of the works.
• The database allows additional filtering options such as publication year or publication language.
Also, as there are many entities that fund research projects in Europe through different programs and calls, we considered including them in the search for related work. The motivation for this is that during the preparation and execution of such projects it is mandatory to carry out searches for related projects. The results in the form of reports and publications are published in different databases and webpages. However, the primary public repository and portal to disseminate information on all EU-funded research projects and their results is the Community Research and Development Information Service (CORDIS), (https://cordis.europa.eu). This database fulfils the following requirements: • Many of the results are publicly available.
• It is a reference database in the research scope.
• It allows using a search string equal or similar to the ones used in the selected scientific databases.
• Provides means to filter the obtained results.

Inclusion Criteria
A set of inclusion criteria (IC) was defined to select those works that are relevant in the scope of the considered related work: • IC1: The document focuses on a systematic review of research projects AND • IC1: The document describes the followed review methodology AND

Query String
To create the search string, we identified the main terms related to the review scope and the possible alternative spellings and synonyms. Based on them the employed query for all the databases was: ( "systematic review method" OR "systematic review methods" OR "systematic review guideline" OR "systematic review guidelines") AND ( projects OR "R&D" )

Review Process
After defining the sources, search string and appraisal criteria, the following steps were followed for the literature review: 1. Collect raw results in two different spreadsheets for the scientific and CORDIS databases (https://bit.ly/3dRkLbn, https://bit. ly/2z2IA1a). After removing all the duplicates across the databases, we obtained 509 results.
2. Analyse the resultant documents based on the title and abstract and the inclusion/exclusion criteria. In those cases where the title and abstract were not sufficient to decide, the authors quickly assessed the entire content of the paper. The resultant candidate papers (118) were added to another two spreadsheets (https://bit.ly/2WFc9Pi, https://bit.ly/2z2IC9i).
3. The documents then passed a quality assessment, looking for those that clearly describe a methodology for performing Systematic Research Projects Reviews. Results were collected in https://bit. ly/3bIAGHE and https://bit.ly/2LzA3FJ.

Results
After following the abovementioned systematic search, appraisal and analysis of documents, it was observed that the majority of papers focused on systematic methods for scientific literature reviews of research documents. However, we did not find any document in the form of research paper nor project deliverable or report in the consulted databases that fully describes a systematic methodology in order to review research projects.

B. Identifying Reviews of Projects that Partially Follow the SALSA Framework
As we did not find any systematic methodology proposal for reviewing research projects, we performed a second literature review. The objective was to gain a better perspective on other authors' approaches to systematic reviews of projects, in terms of how they had followed any of the steps described within the SALSA framework.
We opted for broadening the search and softening the inclusion and quality criteria in order to find proposals that had made a systematic search of projects, even if the purpose is not to propose a unified method of systematization of the review. The search also included the grey literature, provided that it was available. The process undertaken was as follows: 1. We decided to conduct this second search in the Google Scholar database and employed the search string: "systematic review of projects". The search returned 3,390,000 results.
2. We further filtered the results using the advanced search options and limited the output to documents: "with the exact phrase | anywhere in the article: systematic review of projects ". After applying the advanced filtering, we obtained 99 results.
3. The inclusion criteria were limited to documents that focused on a systematic review of research projects, even though they do not fully describe the methodology. Also, we removed duplicates, Scrutinized AMIF and ESF databases (related to integration, social inclusion of migrants and refugees into the labour market) Period was limited to January 2014 through May 2019. Used words in advanced search: e.g. "Migrant", "refugee", "integration"

Not described
Organize projects (calculate % of the total) by keywords, regions, topics, integration level

[13]
Different databases: sectorspecific databases, NGO websites, and peer-reviewed academic literature and gray literature. Used different search terms for each database

Not described
Created two indices (ASI, MSI) and 4 criterions: if criterion is present => scored as 1.0. If referred to without explicit identification => scored as 0.5.
For the ASI, the four scores were summed together, and for the MSI they were multiplied Kruskal-Wallis nonparametric one-way analysis of variance tests to test for differences in ASI and MSI values

[14]
All 152 projects from the CSHGP database Projects that began after October 1, 2000; had some level of effort for malaria, pneumonia, and/or diarrhoea; provided curative interventions; and had project documentation with CCM and contextual detail Tabulated 11 self-made-items (indicators) fulfilled with quantitative and qualitative information Calculated the "indicator yield" across projects (proportion of projects measuring a given indicator) and "indicator density" within projects (proportion of recommended indicators measured by a given project)

[15]
Self-created database of projects through consultation with the study oversight panel Describe a list of criteria to select the appropriate cases, including project characteristics, reviewer, project stage, number of review comments and geographic distribution Define a set of quantitative metrics: benefit from project level evaluation, estimated savings, etc. and qualitative metrics: "extracted from review comments of the project" Probabilistic and regression analysis Review programs from agencies, reports and a Lesson Learned database. Not systematic

Not described
Authors define a set of categories to be identified within the projects Identify the most frequently used categories within the existing practices (in %)

[17]
A systematic review of projects in all county councils in Sweden was performed The final sample consisted of documents from the planning and design process and the target organizations' strategic operational plans. Overall, 45 various documents were reviewed from five building projects that had a budget of over 50 million € (US$5,493,000.00) and were executed between 2010 and 2014 in Sweden Qualitative content analysis. Data from documents were organized into a matrix with five different levels of headings: meaningful unit, condensing meaningful unit, code, subcategory, and category Not described quotes, not available documents and those not written in English, 68 final references were obtained. 4. The resultant documents were read in detail and analysed in order to assess which of the steps of the SALSA framework were systematized and described. The results from this final step can be further consulted in https://bit.ly/3g2Zu0D. Likewise, the Table  I summarises the documents which fulfil the SALSA framework.
It can be observed that none of the studied documents completely describe a systematic process throughout all the SALSA framework. In terms of the search stage, only one document fully describes the database selection process and criteria, along with the employed search string and its motivation [13]. Only two documents, [14] and [15], describe a systematic procedure for the appraisal, synthesis and Analysis steps. However, in those cases the search phase is either performed over all documents of a particular database [14], or performed over a self-created database which construction process is not detailed in the document [15]. On the other hand, in [11] the appraisal and analysis of documents is systematised and properly described. However, the search phase is vaguely mentioned, and the synthesis phase is omitted.
In [16] authors define a set of categories during the synthesis phase and calculate their percentage of occurrence in the considered projects during the analysis stage. However, in this case, the search phase is only partially described and there is no appraisal phase to systematise and validate the selection of considered projects. Finally, the work of [17] follows SALSA methodology during the search appraisal and synthesis stages, but omits analysis of the data obtained.
As a conclusion of the systematic process of reviewing the related work, we can conclude that no works were found that describe a methodology or procedure for the systematic review of research projects. Also, even though there exists extensive literature related to projects' review, it is not fully systematised, and although some works were found which take into account some of the SALSA framework steps, none of them completely describes the SALSA framework.

III. The Method
The systematic review of research projects, also called Systematic Research Projects Review (SRPR), is based on the Kitchenham's adaptation of the systematic literature review (SLR) [6]- [8] and the Petersen's proposal to carry out systematic mapping studies [12], [18]. The high number of projects that are developed annually makes it impossible to analyse all of them. SRPR enables the selection of a set of projects that fulfil a particular criterion; likewise, SLR facilitates the review of scientific papers that would be impossible to handle otherwise. The main objectives that can be achieved through SRPR are: • To identify trends in research projects.
• To identify lacks to define new research projects.
• To justify the innovation of new research projects proposals.
• To collect valuable results that could be applied in another context beyond the project in which they were developed.
This work does not seek to reinvent the protocols of the systematic review, but to adapt them in order to review the compendium of resources, documents, information, which form a research project. Table II shows the main differences at the macro level between a SLR and a SRPR.
The SRPR is divided into four phases with a set of steps inside each one. These phases are quite similar to those defined in the SLR, but the main differences are related to the implementation of the steps. The first and second phases are focused on the definition of the review, and the third and fourth phases are related to the retrieval, appraisal, synthesis and analysis of the research projects. Fig. 1 shows the main phases and steps that compose the SRPR method.

A. Study Definition
The first phase is focused on the preparation of the research projects' review. In particular, it is focused on the objective of the SRPR. This phase does not differ from a typical systematic literature review. It is composed of two steps: identifying the need to conduct the review and defining the research questions.
Before carrying out a systematic review, regardless of whether it is scientific literature or projects, we must ensure that it is necessary to carry out the review. Two questions arise in the first step: • Is there already a systematic review that pursues the same objectives?
• Are the necessary resources (people, time, money) available to carry out the review?
According to Petticrew and Roberts [19], it has no sense to conduct a systematic review that has already been done unless the previous systematic reviews are biased or outdated. These guidelines also apply for research projects.
Regarding the second question, an SRPR usually implies more time and effort per reviewer than a SLR, due to the intrinsic characteristics of the research projects. It is recommended to involve a minimum of two reviewers working together, at least, in the second and third phases.
The second step analyses the objective of the review through the PICOC framework [19] to formulate a set of research and mapping questions. In particular, PICOC stands population, intervention, comparison, outcomes and context: • Population (P): the scope of the projects (i.e. local, regional, national, international) and the main topic of the projects (i.e. mental health, educational technology, gender gap, etc.).
• Intervention (I): the intervention applied in the research projects.
• Comparison (C): to which the intervention is compared. For example, a comparison between various calls for projects or between nationally and internationally funded projects.
• Outcomes (O): what the review seeks to achieve, such as identifying trends or lacks or selecting a set of project results focused on a particular objective.
• Context (C): the context of the research projects must be defined, which is an extended view of the population, including in which sectors are developed, i.e. academia or industry.
Regarding the questions, their definition depends on the objective of the review. Despite this, the SRPR provides a set of meta-questions for defining the mapping questions: • What are the trends?
• What types of institutions are involved in the projects?
• In which countries were the projects implemented?
• Which calls fund this kind of research projects?
• Which years cover the projects?
• How much money was invested in the projects?
• In which contexts are the projects carried out?
• Which kind of outcomes are provided by the projects?

B. Screening Definition
The second phase continues the preparation of the research projects' review. In particular, it covers the protocol definition through the identification of the inclusion, exclusion and quality criteria (screening criteria) and devises search strategy. The main differences between SRPR and SLR during the screening definition falls on the search strategy and the quality criteria.
First, it is necessary to define the inclusion and exclusion criteria in order to apply then during the appraisal of the research projects. In particular, these criteria are applied to the title, keywords, and the summary of the project. For this reason, these criteria have to take into account information that could be available in most of the sources, such as the objective of the projects, the publication dates or the official language of the project.
The search strategy is divided into three tasks: define the search terms, build the search string and select the sources. The definition of the search terms follows the same process as an SLR. The PICOC allows identifying the main terms, and this first cohort is extended with synonyms and words used in the scientific context to describe the same concept. Regarding the search string, it is important to keep it simple to ensure that the search string will be the same in all the selected sources. The databases of research projects do not usually have powerful search tools, so a search string that combines different logical operators and nests may be impossible to use.
The most difficult part of the search strategy is the selection of the sources. The databases of research projects are not always accessible and most of them do not have powerful search tools and do not support metadata, so the search strategy in SRPR depends on the available databases much more than the SLR. Furthermore, there is not an international database of research projects such as Scopus and Web of Science. Usually, the databases are related to particular funding calls. There is a set of recommendation to select them: 1. Identify the scope of the research projects. It is possible to combine different scopes in the same review, but it is important to ensure completeness. For example, it is possible combining national and European projects if we take into account the national projects of all the European countries, not only a part.
2. Identify the main funding calls in the selected scope. Although there are projects that obtain funds by other means, it is not possible to access them unless we get access to the research projects databases inside each institution. It is recommended to include different calls and funding periods in those reviews focused on identifying trends and lacks.
3. Identify the databases with the granted projects for each funding call.
4. Search on the Internet other databases of research projects of the same scope.
5. Navigate through the database, run the search string and apply the following requirements to the identified databases. They are an adaptation of [20]: • It is a reference database in the selected scope.
• It is a relevant database in the research area of this review.
• It allows carrying out searches and downloading the results in some accessible format.
• It is a database available through the authors' institution or an authors' membership to an association.
6. The selected sources will be those that fulfil the requirements and return different research projects using the search string.
7. Identify and document the search filters that will be used in each database. Each database has a totally different search tool, so it is necessary to provide all the filtering details and the search string used in each database in order to ensure the replication and transfer of the review.
Finally, to complete the screening definition, it is necessary to define a set of quality criteria or quality questions. The quality criteria for projects are totally different from the criteria used for scientific literature. The information available about the projects are not always the same. Usually, the databases provide title, summary and funding information. For this reason, we must complete the information provided by the databases with information of the project available on the Internet, such as the project website or scientific papers published about the project. The quality criteria should ensure that the screened projects have enough accessible information to answer the research questions. The following quality questions can be combined with other questions more focused on the objectives and the results of the projects: • Is the website of the project available?
• Are the outputs of the project available?
• Is there more information available about the project than the project summary in the languages you speak?
• Are there scientific publications associated with the project?

C. Projects' Selection
The third phase has only one step, the data extraction, which covers the retrieval and appraisal of the research projects. It is a screening process based on several iterations following the guidelines defined in the previous phase: 1. Search: the search strategy is applied in each selected source to collect the projects according to the search terms and the filters described per each source. In those cases, in which the search tool does not support logic operators, the OR operator will be replaced by a set of individual searches that will be combined manually. The reviewers download the search results and integrate them into a spreadsheet shared between all the reviewers involved in the process.
available information of the project outside the database and to bound the time invested [21]: e) Optionally, launch a search in the main scientific databases using the project title and the reference. Although it is possible to conduct a formal search selecting terms and following a strict protocol, the process would be too long and complex. Moreover, although Scholar search engine is not so rigorous as scientific databases, the result provided should cover a large part of the scientific articles available in the databases.
5. Snowball: it is possible including research projects identified in the information available of the selected projects, usually in their websites. These new projects are included together with the search results and we apply all the previous tasks.
6. Documenting: it is possible to include a PRISMA flow [22] to summarise the screening process.
Regarding the quality criteria, each question is answered with a score between 0 (means the quality criterion is not fulfilled) and 1 (means the quality criterion is fulfilled by the project information available). Moreover, a score of 0.5 is used in those cases in which the criterion is partially fulfilled. Likewise, we will use the symbol "-" in those quality criteria which cannot be inferred from any of the identified resources. Although it is possible to use another scale such as a Likert scale with five levels, we recommend only three levels due to the difficulty to conduct an in-depth analysis of each available document or information of a project. Only those projects which achieve a score over a previously defined minimum will be the selected projects. In a systematic literature review, this minimum should be defined before starting the full review of the papers, but in the review of research projects, it is not possible to estimate a number due to the inherent uncertainty about the information available of each project.

D. Analysis
The last phase is focused on the synthesis and analysis of the selected projects. This phase is no different from the analysis carried out in an SLR. The synthesis is achieved in the step related to the presentation of the results. We extract the main characteristics of each research project using quantitative indicators. In those cases, in which there are qualitative characteristics, a set of dimensions are identified.
Finally, we have to write up the review, including the description and overall assessment of the results found. We have to answer each research and mapping question.

IV. Examples
The guidelines for performing Systematic Research Projects Review were applied in previous works. In particular, two examples are described in this section. First, to identify trends in research projects focused on technological ecosystems in the health sector as a way to identify lacks to develop new products [23], [24]. On the other hand, as a previous step to prepare a proposal about the gender gap in STEM (Science, Technology, Engineering and Mathematics) for a European call [21], [25]. The main characteristics of the method are highlighted in the next sections.

A. Trends in Research Projects
The European Union has a strong investment in R&D and demand side-measures in the health sector. In particular, there is a research area focused on defining and developing technological ecosystems for improving different aspects of the health sector, with a particular focus on the elderly population.
In this scenario, we used the SRPR method to get an overview of the current trends and identify the lacks and opportunities in the development of the technological ecosystem in the health sector. In particular, the mapping questions were the following [24]: • MQ1: What are the trends in the development of technological ecosystems focused on health in Europe?
• MQ2: What is the application domain of the research conducted?
• MQ3: What types of institutions are involved in the project?
• MQ4: How are the stakeholders involved in the technological ecosystems developed?
• MQ5: Which calls fund this kind of research projects?
• MQ6: Which period do the projects cover?
• MQ7: How much money was invested in these projects?
We conducted two reviews. A first review focused on finished research projects which involved different European countries and were funded in health or technology calls. This review includes projects from 2004 to 2018. Moreover, we conducted a second review which updates the previous one with the ongoing projects that started from 2015 to 2018. This allowed testing the replicability of the SRPR method. Furthermore, the comparison between both reviews allows getting an overview of the evolution of technological ecosystems in the health sector.
We conducted the reviews in three databases after a selection process which involved eight sources: Community Research and Development Information Service (CORDIS (https://cordis.europa.eu), Active and Assisted Living (AAL) programme (http://www.aal-europe.eu), and KEEP Database (https://www.keep.eu/keep/search). Noteworthy that some of the sources were identified through a mapping study about cross-border cooperation in healthcare [26], which we found searching on the Internet other databases of research projects of the same scope.
The selected sources comply with the minimum requirements, but there are big differences between the search and download tools. In particular, the AAL projects do not allow downloading or filtering, but its inclusion was justified because it is one of the most important European programs that combine health and technology. Likewise, the search string does not match in all the databases. In particular, KEEP Database does not allow logic operators, so two searches were conducted separately with the different parts of the search string and after the results were combined.
Also, the quality criteria are highlighted. Only those projects which achieved a 6 over 8 score were selected. In addition to the predefined quality criteria to ensure that the project has enough accessible information, a set of criteria focused on the objectives and the results of the projects were included [23]: • Does the project provide a full definition of the ecosystem? (It implements part or the whole ecosystem) • Was (or will be) the ecosystem developed? (Proposal, proof of concept or real system) • Does the website show the activity of the project?
• Does the ecosystem support evolution through the integration of new components?
Regarding the projects' selection, the search provided 718 projects in the first review. After removing duplicates, 707 projects were screened. A total of 102 projects passed to screen full project and, finally, only 19 projects were selected. Concerning the second review, the search provided 368 research projects (344 after removing duplicates). 79 projects passed the inclusion and exclusion criteria and 23 were finally selected after applying the quality criteria. The full data sets are available on http://bit.ly/2uyLeWn and http://bit. ly/2TBm8RH. Finally, we would like to highlight two main threats to validity directly related to the SRPR method. First, the number of projects that reached the final stage after applying the quality criteria were quite low. Although we applied the search protocol to identify the available information of the project, most of the projects provide scarce or none information for answering the quality criteria. On the other hand, the selection of databases based on funding programs introduce a bias in the results because each funding program guides the scope and goals of the financed research. To reduce this bias, the SRPR includes a recommendation during the selection of sources: "It is recommended to include different calls and funding periods in those reviews focused on identifying trends and lacks".

B. Justify the Innovation
The low female participation in all areas of society is one of the Sustainable Development Goals (SDG) of UNESCO. There are some areas in which female participation is lower than others. This is the case of the STEM areas. Among the different objectives of the European Union, increase the number of women in those areas has been a priority in the last years.
The number of European research projects related to gender and STEM has increased due to the priorities of the European Union. The definition of new proposals to foster this research line requires an adequate justification of the added value with regard to other proposals. For this reason, we used the SRPR method to get an overview of the research projects funded by the European Union about the gender gap in STEM areas, both academia and industry, in the last five years. This review covers three research questions to answer following a qualitative analysis of the selected projects [21]: • RQ1: What are the trends in Europe on the study about the gender gap in STEM?
• RQ2: Which kind of outcomes are provided by the projects?
• RQ3: Which kind of solutions or initiatives are developed?
And a set of mapping questions based on the meta-questions provided by SRPR [25]: • MQ1: What are the trends in Europe on the study about the gender gap in STEM?
• MQ2: What types of institutions are involved in the project?
• MQ3: In which countries the project was implemented?
• MQ4: Which calls fund this kind of research projects?
• MQ5: Which years cover the projects?
• MQ6: How much money was invested in these projects?
• MQ7: In which context is carried out the study?
• MQ8: Which kind of studies -diagnosis or intervention -are developed?
• MQ9: Which kind of outcomes are provided by the projects?
In this case, the selection of the databases was also based on the previous experiences applying the SRPR method. In particular, six databases were identified based on the identified scope and funding calls. After applying the database selection requirements, we conduct the review in three databases: CORDIS, which provides projects under the H2020 framework programme; Erasmus+ Project Results Platform (https://ec.europa.eu/programmes/erasmus-plus/projects/); and KEEP, to cover cross-border projects under the different interregional and cross-border programmes. The main problem of Erasmus+ and KEEP databases is that they do not allow logic operators. For this reason, we follow the SRPR guideline related to decompose the search string to carry out several searches with different terms and apply the boolean OR operator manually.
Regarding the quality criteria, we included five criteria related to the implementation and the results of the research projects: • Is the gender gap the main focus of the project?
• Was the study carried out in different countries?
• Does the project carry out any evaluation process focused on the gender gap?
• Does the project provide a toolkit, framework, materials focused on STEM?
• Does the activity of the project continue (or is planned to continue) after the funding period?
In this case, the minimum score was lower than the previous example (5.5 over 9), due to the low number of projects which achieve highest score. Since the quality phase does not depend so much on the quality of the projects themselves but on the information that can be found about them, the minimum limit for selecting a project is defined once the full review of the projects is completed.
Regarding the projects' selection, we collected 580 results from the selected databases and 16 projects were identified as part of the snowball task. After removing duplicates, 435 projects were screened. 84 projects passed the inclusion and exclusion criteria and 31 were finally selected. The full data set is available on http://bit.ly/2IjUJlE. Finally, it is important to highlight the limitation of the quality criteria. The lack of information available about the research projects introduces a big limitation in the results. On the other hand, this limitation serves as a learning outcome to improve access to our research projects.

V. Conclusion
This paper has explained the SRPR method as a way to conduct systematic reviews of research projects, facing the challenges related to analysing a compendium of different heterogeneous information available in different formats. Although there are previous works related to projects' review, the review conducted as part of this paper concludes that those reviews are not fully systematised.
According to Codina [27], a systematic review has four dimensions: systematic, complete, explicitly and reproducible. The proposed SRPR method fulfils these dimensions: • Systematic: it is not arbitrary, biased, nor subjective. The method provides guidelines for examining the best available research projects using the best sources of information.
• Complete: the guidelines to conduct an SRPR provide criteria to select sources that facilitate access to the bulk of research projects in a discipline. Moreover, the project selection phase includes the application of the screening criteria.
• Explicitly: the inclusion and exclusion criteria, the search strategy and the quality criteria are known.
• Reproducible: all the process is documented to ensure that other researchers can follow the steps and compare the results obtained to determine their accuracy or degree of success. SRPR allows including ongoing research projects, so it is crucial to ensure it is replicable to update the review when those projects finish, or new projects are developed.
This proposal will serve as a base for the definition of future research projects, providing a rigorous method to identify the lacks in previous research projects, justify the innovation of the new projects and also reuse the results of related projects previously developed by other research teams.
Regarding the examples described, we can state that the research project reviews that follow the SRPR method are rigorous. According to Onwuegbuzie and Frels [28], by rigorous we mean conducting a literature review that is defensible (i.e. integrates a rationale for decisions of inquiry, strategies, and designs), systematic (i.e. follows a set of guidelines), evaluative (i.e. whereby every step of the process is evaluated for relevance and credibility) and transparent (i.e. documenting beliefs, values, and philosophical assumptions and stances pertaining to decisions).
Finally, a number of caveats need to be noted regarding the present study. On the one hand, the successful application of the method relies on the availability of research project databases. For example, it is not possible to conduct the SRPR at national levels, if the information about research projects funded in public and/or private calls are not available in an accessible repository or database. On the other hand, the method was used in a European context, so it would be interesting to apply it in other regions and funding calls.