Keywords

Introduction

The introductory chapter of this book illuminates the far-reaching centrality of scholarly peers and their importance in assessing the quality of scientific work. The role of scholarly peers in a range of review processes has become institutionalised and integrated into most of the activities in academia (Musselin, 2013; Forsberg et al., 2021 in this book). An interesting aspect of this development is how the initial idea of peer evaluation and assessment of the quality of scientific work has migrated into a range of other academic contexts. In several ways, this migration can be seen to extend the role of peer review beyond the traditional turf of scientific reporting and publishing, potentially changing the premises and conduct as well as our understanding of what a peer evaluation entail.

Nowadays, we can observe peer evaluation and peer assessment as a mandatory and integrated element of scientific research such as in meta-analysis and systematic review. Peer evaluation is described as an essential element of quality assurance of the strictly defined methods and procedures of systematic review (see e.g. Slavin, 1986; Petticrew & Roberts, 2006; Gough et al., 2012; Torgerson 2003).

In this chapter, it is argued that the involvement of scholarly peers deeply embedded in the central stages of the systematic review processes has similarities with traditional peer review processes in academic publishing that aim to ensure the quality of academic work. However, the review process of a systematic review can also be distinguished from a peer review in academic publishing in how the review of manuscripts traditionally aims to formatively contribute to and ensure the quality of future publications. The systematic review process entails a peer evaluation after publishing and for purposes other than publishing. Further, peers are involved in evaluation of studies that are to be included or excluded according to the predefined criteria of the systematic review study. As such, peers in a systematic review can be regarded to make re-judgements of the quality and the relevance of already published work in accordance with the specific scope and predefined criteria for review studies.

The systematic review process places the scholarly peer within the research process where he/she becomes a central part of the scientific method, often contradicting the temporality of conventional ex-ante positioning of peer reviewers in academic publishing. The peer evaluator of systematic review can be considered to be in an x-nunc position (from now on and as long as the process goes, Jibi, 2020) limited to the frame of the systematic review in question. This positioning of the peer in systematic review represents something new and rather different from the intuitively perceived peer reviewer role. This positioning also often represents a breach of the academic principles of anonymity and distance between the researcher and the peer in situations of academic judgement.

The methods and procedures of a systematic review are examples of the migrating functionality of the peer evaluation process for assessing the quality of academic work. With its well-defined methods and procedures, a systematic review frames the involved scholarly peers in a highly specialised way. This framing partly builds on and it has characteristics similar to a peer review and partly differentiates a systematic review from conventional peer review processes, challenging our understanding of the roles of scholarly peers in assessing the quality of academic work. Thus, peers in systematic review processes are an interesting example of how a peer review framed in another context can contribute to a change in the premises and conduct of the peer role. In this chapter, these issues are illuminated by analysing the functions of peers in systematic review and discussing the roles of scholarly peers framed by other academic contexts—such as the systematic review.

The functionality and roles of scholarly peers in a systematic review must be seen in light of the rise of a general societal and policy-driven evidence movement within most fields (Hansen & Rieper, 2009). Within education, this development is reflected in the policy expectations for practitioners and professionals to use evidence when making decisions about teaching, learning and school development (Hansen, 2014; Levinsson, 2013; Sundberg, 2009; Gough et al., 2012; Levinsson & Prøitz, 2017; Prøitz, 2018). The systematic review phenomenon is grounded in ideas about methodological approaches that aim for highly detailed, universal and standardised stages of conduct (Davies & Nutley, 2000). However, with the growing knowledge base on research synthesis, variations in approaches have been acknowledged and problematised (Gough et al., 2012; Levinsson & Prøitz, 2017). Nevertheless, the involvement of peers to ensure the relevance and scholarly quality of primary research included in the systematic review studies is a stable feature across varying approaches (Prøitz 2018). In spite of the extensive and growing body of literature on various approaches to systematic reviews and ongoing debates on methodological and procedural issues, studies on the roles of the peers involved in research synthesis seem to be scarce, warranting closer analysis and discussion of the function of peers in systematic review processes.

Thus, this chapter presents an analysis and discussion of the function of peers (also called field experts, experts or peer reviewers) in scientific quality work of systematic reviews. The analysis draws on literature on traditional peer review in academic publishing and systematic reviews and a document analysis of systematic review technical reports within the field of education. The study is guided by the following questions: What are the functions of scholarly peers in a systematic review? What are their main tasks? What consequences does the analysis have for our understanding of scholarly peers in various types and processes of scientific work?

This chapter is divided into five sections. The study’s thematic and research questions are presented in the first, introductory, section. The characteristics of the systematic review process are described in the second section. The analytical framework is presented in section three, followed by the method and document material described in the fourth section. In the fifth and final section, the results of the analysis are discussed and some concluding remarks are provided.

Peers in Academic Publishing and Systematic Review

For the sake of this study, it is necessary to describe the background and context of peer evaluation and assessment in both academic publishing and systematic review. Throughout the chapter, a choice has been made to use the terms peer and peer work to capture the varied actions of the peers involved in systematic reviews. Here, peer work is considered to cover all activities that scholars perform to ensure academic quality when being involved in systematic review processes as well as those performed by peers in other academic situations.

Peer review in academic publishing has been defined as ‘the process by which research output is subjected to scrutiny and critical assessment by individuals who are experts in those areas’ (Hames, 2012, p. 16). Simply put, the traditional peer review process requires researchers to prepare a manuscript that reports their research and submit this manuscript to a journal for publication consideration in which the peer review process is a central part of the decision.

Based on this definition, the traditional peer review occurs before publishing. The peer review processes can be traced back 300 years to the regulated consultations of publications by experts among the members of the Royal Societies of Edinburgh and London (Hames, 2012; Spier, 2002). However, peer review first became widespread in the twentieth century; today, it has grown into a massive activity in the form of 25,000 peer reviewed journals (Hames, 2012; Ware & Mabe, 2009). Editors and researchers have appreciated how the peer review process has helped strengthen scientific communication through its regulatory characteristics of control and trust in research quality (Ware & Monkman, 2008). Peer review is also, to an increasing degree, criticised for issues related to quality and fairness and abuse and bias, for being expensive, slow and conservative as well as for lacking consistency.

According to Hames (2007, 2012, p.22) a peer reviewer in academic publishing is expected to prevent the publication of bad work, check that the reported research has been carried out well and without flaws in design or method, ensure correct reporting and interpretation of results, ensure results are not too preliminary or speculative, provide editors with evidence to judge the relevance of an article for a journal, provide authors with quality and feedback, improve the quality and readability of articles and maintain the integrity of scholarly record. The expectations neither define how to recognise ‘bad work’ nor exemplify what is meant by ‘research carried out well’ nor stipulate what is meant by correct interpretations, preliminary or too speculative. To a large extent, the essential judgement of quality is left to the scholarly peer based on the individual academic understanding of quality and merits of the qualified peer.

In contrast, systematic review is a rather new invention. The development of systematic reviews can be traced back to the meta-analysis by Glass and Smith in the 1970s, regarded as a cornerstone in the rise of evidence-based medicine (Gough, 2004; Bohlin, 2011). Inspired by Anglo-American success stories of clearinghouses, centres for ‘what works’ and ‘best evidence’ programmes, European governments, researchers and private entrepreneurs have embraced the idea of evidence based practice in various fields (Hansen, 2014; Levinsson, 2013; Sundberg, 2009). This has led to an evidence-based movement calling for systematic reviews in most fields. Systematic review is grounded in ideas about methodological approaches that aim for highly detailed, universal and standardised stages of conduct (Davies & Nutley, 2000). In general, the systematic review process is defined by certain successive steps of scientific conduct. Quality assessment is a central element in most of these steps; although, the use of scholarly peers is a stable characteristic of the method, there are variations in how and when peers are involved in the review process. Mostly the systematic review process contains the following steps: formulate a research question and develop a protocol, define the studies to be included (inclusion criteria), search for studies, screen studies, describe studies (the systematic review mapping can stop at this step or continue towards obtaining the full map and research synthesis using the following steps), appraise the study’s quality and relevance, synthesise the findings (answer the research question) and communicate and engage (Gough, 2007).

Based on a study of approaches presented by agencies developing systematic reviews and their review reports in the field of education, agency-specific variations in procedures of the review process were observed as a general characteristic, as was the employment of peers (Prøitz, 2015). Looking at different examples of the procedural steps used for a systematic review in education, peers can participate in the overall review teams/review groups or serve as field experts, employed by the review team of the review study to support the relevance and quality assessment phases of the review process. Often, the review protocol defining the scope of the review process is established before field experts are involved in the process, but experts can participate in approving the protocol as well as in reviewing the quality of the review steps, the review process and/or the final review report (Prøitz, 2015)

Peers participate in reviewing the protocols and methods. They offer suggestions for revisions and re-submission of protocols and methods. The same peers are also involved in reviewing drafts of the final review reports and providing feedback and suggesting revisions before approval by the review team. Peers can also be a part of advisory groups to the review team of a systematic review and participate in the evaluation of the defined quality and relevance procedure by classifying primary research in accordance with quality standards. Furthermore, peers can also contribute by assessing the evaluation process and suggesting adjustments before participating in the evaluation of the quality of the primary studies procedure. In sum, peers in a systematic review can participate in a variety of procedural steps, they can also serve as members of review teams overseeing the whole process, play an active part in the procedure and be external reviewers of the final review report.

Analytical Framework

In this section, the analytical framework of the study is presented. The peer review role is analysed and discussed by focusing on the status function of peers in the context of the systematic reviews using speech act theory motivated by the work of Searle (1995, 2005). This approach provides an analytical tool to identify what counts as peer review in systematic review processes (Searle, 1995, 2005).

In Searle’s (1995) project, there is a defence of the idea of reality as independent of us as opposed to the idea that all reality is human creation. According to Searle (1995), there are objective facts in the world that are only facts because we believe them to exist. Searle (1995) calls some of these facts ‘institutional facts’ (e.g. money, marriage) as opposed to non-institutional facts or ‘brute facts’ (e.g. mountains, trees) (p. 2). Searle makes a call for the analysis of the role of language in the constitution of institutions as he considers that researchers in social science have taken language for granted and overlooked the building blocks of social reality (Searle, 2005). The creation of institutional facts is enabled by collectively accepted systems of rules (procedures, practices), by which members of a collective impose a specific status function on a phenomenon as an institutional fact, which also gives the phenomenon a specific function through agreement and acceptance. The collective assignment of status and function also involve recognition of something or someone having power by virtue of its institutional status. The creation of an institutional fact requires a collective recognition and acceptance of so-called deontic powers, e.g. rights, duties, responsibilities and obligations. A relevant example for this study is how peer evaluation in varied academic situations is based on collectively agreed upon and recognised powers, which assign the right and duty to express evaluative comments, suggest improvements and make judgement of another researcher’s work with authority for quality assurance required by the peer review function.

According to Searle (1995), a collective’s agreement on giving a specific phenomenon (e.g. peers in systematic review processes) a particular status function can be expressed with the logic of ‘X counts as Y in context C’. Searle described the rules in these systems as having

the form of X counts as Y in C, where an object, person or state of affairs X is assigned a special status, the Y status, such that the new status enables a person or object to perform functions that it could not perform solely in virtue of its physical structure, but requires as a necessary condition, the assignment of the status. (1995, p. 22)

Inspired by Searle, this study investigates the defined status of peers/field experts (X) and what their function ‘counts as’ (Y) in systematic review processes (C). Thus, Searle’s (1995) logic provides a tool to analyse the collectively assigned status function of field experts in systematic review processes as described in systematic review technical reports.

Method

This study draws on a content and document analysis (Bowen, 2009; Cohen et al., 2011) of data extracted from technical reports that describe the method and procedures for determining the relevance and quality of a review involving external peers applied in systematic review processes. The technical reports provide thorough descriptions and rich information about the scope of the review in question, methods, procedural requirements and quality assurance processes where peers are potentially involved.

The review reports that were studied were selected from three different research agencies in Nordic countries (the Swedish Institute for Education Research, the Norwegian Knowledge Centre for Education and the Danish Clearinghouse). All three agencies were established during the last decade, and they form part of government-induced initiatives inspired by the ideas of evidence movement. They all have a certain focus on education and have developed their own agency-specific procedural way of conducting reviews, although they rely on well-known systematic review methods.

The documents were studied based on a three-phase process. In the first phase, 15 selected published review reports of the three Nordic knowledge agencies were skimmed (5 reports for each of the agencies, see “Selected Systematic Review Technical Reports Analysed in the Study” for a complete list of the selected reports). The aim of this phase was to identify the structural patterns of reporting because systematic review reports usually have a rather defined structure, although with an agency-specific variation (Prøitz, 2015). In this phase, mention of peers (e.g. as peer reviewers, external experts, field experts and researchers) in the different chapters, sections or paragraphs of the reports was identified in relation to the different steps of the review process to identify the functions of the peer work. In the next phase, an in-depth reading of the selected reports from each of the agencies was conducted. In this reading, a special focus on the identified passages where peers occurred was employed. In the third phase of the reading, the findings from the second phase were validated across the selected reports of each agency to see if the identified patterns of peer involvement could be characterised as a more general way of conducting the review process. This approach confirmed the existence of overall agency-specific structural patterns of reporting and the involvement of peers in the review processes. The examples presented can be considered as representations of the typical involvement of peers, as described in the 15 technical systematic review reports produced by the three Nordic agencies for reviews within the field of education.

For the analysis, characteristics of the peer status function, as described in the systematic review technical reports, are identified in the studied materials. The observed involvement of peers in the review processes is further discussed in relation to Searle’s (1995) work on institutional facts and, in particular, the assignment of the status function of peers in review processes.

Examples of Peers in three Review Processes

In the following section, the document material is described in detail. The presentation of the material focuses on the use of peers in the review processes as reported in the technical systematic review reports of the three selected agencies.

Swedish Institute for Educational Research

Two external researchers were invited to participate in the review project (Skolforskningsinstitutet, 2019) in a so-called project group consisting of the two researchers and the research agency’s internal staff after conducting a needs assessment (behovsinventering) among stakeholder groups and researchers in the field. A pilot study confirmed the review theme to be relevant and validated the existence of available primary studies. The role of the external experts was to ‘contribute with their understanding of research within the field based on their expert knowledge’ (Skolforskningsinstitutet, 2019, p. 60).

The two experts were further involved in Step 2, which was the relevance review of the systematic review process. In Step 1, the Skolfi staff had completed the first screening of the 9662 articles that were identified in the searches. Based on information in the titles and abstracts, 8646 studies that were considered ‘clearly not relevant’ were excluded. In Step 2, the external researchers individually reviewed the 1016 studies by reading the titles and abstracts. If it was unclear if a study was relevant, it was included in the next phase. Thus, the researchers excluded 815 studies in this phase. In Step 3, the two researchers individually reviewed the 201 studies left after a full reading of the text. In cases where the researchers disagreed, the studies were discussed and disagreement was resolved by consensus. In Step 3, 151 studies were excluded. In Step 4, the researchers conducted a collective relevance and quality review of 50 studies. The quality review was conducted with the support of the Skolfi guidelines for quality review (which was missing in the document, but which can be found on the Skolfi web pageFootnote 1). In this process, another 35 studies were excluded; thus, 15 articles were included in the study as being relevant and of ‘good quality’ (Skolforskningsinstitutet, 2019, p. 63).

The external researchers were also involved in the data and result extraction process in which they described the purpose of the studies and their results in writing on an A4 page. These writings were later used in the process as guidelines for the internal project group’s understanding of the results of the different studies. The external researchers were also involved in writing the report. When the report was finalised, it was first read by internal staff and then by two different external researchers within the field that were hired only for this purpose. The first two external field experts are presented in the report with their full name, title and an extensive biography; the other two experts reading the report were un-named. The Swedish Institute for Educational Research has the most standardised description of the procedures regarding the involvement of external researchers/peers described in every report under a separate chapter with the heading: Method and Conduct.

The Norwegian Knowledge Centre for Education

The review produced by the Norwegian Knowledge Centre for Education (Kunnskapssenter for Utdanning [KSU]) is a so-called rapid review characterised as ‘a format developed to do reviews quickly while at the same time ensuring the same quality criteria as for systematic review and has the same requirements for systematization and transparency’ (Lillejord et al., 2018, p. 10). The rapid review was based on guidelines and tools developed by the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI) at the University College London. Search resulted in 2542 hits. All references were imported into the EPPI-Reviewer 4 programme, which was developed to handle large amounts of data. The process of screening the articles consisted of three steps. In Step 1, two un-named researchers performed the first screening by reviewing titles and abstracts according to three, predefined thematic inclusion and exclusion criteria. In that step, 1685 studies were excluded leaving 53 articles with potential relevance for the review study. In Step 2, two researchers independently reviewed the quality and relevance of the studies. The applied quality criteria are partly described by the concepts of validity, reliability and generalizability and partly by three questions to be answered: Is the research question clearly defined? Are the research method and the research design specified? Is there coherence between the research question and the results? Each point and question are to be reviewed as high (explicit and detailed description of the method, data collection, analysis and result; the results have clear support in the findings), medium (satisfying description of the method, data collection, analysis and result; the results are partly supported in the findings) or low (weak description of the method, data collection, analysis and result; results have weak support in the findings). When in doubt, the articles were presented to the project group to make a final decision. After Step 2, 33 studies were included in the research mapping for analysis. In Step 3, the articles were prepared for synthesis and the content of the studies was mapped by methods for data collection, data analysis and country. After mapping, a number of un-named researchers read the full text of the articles and every study was described in a short summary that helped clarify how it could contribute to the research question of the systematic review.

The report says little about the researchers involved in Step 2 and Step 3 of the review process and about their involvement and how they worked. Thus, some differences were found between the selected reports in terms of whether the involved peers/researchers/research group are named or whether their involvement is specified. For example, it seems that this varies with the magnitude and ambitions of the review; larger studies provide more details on the involvement particularly of research groups, but mostly in general terms. This is in contrast to the example of the Swedish report, where the full name and the individual title, experiences and qualifications of the two external researchers involved in Step 2 were presented, while the peers used in the review of the final report were not named.

Danish Clearinghouse for Educational Research (DCU)

The process of developing the research mapping is described as an example of the DCU standard procedure, which consists of using the software tool EPPI-Reviewer version 4.7.0.0 developed by the EPPI-Centre. The mapping was conducted as a collaboration between staff at the DCU and the members of the review group. The review process, including all communication between the DCU staff and the review group, was documented (Bondebjerg et al., 2016). The review process consisted of several phases; members of the review group were involved in the last phase of the process which entailed coding and review. The report describes a collaborative effort where DCU staff filled in the forms of the EPPI-Reviewer and sent them to a member of the review group. The review group member then filled in his or her review of the study, including potential corrections or additions to the DCU staff coding. In the end, the two reviewers agreed on how much evidence weight each individual study should have in the study.

The reported study identified 2409 references in the search process; 197 references were excluded due to duplication, 2122 were screened by title, abstract and full text, 2062 studies were excluded in the first screening, 150 references were coded and reviewed in the second screening. Finally, 144 studies were included. The screening and exclusions that occurred before the second screening were solely done on the basis of relevance by DCU staff. Assessing the quality of the studies was not part of the first screening process. The 144 included studies were closely read, coded and reviewed in the EPPI-Reviewer. Based on the codes, every study was given an evidence weight (high, medium or low) characterising the degree to which the individual study fulfilled general scientific standards for empirical research; thus, 63 studies were reviewed to be of high or medium evidence weight. The quality review considers if the study actually investigates the issue it is meant to evaluate, if there is coherence between the premises of the study, the data and the conclusion and if the study achieves its aim. It also includes ethical criteria in data collection and selection, and the way the relationship between the empirical data and the conclusion is described. The quality review also considers the generalisability study. The report refers to the DCU research quality guidelines, which are written in a separate document.

The three example reports illustrate varying degrees of thoroughness and transparency in the reporting of who the peers/external experts were as well as their roles and involvement in the review process. In the Swedish example, the peer/experts involved in the process were presented in a separate section with their full name and biography, while the identities of the peers/external experts employed to read the final report remained anonymous as in conventional blind peer review. In the Norwegian case, the names of the peers vary, and little information is given about the ‘researchers’ that were involved and the extent of their involvement, including whether the researchers were external or internal. However, in larger studies, the review group is usually presented in full with names and affiliations on the first page of the Norwegian reports. In the Danish report, the external researchers are named as part of the review group without being emphasised particularly. In all three cases, the peers/experts were involved in the later stages of the review process after the initial screening processes, which were often performed by agency staff. In the Swedish and Norwegian cases, the peers/experts were involved in reviewing titles and abstracts, as well as in the phase of reading the full text of the articles considering both issues of relevance and quality. The Danish peers/experts mainly reviewed the coding made by the DCU staff in the EPPI-Reviewer system, and they added their review of the quality of the articles.

Peers in the Systematic Review Processes

Overall, there are several features that can be regarded as characteristic of the work of peers in the systematic review processes studied. First, general principles of scientific quality such as transparency, validity and reliability for quality assurance seem to be underlying elements throughout the work. Second, the peer evaluations are framed by the scope of the systematic review, the method, the procedures, the review protocol and its defined inclusion and exclusion criteria. As such, the peer work of ensuring scientific quality is not only framed by general principles of scientific quality but also, and more strongly, defined by specific and strict principles of method and procedure. The principles can be interpreted as devices for quality assurance that both secure transparency and delimit the space available for professional judgement from going outside the scope of the systematic review. The peers in a systematic review are employed to maintain the quality of the systematic review related to the scope and methods of the review in question.

This issue can be further interpreted by the work of Searle (1995) and his ideas about how larger groups of people assign status functions, such as in this study the scholarly community more or less worldwide, have agreed on the idea of peer review as a sound way to ensure scientific quality in academic reporting and publishing in general and how this idea serve a somewhat similar function in a systematic review process. However, the logic of X counts as Y in a context C also helps identify the status function of peers in systematic review as something different from the conventional peer work when seen in relation to the frames of reference and context.

Following this line of thinking, the peer review process in, for example, academic publishing is assigned its status function through the expectations of being an anonymous/blind guarantor of academic quality based on individual scholarly merit providing academic judgement, critique and formative advice. In a systematic review, the peer can be considered to have been assigned status function as a known guarantor of academic quality and relevance based on individual scholarly merit as defined by the scope, purpose and procedures of the systematic review in question. (see Table 12.1)

Table 12.1 The status function of peers in academic publishing and in systematic review
Full size table

The significance of the differences can be further illustrated by the difficulties that would occur if peers in a systematic review were to take on the role/status function of the peer in publishing. Their function as producers and guarantors would fall outside the focus of the review for the publishing frame, and it would most likely lose status function within the openness of framing the peer review for academic publishing. In consequence, to a large extent, peers in a systematic review are peers primarily seen in relation to the method, scope, purpose and procedure of the review. In contrast, for peers in academic publishing, the scholarly status function can be more broadly defined for a larger field of scholarly expertise, where the peers have a certain responsibility to uphold the ‘record’ of the field and provide help, support and advise to authors, mainly through processes characterised by anonymity and distance between the researcher and the peer reviewer. Similarly, this status function of the peer in the publishing context would not be functional within the framing of a systematic review.

Another characteristic of the systematic review process, displayed by the presented material, is the set of procedures that are applied when the peer evaluators disagree. Consensus on the quality and relevance of reviewed work is important for the overall quality of the systematic review, and specified practices of conduct are defined in the method to reach a common agreement among peer evaluators. In the material, approaches for reaching agreement among the involved reviewers were described in the Swedish and the Danish examples. This is in contrast to the ideal of a blind review in a conventional peer review for academic publishing where disagreement between reviewers is partly left to be resolved by the author in the manuscript revision and partly to be resolved by the editorial decision and advice given to the author and sometimes by involving yet another reviewer to obtain a third opinion.

The material also displays how the peer/expert in a systematic review is mostly involved after the scope and purpose of the study has been set by the review team and also often after the groundwork of searching and screening of primary studies has been completed by the agency staff. This issue of temporality illustrates yet another and central feature of the peer work of systematic review and it is probably where peer work in systematic review distinguishes itself mostly from the traditional peer evaluation. The formative aspect of ex-ante peer evaluation is a central academic principle that underscores the ambition and importance of collegial sharing, critique, correction and revision for improvement of research. The peer evaluation in the systematic review processes does not aim for such formative purposes but it can be considered as an integrated ex-nunc judgement of another kind and for other purposes, where published studies are measured up against specified criteria of relevance and selected on the basis of being the best fit with predefined criteria, including aspects of quality. The aspect of temporality highlights issues regarding the professional judgement of the involved peers in systematic review. In Chap. 3, Vanderstraeten interestingly documents how the reviewer/author and editor roles have changed from being blurred and intermingled to becoming more specialised, standardised and pronounced as a result of historical contingencies that defines the grounds for scientific research. Looking at the peer reviewers in this chapter illuminates how experts in systematic review today also seem to work under resembling blurred lines, for example through their roles as experts evaluating published work and contributing to the development of new systematic review publications at the same time. This issue raises questions relevant for all peers regarding what the peer work is about, and with reference to the thematic of this book, what peers in varied academic contexts including systematic review are ‘gatekeepers’ of? In the context of systematic review, peers are not only making re-judgements of already reviewed and published research they also function as gatekeepers of the given standards, guidelines and procedures of the review method.

Concluding Remarks

In this study, we have seen how the involvement of peers in systematic review processes makes use of peers resembling those in a traditional academic peer review process to ensure academic quality. We have also seen that there are differences between the roles and the status function of peers related to the framing and purpose of a peer’s ‘guarantor role’. The study highlights the issues of the role and status of peers in varied academic contexts. It also highlights how the peer review role change with changing temporalities and different devices that provides different spaces for professional judgement.

As such, the analysis lays the groundwork for a debate on peers in different contexts framed by different processes with different purposes, and it questions whether a peer review is the same when the premise of the scholarly activity changes. This study also highlights the difficult question of the function of the peer reviewer in-between being the anonymous and distant person ensuring and guaranteeing scientific quality and being the one to openly and actively participate in the formation of a scholarly product while also playing the peer role. This question is highly relevant considering the more recent developments and debates on the need for more open peer review processes in academic publishing, changing the premises of the conventional activity.

With this backdrop, this study calls for a stronger framework or, potentially, a typology distinguishing between varied forms of peer work to clarify the differences between the roles of peer reviewers in different academic activities, considering the migration of the use of peers in a range of academic work.