Linking policy and practice in monitoring socially responsible research and innovation (RRI): A conceptual framework to evaluate progress through the UNESCO-led Recommendation on Science and Scientific Researchers

This paper sets out a high-level conceptual framework for monitoring the development of socially responsible research and innovation (RRI) systems linked to the global policy instrument called ‘the Recommendation on Science and Scientific Researchers’ (RSSR). RRI is an umbrella concept, rooted in EU policy and research, that encompasses a range of topics that overlap directly with the RSSR. These topics include social inclusion in scientific research, improved scientific transparency and governance and expanded public engagement with science. The RSSR is an international accord ratified by 195 United Nations Educational Scientific and Cultural Organization (UNESCO) Member States in 2017, updating an earlier version of the instrument. This UNESCO-led initiative offers a globally inclusive and agreed structure for advancing RRI-related principles. A key feature of the RSSR initiative is its permanent structure of quadrennial monitoring to assess implementation of its principles. Here, a conceptual framework is presented to clarify the specific dimensions of RRI embedded in the 10 key priority areas for this quadrennial monitoring process. The paper explicates these dimensions and the underpinning policy language from the 2017 recommendation with the aim of supporting UNESCO Member States and research stakeholders globally to design appropriate evaluation methods. This conceptual framework is intended to support development of globally aligned measurement of RRI policy and practice that allows research and policy stakeholders from each world region to learn from each other. Fostering mutual learning on a global scale will help to enable evidence-based practice in the context of RRI to improve outcomes and mitigate the limitations of well-meaning but ineffective policies and practices.


Amendments from Version 1
There were two reviews containing critical comments.The first raised concerns (1) about structure, saying 'the reader receives almost no information about what the monitoring framework is about.This needs to be added to the main body of the text...The key aspects of the framework one would expect in the main body of the text.'This was how I initially structured the paper but ORE editorial staff required me to remove the bulk of the framework content and move it into the supplemental material.So no change can be made here.(2) 'The introduction should be shortened to about a quarter of the text'.[similar point mentioned by the second critical reviewer] I have introduced an additional heading to achieve this by splitting up the section currently titled 'Introduction'.(3) 'The conclusion restates what has been written in the introduction.Redundancies need to be removed.'[also mentioned by the second critical reviewer] I have done this.(4) The last paragraph talks about ecosystem research and COVID-19.As a reader, I got lost.This had not been mentioned before, why now?'.[also mentioned by the second critical reviewer] I have removed and replaced this content.
The second reviewer that provided critical comments noted some overlapping issues: (1) "It is important to frame the 'RRI discourse' on the text and to illustrate which synergies with RSSR have."This has now been done in the introduction.(2) "Section three will benefit from a further clarification about the 'supporting function' that this document has."This has been done in the conclusion now.(3) "The text will benefit from ' citation diversity'.Especially for situating the concept of RRI into the text and helping to the reader to understand the synergies between RRI and RSSR."Additional citations added as suggested.
Any further responses from the reviewers can be found at the end of the article and Cultural Organization (UNESCO), and the underpinning policy language from the 2017 recommendation.This elaboration of the existing guidance from UNESCO for the monitoring process is aimed at supporting UNESCO Member States and research stakeholders globally to design appropriate evaluation methods.This is because good evaluation methods require clear elaboration of the outcomes that are being targeted for measurement (Jensen, 2014;Jensen & Laurie, 2016).Such evaluation can lay the foundations for long-term international mutual learning at the level of best practices in RRI advancement and evaluation, linked to the RSSR's quadrennial cycle of stakeholder consultation, monitoring, and reflection at the national level.This monitoring process (UNESCO, 2021) is a legal expectation for each UNESCO Member State to provide national reporting on progress towards the RSSR's full implementation with appropriate substantiation, along the same lines as other UN conventions (UNESCO, n.d.).Indeed, having comparable assessments conducted across countries and time can be highly useful for policymakers as they consider the common global standards embedded in the RSSR, along with analysis of local research and innovation policies and practices that can deliver on those standards.

Context for conceptual framework
This conceptual framework is aimed at supporting each government and scientific community across the UNESCO Member States to take necessary actions for effective evaluation and reporting on RRI dimensions during their ongoing engagement with this quadrennial process.The tone and structure of this open letter is designed to be helpful to these research and policy stakeholders as they come to grips with the RSSR and its monitoring process.To aid the process of measuring progress in addressing the different dimensions included in the RSSR, this paper provides a conceptual framework with a detailed identification of the specific elements of the 10 key priority areas for monitoring the RSSR.The 10 priority areas that have been identified by UNESCO and confirmed by Member States in the UNESCO Executive Board in March 2020 1 (UNESCO, 2020) as the initial focus for RSSR implementation and monitoring are outlined in Table 1.The specifics of how this set of 10 key priority areas was initially fashioned have not been publicized by UNESCO beyond indicating that the goal was to achieve simplification to ease the burden of the monitoring process.
This open letter takes these confirmed key priority areas for the RSSR as a starting point and uses a close reading of the original RSSR to break each priority area down into its component dimensions to clarify where measurement is needed to feed into the long-term monitoring process for the RSSR (see supplemental materials for details).To further guide UNESCO Member States and research stakeholders in their

Introduction
The present text puts forward an overarching conceptual framework to support monitoring of an international accord adopted by 195 governments, representing a global commitment to socially responsible research and innovation (RRI).This accord sets out a wide range of principles for regulatory action on RRI, explicitly committing governments to take pro-RRI actions.The UN-organized accord is called the 'Recommendation on science and scientific researchers (2017)' (RSSR) (UNESCO, 2017).This paper aims to bolster the RSSR's role as a global vehicle for RRI by providing a conceptual framework for its associated quadrennial monitoring process.
RRI is an umbrella concept rooted in EU policy and research, that encompasses a range of topics that overlap directly with the RSSR (Gerber et al., 2020).These topics include social inclusion in scientific research, improved scientific transparency and governance and expanded public engagement with science.The paper explicates the dimensions of the RSSR that have been identified as priorities for measurement by the United Nations Educational Scientific 1 This document is available from the Executive Board documents following the reference link on the UNESCO site, listed under the 209th session -2020, 209 EX Main Series.consideration of relevant evaluation measures and indicators, guidance notes, and direct quotations from the RSSR are included in grey font in the full framework document (see underlying data : Jensen, 2021).These notes are focused on dimensions where relevant measurement options may be more ambiguous or needing elaboration to identify the relevant aspects to target for RSSR reporting.This presentation of the 10 key priority areas for RSSR implementation and monitoring uses verbatim language from UNESCO policy documents extensively to provide assurance of the alignment of the conceptual framework to the underpinning policy instrument.The conceptual framework presented is one of a series of publications intended to guide UNESCO Member States through the process of evaluating RRI progress in science systems (Jensen, 2020).This evaluation process is required by the RSSR, but the specifics of how to assess and improve the socially responsibility of science systems in line with this policy instrument are not spelled out by UNESCO or the UNESCO Executive Board.This means that many countries' representatives are left without detailed advice about how to interpret, implement, and evaluate the RSSR in their national systems.In part, this ambiguity is by design because it allows for context-appropriate adaptation of general RRI principles.Indeed, national governments are encouraged, but not required, to convene 'working groups' to provide diverse stakeholder voices, including representatives from scientific bodies, industry, citizen groups, etc., for this process of adaptation.However, greater elaboration and clarification of the components of the monitoring framework can streamline the initial steps of coming to grips with the policy and its dimensions, thus improving the experience for national governments and boosting the value of the monitoring exercise.

Development of the conceptual framework
The conceptual framework presented in this paper was developed during the responsible research and innovation networked globally (RRING) (rring.eu)project to guide the project's work in conducting three national case studies in South Africa, Lithuania, and Serbia focusing on the monitoring process for the RSSR.UNESCO was a formal partner in this research and innovation action, and its lead representative on the project, April Tash, provided critical feedback during the development and application of the conceptual framework in South Africa, Lithuania, and Serbia from approximately June 2020 until April 2021.A range of global studies on RRI undertaken during this period under the auspices of the RRING project (e.g., see Foulds et al., 2023) highlighted the strong overlap between RRI as a concept and the RSSR as a live policy document with broad-based support.
The framework was constructed based on a close reading of the RSSR full policy text (UNESCO, 2017) and the distilled 10 key priority areas, with the aim of establishing a robust conceptual foundation for identifying relevant sources of evidence to include in formal UNESCO Member State quadrennial reporting against the RSSR.The framework was developed by the author of this open letter by applying logical deduction, separating out the elements of compound sentences to add clarity, noting cross-referencing in UNESCO policy documentation where available, and matching the language and intentions of the key priority areas with details in the RSSR full policy text.Ultimately, this framework is built on logical argumentation, with full transparency to allow others to come to different conclusions or considerations about the linkages between the two documents.The review and implementation process did not result in significant changes to the framework.This policy analysis document integrates these two distinct documents, with the aim of clarifying for Member States how the full policy text can be linked directly to each aspect of the 10 key priority areas.Development of this conceptual framework was prompted by feedback from the aforementioned national case studies that the 10 key priority areas on their own were not sufficiently elaborated to clarify which kinds of evidence where relevant for which key priority area.

Using this conceptual framework
While this open letter may be relevant to anyone interested in evaluating RRI on a global scale or the RSSR policy instrument, its specific audience is UNESCO Member States and the working groups they convene to deliver expert input into the long-term RSSR monitoring and reporting processes.In particular, UNESCO Member State government representatives and working groups are encouraged to consider the categories indicated in the framework (Jensen, 2021) to selfassess the comprehensiveness of the available evidence that has already been documented.This document is intended to be used as a worksheet by the informally constituted working groups that some countries convene so that they can prepare reporting on evaluation measures and indicators relating to the RSSR that are as comprehensive as possible.
It is understood that practical constraints will mean that most UNESCO Member States cannot address all aspects of the 10 key priority areas in full, comprehensive detail.For this reason, this conceptual framework is designed to be used selectively to target aspects of the RSSR where clarification and orientation would be helpful.That is, the framework is designed to allow national working groups to pick and choose which key priority areas they would like to consider further at any given point.This is a supporting document to aid the monitoring process and does not replace direct engagement with the key priority areas and the full text of the RSSR.

Conclusion
Recent years have seen a surge in public support for science (Jensen et al., 2021).Research systems need to improve to be worthy of this public trust.To do this, we need to take an evidence-based approach to advancing the principles, such as inclusive and transparent research processes, that RRI and the RSSR share (Jensen & Gerber, 2020).Establishing robust methods for evaluating progress in the development of more socially responsible research and innovation systems is essential to ensure progress.The RSSR is a useful means of advancing this effort, offering a ready-made agreement of many countries with monitoring already included.The monitoring process will continue in perpetuity, thus making it an excellent focus for aligning global efforts to create improved science systems.The conceptual framework included in the supplementary materials for this letter offers a tangible structure for advancing this global alignment into the future.

Simon Fielke
Sustainability Pathways, CSIRO Environment, Dutton Park, Queensland, Australia This letter is labelled a conceptual framework but there is no conceptual framework.The supplementary document is more of an expansion of the ten key themes rather than what I would consider an operational conceptual framework.
There are many references to the authors work (over half the reference list) and I am sure there are other relevant works concerning the science of responsible innovation as a process of practices -eg the original conceptualisations (Owen, Stilgoe, et al).
Is the rationale for the Open Letter provided in sufficient detail?(Please consider whether existing challenges in the field are outlined clearly and whether the purpose of the letter is explained) -No The Open Letter suggests a conceptual framework that is not presented in the text nor in an appropriate format for me to consider it a conceptual framework.I appreciate that the format has only allowed limited representation of the conceptual framework in text, but that suggests to me this is the wrong format if the contribution of the letter cannot be made within the letter.

Does the article adequately reference differing views and opinions? -No
It is mainly self-citations when there are many relevant and seminal responsible innovation contributions overlooked -i.e.where did such thinking and practice come from and why is it important?
○ Are all factual statements correct, and are statements and arguments made adequately supported by citations?-Partly See above comments -where is the conceptual framework built from and how are the ten themes operationalizable?

Introduction
The introduction offers an adequate overview of the international accord RSSR and its alignment with the RRI concept, emphasizing the importance of monitoring and evaluation.A clear connection to EU policy and research regarding RRI is presented, but it may still need a bit more clarity in framing the relationship between RRI and RSSR as per reviewer two's comments.

Context for conceptual framework
This section details the framework's design, tone, and intended audience.While it is informative and extensive, it might still benefit from a brief summary or visual representation (like a diagram) of the framework itself in the main body to address reviewer one's concern about structure.

Development of the conceptual framework
The development process of the framework is explained, including the involvement of UNESCO and the logical deductions applied in its creation.While comprehensive, the absence of the detailed framework in the main text might still leave some readers wanting a more concrete understanding of what it contains.

Using this conceptual framework
This section provides practical guidance on how to use the framework, targeting specifically UNESCO Member States.It's clear, informative, and engages the reader with a specific call to action.

Conclusion
The conclusion highlights the need for evidence-based approaches to enhancing RRI principles, emphasizing the importance of the RSSR and the ongoing monitoring process.The author has addressed the initial concern about redundancies.

Author's Response
The author's response indicates they have taken the initial reviews seriously, making adjustments where feasible and explaining why other changes were not made.The clarification regarding the removal of the framework's content due to editorial requirements is important but leaves an unresolved concern about the structure.

Recommendations for Improvement
Clarify the Relationship Between RRI and RSSR: As suggested by reviewer two, the 1. introduction should provide a clearer framing of how RRI aligns with RSSR.
Include a Brief Overview or Visual of the Framework: While it's clear the details of the framework were moved to supplemental material due to editorial guidelines, a brief summary or visual representation could enhance reader comprehension without overwhelming the main text.

Conclusion
The revised manuscript has made efforts to address the concerns raised by the initial reviewers.While some improvements have been made, further refinement could enhance the clarity and comprehensibility of the paper, especially in the context of the relationship between RRI and RSSR, and the visualization or summarization of the framework.The paper represents an essential contribution to the understanding of RRI and offers a valuable tool for UNESCO Member States and others in their monitoring efforts.
Is the rationale for the Open Letter provided in sufficient detail?(Please consider whether existing challenges in the field are outlined clearly and whether the purpose of the letter is explained) Yes The topic that addresses this letter is relevant and timely and it tries to engage with ongoing efforts and strategies for strengthening science-society interactions.The open letter pays attention to the synergies between RRI and the UNESCO "Recommendation on science and scientific researchers' (RSSR) that was ratified by 195 UNESCO Member States in 2017.The article is well-written, and the language employed is accessible.

Does the article adequately reference differing views and opinions? Yes
However, there are several issues that demand attention from the author to improve this open letter.Some of them are related with the content whilst others are more related with the format.In this sense, I tend to agree with the other prior reviews in several issues.I propose to the author several changes in order to strength this manuscript: It is important to frame the "RRI discourse" on the text and to illustrate which synergies with RSSR have.It is hard to understand for a not-specialized reader.

○
Eliminate redundancies and repetitions.Especially in the conclusion which does not point to next steps or future challenges ahead.

○
The introduction is too long for a short letter like this.It needs to be shortened around half (or even more) and incorporate the RRI concept which is not explained at all in the paper.

○
Section two of the letter should provide more details about how the framework was set up.Specially of interest for the reader should be to expand the 10 points of the figure into subsections (more space for the figure will be welcome in the text).
○ Section three will benefit from a further clarification about the "supporting function" that this document has.

○
Mentions to COVID-19 at the end of the text are confusing for the reader.It should be erased.
○ The text will benefit from "citation diversity".Especially for situating the concept of RRI into the text and helping to the reader to understand the synergies between RRI and RSSR.

○
The changes proposed here demand significant work on this paper (major changes), but I think that the letter will benefit from this work to become a valuable contribution on this matter.Reviewer Expertise: RRI, RI, science-society interactions, STS I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Is the rationale for the
Although the article's language is accessible with subject-specific terms such as RSSR explained and abbreviations given, the concept of RRI could have benefited from an in-depth explanation of what it is and what it entails particularly for stakeholders unfamiliar with the term or with its practices.Further, in as far as RRI is concerned, the paper could have done well to offer differing views and opinions of RRI which may not necessarily be a universal concept that is understood and practiced in the same way across the 195 UNESCO member states.As such, different views and opinions, particularly in as far as its understanding and practice in different geographical locations are concerned would have been valuable.Therefore, given that there are 195 member states who may or may not fully understand the rationale or concept of RRI or who no doubt may practice it differently, a presentation of competing assertions would have been an ideal consideration in the article.By so doing, the article would have given insight to potential challenges in the field, therefore providing possible considerations on how to address evaluation challenges and potential mutual learning points in the monitoring process of RSSR.Further, such considerations could offer a much deeper understanding of the challenges that have prompted the need for the incorporation of socially responsible research and innovation systems when it comes to RSSR.This is important, particularly for stakeholders unfamiliar with RRI.
The article makes a great point about UNESCO's omission, albeit deliberately, in providing specifics of how to assess and improve the social responsibility of science systems in line with the RSSR policy instrument.Although the intention was to allow context-appropriate adaption of RRI, it would still have been beneficial to have some criteria or as indicated by the author specifics on the assessment and RRI systems.This could avoid unwieldy interpretations, implementations, and evaluations of the RSSR to the extent that the varied interpretations become challenging in their address of RSSR policy, especially in terms of comparability between member states.Similarly, the fact that member states can use the conceptual framework selectively to target aspects of the RSSR where clarification and orientation would be helpful may not bode well for comparability between member states.Granted, this is intended to be helpful for those member states who may not be able to address all 10 key priority areas in full.Despite this, it would have been beneficial to have compulsory aspects of the RSSR that each member state would need to address as a way of ensuring uniformity while leaving some as optional in the event of a failure in addressing all 10.That said, this is a timely article that is useful in its contribution to the importance of having socially RRI systems in RSSR policy.
Is the rationale for the Open Letter provided in sufficient detail?(Please consider whether existing challenges in the field are outlined clearly and whether the purpose of the letter is explained) Partly

Does the article adequately reference differing views and opinions? No
Are all factual statements correct, and are statements and arguments made adequately supported by citations?Yes Is the Open Letter written in accessible language?(Please consider whether all subjectspecific terms, concepts and abbreviations are explained)

3.
Role of science in national policy and decision-making, international cooperation, and development.4. Promotion of science as a common good. 5. Inclusive and non-discriminatory work conditions and access to education and employment in science.6.Any scientific conduct is subject to universal human rights standards.7. Balancing the freedoms, rights, and responsibilities of researchers.8. Scientific integrity and ethical codes of conduct for science and research and their technical applications.9. Importance of human capital for a sound and responsible science system.10.Role of Member States in creating an enabling environment for science and research.
Are all factual statements correct, and are statements and arguments made adequately supported by citations?Yes Is the Open Letter written in accessible language?(Please consider whether all subjectspecific terms, concepts and abbreviations are explained) Yes Where applicable, are recommendations and next steps explained clearly for others to follow?(Please consider whether others in the research community would be able to implement guidelines or recommendations and/or constructively engage in the debate) Yes Competing Interests: No competing interests were disclosed.Reviewer Expertise: RRI, value sensitive design, and ethic-by-deisgn approaches to novel and transformative technologies I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.Version 1 Reviewer Report 21 October 2022 https://doi.org/10.21956/openreseurope.15387.r30263© 2022 Tabares R.This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Raul Tabares Basque Research and Technology Alliance (BRTA), Derio, Spain Dear author, Thank you for submitting your work to Open Research Europe.
Open Letter provided in sufficient detail?(Please consider whether existing challenges in the field are outlined clearly and whether the purpose of the letter is explained) Partly Does the article adequately reference differing views and opinions?No Are all factual statements correct, and are statements and arguments made adequately supported by citations?Partly Is the Open Letter written in accessible language?(Please consider whether all subjectspecific terms, concepts and abbreviations are explained) Yes Where applicable, are recommendations and next steps explained clearly for others to follow?(Please consider whether others in the research community would be able to implement guidelines or recommendations and/or constructively engage in the debate) Partly Competing Interests: No competing interests were disclosed.

the rationale for the Open Letter provided in sufficient detail? (Please consider whether existing challenges in the field are outlined clearly and whether the purpose of the letter is explained) No Does the article adequately reference differing views and opinions? No Are all factual statements correct, and are statements and arguments made adequately supported by citations? Partly Is the Open Letter written in accessible language? (Please consider whether all subject- specific terms, concepts and abbreviations are explained) Yes Where applicable, are recommendations and next steps explained clearly for others to follow? (Please consider whether others in the research community would be able to implement guidelines or recommendations and/or constructively engage in the debate) Partly Competing Interests:
No competing interests were disclosed.
○Is the Open Letter written in accessible language?(Please consider whether all subject-specific terms, concepts and abbreviations are explained) -Yes ○ ○ Is

have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.
© 2023 Umbrello S. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.