Global indicators framework for socially responsible research and innovation (RRI): Aligning standards to monitor public and researcher perspectives with the UNESCO Recommendation on Science and Scientific Researchers

As calls for more socially responsible research and innovation (RRI) policies and practices grow more insistent, the need for high-quality indicators that can be used to evaluate progress is becoming increasingly important. Given the global nature of science, such indicators need to be relevant to countries across all world regions. Moreover, the methodological quality of indicators is critical to provide a strong foundation for long-term comparative measurement of the impacts of different kinds of policy intervention. There is a practical challenge here, given the uneven mechanisms for data collection and analysis available in different countries. There is also a geopolitical challenge in gaining buy-in from countries with very different, and sometimes competing, agendas. Here, the 2017 UNESCO-led Recommendation on Science and Scientific Researchers is highlighted as an existing vehicle that can enable cooperation on globally comparative measurement of socially responsible research and innovation. In particular, the quadrennial monitoring of the implementation of this wide-ranging global policy instrument that has been ratified by 195 countries affords a unique opportunity to add value for these countries by linking RRI to the 2017 Recommendation while establishing benchmark indicators for RRI more generally. As a practical and methodological contribution to the global community of science and innovation policymakers, researchers and research and innovation stakeholders committed to socially responsible research, this report contains detailed survey questions and response options focusing on public opinion and individual researchers’ level of measurement. It provides details of sources of benchmark survey data that have readily available open data that can be used to benchmark the development of socially responsible research and innovation over time from the vantage points of the public and researchers around the world. The aim of this kind of science ecosystem-level indicators framework is to enable evidence-based practice in socially responsible research and innovation.


Amendments from Version 1
Changes have been made to address peer reviewer comments.For example, Reviewer 1's comment, 'Since this is an essay, I recommend that the Introduction is supplemented with an explanation/definition of socially responsible research and innovation' has been addressed.In addition this reviewer's comment, 'The main body of the text consists of short descriptions of various data sources/surveys that, according to the author, may be linked to socially responsible research and innovation.The attempt to link particular surveys to specific aspects/dimensions of RRI is desirable.It would indicate clearly how a particular survey/dataset is linked to the concept of RRI.Moreover, some sort of a concise synthesis of all presented surveys would be helpful' -this comment has been addressed by shifting this information into a table format to make it more readable.Reviewer 2 seems to be looking for a more traditional, long-form academic essay with citations to big names in the RRI field (perhaps with the idea this would be aimed at academics as a primary audience).In my view, this would be a different paper than the practically-oriented manuscript I have prepared with a focus on the UNESCO monitoring process for the Recommendation on Science and Scientific Researchers.

Introduction
Responsible research and innovation (RRI) has been a key priority for the European Union for many years.Efforts to advance socially responsible research and innovation aim to get science and technology to take into account societal needs and ethical considerations, while driving engagement and dialogue with relevant parties who may be affected.It involves incorporating public concerns and perspectives into scientific and innovative processes to ensure beneficial outcomes and foster societal acceptance.In practice, RRI involves an insistence that research and innovation work in synchrony with society, addressing its diverse needs, whilst prioritising ethical transparency and inclusivity.It strives to create an ecosystem where science and society co-evolve, with the former respecting the latter's values, aspirations and reservations.In recent years, the need for high quality measurement approaches to evaluate responsible research and innovation policies and practices has become increasingly apparent.Establishing globally relevant and usable indicators for RRI is challenging but essential, given the global nature of science.Ensuring these indicators are methodologically robust is important but must be balanced against practical constraints facing measurement initiatives in this domain (Jensen, 2020a;Jensen, 2020b;Jensen & Lister, 2015).This is a daunting given the uneven mechanisms for data collection and analysis available in different countries (e.g., Heras et al., 2016;Heras & Ruiz-Mallén, 2017).
The geopolitical challenge of gaining consensus and buy-in from countries with very different and sometimes competing agendas to align their RRI indicators can be addressed in part by tethering those indicators to a shared global policy instrument.Here, the 2017 UNESCO-led Recommendation on Science and Scientific Researchers (RS/SR) (UNESCO, 2017) is highlighted as a valuable existing vehicle to enable cooperation on globally comparative measurement of socially responsible research and innovation.Numerous aspects of RRI that are enshrined in the RS/SR have implications for public views on the role of science.Therefore, it is worth considering including an indicator dimension that focuses on the public aspect of the RS/SR priority areas.The framework focuses on aligning the public opinion level of measurement to benchmark global survey measures, where feasible, such as the Wellcome Global Monitor (WGM) (wellcome.org/reports/wellcome-globalmonitor-mental-health/2020wellcome.org/reports/wellcomeglobal-monitor-mental-health/2020).
In particular, the quadrennial monitoring of the implementation of this wide-ranging global policy instrument that has been ratified by 195 countries affords a unique opportunity to add value for these countries by linking RRI to the Recommendation while establishing benchmark indicators for RRI more generally.As a practical and methodological contribution to the global community of science and innovation policymakers, researchers, and research and innovation stakeholders committed to socially responsible research, this essay contains specific, detailed survey questions and response options focusing on the public opinion and individual researcher level of measurement.It provides details of sources of international survey data that are readily available for secondary analysis to benchmark the development of socially responsible research and innovation over time from the vantage point of the public opinion.
10 key priority areas were identified and agreed by UNESCO Member States as a practical way to simplify and focus RSSR implementation and long-term monitoring, given the sprawling coverage of the full policy text (see Jensen, 2022a).This document outlines benchmark survey indicators and concomitant open data sources pegged to these 10 key priority areas that can be used to evaluate the development of RRI principles that are integral to the RSSR policy instrument.This measurement approach also allows for the fact that many of the principles included in the RSSR are already in evidence to some extent at different levels of national research systems, yet public opinion on these topics is not well established.

Public opinion level of measurement for global RRI indicators
This manuscript goes beyond the current state of the art by presenting a framework of indicators that can be used to gain an understanding of RRI progress at the public level.This public level of measurement is often overlooked in a focus on national statistics and research and innovation per se, but the public's perspective on progress towards ever more socially sustainable science and innovation is essential to the long-term health of research and innovation ecosystems globally.Providing RRI indicators at this level of measurement offers specific ways to improve on existing RRI indicators and better support evidence-based practice in this domain (Jensen & Gerber, 2020).
The checklist icon (left) in this document highlights questions linked directly to the formal national reporting (UNESCO, 2021) for the UNESCO-led Recommendation on Science and Scientific Researchers RSSR (UNESCO, 2017).Answers to these overarching questions guide the reader to parts of the document likely to be most relevant.This is designed to highlight relevant RRI measures and indicators that have clear links to the long-term monitoring of the RSSR's implementation.
Direct quotations from the RSSR are included in grey font in the underlying data (Jensen, 2022b) to help clarify the different aspects of the categories included in the 10 key priority areas.
Here, indicators are documented that can provide evidence relevant to the public opinion level of responsible research and innovation indicators.Here, these indicators are organised according to the 10 key priority areas for the UNESCO RS/SR.This section of the report describes the sources of global survey data that have been used to provide these indicators.This international science survey of science and society topics includes the extent of public trust in scientists, consumption of science news, views about science policy, and government investment in science, as well as a range of other related topics (pewresearch.org/science/dataset/international-sciencesurvey).

World Values Survey (WVS)
The world values survey was conducted between 2017 and 2020 (worldvaluessurvey.org/wvs.jsp).worldvaluessurvey.org/wvs.jsp).The country coverage s extensive, including 49 nations from a wide range of geographically dispersed regions.
In this dataset, there are relevant survey questions to RRI and the RS/SR, including information about public values relevant to science.

Special Eurobarometer 'public perceptions of science, research and innovation'
A special Eurobarometer survey was carried in 2014, focusing on European Union (EU) citizens' attitudes about scientific research and related issues This dataset covers public attitudes about science, research and innovation with samples in countries across Europe.

3M State of Science Index Survey
().With an average sample size of 1,000 for consistency, this public attitudes survey has been gradually adding additional country coverage over time, and already spans different world regions.
The 3M annual state of science index measures science attitudes in 14 countries (dataset), focusing on various aspects of public views about science.This dataset covers a relatively small number of countries but offers good coverage for those countries.

SFI Science in Ireland Barometer survey
An example of a national-level science attitudes survey, this survey was conducted in 2020-2021 in Ireland.It followed robust methodological procedures in the set-up, piloting/validation, and implementation of the survey design..This survey dataset covers a wide range of RRI-related topics, including public views about science's inclusiveness, whether the benefits of science are widely shared and gender equality in science.

Individual researcher level of measurement for global RRI indicators
This essay goes well beyond the state of the art by presenting a framework of indicators that can be used to gain an understanding of RRI progress at the individual researcher level.These indicators are mapped directly to the formal national reporting (UNESCO, 2021) for the UNESCO-led Recommendation on Science and Scientific Researchers (RSSR) (UNESCO, 2017).
This report documents indicators that can provide evidence relevant to the individual researchers' level of socially responsible research and innovation indicators (Jensen, 2022b).Here, these indicators are organized according to the 10 key priority areas for the UNESCO RSSR.This section of the report describes the sources of global survey data that have been used to provide these indicators, and that research and science policy stakeholders around the world can use to assess their current status with responsible research and innovation.

RRING survey on socially responsible research/innovation.
A Global monitoring and evaluation of RRI-related outcomes is rife with complicated practical and methodological challenges, given the diversity of priorities, interests, resource levels, and capabilities for undertaking measurement exercises in this domain internationally.RRI indicators should be globally relevant, given that science itself is a global enterprise.In addition, it is vital that indicators deliver precise results because they will be used to determine which kinds of RRI policies and practices are most worthy of further investment and development.Methodological rigour is also important to establish a good basis for comparative analyses (Jensen, 2020a;Jensen, 2020b;Jensen & Lister, 2015), , looking across different research and innovation systems to reveal the factors affecting the efficacy of RRI interventions.
This manuscript indirectly addresses the need for methodologically sound measurement approaches for RRI monitoring and evaluation at the level of individual researchers and public opinion.This is because standardised measurement, applied at a regular interval, will likely deliver higher quality data than isolated one-off efforts.Indeed, appropriate methods of monitoring and evaluation are achievable through the application of relevant social research principles: "Evaluation is just one type of research framework, which focuses on whether a set of objectives have, in fact, been achieved.There is every reason to expect both knowledge and practical guidance to emerge from the same well-designed impact evaluation."(Jensen, 2014: C04) Beyond the social research aspect of evaluation, however, the practicalities of implementing an indicators framework inevitably affect methodological rigour.Who will ensure that indicator data are collected?How will the quality of that data be assured?What will motivate nations or organisations to collect, collate, analyse and report such data?This manuscript offers an indicators framework that comes with practical solutions to these challenges for addressing the public opinion and individual researcher level of measurement (Jensen, 2022b).
The proposed use of a range of relevant data sources containing global researcher survey data provides an opportunity for methodological triangulation between different surveys and question types.These international survey data sources (and others) can be used to compare results between countries with the aim of identifying good practices worthy of wider implementation.
If the type of international surveys noted in this essay are used, then opportunities for useful empirical insights emerge that can reflect back on RRI's conceptualisation itself.To effectively mobilise evidence-based practice in the context of RRI, a wider community of international research policy, practice and evaluation must emerge to bring diverse perspectives into dialogue (e.g., see Jensen & Gerber, 2020).Such efforts can greatly enrich the RRI monitoring and evaluation process by taking into account the different socio-historical and cultural characteristics of each country.
The rationale for introducing high quality public indicators to monitor RRI globally is to enable evidence-based practice in the field of socially responsible research and innovation (Jensen, 2014;Jensen & Gerber, 2020).Without high-quality measurement in place (Jensen & Laurie, 2016), it is difficult to accurately pinpoint the policies that are effective in developing healthier research and innovation ecosystems (cf., Lindner, 2016).Moreover, flying blind without such measurement increases the risk that well-intentioned but ill-conceived RRI policy and practice interventions that have a negative impact in practice go undetected, unchecked and unreformed.
This manuscript provides a concrete proposal of survey questions that can be used to measure key aspects of socially responsible research and innovation at the level of public opinion.This level of measurement is important, given that the public ultimately provides funding and political support for long-term research and innovation funding from the government, and should be a major downstream beneficiary of research and innovation.The advantage of these particular questions is that they already have data available across many countries and/or have been put through methodological testing to ensure that they are robust.The author has decided not to address any of the comments raised in the extensive report we wrote previously (as reviewer 2) by stating that it would be a different paper should they do so.We are therefore unable to comment on the amendments as nothing we raised has even been commented on, changed or revised.In light of this, our view also does not change.
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: All three reviewers work in the area of computer and technology ethics and have strong backgrounds in RRI.
We confirm that we have read this submission and believe that we have an appropriate level of expertise to state that we do not consider it to be of an acceptable scientific standard, for reasons outlined above.

Version 1
Reviewer Report 20 October 2022 https://doi.org/10.21956/openreseurope.15350.r30259 © 2022 Haworth P.This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Penny Haworth
South African Institute for Aquatic Biodiversity Penny Haworth (SAIAB), Grahamstown, South Africa With regard to the presentation of a persuasive argument supported by evidence, the references supplied suggest a broad base of reading into, as well as some significant contributions to, the available literature on the topic.
As far as my personal knowledge of the current literature goes, discussion points appear well supported.
The final review question: 'Does the essay contribute to the cultural, historical, social understanding of the field?' is far-reaching in its focus and the debate is in a formative phase.
As a reviewer who is not an academic, my knowledge of the discipline is limited and my opinion is based more on experiential learning and experience in various contexts since the early 1980s first as a teacher, then museum educator and in the past 18 years consolidating that experience and applying it as a science communicator.Historically, it appears that the period covered by this article goes back only as far as the UNESCO-led RSSR (2017), although some of the survey information used was gathered prior to this (2014).It depends how far reaching this essay wants to be in its historical scope, but it could be informative to remember that museums and science centres have been grappling with ways to improve public understanding of science through interpretative design and the use of suitable language in interactive displays and exhibitions at least since the late 1980s.
Whilst the terminology has changed and RRI is now used (to a degree) to capture the idea of the social relevance and applicability of research and innovation, a return to some of the earlier literature regarding living museums and the development of interpretative frameworks to enable more effective engagement with pubic audiences may be useful in developing meaningful indicators for a broad spectrum of policies and practices.
Similarly, whilst the essay mentions the need for "globally relevant and usable indicators", it does not make clear reference to indigenous knowledge systems (IKS), which in third world and developing countries are beginning to be sought out and integrated into research and innovation.Perhaps the integration of IKS into RRI and contributions of RRI policies and practices to sustainable development could become indicators in their own right.
Is the topic of the essay discussed accurately in the context of the current literature?Yes

Is the argument persuasive and supported by appropriate evidence? Yes
Does the essay contribute to the cultural, historical, social understanding of the field?Partly Competing Interests: No competing interests were disclosed.
Reviewer Expertise: As stated in my report, as reviewer who is not an academic, my knowledge of the discipline is limited and my opinion based on experience in various contexts since the early 1980s as a teacher, museum educator and science communicator.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.It is unclear how the paper goes beyond the state of the art despite the author's assertions, and the current state of the art is not described or discussed.There is little discussion or critique, and the paper does not adequately discuss benchmark survey data in the context of making recommendations to evaluate processes that utilise RRI at its core.The abstract mentions a framework but does not provide a recognisable framework or approach within the paper itself.
The whole premise is the importance of evaluation of RRI practices and policies, but the paper fails to provide context, core principles of RRI or to reference key authors in the field such as von Schomberg, Owen, Stilgoe, Stahl, etc.
A key criticism is that the author draws much of their perspectives from their own published work and therefore does not draw adequately on the wealth of discourse and literature currently available.
The paper assumes that RRI should be deployed in a global context but provides little justification, and does not consider the global implications of implementing a fundamentally Euro-centric framework -given that the UNESCO RSSR referred to throughout this essay states an aim towards a set of 'shared values', it is difficult to reconcile this with the inference that those shared values should, by the nature of RRI, be European in nature.The essay does allude to challenges of different and competing national research agendas but offers no explanation of these differences nor competition/competitiveness.
There are also some inconsistencies and inaccuracies.For example, the author states RRI is integral to the RSSR policy instrument and "linking RRI to the 2017 Recommendation".However, later they take the position that the author-defined 10 key priority areas, "identified and agreed by UNESCO Member States" i.e., the RSSR Policy instrument, should be used to evaluate the development of RRI principles.However, Annex II, Item 7.4 of the agenda of the 37th UNESCO session, (Paris 2017), makes no mention of these 10 priority areas in its 48 recommendations over seven sections.Further, the enshrinement of 'aspects' of RRI in the RSSR is contestable as Annex II Recommendation on Science and Scientific Researchers of the General Conference of UNESCO (pp117-127), makes no specific mention of RRI.The basis for recognition or understanding of "the principles included in the RSSR," by the public, (layman), deemed "not well established" is not justified, and the essay argues for an indicator framework to monitor or measure such a perspective, but does not indicate how such an exercise will offer "specific ways to improve on existing RRI indicators".
The paper is somewhat situated in current discourses of the literature although it takes a contrary position.For example, its advocacy for "good metrics to measure progress" contradicts, without argument or justification, a Global Research Council (2021) finding of "a shift away from reliance on metrics towards more qualitative or mixed-methods modes of assessment" amongst the global signatories to responsible research assessment (GRC Publications | Global Research Council).Furthermore, the author's recommendation for the use of "existing metrics" to save resources and improve quality is unfounded.
The essay engages with a number of surveys around the topic of RRI indicators, although it is not possible to access the linked data and the checklist icon does not highlight any questions and provides no guidance to the reader to parts of the document likely to be most relevant.
The surveys take a diversity of approaches, but it is unclear how much overlap there is between the countries/regions covered, and whilst this is an important issue to address, without understanding which group/country/demographic each survey was tailored to, it is difficult to assess the use of these in the context of the 195 UNESCO RSSR signatories.
From a preliminary view, it appears that European countries feature (somewhat unsurprisingly) heavily in the selected public opinion surveys on RRI progress, and the suitability and/or effectiveness of these surveys being translated globally is ill-explored here.
Justification for the selection of the six sources of global survey data are absent, as are explanations of its relevance either to RRI or RSSR.Why just the six (at public level)?It is notable that whilst the WGM is relied upon as 'truly global', the author does not clarify that the 2020-21 survey covered a reduced scope of 113 countries, rather than the 140+ countries in the 2018 version -given 195 countries signed up to the UNESCO RSSR, this does not, in fact, feel 'truly global'.Further, the use of the 2014 Eurobarometer report feels outdated, given the development guidelines for any kind of literature review to be included, and indeed that was never intended here.The 10 key priority areas were officially adopted by the UNESCO Executive Board as a practical way forward for distilling the much longer original ratified text of the Recommendation on Science and Scientific Researchers.This was primarily aimed at making the monitoring process (which runs on a four-yearly cycle) more manageable for Member-States.The details are now available on the UNESCO website for the Recommendation on Science and Scientific Researchers, with more information coming over the next year or two.
Competing Interests: No competing interests were disclosed.

Łukasz Nazarko
Bialystok University of Technology, Bialystok, Poland Operationalising Responsible Research and Innovation paradigm is a necessary step towards the maturity of this concept.Some progress has already been made in this area -more with regard to public entities (research institutes, higher education institutions, funding agencies), less with regard to industry (enterprises, private labs).
The submitted essay looks at the possibility of developing a framework (a set of indicators) to monitor progress toward the socially responsible innovation at the global level, from the perspective of the public and individual researchers.Author proposes to make use of the already existing vehicle of the 2017 UNESCO-led Recommendation on Science and Scientific Researchers (RSSR).As such, this proposal is rational and offers an approach that addresses a frequent problem of "reinventing the wheel".
Since this is an essay, I recommend that the Introduction is supplemented with an explanation/definition of socially responsible research and innovation.The main body of the text consists of short descriptions of various data sources/surveys that, according to the author, may be linked to socially responsible research and innovation.The attempt to link particular surveys to specific aspects/dimensions of RRI is desirable.It would indicate clearly how a particular survey/dataset is linked to the concept of RRI.Moreover, some sort of a concise synthesis of all presented surveys would be helpful.What RRI dimensions are covered?What dimensions are not yet covered?
The chosen form of the submission (essay) imposes a certain discipline on the author.One has to present thoughts in a coherent and non-trivial way, without pondering on details and nuances in order to keep the text short.The author has been successful to this end.
Eight out of 13 referenced works are the author's own publications.That would be a problem with a research paper but is acceptable in an essay, especially given that the references are relevant.Reviewer Expertise: RRI, Technology Assessment, Innovation Policy I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Reviewer
Report 27 June 2022 https://doi.org/10.21956/openreseurope.15350.r29465© 2022 Wilford S et al.This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Sara Wilford DeMontfort University, Leicester, UK Mayen Cunden De Montfort University, Leicester, UK Tonii Leach De Montfort University, Leicester, UK This paper does not 'read well' and jumps between ideas without discussion, preamble or context.

Reviewer Report 12
April 2022 https://doi.org/10.21956/openreseurope.15350.r28820© 2022 Nazarko Ł.This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
the topic of the essay discussed accurately in the context of the current literature?Yes Is the work clearly and cogently presented?Yes Is the argument persuasive and supported by appropriate evidence?Partly Does the essay contribute to the cultural, historical, social understanding of the field?Yes Competing Interests: No competing interests were disclosed.

Organisation for Economic Co-operation and Develop- ment (OECD) International Survey of Scientific Authors.
scientific authors from all over the world and across all disciplinary areas, representing to a varying extent the subset of the research population engaged in scholarly publication work, including those in the business sector.
Heras et al., 2016;Heras & Ruiz-Mallén, 2017)esponsible Research and Innovation Networked Globally (RRING) project (zenodo.org/communities/rring).The survey was open from 1 st October 2019 to 20 th December 2019.Aiming to get a deeper insight into the practices and policies of responsible research and innovation (RRI) across the world, this study was conducted across 20 countries.Diversity was ensured across factors such as the research and development expenditures, per capita income levels, etc., while selecting the countries to prioritise for saturation sampling (a mix of locally organised data collection in selected countries and an overall open call for responses, with email-based participation requests, was used).A sample of scientific authors listed as corresponding authors received participation requests directly from the OECD and were asked to report on their use of a broad range of digital tools and related practices, in addition to another key demographic and career information.Responses were collected for a total of approximately 12,000ConclusionRRI initiatives strive for research and innovation to chart a course where scientific and technological developments are intrinsically entwined with societal values, needs, and ethical considerations.The purpose is not only to produce knowledge or inventions, but to do so in a way that respects the environment, the wider public, and the fundamental ethical standards that bind us all together.The EU has invested considerable time and resources in promoting socially responsible research and innovation policies and practices over the last decade.This has resulted in ever-greater demand for measurement frameworks and indicators capable of assessing whether this intervention is delivering improvements in research and innovation systems (e.g.,Heras et al., 2016;Heras & Ruiz-Mallén, 2017).
These survey questions are being offered up to start a wider conversation about how best to evaluate progress in the development of healthy and socially responsible research systems.Further development and refinement of the framework presented in this essay will certainly be needed.The long-term journey towards socially responsible research and innovation systems will benefit from such improvements to monitoring and evaluation methods.
Data are available under the terms of the Creative Commons Attribution 4.0 International Public License (CC-BY 4.0).Sara WilfordDeMontfort University, Leicester, UK Mayen Cunden De Montfort University, Leicester, UK Tonii Leach De Montfort University, Leicester, UK