Monitoring Open Science policy using a regional CRIS – the Flanders case with FRIS

for EuroCRIS conference “LINKING RESEARCH INFORMATION ACROSS DATA SPACES”, 15th International Conference on Current Research Information Systems (CRIS2022), Dubrovnik, Croatia, May 12-14, 2022 (https://eurocris.org/cris2022-conference-call-proposals) track 2: II. Measuring research, its quality and impact


Introduction
This contribution gives an overview of the process to set up a policy monitor for Open Science in Flanders using FRIS, the regional CRIS of Flanders, as a primary datasource. With this contribution we want to share the experience of the Flanders case. We describe the process and elaborate on the outcomes.

Open Science policy initiative in Flanders
In december 2019, the Flemish government decided to support an Open Science initiative in Flanders which formulates a joint vision on Open Science and EOSC for knowledge institutions in Flanders 1 . It provides support to the Flemish knowledge institutions with a recurrent budget of 5 million euro's. With this budget the stakeholders can invest in data-infrastructure and engage the necessary human resources needed, like data stewards. The roll-out of the initiative is managed by the Flemish Open Science Board (FOSB), a committee representing the knowledge institutions, the research funders and the Flemish government. This FOSB will steer and foster the necessary transition towards Open Science and EOSC making publications open and datasets as FAIR as possible and as open as possible and as closed as necessary. The FOSB reports on the progress of the initiative to the Flemish government.
In this initiative, FRIS (Flanders Research Information Space), the regional CRIS of Flanders, is put forward to act as a hub for metadata on datasets towards EOSC hence contributing to the disclosure of Flemish research data. By making the information about research in Flanders openly available, FRIS contributes to the Open Science goals by making research more findable and accessibly. In addition, FRIS is appointed as the primary data source to monitor progress in Open Science. In this contribution, the use of FRIS for the Open Science monitoring is highlighted.
FRIS, the regional CRIS of Flanders FRIS, the Flanders Research Information Space, is a regional CRIS, capturing information on publicly funded research performed in Flanders. The FRIS-platform is connected with the CRIS-systems of all Flemish universities and many other knowledge institutions. These connections enable research information to flow automatically from the institutions to FRIS. As soon as a knowledge institute changes something in its system, an automated update is sent to FRIS. The exchange format of the data is CERIF, the Common European Standard for Research Information 2 , version 1.5.
The coverage of research infomation in FRIS keeps expanding, as well in number of connected knowledge institutions as in the span of research information objects it covers. Today FRIS contains information on more than 40.000 researchers in nearly 2000 research groups of about 60 knowledge institutions, nearly 50.000 projects, and more than 520.000 publications. In the course of 2022 FRIS will cover information on additional research information objects: patents, datasets and research infrastructure.
The (meta)data supplied are made accessible via the FRIS portal (www.researchportal.be/en) and through open API's.

FRIS and monitoring
Monitoring of research activities FRIS contains a good overview of the research activities of publicy financed research performed by knowledge institutes in Flanders that are connected to the FRIS-platform. The Flemish administration uses this information for policy monitoring, by tracking how research funding is growing through the years, how it is spread across research disciplines, what the outcomes are in terms of publication outputs etc. The information in FRIS also helps to highlight which research groups or researchers are involved in very specific research topics. It is also used to answer questions asked by the parliament on research policy questions like "how much budget did the Flemish government spend on alzheimer research and which projects are currently running?". Using FRIS, answers can be generated quickly without having to bother the knowledge institutions and ask each of them to search for the information. Thus the administrative burden is reduced on the side of the knowledge institutions (the institutions deliver once to FRIS, the administration re-uses it often).
In addition, the Flemish government recently decided to retrieve the figures for the reporting of the funding it provides to universities and schools for higher education completely and directly via FRIS. More specifically this is for the instruments BOF (Special Research Funds, these are funds for fundamental research for universities) and IOF (Industrial Research Funds, these are funds for applied research for universities and schools for higher education). This decision implied quite a huge change in the reporting process of the institutions and it took a some time from the moment it was decided (new legislation in force from 2019 on) until it was fully implemented (the first trialruns were done in the beginning of 2022). The transition process involved clarifying clearly what information was expected from the institutions, making agreements about definitions of concepts used in the reports, agreeing on the details of what was to be included or not, what was to be attributed to which category etc., implementing the necessary extra attributes in the datamodel (as well on the side of FRIS as on the side of the CRIS's of the institutions), gathering the data, running trials, collecting feedback of the institutions, perfecting the report and the process and repeat the latter steps in an iterative way. This has been a huge and time-consuming effort for the institutions, but once the process will be optimised, it will be of great benefit for them. The process will then be largely automated as project data will be gathered and integrated in the report automatically by FRIS, so it will reduce the administrative burden on their side. The benefit for the administration is that it will receive data of much higher quality than before and these data will be better comparable from one institution to another because of the better definitions and better understanding by the institutions.

Monitoring of Open Science
When the Flemish government launched its Open Science Initiative in 2019, FRIS received a new task in policy monitoring as it was appointed as the tool to monitor the progress of Open Science. So the FRIS-team was closely involved in the process of setting up the Open Science metrics.

Setting up the measurement framework for Open Science
The process of setting up the measurement framework was quite laborious, starting with the launch in 2019 of the initiative and installing the FOSB, defining the KPI's, setting goals, extending the research information models to include datasets and all additional attributes needed for the metrics, expliciting in detail the metrics based on the data in FRIS and finally executing the measurements.
Defining the Open Science KPI's The FOSB was asked to develop KPI's to monitor progress of the Open Science initiative. The following KPI's were defined: -ORCiD KPI: researchers that receive public funding should have an ORCiD -DMP KPI: projects that receive public funding should have a Data Management Plan (DMP) -Open Access KPI: peer-reviewed journal articles resulting from publicly funded research should become available in Open Access -FAIR KPI: Research data underlying journal articles resulting from publicly funded research should become as FAIR (Findable, Accessible, Interoperable and Reusable) as possible -Open Data KPI: Research data underlying journal articles resulting from publicly funded research should become openly available.
The definition of these KPI's was thoroughly discussed in FOSB taskforces and working groups, thus ensuring to envolve all the relevant stakeholders and get them on the same page.
Especially for the KPI's involving datasets, i.e. on FAIR and Open Data, this proved to be quite challenging as the gathering and registration of datasets currently is only in its primary stage. It is very difficult to talk about monitoring of data that are not yet or only partly captured. Institutions do not yet have a complete overview of the datasets that are produced by their researchers nor do they have much experience with policies on how to store, manage or capture datasets. Big efforts are currently going on to make progress on this in the months and years to come and the Open Science initiative is a big driver to do so. In the meantime however, some KPI is needed to monitor progress starting right now. For the KPI on Open Data, a best effort was made to create a definition, realising that this definition is not perfect yet and that it should be revised once the research community reaches more maturity with respect to the management of datasets and more systematic registration of datasets has become a reality.
For the KPI on FAIRness of data on the other hand, there was no general agreement yet on how to measure this. Since there was no mature measurement framework available on the international scene, it was decided to put this KPI on hold, until an measurement framework becomes available. Flanders will define this KPI in line with the European definition.
In addition to the definition work, effort was also put in defining the final goals to be reached by each knowledge institution for the KPI's and the timeline required to reach these goals. For each KPI, three possible timelines were set out varying in degree of ambition: in the most ambitious track, the goals have to be reached in a shorter timeline than in the least ambitious one. A zero-measurement will determine which one of the tracks a knowledge institution has to follow.
The final definitions, goals and timelines were documented and approved by the Flemish government 3 . Attention was also drawn to the requirement of the Flemish government for the KPI's to be monitored as much as possible using data in FRIS.

Extending the research information datamodel with the information needed for the metrics
The research information model then had to be extended with the attributes that are needed for the metrics. For datasets, no research information model was available yet within FRIS, so the FOSB Working Group Metadata & standardization, created a common metadatamodel for research datasets, based on de DataCite-model but extended with the properties needed for FRIS and for the Open Science metrics. This metadatamodel 4 includes all the attributes that will be registered by all knowledge institutions in Flanders, it includes info on the mandatory character and lists definitions about the terms used.
This model was then used to extend FRIS with an object for dataset (using the cfResProd entity), and extended with a few FRIS-specific extensions that were not available in the used cerif-version, but necessary to include in FRIS because of the legal requirements and the metrics for Open Science, like information on Open formats and Opt-out reasons. Also for other objects, some extra attributes were added for the KPI-metrics, like a tag for DMP on projects, and an Open Access label for publications. The current scope of the FRIS research information and how it can be delivered to FRIS via xml is documented extensively in the FRIS Integration Guide 5 .
Expliciting and validating the metrics of the KPI's methodology The next step was to explicit how the metrics of the Open Science KPI's were going to be translated into the selection of metadata in FRIS. To this end, the FRIS-team wrote down very detailed specifications on how the measurement of the KPI would be done based on data in FRIS (and some external sources) and published this into Technical Forms.
Making this measurement methodology in FRIS so explicit has some advantages: -It creates perfect transparency on how the indicators are measured. The results can be reproduced by using the data disclosed in FRIS and some external sources. -It gives consistent results since data from all knowledge institution are processed in exactly the same manner. The prerequisite is of course that all providers have a clear understanding of the definitions of the detailed data that are sent to FRIS; -For institutions that already provide data towards FRIS, it reduces administrative burden to gather and calculate the data for the KPI-measurement as this will be done automatically in FRIS; -For knowledge institutions that do not (yet) provide data to FRIS, they can use the methodology described in these technical forms to calculate the metrics based on their internal data using an methodology that is as comparable as possible. This increases data quality and comparability of the data gathered throughout the stakeholders.
These forms were distributed among the stakeholders of the FOSB and their feedback was gathered in order to further clarify and optimize the measurement methodology. A lot of effort was put into this process because it is important to have a good common understanding in the way progress will be tracked so that all stakeholders move in the same direction.
During the discussions it became clear that the KPI's and their definitions are not yet perfect. They have limitations in the sense that they do not yet reflect the full complexity of the reality e.g. the KPI on Open Data was limited to "datasets that are underlying peer reviewed publications". It is clear that there are many more other datasets, not complying to this restricted definition, that are equally valuable with respect to Open Science, but the scope of the definition was narrowed because of practical considerations whereby a step-by-step approach seemed more achievable, rather than trying to capture the full complexity from day one.
The purpose of the KPI's is to move all stakeholders towards Open Science, and monitoring this with metrics that are a "best effort" for now, realising the metrics is not perfect. The main goal is to make progress in Open Science. In 2023 the Open Science Initiative of the Flemish government will be evaluated giving also the opportunity for the metrics to be perfected. For now, the measurements will be done using this first methodology that was agreed upon.
The final version of the Technical Forms was communicated to and approved by the FOSB. These documents and all the annexes they refer to, are made publicly available on the FRIS Open Science webpage https://researchportal.be/nl/open-science.
The indicators will be measured in FRIS in the end of April of each year, including all the details behind the figures, and sent to the knowledge institutions. If they notice incorrect or incomplete data, they can send an update of the corrected data to FRIS and ask for an update of the KPI-measurement. The knowledge institutions incorporate these KPI-results into their yearly report on Open Science to the government and can add comments.
In 2023 the Open Science initiative will be evaluated and the Flemish government will then decide on a potential prolongation of the initiative.

Conclusion
In Flanders the government adopted an ambitious policy initiative to promote Open Science and it appointed FRIS, the regional CRIS, as the main tool to monitor the progress.
Using a CRIS as a monitoring tool for a regional policy initiative has some advantages like good transparency, consistency, comparability of the results and less administrative burden for the institutions once the process for gathering and sending the information is in place.
The preparatory process to define concepts, create definitions, discuss about the scope and the goals, was quite laborious and time-consuming, and was absolutely useful and necessary to get a good common understanding of the context, yielding better metrics and better alignment of the stakeholders towards the same goals.
The current metrics framework is not an endpoint, but only the beginning. It is a start to improve the way we will monitor Open Science as we gather more insights on the way. The metrics help to get all stakeholders pulling in the same direction towards more Open Science. We are looking forward to the challenges ahead of us!