Euro-VO - Coordination of Virtual Observatory activities in Europe

The European Virtual Observatory Euro-VO has been coordinating European VO activities through a series of projects co-funded by the European Commission over the last 15 years. The bulk of VO work in Europe is ensured by the national VO initiatives and those of intergovernmental agencies. VO activities at the European level coordinate the work in support of the three"pillars"of the Virtual Observatory: support to the scientific community, take-up by the data providers, and technological activities. Several Euro-VO projects have also provided direct support to selected developments and prototyping. This paper explains the methodology used by Euro-VO over the years. It summarizes the activities which were performed and their evolutions at different stages of the development of the VO, explains the Euro-VO role with respect to the international and national levels of VO activities, details the lessons learnt for best practices for the coordination of the VO building blocks, and the liaison with other European initiatives, documenting the added-value of European coordination. Finally, the current status and next steps of Euro-VO are briefly addressed.

at the European level coordinate the work in support of the three "pillars" of the Virtual Observatory: support to the scientific community, take-up by the data providers, and technological activities. Several Euro-VO projects have also provided direct support to selected developments and prototyping. This The digital revolution is bringing a change in paradigm in the way science is done. Astronomy has long been at the forefront for widespread sharing and re-use of data, with a new step towards interoperability and data integration through the development of the astronomical Virtual Observatory (VO). The concept of a Virtual Observatory is that astronomical data worldwide should be accessible through common protocols, with interoperability standards for metadata and tools so that access and use of the data become seamless. VO is embraced as a world-wide community-based initiative that is transforming and restructuring the way astronomy research is done.
The VO has now reached a significant level of maturity and provides an operational framework for interoperable and efficient access to world-wide astronomical data and services. The construction of the VO has followed a predominantly bottom-up, incremental, pragmatic approach that has been carefully matched to the needs of the astronomical community. This has been supported by national VO initiatives and data centres, with a significant contribution from the European VO initiatives. Having reached this level of maturity, it is now timely to write its history and to describe how it was built and the lessons learnt over the years. This paper is focussed on the European Virtual Observatory, Euro-VO. The main role of Euro-VO has been to coordinate European VO activities, with also for some of the European projects support to technological activities. This paper will highlight the main VO building blocks and lessons learnt on how to coordinate them.
The astronomical VO was conceived from the beginning as a global endeavour, and European participants were among the founding members together Formal participation in the VO is based on national and international collaborations gathering development teams and data providers, as stated in the "Guidelines for participation of the International Virtual Observatory Alliance" (Hanisch et al. 2010 [1]). VO initiatives from several European countries are IVOA members, as is the European Space Agency (ESA), which is an intergovernmental organisation. The potential usefulness of the coordination of VO activities at the European level was recognised very early. A mechanism was needed to bring the relevant parties together.
Funding by the European Commission through its successive Framework Programmes (FP) has a strong structuring role for astronomy in the European countries, as well as the two intergovernmental organisations ESA and ESO which pool resources from their member countries to realize large space-and ground based programmes. It thus appeared natural to set up a coordination of the VO activities at that level, which soon became the European Virtual Observatory Euro-VO. Inclusive federation of complementary expertise with a light organisational structure and an evolving partnership has been the foundation of the Euro-VO. As a starting point, the two intergovernmental organisations ESA and ESO brought in experience on how to deal with observational data from large ground and space-based telescopes, Strasbourg astronomical data centre CDS had anticipated several aspects of the Virtual Observatory (for instance Aladin already provided access to distant observatory archives, and employed a registry of resources (Fernique, Ochsenbein and Wenger 1998 [2])), and the UK VO project AstroGrid had just started. These were the main partners of the first European project. Other VO initiatives progressively started in other countries, and Italy, Spain and Germany (respectively the Italian Virtual Observatory VObs.it, the Spanish Virtual Observatory SVO and the German Astrophysical Observatory GAVO) joined Euro-VO.
The Astrophysical Virtual Observatory (AVO) project was successfully submitted for funding to the European Commission in 2001 (Quinn et al. 2002 [3]).
This allowed a close collaboration between the national initiatives, ESA, and ESO. As expected, the European initiatives brought their complementary expertise, and coordination and collaboration allowed them to reach a critical mass for the different aspects of the activities. We found that activities developed in collaboration at the European level worked well as templates for activities at national level, and technical collaborations between teams from different countries strengthened the impact of the work of the individual teams.
It was soon realized that the VO aim of providing seamless access to the wealth of astronomical resources, was well aligned with key objectives of the European Union. One of the main impacts of the VO is to offer an equal opportunity to access the best data and tools to scientists from all countries, and in particular to astronomers of all European countries, including peripheral and outermost regions. All data and service providers, large agencies, and small teams willing to share their knowledge, can contribute. The VO is thus a powerful vehicle for the integration of the national European astronomical communities into a single community, which is actually world-wide.
The European project also gave good visibility to VO development across Europe. This has been very important to get the VO taken into account in the European strategy for Astronomy [4] established in 2008 by the Astronet ERA-NET 1 , which gathers the astronomy funding agencies.
The paper is organised as follows: Section 2 briefly describes the suite of European projects which supported the development of Euro-VO over the years, and the evolution of the activities while the project was maturing; Section 3 explains the specific role of Euro-VO in the global VO landscape, and its positioning with respect to the IVOA and the national initiatives; Section 4 details the lessons learnt from the building of Euro-VO, and identify best practices for coordinating VO activities; Section 5 explains the Euro-VO role in a wider European context, its liaison with "nearby" disciplines and projects which work on generic building blocks of the data infrastructure. Section 6 concludes by discussing the current status of Euro-VO and the next steps.

The European VO projects: building the European contribution to the Virtual Observatory
The VO is one of the research infrastructures of astronomy, but it was a novel concept when it emerged and there was no pre-existing model to follow to develop it. The European projects progressively went through different phases, from R&D and proof of concept to operations, and defined the activities required to develop and maintain the European VO. The projects concerned are briefly described in this section, to show how the current Euro-VO strands of work progressively took shape. More details on how the activities were performed will be given in Section 4 which explains the lessons learnt. 1 ASTRONET was created in 2005 by a group of European funding agencies in order to establish a strategic planning mechanism for all of European astronomy. As of 2014, the ASTRONET consortium has eleven participants, twenty-one Associates and two Forum members.

The early stages of Interoperability
The first key contribution of European funded projects to the construction of the VO was through the Interoperability Working Group set up by the OP-TICON Thematic Network 2 in 2001, that was led by CDS. The global nature of the VO was understood from the start, and this Working Group included participants from the USA and Canada. It was the first international forum to discuss astronomical interoperability standards in the VO context.
A meeting held in Strasbourg in 2002, January 28th-29th, was the first exchange of views between the emerging VO projects which were just beginning to be funded (Europe, UK and USA). The meeting was a starting point for Europe-wide collaboration in the development of the VO, with representatives from France, Germany, Italy, the Netherlands, Spain, UK, ESA and ESO, going beyond the initial European VO partnership set up a few months before in the first VO project AVO, which will be described below. Several of the participants of this initial meeting have since played a key role in VO initiatives that were set up later in their country or Agency.
The Strasbourg meeting allowed a detailed discussion of a standard format for tabular data that was initially proposed in collaboration between the CDS and US-NVO. This was the starting point for VOTable (Ochsenbein et al, 2004 [5]), an XML-based standard for the exchange of tabular data, which became the first VO standard in March 2002. VOTable was derived from the astrores XML format (Ochsenbein et al., 2000 [6]), itself modeled on the FITS table format (Cotton et al., 1995 [7]) 3 . VOTable was taken up by the IVOA after it was created and is currently at Version 1.3. It has been (with the pre-existing FITS standard) a key asset for the development of the VO, allowing data exchange and thus a first level of interoperability from the early stages, and it has been 2 Fifth Framework Programme, HPRI-CT-1999-4002 3 VOTable overcomes limitations of FITS binary table usage in a distributed-computing environment, in particular the dataset size is not required in the header, which is an issue for remote data streams. It also allows expression of passwords or other identity information.
VOTable can be used either to encapsulate FITS binary tables or to re-encode the metadata.

The Euro-VO projects
European funding is organised in cycles, the so-called Framework Programmes of the European Commission (EC). Euro-VO 4 was supported by several framework programmes, first in the Research, then in the e-Infrastructure programmes.  Table 1 provides information about the EC-funded projects including dates and partners.

The early phases of Euro-VO
The early phases of Euro-VO are described in Genova (2009) [8].

9
AVO led to the understanding that the coordination of VO activities relied on a "triangle": support for scientific users, support for data providers, and technological activities. This structured the projects in the following phases (Padovani, 2006 [10]). the level of effort required to publish their data in the VO framework.
EuroVO-DCA performed the first census of European data centres. It also included explicitly for the first time liaisons with the astronomical community in European countries beyond the project partners, which has been a standing item of all the projects since then. Collaborations and VO Days organised with the Czech Republic, Lithuania, Poland, etc served to raise awareness and in some cases to create a national proto-VO.

Transition to operations
The Euro-VO transition to operations has been supported by three projects under the Seventh Framework Programme. the technical needs to maintain the VO framework" and "Outreach towards education and the general public interested in astronomy". These areas cover the different aspects of the VO-related work and they gather elements to feed the sustainability assessment.
The following sections summarize the findings of the projects on the role of Euro-VO in the VO landscape and lessons learnt for how to most effectively perform the different kinds of VO-related activities.

The role of Euro-VO in the VO landscape
Euro-VO is an element of the VO landscape which comes between the inter- coordination of the resources, which are provided by individual institutes and laboratories; in others, coordination is provided directly by a national funding agency. Many combinations of the above schemes are possible but the common characteristic is that all partners are strongly committed to VO development.
The balance between the implementation of VO in data centres and technical work on the VO framework is also different for the different partners.
Euro-VO has proven to be indispensable for coordination and cooperation, and for the exchange of information and the sharing of expertise, strengthening the weight of the individual European projects in international cooperation.
Individual projects keep their own aims, but learn from each other and build a global strategy, and duplication of efforts is minimized. This helps national activities to be better focused and to have much greater impact.
The European level also brings critical mass for the activities, in particular for support activities for scientists, data providers and education. The tutorials and support activities are developed at the European level, and then used as templates for national activities. Important collaborations between teams from different countries were also nurtured by the technological discussions within Euro-VO.
During the last few years, the European projects were funded to support solely coordination activities. The previous projects came also with support for service activities and technological development. This was instrumental in focusing the activities in key elements of the VO, the essential initial standards and the tools which are now the core VO framework, besides the proper interests of the national and agency initiatives. In turn this empowered the European teams to influence the development of the global VO. There are currently 39 standards recommended by the IVOA, among which only two have no European co-author, and 22 have at least one European editor.

Lessons learnt for best practices for VO activities
As confirmed by the Euro-VO projects, the activities required to maintain the Virtual Observatory are the following: • Development of standards and protocols, and their international agreement; • Construction of "glue" software components -registry, workflow, user authentication, virtual storage; • Uptake by data centres, who need to "publish" to the system, i.e. to write VO compliant data services connected to their holdings; • Construction of tools to effectively take advantage of seamless access to data, some of them able to act as VO portals for certain types of data; • Support for the scientific community in its uptake of the new framework.
The three main components are well summarized in Figure 2, which shows the IVOA "level zero" architecture as stated in the IVOA Architecture document (Arviset et al. 2010 [12]), with data providers (including modelling data) at the bottom and users (including machines) on the top, and in the box in the middle the elements of the Virtual Observatory framework: the VO provides a technical framework for the providers to share their data and services ("Sharing"), and allowing users to find ("Finding") these resources, to get them ("Getting") and to use them ("Using"). To enable these functionalities, the establishment of some core astronomically oriented standards ("VO Core") is also necessary.
Euro-VO activities over the years led us to define best practices for the main strands of work: support to the scientific community, support to data providers, and coordination of technological work. They are detailed in the following subsections.

Support to the scientific community
Euro-VO considered from the beginning that providing support to the community includes gathering its requirements. The Science Advisory Committee was set up at a very early stage and regularly met to discuss science needs, comment on the Euro-VO activities and propose directions for future work.  [16]). Members of the science community were invited to discuss their scientific needs with VO scientists and developers, and VO teams presented the current status of the VO in these domains. This was at a stage when the core standards and tools were being developed and it was important to gather detailed science needs.
For a more direct support to individual researchers, several kinds of activities were tested. The first projects included Calls for scientific proposals, the so- The template defined by the Euro-VO "Hands-On Workshops" has been used in national and regional events. There is a real interest in the scientific community, as demonstrated for instance by the remarkable success of the "VO...  [19]). The Spanish VO SVO has also been organising high impact national VO schools 12 since 2009. The eight schools gathered more than 200 participants (which represents more than one third of the census of Spanish astronomers). One year after the school, participants are contacted asking about the role VO-tools play in their daily research. In most of the cases positive answers are received (they do really use VO-tools and services) and, in some cases, they have published VO-science papers in which they did the VO analysis independently (without support of SVO staff).
European astronomers are currently playing a leading role in the VO science field. The Data Centres were asked to identify the types of data and services they provide according to three main categories: "Observational Archives and Data Products", "Services, Tools and Software Suites", and "Theory Archives or Services". The Venn diagram in Figure 3 shows the relative proportions of the responses in each category, and the strong overlap between these categories with many Data Centres indicating that they provide multiple types of services.

Support for data providers
The Census is used to gather information on how data providers use the VO framework, and about their feedback and requirements. trained in the usage of data publishing tools, and were invited to apply them to their own data holdings. Several participant groups and individuals eventually posted VO-compliant data and services, and some engaged as active IVOA participants. There was initially a good response to these workshops, but at the third workshop fewer participants applied and it was felt that this was the end of a cycle. The individual VO initiatives continued to give support to data providers on request, and many potential VO contributors also attended the IVOA meetings and events organised at the national level to learn about the VO framework.
The good level of dissemination of knowledge about the VO in the European data provider community was demonstrated by the Data Centre Forum organised by the CoSADIE project, which gathered 40 participants in Heidelberg in June 2013. They were invited to explain how they use the VO framework, and many talks were presented, showing a high level of awareness and take-up.
Data providers would welcome a regular forum to share their experience of using the VO, provide feedback and get support from the VO teams. It was emphasised that making data centre holdings available in the VO increases visibility and impact. Another important motivation for data providers to engage in VO take-up is that they are interested in having their data made useable in VO tools such as Aladin for visualization or TOPCAT for tabular data: they also use the VO because it saves time and effort and provides valuable tools. The help provided by VO teams was identified as an important factor for take-up, especially for the smaller data centres. Some also use VO publishing tools such as, currently, DACHS, SAADA, VODance, developed respectively in Germany, France and Italy, and the suite of tools developed by the Spanish VO (SVO-Cat, MySpec-MyImg) 13 . The most appropriate publishing tools depends on the characteristics, in particular of the complexity, of the data. VO publishing workshops are also organised at the national level, for instance by the French 13 Publishing tools have been developed since the early stages of the VO, for instance in the EuroVO-AIDA project, but most of the first generation of tools is obsolete. Large astronomy infrastructures also participated in these events, but the situation is a bit different for them. It is better and better understood that a close collaboration of the VO with the large projects is essential: they bring new kinds of data which may require specific developments and adaptations of the VO standards and tools. This has been identified as a key factor to ensure VO relevance and thus its sustainability in the medium term, and establishing links with these projects has been a target for the European VO projects. The IVOA has been organising Focus sessions with large current and future projects since its meeting in Heidelberg in 2013, and has set IVOA priorities with the large recent and future projects in mind -these are currently multi-dimensional data (ALMA, VLT and E-ELT, LOFAR, SKA) and the time domain (LSST).
European members of the IVOA Committee on Science Priorities have been among the driving forces of this evolution.

Why and how to organize technical collaboration at a continental level
The core coordination activity for the development of the VO framework of standards and tools is obviously at the IVOA level: the Working Groups and Intererest Groups, overseen by the Technical Coordination Group (TCG).
IVOA organise the technical activities, and priorities are set up by the Executive Board. Also, as explained in Section 3 most of the technical work is performed by the national and agency VO initiatives.
The Euro-VO added-value for the technological activities is that it allowed the sharing of knowledge and the establishment of close collaboration in common technological activities, as well as the building of links between tools, which are one of the core assets of the VO interoperability framework thanks in particular to the SAMP 14 standard (Taylor et al. 2012 [21]). As explained in the preamble of SAMP "IVOA members have recognised that building a monolithic tool that attempts to fulfil all the requirements of all users is impractical, and it is a better use of our limited resources to enable individual tools to work together better".
Euro-VO fully endorsed this philosophy from the beginning and its partners put emphasis in the development of tools allowing access to, visualisation of and usage of VO-enabled data such as Aladin (VO portal for images), TOPCAT (Management tool for tabular data), VOSpec (visualisation and management of spectra), VOSA (Spectral energy distribution building and analysis).
Technical cooperation and coordination has been organised since the VOTECH project through bi-yearly meeting of the European technical teams involved in the VO development, the Euro-VO Technology Forums. These meetings allowed the sharing of information about VO status and technical developments in the different countries, the establishment of cross-border collaborations, and also "hands-on session" (Hack-a-thon) for testing or prototyping software. IVOA standards in development could also be discussed, especially when diverging opinions had shown up between European teams or when most of the interested participants were Europeans, allowing more efficient work in the IVOA context.
Additional face-to-face meetings were also organised on specific topics, for instance theory -to which Euro-VO has been devoting special attention since the EuroVO-DCA project. These meetings were instrumental in building a Europewide technical VO community, including staff who were eventually not able to travel to the IVOA Interoperability meetings, in particular junior staff, temporary contractors and interns, who could get an early exposure to working in an international context.
The Euro-VO projects which provided support for technological activities also produced a detailed description of the VO architecture, and important advances in the VO framework, by allowing the partners to focus technological work on key standards and tools and to assess new technologies. and in web browsers, and is also intended to form a framework for more general messaging requirements.

Euro-VO in a wider European context
Euro-VO paid particular attention to liaisons with "nearby" disciplines. Several national initiatives, for instance the French and Italian ones include representatives of other disciplines, in particular planetary or heliospheric studies, or astroparticle physics, as well as representatives of the atomic and molecular physics (since atomic and molecular data are heavily used to interpret observations in several fields of astronomy). Several of these disciplines also set up European projects dealing with data aspects, for instance among others Euro-Planet, HELIO (and other projects assessing the sharing of heliospheric physics data) and VAMDC. The other disciplines assessed solutions developed for the astronomical Virtual Observatory with respect to their own needs. In some cases it did not fit but some of the VO standards and tools were re-used and customized in other disciplinary contexts, for instance standards such as the Registry of Resources, VOTable or the Table Access Protocol, or tools such as Aladin and TOPCAT.
The specific requirements of the Cerenkov Telescope Array CTA, a project common to the astronomy and astroparticle domains, are also tackled by Euro-VO and the IVOA to include these data fully in the Virtual Observatory context. They are for instance particularly useful to understand how to deal with "provenance" in the VO.
More generally, Euro-VO was one of the early examples of a European project dealing with the sharing of scientific data in a disciplinary context, and EuroVO-AIDA was one of the seven projects selected in the first Call for Proposals of the Seventh European Framework on the Scientific Digital Repositories topic, in 2008. The astronomical VO has since then been presented in many meetings in Europe, as an example of an operational, world-wide disciplinary data framework. All disciplines have different cultures and organisations, and there is no unique solution to build the disciplinary part of the data infrastructure. But Euro-VO and the IVOA have explored many of the aspects of building a disciplinary framework and they share lessons learnt.
Liaison with projects which deal with the generic aspects of the data infrastructure was also sought by Euro-VO. Astronomy was one of the thematic topics of the first European project 15 devoted to the European support for the emerging Research Data Alliance RDA. The aims were both to share lessons learnt in the building of the IVOA, and to explore possible liaison with generic interoperability projects, in particular EUDAT 16 (European project which builds generic building blocks for the scientific data infrastructure). These were also among CoSADIE goals.
The Virtual Observatory framework has never been an isolated island, and generic building blocks were used whenever possible. In particular, the IVOA Registry of Resources (Hanisch et al., 2007, [22]) is compliant with OAI-PMH, the Open Archive Initiative -Protocol for Metadata Harvesting widely used by the digital library community (and also by EUDAT). Also the standard format for IVOA vocabularies (Derriere et al., 2009 [23]) is based on the W3C's Resource Description Framework (RDF) and Simple Knowledge Organization System (SKOS). As stated in the IVOA Recommendation for vocabularies, "by adopting a standard and simple format, the IVOA will permit different groups to create and maintain their own specialised vocabularies while letting the rest of the astronomical community access, use, and combine them. The use of current, open standards ensures that VO applications will be able to tap into resources of the growing semantic web." The aim of Euro-VO liaison with EUDAT and the RDA is also to make sure that the generic framework is able to interoperate with existing disciplinary frameworks, and that our specific requirements are taken into account. 15 This project (2012-2014), initially called iCORDI for International Collaboration on Research Data Infrastructure, soon became RDA/Europe. 16 www.eudat.eu

Conclusion
As shown throughout this paper, the three main pillars to build and maintain VO coordination are support for scientific users, support for data providers, and coordination of technical activities. At the beginning of the Euro-VO endeavour, this was described by a proposed organisation with three entities, the Euro-VO Facility Centre (in support of the scientific users), the Euro-VO Data Centre Alliance and the Euro-VO Technology Centre, each led by one or two pre-defined partners, with distributed participation (Padovani, 2006 [10]; Genova, 2009 [8]). Euro-VO partnership evolved along the years, and the current system is more versatile, with a Memorandum of Understanding between five organisations and institutes currently representing the national initiatives which share the core VO activities in Europe 17 in close liaison with ESA, and possibly evolving responsibilities for each partner defined in common agreement.
The Euro-VO project has been linking together the European national and agency VO initiatives and increasing their effectiveness. It succeeded in building a European-wide VO community of scientific users, data providers and VO teams, in making astronomy visible amongst the European disciplinary data infrastructures, and in having the Virtual Observatory recognized as a Research Infrastructures of astronomy in the European Roadmap established by the Astronet ERA-NET in 2008 [4]. It has proven to be indispensable for cooperation and collaboration.
One of its products is the assessment of all the elements needed for medium term sustainability, and work is on-going, in particular with the funding agencies represented in Astronet, to understand how these elements can be put in place.
One clear evolution is the need to establish closer links with the large projects of the discipline. This is the aim of the next Euro-VO project, which enters as Astroparticle ESFRI facilities". ASTERICS WP4 will establish a close collaboration between the Euro-VO and infrastructure teams. It will develop seamless access to the data products of the ESFRI projects and pathfinders, and other large infrastructures such as the European Gravitational Observatory EGO, in the VO framework. The tasks again cover support to data providers, support to scientific usage, and technological development.
Another clear strand for evolution is a closer collaboration with, and effective participation in, projects which work on the generic framework for data sharing, driven in particular by the very fast emergence of the Research Data Alliance as a major global initiative in scientific data sharing.