Solving the grid interoperability problem by P-GRADE portal at workflow level
Introduction
Current production grid resources are provided as non-interoperable grid islands that cause difficulties when scientific experiments overgrow the capabilities of one specific grid. Different production Grids are based on different grid middleware, use different tools and policies for authentication and authorisation, describe and submit jobs differently, and provide data services based on a variety of protocols and access mechanisms. This makes the utilization of resources from several production grids rather cumbersome. No wonder that the Grid community devotes quite a lot of effort nowadays towards grid interoperability. However, most of these research efforts concentrate at middleware level solutions where the introduction of widely accepted standards is required. This could take quite a long time leaving the grid user community with non-interoperable solutions in the meantime.
In the current paper we show that interoperability can be solved at a much higher level, namely at the workflow level that could be part of a Grid portal. Indeed, P-GRADE Grid portal [8] is the first Grid portal that tries to solve the interoperability problem at the workflow level with great success. It means that the components of a workflow can be executed simultaneously in several Grids. In this way the user can exploit more parallelism than inside one Grid provided that the same workflow is executed with many different parameters as a parameter sweep application. More than that the workflow level completely hides the low level Grid details for the end-user who need not learn the low level Grid commands of different Grids. In this way porting workflow application between different Grids can be done with minimal user efforts. It also eliminates application porting work when a production Grid moves to a new Grid middleware. This happened, for example, in EGEE where the LCG middleware is changed to gLite. Users who developed their application under the P-GRADE portal at the workflow level can use their applications in the gLite-based EGEE Grid as well without any modification of the application. Similar changes of technologies are foreseen for many production Grids including UK NGS [10], US OSG [14] and US TeraGrid [15]. These Grids currently use GT2 technology but in the near future they will move most probably to GT4. Since P-GRADE portal already supports both GT2 and GT4, users of these Grids will be lifted from tedious application porting activity if they use the workflow environment of P-GRADE portal.
Since second generation Grid middleware like GT2 supports job submission activities meanwhile third generation Grid middleware like GT4 provides service-oriented usage of the Grid, P-GRADE portal supports both job submission and service invocation inside the workflows. For the purpose of service invocation P-GRADE portal was integrated with the GEMLCA legacy code architecture [9] that was developed by the University of Westminster to turn legacy codes into Grid services with minimal user efforts without touching either the source or binary code of legacy applications.
In the paper we describe those features and techniques that are provided and used by the P-GRADE portal to solve the Grid interoperability problem. Section 2 gives an overview of state-of-the-art grid interoperability work. Section 3 explains those features and techniques in P-GRADE portal that are provided for job submission. These include techniques for handling multiple Grid certificates, for connecting and tailoring the portal for several Grids, for providing access to various Grid information systems, for providing access to various Grid brokers and for assigning workflow components to various Grids. Section 4 describes how the integrated P-GRADE GEMLCA portal can support service invocation both in the second and third generation Grids. Finally, Section 5 shows how P-GRADE portal can connect various Grids (UK NGS [10], EGEE [11], WestFocus [12]) and various grid technologies using jobs and GEMLCA services in a mixed way.
Section snippets
Related works
A major coordinator of grid interoperation efforts is the GIN Community Group of OGF [23]. The GIN is carrying out several demonstrations and investigates current interoperation of middleware solutions in its 4 identified areas [24], [25]. In conjunction with the GIN activities several research projects are investigating the middleware level interoperation of current production Grids. The OMII-UK (Open Middleware and Infrastructure Institute) [26] team at Southampton has investigated the
Supporting features and techniques in P-GRADE portal for job submission
P-GRADE Portal is a set of high level middleware services and Web portlets running in Gridsphere [5] portal framework. The job submission interoperability is supported by several functions in P-GRADE portal:
- •
Grid setting management
- •
Certificate management
- •
Workflow definition and management
- •
Information system access
A Grid is typically identified by its information system and hence when the P-GRADE portal is installed, the portal administrator can connect the portal to several Grids defining their
Extending the P-GRADE portal towards service-oriented grids by GEMLCA
GEMLCA (Grid Execution Management for Legacy Code Applications) represents a general architecture for deploying legacy applications as Grid services without re-engineering the code or even requiring access to the source files. GEMLCA adds an additional layer on top of a service-oriented Grid middleware, like Globus Toolkit version 4 (GT4). The GEMLCA layer is responsible to hide the legacy nature of the application. It communicates with the client through SOAP-XML messages, gets input parameter
Connecting Grid Generations and Technologies with the P-GRADE Portal and GEMLCA
Sections 2 Related works, 3 Supporting features and techniques in P-GRADE portal for job submission described those features of the P-GRADE portal and GEMLCA that enable the submission of workflows into multiple Grids based on different underlying technologies and middleware. Here we present those experiments and demonstrations that were specifically designed to illustrate how the previously described features can be combined in order to make different Grid solutions interoperable at the
Conclusion and further work
We have shown in the paper that P-GRADE portal extended with GEMLCA technology can serve as a bridge between different production Grids that are based on different Grid middleware technologies. The obvious advantages of such a bridge are the following:
- 1.
End users can access any of these Grids from their workflow.
- 2.
Virtualization, resource sharing and collaboration can be realized through the boundaries of different production Grids.
- 3.
Porting the applications between production Grids does not require
Acknowledgements
The research work presented in the paper was carried out under the FP6 Network of Excellence CoreGRID funded by the European Commission (Contract IST-2002-004265) and under the SEE-GRID-2 project funded by the European Commission (Contract number: 031775). This work makes use of results produced by the Enabling Grids for E-sciencE project, a project co-funded by the European Commission (under contract number INFSO-RI-031688) through the Sixth Framework Programme. EGEE brings together 91
Peter Kacsuk is the Head of the Laboratory of the Parallel and Distributed Systems in the Computer and Automation Research Institute of the Hungarian Academy of Sciences. He received his M.Sc. and university doctorate degrees from the Technical University of Budapest in 1976 and 1984, respectively. He received the kandidat degree from the Hungarian Academy in 1989. He habilitated at the University of Vienna in 1997. He received his professor title from the Hungarian President in 1999 and the
References (44)
- et al.
Grid resource brokering algorithms enabling advance reservations and resource selection based on performance predictions
Future Generation Computer Systems
(2008) - The GRIP project web site,...
- Globus Team, Globus Toolkit 4.0 Release Manuals,...
- The gLite website,...
- et al.
UNICORE: uniform access to supercomputing as an element of electronic commerce
Future Generation Computer Systems
(1999) - J. Novotny, M. Russell, O. Wehrens, GridSphere, An Advanced Portal Framework, in: Conf. Proc. of the 30th EUROMICRO...
- Condor Team, University of Wisconsin-Madison: Condor Version 6.4.7 Manual,...
- The NorduGrid web page,...
- et al.
Multi-Grid, multi-user workflows in the P-GRADE Grid Portal
Journal of Grid Computing
(2005) - et al.
GEMLCA: Running legacy code applications as grid services
Journal of Grid Computing
(2005)
A modular meta-scheduling architecture for interfacing with pre-WS and WS Grid resource management services
Future Generation Computer Systems
Cited by (33)
Scientific workflows: Past, present and future
2017, Future Generation Computer SystemsRapid development of scalable scientific software using a process oriented approach
2011, Journal of Computational ScienceCitation Excerpt :Applications created with this tool are compiled to MPI or PVM and can deal with grid systems. A later project named P-GRADE portal [6] presents a platform for handling dependencies between grid jobs and enables a single application to use multiple grid middleware architectures. The P-GRADE portal has left the CSP style programming and is now using directed acyclic graphs for workflow programming.
Interoperability of BOINC and EGEE
2010, Future Generation Computer SystemsJobresource matchmaking on Grid through two-level benchmarking
2010, Future Generation Computer SystemsCitation Excerpt :Another approach to supply some kind of interoperability is provided by Grid portals, which operate above the middleware and hide some middleware specificity to Grid users. P-Grade [64] is a portal providing workflow management features which leverages on the brokering functions of GTbroker and LCG2 brokers, allowing users to develop and execute multigrid workflows. Though a GUI aids users to express job descriptions and requirements on resources, without the need of tackling JSL notions, the system is not completely transparent to users that have to know in advance the broker (hence the JSL) which manages the submission.
Federation of TeraGrid, EGEE and OSG infrastructures through a metascheduler
2010, Future Generation Computer SystemsCitation Excerpt :This is exactly the aim of the Grid Interoperation Now [5] (GIN) group of the OGF, in which some of the projects mentioned from now onwards participate. Following this interoperation philosophy, we can note efforts to interoperate different grid middleware (for example, Globus and UNICORE [6]) and approaches based on portals and workflows [7]. In this line, it is also interesting to see efforts like InterGrid [8], which propose the creation of InterGrid Gateways (IGGs) to interconnect the existing different grid islands, or GridX1 [9], which can be accessed as another resource of LCG.
GMBS: A new middleware service for making grids interoperable
2010, Future Generation Computer SystemsCitation Excerpt :The Security and Accounting mechanisms are not implemented in the GMBS, either. The security solution applied in this service is the same as described in [18] for the multi-grid P-GRADE portal and hence we do not give more details here on the security issues. There are two scenarios how to use GMBS:
Peter Kacsuk is the Head of the Laboratory of the Parallel and Distributed Systems in the Computer and Automation Research Institute of the Hungarian Academy of Sciences. He received his M.Sc. and university doctorate degrees from the Technical University of Budapest in 1976 and 1984, respectively. He received the kandidat degree from the Hungarian Academy in 1989. He habilitated at the University of Vienna in 1997. He received his professor title from the Hungarian President in 1999 and the Doctor of Academy degree (D.Sc.) from the Hungarian Academy of Sciences in 2001. He has been a part-time full professor at the Cavendish School of Computer Science of the University of Westminster and the Eötvös Lóránd University of Science since 2001. He served as visiting scientist or professor several times at various universities of Australia, Austria, Canada, England, Germany, Spain, Japan and USA. He has published two books, two lecture notes and more than 200 scientific papers on parallel and Grid computing. He is co-editor-in-chief of the Journal of Grid Computing published by Springer.
Tamas Kiss is a Senior Lecturer in Database Systems at the Department of Information Systems and Computing, and a researcher at the Centre for Parallel Computing at the School of Informatics, University of Westminster, London. He led the design and development activities resulting in the Grid Execution Management for Legacy Code Architecture (GEMLCA) solution within the UK EPSRC founded OGSA Testbed project. He contributes to the CoreGrid Network of Excellence project where he leads the Legacy Code Wrapping and Deployment Methodologies Research Group within the Institute on Grid Systems, Tools and Environments. He also co-ordinates University of Westminster research activities in the framework of the Westfocus Grid Alliance Project that aims to applying the benefits of Grid computing to real-world industry applications. He has extended experience in teaching in higher education and giving Grid tutorials, lectures and hands-on sessions (e.g. GEMLCA/P-GRADE portal courses organised by the UK National e-Science Center (NESC) and EGEE). He co-authored one book and more than 40 scientific papers in journals and conference proceedings and as book chapters.
Gergely Sipos is a research fellow at MTA SZTAKI (Computer and Automation Research Institute of the Hungarian Academy of Sciences). He received his M.Sc. degree in Information Engineering from the University of Miskolc, Hungary in 2003. In the same year he joined the Laboratory of Parallel and Distributed Systems of MTA SZTAKI and started PhD studies on grid computing at the University of Miskolc. His research interest includes service oriented computing, grid workflows, high level grid programming and collaborative environments. He has published several book chapters, journal papers and over 30 conference papers on these topics. Since 2005 Gergely Sipos performs research on grid portals, legacy code support and grid workflow management and actively contributes to the training activities of several international grid projects.