Evaluation of global online training activities of the International Training Centre of the International Labour Organization (ITCILO) in 2020

: This paper reports on an evaluation of the online training activities of the International Training Centre of the International Labour Organization (ITCILO) in Turin (Italy). The purpose of this evaluation is to assess the design, implementation and quality of training activities of ITCILO that were delivered in an online distance learning mode since the outbreak of COVID-19 in 2020. The methodology for this evaluation included quantitative and qualitative evaluation methods to provide conclusions and recommendations from the findings, substantiated with statistical data and case studies documenting good practice. 1.284 responses from 151 different countries were collected from a participant's survey, and in-depth interviews were conducted with ITCILO's staff members (27), institutional clients (2), and participants (7). The results of the evaluation are reported in terms of training outreach, learning effectiveness, learner preference of the international online training activities. For example, the number of enrolments on the Centre’s training activities increased dramatically, caused by the rapid adoption of online training activities during the Covid-19 Pandemic. Subsequently, 75 % of the participants said they would prefer digital training activities (blended or fully online) in the future. However, internet connectivity is still a problem in many countries: 50 % of participants from Africa, Asia, the Middle East, and Oceania reported they had regular issues with internet connectivity that disrupted their learning. Based on the findings, ten recommendations for the further development of ITCILO's online training activities are presented.


Introduction
The International Training Centre of the International Labour Organization (ITCILO) is the training arm of the ILO, the Specialized Agency of the United Nations which promotes social justice and human rights in the world of work. The Centre delivers face-to-face and online training activities for individual learners plus a range of capacity development services for institutional beneficiaries (including management consultancies, event management services, training product development, knowledge management services and design of communication and advocacy campaigns), in support of decent work and sustainable development.
The ITCILO has seen a massive shift towards fully online distance learning (ODL) in 2020 during the COVID-19 Pandemic. While ODL training accounted for only 4% of ITCILO's activities in 2019, it grew to up to 48 % by the end of 2021. This paper reports on the evaluation of ODL in the context of International capacity development training.The purpose of the evaluation was to provide a comprehensive account of the effectiveness of such an accelerated shift and draw useful insights and recommendations to the Centre (and, more broadly, adult training contexts) to make better use of the online medium in the future. The focus of this evaluation was not placed on the evaluation of single courses or content areas but the overall management, design and impact of online training activities. Given the significant role that the Centre plays in the field, it is worthwhile to share the evaluation results with other relevant stakeholders outside the Centre. Also, compared to a large volume of research outcomes drawn from the evaluation of emergency remote teaching activities in other educational contexts such as universities and schools, there is an even more pressing need to share and reflect on our findings with those who may benefit from it.

Theoretical and Methodological Framework
The evaluation was undertaken using a mixed-methods approach and a theoretical foundation drawn from the well-developed field of distance education as follows:

Desk-based review of institutional data
In a first step, a set of selected institutional documents (e.g., Strategic Plan of the ITCILO for 2018-21, Programme and budget proposals of the ITCILO for 2020-21, Interim implementation report 2020) were reviewed in order to establish a solid contextual understanding of the Centre and its online training activities. The design of 20 selected training activities delivered online in 2020 were examined. The 20 courses were chosen based on their representativeness of the topic, delivery mode (i.e., stand-alone webinars, communities of practice, virtual reality, tutor-based, self-guided distance learning), languages (English, Spanish, French) and cost (i.e., free, tailor-made and sponsored, fee-based open courses). A range of available course data (e.g., course evaluation results, knowledge acquisition test results) was reviewed.

Quantitative data collection and data analysis
An online participant survey was sent to 9,053 individuals enrolled in one or more of the 20 online courses. 956 surveys in English,240 in Spanish,and 88 in French (N = 1,284) were collected between 10 th June and 9 th July 2021, resulting in an overall response rate of 14 %. About two-thirds of respondents were male (66.3 %), and onethird were female (32.3 %). Two participants indicated diverse (0.2 %), and 26 (2.0 %) did not reveal their gender. Enrolments are reported for all the 20 courses.
The survey comprised five sections. Participants' demographics were collected in section A. In section B, the validity of the training design to support a meaningful online learning experience was evaluated using the Community of Inquiry (CoI) framework developed by Garrison, Anderson, and Archer (2000), which is an empirically tested and widely used model to examine the educational experience in online distance learning. An educational CoI is a group of individuals who collaboratively engage in purposeful critical discourse and reflection to construct personal meaning and confirm mutual understanding. This process of creating deep and meaningful learning is facilitated through three interdependent elements, i.e. teaching presence, social presence, and cognitive presence. The questions in section B of the survey were based on a self-rating instrument was developed by Teng, Chen, and Leo (2012).
Decades of experience in distance education have shown that learner support is the critical link to avoid drop-out and failure in distance learning courses (see Zawacki-Richter & Naidu, 2016). Models of learner support distinguish between pedagogical support of learning and teaching and technical (or administrative) support. The dimension of learning support was covered by the items based on the CoI framework in section C of the survey. The technical support dimension was addressed by adopting Lee et al.'s (2011) survey on students' perception of support and course satisfaction. Section D focused on the different delivery modes in online distance learning. The different modes of training delivery have a significant impact on the constituent elements of the so-called "Golden Triangle" of the provision of distance learning opportunities, i.e. access, quality, and costs (see Guri-Rosenblit, 2014). Finding a balance between these three elements paves the way for training providers to reach and serve their target groups by widening access and scaling up distance training activities.
The digital media and tools used for synchronous and asynchronous interaction play an important role. For example, the integration of synchronous videoconferencing sessions on a regular basis throughout a course helps to avoid a feeling of isolation and to build a sense of community among the course participants as well as between the instructor and the learners. Since learning is a social exercise, interaction among course participants and personal support by the instructor is a clear indicator for high quality distance learning. However, synchronous meetings reduce the flexibility and independence from time and space, and thereby access for those who are unable to attend at a certain time. On the other hand, online interaction (synchronously or asynchronously) has to be facilitated and guided by an instructor or tutor, which raises the costs of the training activity and limits opportunities for economies of scale. Thus, the participant's perceived demands for the different modes of delivery as well as for synchronous and asynchronous interaction were explored. Finally, the training activities outcomes and application to the work context and overall course satisfaction were evaluated in section E of the survey.

Qualitative data collection and analysis
As a means of triangulation, a series of interviews were conducted to support and confirm the results of quantitative evaluation. Semi-structured interviews were conducted with 27 staff members at the Centre to draw insiders' perspectives. Interviewees shared their opinions about the quality of different online training activities concerning the Centre's strategic plans as well as individual learners' experiences. In addition, two semi-structured interviews with institutional clients of the Centre and two focus group discussions with formal training participants The Tenth Pan-Commonwealth Forum on Open Learning (PCF10) September 14-16, 2022, Calgary, Canada.
(n=7) were conducted. All interviews held on Zoom were recorded and transcribed. A thematic analysis, recording both similarities and differences among collected perspectives and opinions, was conducted to draw informative and accurate evaluation outcomes.

Distribution of participants
According to ITCILO's Interim Implementation Report 2020, the number of enrolments on the Centre's training activities increased dramatically, caused by the impact of the COVID-19 Pandemic. The number of distance learners grew from 9,800 in 2019 to 52,000 in 2020, compensating for the sharp decrease in face-to-face participants from 10,700 to 1,800. With online training, a wider and more diversified audience can be reached. Especially, participants from middle-income countries can take advantage of digital learning solutions avoiding costs for travel and accommodation. As shown below, more than 50 % of all online participants come from Asian and African countries.

Access to technology and tools
The participants of the 20 selected courses seem to be well equipped with technical devices and tools to access ITCILO's online courses: 90.4 % agreed or strongly agreed to the statement that they had full access to the technology and tools required to participate in online learning (M = 4.18 on the 5-point scale).
92.4 % reported being able to freely choose and use different devices (PCs, laptops, mobile phones, tablets) to pursue online learning. However, some mentioned difficulties in accessing the online courses via mobile devices due to the design not being suitable for small screens or mobile Internet connectivity.

Internet connectivity
Nevertheless, the survey responses clearly show that Internet connectivity is an issue. The quality and reliability of Internet connectivity vary across countries and regions. About 50 % of participants in Africa, Asia, the Middle East, and Oceania agreed or strongly agreed that they had regular issues with Internet connectivity that disrupted their learning (see Figure 2). Interestingly, the Internet connections in Latin America seem to be better with only 27 % of participants, who had regular problems with the Internet. Not surprisingly, the best Internet connectivity is available in Europe and North America.
In areas with low bandwidth and unstable connections, asynchronous communication is preferable because participants can log-in, communicate and download learning material at a convenient time when the Internet is available. In contrast, synchronous video-conferencing (e.g., in webinars) require much more bandwidth and a stable connection. In many cases, participants have to turn off their video to be able to join the conversation.

Course satisfaction and knowledge application
An impressive 98.3 % of sampled participants would recommend the training to their colleagues. 83.0 % reported back that the online training as a whole was good or very good (M = 4.2, n = 943), and 77.6 % said that the effectiveness of the online training format was good or very good on a 5-point scale (M = 4.1, n = 943). The learner survey results (and course evaluation results) suggest that many have found their engagement with the activities beneficial, contributing to their professional practice and development. All mean scores in Table 1 are well above four, and the vast majority of participants agreed or strongly agreed that the course was relevant (95.4 %), translated theory into practice (92.3 %) and that they can apply what they learnt in their work setting (94.3 %). When they were asked to provide concrete examples of knowledge application (i.e., "Can you give a concrete example on the way in which the course itself has been of practical use for achieving results in your work?"), 48.6 % (n = 524) of the respondents offered examples that include: • I developed a gender policy for my workplace.
• I have changed all the fire management system on my current job.
• I am introducing a program called RIDE Recreational Areas Improvement Drive based on ILO's Participatory Approach on tackling safety issues at work.
Such knowledge transfer seems to be more evident in tutor-guided courses that also provide opportunities for interaction among participants who can bring in their experiences from their work context. Excluding the two self-guided distance learning courses increases the measure to 54.6 % (n = 361). Although this figure is still much lower than the baseline of face-to-face training of 2018-19 (71.4%), given the challenging circumstances of most workplaces during the Pandemic, the result is still positive.
The participants reported that they made large or very large improvements in terms of their competencies (85.6 %) and job performance (69.0 %) as a result of the training activities (see Table 2). Although the available data is minimal, the Knowledge Acquisition (KA) results of the three training activities also reassure the positive progress made by learners. For example, the KA result of one of the selected training activities shows the meaningful difference between the pre-test result (average 4.77) and the post-test results (average 7.60). It is important to note that not all immediate training objectives can be quantitatively measured by the KA tests or even by student satisfaction responses. Nevertheless, these figures certainly suggest the effectiveness of the reviewed online training activities.

Teaching, social, and cognitive presence in online courses
To evaluate the learning experience in ITCILO's online training activities, the participants' perceived sense of teaching presence, social presence, and cognitive presence were measured using a 5-point scale (1 = strongly disagree, 5 = strongly agree, and not applicable). The scales are not applicable to self-guided distance learning courses, where participants only interact with the presented learning material, but not with a tutor, training facilitators or other course participants. Thus, 291 surveys were collected.
The results indicate that ITCILO's course designers and facilitators managed to deliver highly engaging, interactive, and supportive online courses that provided opportunities for rich and deep learning experiences, with average ratings of the teaching presence of 4.52, the social presence of 4.32, and the cognitive presence of 4.35. Especially the course tutors and facilitators are to be commended for their proactive and clear communication and guidance right from the beginning of the COVID-19 Pandemic. Furthermore, the findings underline the importance of social interaction for deep learning with a strong positive correlation between social and cognitive presence (rs = .82, p < .001).

Technical issues and support
However, many respondents faced technical issues in participating in the online training activities: Overall, 35.5 % agreed or strongly agreed that they had many technical problems in their courses. Major problems were reported by participants from the Middle East (46.3 % agreed or strongly agreed), followed by Asian (39.5 %) and African countries (37.5 %).
The Tenth Pan-Commonwealth Forum on Open Learning (PCF10) September 14-16, 2022, Calgary, Canada. 3 0 0 0 Note: 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree Given these challenges, it is very important to provide technical support. On a scale from one to five, the mean scores for the items related to technical support and guidance are both below four. Thus, there might be room for improvement regarding the information for participants where to find help and the response time of technical support.

Demand for online training
Only one quarter (n = 234; 24.8 %) of the respondents said that they would prefer to go back to fully face-to-face training again; the majority (n = 364; 39.6 %) wants blended learning courses, closely followed by participants (n = 337; 35.7 %) who would even prefer fully online distance learning courses (see Figure 3). Thus, three quarters of the participants would prefer a digital format of some kind in the future. However, looking at different countries and regions reveals a more nuanced picture. Only in Oceanian countries, the majority of participants would prefer face-to-face training in the future. In Africa, Latin America, and Europe, most participants want blended learning courses, and in Asia and the Middle East, fully online distance courses would be the first choice. These differences might be associated with issues related to technical infrastructure, Internet connectivity, and access to digital devices.

Asynchronous vs. synchronous communication
Participants were also asked if asynchronous computer-conferencing (e.g., communication via a discussion forum), asynchronous video content (e.g. a recorded guest lecture or video presentation) as compared to synchronous video-conferencing (e.g., a webinar via Zoom) were used too often (1), just enough (2), or not often enough (3).

Learner suggestions
The participants gave recommendations to improve the quality of online training activities, which can be categorized into i) managing workload (e.g., "I thought the material and assignments were bulky compared to the allocated time" and "The course load was too much to take on a regular working day. I had difficulties catching up with my workload. Study leave could be given equal to the number of hours that the course requires."), ii) increasing practicality (e.g., "Have more tutorials and video simulations in the learning materials" and "Increase time or follow-up course which is hands-on/practical in nature with support from experts."), and iii) increasing accessibility (e.g., "I was not able to participate in any of the live webinars due to the time difference. All webinars were set at 3:00 a.m. my time. Those differences should be taken into consideration when defining the schedules for the webinars", Online learning is great. I suggest the price will be reduced to help many, especially those within the remote parts of Africa, get access", and "Make platform more easy and accessible).

Conclusions and Recommendations
During the Pandemic, the Centre reached a wider and more diversified audience with ODL activities. Especially, participants from middle-income countries could take advantage of digital learning solutions avoiding costs for travel and accommodation. Participants from 151 different countries responded to the participant's survey, and most of them indicated that they had adequate access to technology to participate in online training. However, internet connectivity is still a problem in many countries. 50 % of participants from Africa, Asia, the Middle East, and Oceania reported they had regular issues with internet connectivity that disrupted their learning.
In spite of sudden urgency of adopting a fully online training format , the online training activities clearly strengthened the capacity of ILO constituents and other ILO development partners. 98.3 % responded that they would recommend the training activities to their colleagues. Participants perceived courses that provide tutorguided opportunities to use new skills in their work settings and share their experiences with other participants more effectively. However, it is important to note that 35.5 % faced a range of technical issues, requesting more support. 75% of the participants said they would prefer digital training activities (blended or fully online) to faceto-face ones in the future. Participants tend to slightly prefer asynchronous content presentation and communication that allows for higher levels of flexibility and accessibility.
There are some critical recommendations drawn from these findings. Firstly, the Centre needs to develop a systematic course design framework and an effective operational model, taking into account the full spectrum of target groups, content areas, technological tools, pedagogical methods. It is essential to manage both learners' and facilitators' workload in online settings as participants (and facilitators) often juggle different professional and personal responsibilities. Secondly, the Centre focus more on tutor-based distance learning that facilitates interaction between tutors and learners as well as among participants and provides more practical learning opportunities.
Most importantly, despite the increased outreach, there is still room for further improvement of "authentic" accessibility in a multi-dimensional sense. Temporal accessibility can be improved by accommodating time differences among distance learners. Technological accessibility, influenced by each learners' Internet connectivity and technical ability, can be improved by making the right balance between synchronous and asynchronous online learning (with alternatives available) and improving its technical support provisions to help participants smoothly join and navigate their online courses.