E-learning programs in oncology: a nationwide experience from 2005 to 2014

E-learning is an established concept in oncological education and training. However, there seems to be a scarcity of long-term assessments of E-learning programs in oncology vis-á-vis their structural management and didactic value. This study presents descriptive, nationwide data from 2005 to 2014. E-learning oncology programs in chemotherapy, general oncology, pain management, palliative care, psycho-social-oncology, and radiotherapy, were reviewed from our databases. Questionnaires of self-perceived didactic value of the programs were examined 2008–2014. The total number of trainees were 4693, allocated to 3889 individuals. The trainees included medical doctors (MDs; n = 759), registered nurses (RNs; n = 2359), radiation therapy technologists (n = 642), and, social and health care assistants (SHCAs; n = 933). The E-learning covered 29 different program classifications, comprising 731 recorded presentations, and covering 438 themes. A total of 490 programs were completed by the trainees. The European Credit Transfer and Accumulation System (ECTS; 1 ECTS point equals 0.60 US College Credit Hours) points varied across the educational programs from 0.7 to 30.0, corresponding to a duration of full-time studies ranging between 15 to 900 h (0.4–24 weeks) per program. The total number of ECTS points for the trainee cohort, was 20,000 corresponding to 530,000 full-time academic hours or 324.0 standard academic working years. The overall drop-out rate, across professions and programs, was 10.6% (499/4693). The lowest drop-out rate was seen for RNs (4.3%; P < 0.0001). Self-reported evaluation questionnaires (2008–2014) were completed by 72.1% (2642/3666) of the trainees. The programs were overall rated, on a 5-categorical scale (5 = excellent; 1 = very inferior), as excellent by 68.6% (MDs: 64.9%; RNs: 66.8%; SHCAs: 77.7%) and as good by 30.6% (MDs: 34.5%; RNs: 32.4%; SHCAs: 21.5%) of the responders. This descriptive study, performed in a lengthy timeframe, presents high-volume data from multi-professional, oncological E-learning programs. While the E-learning paradigm, across professions, seems to have been well received, it is imperative that prospective studies, benchmarking against traditional training methods, are carried out, examining the hypothesized didactic value of our E-programs.


Background
E-learning, also called computer-based learning, online learning or web-based learning, is a ubiquitously used technology in higher education [1][2][3]. E-learning comprises internet-based, interactive and asynchronous teaching and learning tools. A number of open universities, some even 'mega-universities' with more than 100,000 students, have adopted these as standard education techniques. US National Center for Educational Statistics, in the 2011 report, states that 30% of all students with bachelor's degrees (n = 860,000) were enrolled in distance education courses with 75% of these taking their entire postgraduate program online [4,5]. Probably, more than 80% of U.S. doctoral/research institutions have some form of online offering, either courses or full programs [6].
A number of advantages of E-learning programs, visá-vis traditional education programs, have been hypothesized: uncoupling of education from time and place, standardization of instruction and assessment, ease of documentation of learner behavior, student control of the education experience and increased educational costeffectiveness [7]. A recent meta-analysis [8] indicates that no-intervention studies of internet-based learning, using pre-and post-tests, demonstrated augmented effectiveness in relation to acquisition of knowledge (factual or conceptual understanding) and skills. However, in studies comparing internet-based learning with non-internetbased traditional methods there only seems to be a small educational benefit from internet-based learning [8,9]: a likely explanation is the large heterogeneity and variance in the data, characterizing the studies.
Oncology is indeed a challenging subject matter not only for any trainee, but also for the educator [2]. The E-learning programs may overcome some of the difficulties seen with traditional learning programs by allowing flexibility in time, place, and pace, for the clinically working trainee and educator [10]. There is a paucity of detailed, descriptive long-term data regarding management and outcome of E-learning programs. This study presents outcome data from our E-learning programs in oncology: chemotherapy; general oncology; pain management; palliative care; psycho-social-oncology; and radiotherapy, covering a span of 10 years, and including various health-care professions: medical doctors; nurses; radiation therapy technologists; and social and health care assistants.

Ethics
The E-programs and the associated data-bases complied with Swedish laws and regulations stipulated in the Personal Data Act (1998:204), aiming to prevent the violation of personal integrity in the processing of personal data [11]. Since the study was retrospective and registrybased, an application to the ethical committee was not considered necessary.

Organization behind the E-learning programs
The organization behind the E-learning programs (E-programs) was established in 2002 by the authors (JD, SS), affiliated to the Department of Oncology, Clinical Sciences, Lund University. Initially, the E-programs were made and driven by local specialists: oncologists, radiation therapy technologists, medical physicists and anesthesiologists. After some years of benchmarking, both regional and national teaching resources were successively recruited into the E-programs. The supplier of the E-programs was the Department of Oncology, University Hospital of the Southern Region, in collaboration with the Medical Faculty at Lund University.

Commissioning parties
The primary commissioning parties were local (n = 17), or regional, university-based (n = 7), oncological departments in Sweden.

Financial aspects
The trainees' tuition fees were reimbursed by their affiliated oncological departments, in turn recompensed by the respective County Council [Sweden comprises 21 County Councils (across six Health Care Regions) responsible for financing and providing health care managed by the Ministry of Health and Social Affairs]. In 7 out of 20 E-programs, co-financing with the Swedish Medical Association, the National Board of Health and Welfare, the Association of County Councils, and the National Agency for Higher Vocational Education, were agreed upon.

Structure of the education
The structure included four hierarchical levels: Educational Fields, Educational Sub-Fields, Programs, and Modules (Fig. 1). The Educational Fields covered Chemotherapy, Oncology, Radiotherapy and Symptom Therapy. The Educational Sub-Fields covered for the Educational Fields, Oncology: Basic Oncology, Specialized Oncology; Radiotherapy: Basic Radiotherapy, Image Guided Radiotherapy; and for Symptom Therapy: Palliative Care; Pain Management; Psychosocial Oncology. The Educational Field, Chemotherapy, however, did not accommodate any Educational Sub-Field (Fig. 1).
The Programs, included a categorization of topics based on trainees' profession, e.g., Basic Oncology for social and health care assistants, Radiotherapy for registered nurses, and, Radiotherapy for medical doctors. The trainees' professions and required proficiency levels across Educational Sub-Fields, and the duration of individual E-programs, are presented in Table 1. A detailed description of an E-program, representative in Radiotherapy for medical residents in oncology is illustrated in Table 2. Illustrations of E-programs from each of the four Educational Fields are presented in Table 3. Finally, Modules constitute the basic educational building blocks, usually aggregated in a number of themes (Table 2).

Professions
Health care professions included were, registered nurses (RNs), medical doctors (MDs), social and health care assistants (SHCAs; 7% of SHCAs belonged to administrative personnel), and radiation therapy technologists (RTTs). The trainees' proficiencies were classified into basic, specialist-in-training or specialist levels (Table 1).

Recruitment
Three different recruitment paths were utilized. Most commonly an educational contract between the trainee's oncological department, mainly university-based departments, and the supplier of the E-program, was established, stipulating the educational requirements. Recruitment by advertisements in journals, flyers or web-sites for health care professionals were also utilized. A substantial number of trainees were also recruited by word-of-mouth and selected from waiting lists. Trainees  were affiliated to anesthesiological, medical, oncological or surgical departments.

Tutors and producers Professions
Tutors were senior ranking clinical oncologists and anesthesiologists, many with an academic affiliation. Producers of the educational material included specialists from various disciplines: advanced practice registered nurses, medical doctors, physicists and psychologists, almost all with senior clinical background in addition to an academic affiliation.

Basic structure of the module Design
The design of the Module was based on theory-derived principles of educational practice, following the recommendations on web-based learning by Cook and Dupras [12]. The goals and objectives of each Module were pre-specified and presented at an early stage to the trainee, the IT-platform was successively tailored to the needs of the trainees and the tutors, appropriate multimedia and hyperlinks were used and an active learning-approach was encouraged (self-assessment, reflection, self-directed learning, problem-based learning, learner interaction, and feedback). The IT-platform underwent substantial changes and improvements during 2005-2014, reflecting the general technical progress in the IT-field. To address the need for real-life, clinical problems, case-based training scenarios were incorporated into the Modules.

Educational material
The specialists either by themselves, depending on their IT-proficiency, or, in collaboration with the IT-technical staff (SS, JD) produced lectures, usually in the format of multimedia presentations (PowerPoint with voiceover speech). Since 2006 a web-based "authors' tool box" facilitating the production has been available for import of images and audio-files. The Module is based on one or more presentations (Table 2), Additional file 1 (e.g., scientific literature, guidelines, images, animations), links to relevant web-sites, web-based tests intended to establish performance status (MCQ, short essays) and the trainees' web-based self-evaluation of the training quality of the Module. In addition conventional or online education materials like textbook chapters sometimes were employed by the tutors. From 2005 to 2011 the educational material was CD-based, but thereafter online login procedures were employed.

Educational objectives
Where a core curriculum stipulated by national consensus and based on national guidelines, were available, this was applied for each respective Educational Field, e.g., Radiotherapy, and correspondingly for each hierarchical level, i.e., Educational Sub-Field, Program and Module.

E-based interactivity
Interactivity was considered essential in encouraging the trainee's active learning process [12], including selfassessments, self-directed learning often based on problem-solving issues and interaction with the tutor. The trainee's activity on the technical platform was logged, and information on latest logins and lectures viewed, was available for the tutors. In order to augment self-assessment and self-directed learning a procedure called selfevaluation was used in essay exercises. After submission of the essay the trainee automatically received a complete essay report pre-fabricated by the tutor, and was then asked to re-submit the report after considering necessary changes from the original response, at the discretion of the trainee. The tutor then reviewed the re-submission and gave an individualized feedback on the essay report to the trainee. This simple self-evaluation formative measure seemed to increase the didactical value both for the trainee and the tutor, in addition to decreasing the workload of the tutor. Further, it also helped to pin down the learning objectives, facilitating the awareness of the trainee that the core goals had been achieved. But most importantly, it gave the trainee a learning opportunity and helped to the tutor's recognition of didactic misunderstandings. E-based interactivity between fellow-trainees was based on asynchronous fora.

Physical meetings
During the first years of the E-programs a requirement for physical meetings, vis-á-vis virtual meetings, became apparent, and, we, empirically incorporated 2-3 compulsory meetings of 1½-days duration for every 4-6 months of participation in the E-program. Furthermore, it was required that the trainees completed all modules and seminars in order to get approval of the course. The rationales behind these meetings were based on practical, didactic and social aspects. First, when the E-programs were initiated 2003-2005, conventional learning methods, were still considered "the gold standard", at least across the targeted professions and the age groups, and therefore a noticeable demand for physical meeting existed. Second, although the general acceptance of E-learning methods has increased dramatically during the last decades [13], it has been our experience that the physical gatherings are still justified since the trainees' active learning process is stimulated. Third, needless to say, it consolidates the social networking, important since trainees often may live at considerable distance, sometimes up to thousand miles from each other. Fourth, a number of studies seem to indicate that the drop-out rates in E-programs decrease by use of physical meetings.

Technical aspects
An extended E-support for trainees and tutors was instituted from the beginning of the E-programs, and was quickly considered a prerequisite for successful implementation. A number of the trainees the first years generally demonstrated a lack of prowess and routine in IT-issues, requiring basic support beyond the supplied instruction manuals. During 2005-2011 the bulk of the programs, as previously mentioned, were CD-based requiring installation routines sometimes associated with technical incompatibility problems across systems and drivers. An IT-engineer (SS) and a highly qualified technician (JD) were at all times available, particularly important, during server-malfunctioning problems. The maintenance, development and improvements of the platform were by the engineer and an IT-assistant.

Learning objectives
After the completion of each Module the trainee evaluated how well the training had achieved its objective by a number of standardized questions including simple categorical rating scales. At completion of the program the trainee was also asked to rate qualitative aspects of the education material. These evaluations were continuously used by the tutors as an important feed-back mechanism on the didactic quality of the E-programs.

Other
Drop-out rates, across professions and programs, were calculated from our database as:

Data collection
Data were collected from our central database, from January 1, 2005 to December 31, 2014, containing information on participants' initials, profession, zip-code, program affiliation (entry and completion date), tutor affiliation and response to questionnaire on learning objectives. In addition, information regarding the tutors' and producers' profession, academic education and program affiliation was retrieved from the database. A technical database was accessed analyzing the completed number of Programs and Modules, and, the technical structure of the Modules in regard to number of frames, the duration of the oral presentation, the cumulated frame time for the available lectures and use of video sequences.

Educational structure
An overview of the educational structure is presented in Table 1  The 29 different Program classifications, contained a total of 731 Modules covering 438 themes (Fig. 1). The trainees completed a total of 490 Programs. Each Module presentation contained in median 18 frames (IQR 11;26) with an oral presentation time of each frame of 0.68 min (IQR 0.38; 1.12), corresponding to a total time for each presentation of 12 min (7.48; 17.68). The cumulated frame time for all available lectures was 10,605 min (177 h). Video presentations were available in 30 Modules with a total number of video sequences of 64.

Demographics Trainees
The annual number of trainees, across professions, from 2005 to 2014, are presented in Fig. 2. A total of 4693 trainees completed the Programs, while the total number of individuals were 3889, since 20.7% of the trainees participated in more than one Program. The trainees' geographical distribution across the six Swedish health care regions, is illustrated in Fig. 3. The bimodal relationship between number of trainees (n = 3926) and duration of courses (ECTS-points) is indicated in Fig. 4. The percentage of trainees attending courses corresponding to ≤4 ECTS-points was 61.9%, and to ≥7.5 ECTS-points 38.1%. The total number of ECTS-points for the trainee-cohort was 19,438, corresponding to 534,545 full-time academic hours, equaling 324.0 standard working years (Joint Costing and Pricing Steering Group for Higher Education Institutions, U.K.) [14]. The distribution of professions of the trainees, across Educational Fields, is illustrated in Table 4.

Self-reported evaluation of learning objectives
The Programs were, across professions, overall rated as excellent by 68.6% and as good by 30.6% of the responders. SHCAs demonstrated significantly higher overall ratings of the Programs than RNs and MDs (Tables 6, 7).

B. Will you be able to use what you've learnt in everyday clinical practice?
The clinical applicability of the Programs was rated as "to a very high degree" of 45.2% and "to a high degree" of 49.2% of the responders. MDs demonstrated significantly lower ratings compared to RNs, while no differences compared to SHCAs were seen (Tables 6, 7).
C. Would you recommend the program to a colleague in a similar situation as yours? The recommendability of the Programs was rated as "to a very high degree" of 69.5% and "to a high degree" of 26.6% of the responders. Interestingly, RNs and SHCAs rated the recommendation value of the programs significantly higher than the MDs (Tables 6, 7).

D. How did you experience the workload during the program?
The workload was experienced as "appropriate" to "very low" by 62.4% and as "very high" to "high" by 37.6%. Interestingly, the workload during the programs was perceived as relatively higher by the RNs and SHCAs, compared to MDs (Tables 6, 7).
E. Evaluate the importance of the recommended literature in the program. F. Evaluate the importance of the recommended scientific publications in the program.
The importance of the recommended literature and the scientific publications (e.g., clinical studies, chapters from textbooks) was considered of "very high importance" to "high importance" by 44.8%, and of "moderate importance" by 25.2% of the responders. Both RNs and SHCAs evaluated the relative importance of the recommended literature significantly higher than the MDs (Tables 6, 7). In contrast, there were no statistical differences across professions in regard to the value of the recommended scientific publications (Tables 6, 7).

G. Evaluate the importance of the lectures in the program.
Correspondingly, the importance of the recorded lectures was considered of "very high importance" to "high importance" by 89.8%, and of "moderate importance" by 6.6% of the responders. RNs and SHCAs rated the  Correspondingly, the importance of the IT-links (H), the exercises and feedback from the tutor (I) and of the physical meetings (J) were considered of "very high importance" to "high importance" across professions, by 43.8, 83.9 and 91.7%, respectively. The importance of the links in the program was rated significantly higher by the SHCAs than by MDs and RNs. The importance of the exercises and the feedback was rated significantly higher among SHCAs than other professions (Tables 6, 7). Although all the professions rated the physical meeting(s) in the program of "very high importance" or "high importance", with no statistical significance across professions, the relative ranking was highest for SHCAs (86.6%), followed by RNs (82.5%) and MDs (72.5%) ( Tables 6, 7).

Discussion
This study presents nationwide experience with multiprofessional, educational E-programs in oncology, during a 10-year period, including nearly 5000 participants. Selfreported outcomes, assessed at completion of the education revealed a high overall contentment, and perceived clinical usefulness of the E-programs, across professions. However, the descriptive design of study does not allow any firm didactic conclusions to be drawn about the E-programs, but the study is at best considered of hypothesis-generating nature, presenting valuable highvolume data for future more scientific rigorous research.
Clinical oncology is an expanding branch of medicine, and does not only include the traditional disciplines as chemotherapy, hormone therapy, immunotherapy and radiotherapy, but also a number of adjuvant specialties, such as palliative medicine, oncological pain management, psychosocial oncology, supportive care and surgical oncology. Educational measures in clinical oncology thus require both multi-disciplinary and multi-professional approaches. The present learning paradigm reflects this well, not only mirrored in the different professions and proficiency records of the trainees, but also in the professional records of the tutors and producers. The educational material presented in our E-programs benefits by the homogenous design across professions and disciplines, facilitating clinical cross-professional and cross-disciplinary collaboration. The statistical significant differences in attitudes and perceived utility of the E-programs between the professions (Table 7), are interesting, indeed, but the authors have not been able to recover any systematic analysis in the literature of differential attitudes towards E-learning, between health care professions. Although very pertinent for detailed didactic, scholarly discussions, these observations await further explorative analyses.
The main advantage of the study are inclusion of a large number of participants (n = 4693) attending a considerable number of E-programs (n = 490), with a study duration of individual educations ranging from days to months (full-time: 0.4-24 weeks). The average drop-out

Table 5 Distribution of professions of the tutors, the producers (producing education material) and the combined producer-tutors
The number of PhDs for each profession indicate the academic affiliation. Medical doctors (MD) and registered nurses (RN) comprised 53.1 and 9.1%, respectively. Other professions (26.8%) included a legal counselor (n = 1), dental hygienist (n = 1), dieticians (n = 2), high school teacher (n = 1), pharmacist (n = 1), physiotherapists (n = 3), psychologist (n = 1), radiophysicists (n = 19), medical researchers (n = 3), and, social and health care assistants (n = 2)   rate of 10.6% was considerably lower than previously described in E-programs, ranging between 25 and 60% [15,16]. A course retention near 90% is an indirect measure of the contentment and the perceived didactic value of the E-programs and thus is a critical measure of training efficiency. While, the low drop-out rates likely are of multifactorial origin, detailed prospective studies of our E-learning programs are needed to explain these favorable results, particularly regarding the inter-professional differences observed, i.e., significantly lower drop-out rates for RNs and RTTs, compared to MDs and SHCAs. However, the interesting, but disappointing observation of a highly significant linear annual increase in overall, relative drop-out rates probably is likely explained by a rapid expansion of the programs going from 150 trainees 2005, to 700 trainees 2014 (Fig. 2), perhaps indicating an organization in "growing pains. " The hierarchical structure, considered important during the early development of the E-learning programs, was successively replaced with a more horizontal management structure, due to an increased number of programs, trainees, tutors and producers, also reflected in the wider geographical catchment area. An important aspect in development and designing of E-learning programs is a tendency towards the use of a "richer" medium over time, progressing from text to a graphical platform, and from audio to video recordings, in an attempt to increase the effectiveness of the programs [17,18]. One study indicated that a "richer" medium content positively correlated with the concentration efforts of the user, but ambiguous results were obtained with the perceived usefulness of the program [17]. Another study observed that the relationship between the "richness" of media choice and the effectiveness was moderated by the learning domain of the program and the learning styles of the user [18]. In the design of our E-learning programs from 2005 to 2014, transitions from audio to video recordings were not implemented to any higher degree since our empirical data indicated that no gain in didactic quality nor trainee satisfaction was obtained by use of a "richer" medium. More studies examining the influence of different media for contents of e-learning programs, regarding the principal outcomes educational efficiency, perceived usefulness, satisfaction scores and cost-efficiency measures, are clearly needed [18].

Limitations
First, an important limitation of this study is that the major outcome, self-reported evaluations only contained data from 2008 to 2015, and, further that these data only comprised 72% of the total number of trainees in this period. Although emphasis on quality assessment from the beginning was considered essential, a number of different evaluation questionnaires were used during the years, impeding the quality of data sampling. Another reason for the incomplete data collection is that a number of programs were locally managed and supervised, and thus, not infrequently, beyond our quality control. Second, this retrospective study does not include any control group: a meaningful comparison with other programs, although highly relevant indeed, is therefore not possible at this stage of research. This makes it even more difficult to draw conclusions about the potential advantages of E-learning compared to face-to-face lectures or even computer-based instructions. Comparisons between E-learning and traditional classroom teaching have been evaluated in several studies [19][20][21][22], but the published experience in oncology is rather limited [23]. However, the systematic review and meta-analysis by Lahti et al. [9] did not demonstrate any statistical differences between E-learning and traditional learning groups, regarding knowledge, skills or satisfaction. In a recent randomized controlled trial, comparing live lecture, internet-based and computer-based instruction [24], it was observed that even if the interactive Internet-based instruction is a difficult and time-consuming teaching method, it was recommended to integrate the method in medical teaching. Further, in a prospective randomized controlled study [25] including physiotherapy students participating in an oncology course, comparing traditional classroom with E-learning, concluded that the use of E-learning in oncology is a feasible method of teaching. Moreover, Alfieri et al. [10] showed that the use of interactive E-learning for radiation oncology is an effective method to improve the radiologic anatomy knowledge and treatment planning skills of radiation oncology residents. Another advantage, which should be mentioned, is that E-learning may increase the cost-efficiency since the same E-learning program can be transmitted to a larger number of students [26], reducing the demand for a classroom teacher, offering more flexibility [22]. Third, only self-reported outcomes, vis-à-vis objective outcomes, are available, and obviously, subjective responses are vulnerable to a number of biases [27]. The printed questionnaire was a structured interview, administered to the trainees during a physical meeting with the tutor(s) upon completion of the program. The questionnaire was answered independently and anonymously by the trainee, and the scoring sheet was collected manually by the tutor. Acquiescence bias, referred to as "yea"-saying or "nay"-saying, i.e., a stereotype response defined as a tendency to agree with attitude statements, regardless of content [27] may have confounded the outcome. In addition, acquiescence bias is likely augmented by our use of similar categorical rating scales among items not conceptually related. Extreme responding bias, a behavior only selecting anchor values, as well as leniency bias, a behavior projecting social relationships upon the evaluation, could also contribute to the skewed distribution seen in response ratings. On the other hand, it could be argued that the self-reported evaluations were accumulated from a large number of participants, professions, programs, covering different topics and time spans, securing a broad platform of experience. The long-term impact of E-learning on clinical practice has been examined in a number of controlled randomized studies [28][29][30][31]. Some of these studies observed improvement in the knowledge, skills and clinical behavior of the personnel [30,31], while others only demonstrated slight, if any advantage over conventional learning [28,29]. This is corroborated by two recent systematic reviews which concluded that there was insufficient evidence regarding the effectiveness of E-learning on healthcare professional behavior or patient outcomes [9,32]. Fourth, the learning experiences have not yet been tested in a clinical benchmarking scenario against conventional on-location programs.

Conclusions
This descriptive study presents high-volume data and covers 10-years of nationwide multi-professional and multidisciplinary experience with oncological E-learning. The consistent high fulfillment ratings in learning objectives, the low drop-out rates and the wide geographical catchment area suggest that E-learning programs with contemporary techniques are feasible and complementary pathways to improve pre-and post-graduate training in oncology. Further, the contemporary demands placed on the health workforce regarding a professional responsibility of maintaining the medical competence in practice, E-learning may play an important role in overcoming this challenge [32]. Finally, these hypothesis-generating data demonstrate that our educational paradigm has been well received across disciplines, professions and proficiency levels. However, it is also evident that prospective, high-volume comparative studies, ascertaining the objective didactic value of the E-programs, are needed.