International Review of Research in Open and Distributed Learning IRRODL Evaluation of Student Learning in Remotely Controlled Instrumental Analyses

The Canadian Remote Sciences Laboratories (CRSL) website (www.remotelab.ca) was successfully employed in a study of the differences in the performance and perceptions of students’ about their learning in the laboratory (in-person) versus learning at a remote location (remote access). The experiment was completed both in-person and via remote access by 70 students, who performed essentially the same, academically, in the two modes. One set of students encountered the in-person laboratory first and then did the remote laboratory, while the other set of students did the activities in the reverse order. The student perception survey results (n = 46) indicated that the students found both experimental scenarios to be at appropriate levels of difficulty, clear to understand, and did not overall prefer one way of completing the experiment over the other. However, they felt that they learned more about the theory of the experiment, more hands-on skills, and more about the operation of the instrument when they performed the experiment in the laboratory in the presence of an instructor. They also believed that they learned more about the instrument operation from their laboratory partner when they completed the experiment in the laboratory, but learned more from their partner about the operation of the instrument software when they completed the procedure from a remote location.


Introduction
Remote access to modern chemistry laboratory equipment is important to chemical industry and chemistry education (Kennepohl, 2010).Most industry analyses are now completed using software controlling automated instruments, which affords the opportunity to operate chemistry instruments remotely; either in hazardous environments or by on-line monitoring in remote locations of a chemical production facility.
Chemistry instructors at universities and colleges are beginning to take advantage of this opportunity as well (Barot et al., 2005;Szalay, Zeller, & Hunter, 2005;van Rens, van Dijk, Mulder, & Nieuwland, 2013;Saxena & Satsangee, 2014).There are at least two reasons for this.First, since many chemical technologists will now be operating laboratory instruments in industry via a remote connection (either LAN or WAN), it is valuable to include remote-access activities in the curriculum (Baran, Currie, & Kennepohl, 2004).
Second, remote access to laboratory instruments provides educational institutions the ability to deliver distance education on these instruments, as well as instrument sharing and after-hours access for students on equipment with high scheduling demands (Albon, Cancilla, & Hubball, 2006).Initial studies on remote laboratories for teaching over the Internet focused primarily on the technology, feasibility, and the ability to have the learner connect with and control an experiment remotely.The current emphasis is more on pedagogy and learning design to determine how to facilitate a high level of student learning and skillsdevelopment at a distance (Azad, Auer, & Harward, 2011).To provide some insight into learning in a remote laboratory environment we look to another related mode of laboratory delivery that has been more thoroughly studied-namely the computer simulation.
For many years, the use of computer simulations to create virtual laboratories has been extensively explored and is relatively well understood in the sciences (Smetana & Bell, 2012).In chemistry, recent reports have looked at differences in how students learn when using animations and simulations (Kelly & Jones, 2008;Akaygun & Jones, 2013).These tools aid in student understanding of the behaviour of chemicals without the cost and laboratory time required for in-the-laboratory exploration.The literature discussion has moved away from arguments of which mode is better (in-person versus simulation) towards the realization that an appropriate combination of the two often provides richer learning opportunities than either mode on its own.For many years, the natural assumption for the order of this combination had been to treat the simulation as a pre-lab exercise followed by the real laboratory experience.However, the increased benefit has been seen irrespective of the order of mode (Jaakkola & Nurmi, 2008;Zacharia, Olympiou, & Papaevripidou, 2008) and one group has even reported that the best combination in their teaching laboratory was to do the real experiment first before doing the simulation (Smith & Puntambekar, 2010).
In a similar fashion to computer simulations, would blending remote laboratory work with onsite laboratory work afford any benefits and would the order in which a student is exposed to onsite versus remote laboratories be different from a learning perspective?
It is important to note that in contrast to computer simulations, remote control of laboratory instruments allows students to analyze real samples, on real instruments with resulting real behaviour.Chemical educators are interested in the effects that remote-access instruction in instrumental analysis has on student learning and students' attitudes to learning.However, research in both remote laboratory design (Cagiltay, Aydin, Aydin, Kara, & Alexandru, 2011;Lindsay, Murray, & Stumpers, 2011) and evaluation of student learning (Ma & Nickerson, 2006;Nickerson, Corter, Esche, & Chassapis, 2007;Elawady & Tolba, 2009) in different laboratory modes is meagre in chemistry and is being led by disciplines, such as computing science and engineering.Indeed, a world-wide remote controlled laboratory inventory indicates about 60% are sites from robotics, computing, and engineering fields, 30% from physics, and 10% from all other science disciplines (Gröber, Vetter, Eckert, & Jodl, 2007).
The framework of teaching and learning in chemistry is steeped in constructivist theory (Bailey & Garratt, 2002).This is especially true of the laboratory component which attempts, among other things, to bring together and mutually strengthen theory and practice.Three important principles are assumed in this construct namely (1) there is a direction to learning which continually builds on previous knowledge, (2) knowing and doing are intimately linked so the laboratory activity provides situated cognition, and (3) student perception can influence the construction of knowledge.In this paper, we report an investigation of the differences in learner performance and perceptions in the laboratory (in-person) compared with a remote location (remote access) in equivalent analytical chemistry experiments.We also examine whether the order of the mode of the experiment the student first encounters (in-person or remote access) plays a role in learning or learner attitude in the laboratory.Student performance was tracked and students were surveyed as to their perceptions of ease of operation, quality of support materials, how much they felt they learned in both scenarios, and how much they felt that they learned from their laboratory partner in both scenarios.

Methodology
This study applied for and received approval from both the NAIT and Athabasca University Research Ethics Boards.It was designed to develop a laboratory experiment that could be performed both in the laboratory and by remote control, and to investigate student performance and perceptions as to how they learn differently under the two conditions.
Students (n = 70), with the exception of two individuals, worked in self-selected pairs.Some student pairs would complete the in-laboratory exercise first and then complete the remote-access exercise.The other student pairs completed the remote-access exercise first and then the in-laboratory exercise.The order of the remote-access versus in-laboratory work was assigned randomly.Pair performance (based on laboratory report grades) was tracked and any operational issues with using the remote environment were noted by the instructors.In addition, all students were asked to voluntarily complete two surveys that measured their perceptions and comparisons of the two experiences once both remote and in-person experiments had been completed.No student personal information was collected in this study and the surveys were anonymously coded by the students such that the researchers could link the surveys to the same participant, but not identify the participant.The survey tools asked the students to identify whether or not they had completed the in-laboratory exercise or the remote-access exercise first.Student perceptions were measured through the use of Likert Scale questions and comment boxes.The survey tools are provided in the Supporting Materials.
The data collected from individuals (n  46) by the on-line survey tools (Fluid Surveys, free version) was tabulated by the authors using Microsoft Excel®.Systematic analysis and conclusions in the student survey portion of this study were applied to the results of the Likert Scale questions only.Students' written comments and other qualitative data in the survey were not systematically analyzed, but were summarized and included in the results section.Statistical tools in Excel (t tests) were used to test hypothetical relationships.

Remote Access Environment
The NAIT Chemical Technology program has a website that allows remote access to a variety of chromatographic and spectroscopic instruments including gas chromatographs (FID, MS, PFPD, TCD detectors), a liquid chromatograph (triple quad MS detector), as well as NMR, ICP-OES, and ICP-MS spectrometers.All of the accessible instruments are equipped with autosamplers.Access to an instrument is controlled by a website administrator, who grants appointments and log-in credentials to qualified operators.The website also includes facilities to allow users to request bookings, and to view and manage personal bookings.To control an instrument through the website, the client accesses the site (www.remotelab.ca)through their browser.The request is sent to an F5 switch and redirected to a RemoteLab web server (Figure 1).The RemoteLab server uses Active Directory as the authentication method for access to the website.If the client's username and password are valid, they are allowed to proceed.If the client has a valid reservation on an instrument, the RemoteLab website sends the request to a VMWare server.If on the first time a client accesses an instrument they do not have the VMWare software installed on their device, the VMWare server begins to initiate installation of the software on their computer.The VMWare server manages the client's access to the laboratory computer from this point on (Figure 1).

Experiment
The Inductively Coupled Plasma -Optical Emission Spectrometer (ICP-OES) instrument was chosen.
Experiments and learning support materials were developed for the students included the laboratory experiment procedures, as well as two tutorials (with animations) and study games.The laboratory experiment procedures included introductory theory concepts as well as the instructions for completing the laboratory exercises.
The in-person experiment involved studying the interference effects of iron on tin analyses.In this experiment the students prepared iron (III) nitrate (Fe(NO3)3•9H2O) and tin (IV) chloride (SnCl4•5H2O) in an acidic matrix.The students also prepared analytical solutions containing just tin, and various concentrations of tin and iron.The analytical solutions were designed to illustrate the instrumental results when only tin was present versus solutions where varying amounts of iron as an interferent was present.In a second part of this experiment the students utilized a software algorithm that allows the iron to be listed as an interferent.In this section the instrument software monitors the emission of both tin and iron individually and calculates a correction factor to remove the effects of interference by the iron.The students repeated the analysis of the samples in this manner and observed the new values for tin concentration (either tin wavelength) due to the correction factor applied by the software.The students also used the instrument data to calculate their own correction factor and compare their factor to the one reported by the software.
In the remote experiment, essentially the same procedure was followed.The same solutions were prepared, and the same instrumental analyses were performed.The main difference was that once the solutions had been prepared and loaded into the instrument autoanalyzer trays, the students then went to a remote location and ran the instrument software through the Canadian Remote Sciences Laboratories website.
Once the operation of the instrument was complete, the students returned to the laboratory to clean up their samples, place the instrument into a standby state, and to collect their instrument printouts.The students had the option of using a Chemical Technology program notebook computer or one of their own.

Results
The information acquired from the study can be grouped into three categories: (1) instrument/website operation, (2) student performance, and (3) student perceptions.

Operational
A few issues arose during the remote experiment preparation and student participation phases.First, the booking system of the CRSL website only allows one user ID to be logged into the computer operating the laboratory instrument at one time.It also does not have the capability of automated scheduling.This would mean manual reservations with separate IDs would need to be made centrally by a site administrator to allow for completely secure appointments of the instrument.For the purposes of this initial study, we chose to use a single generic login ID for all student pairs to streamline the reservation process.Second, due to security concerns at NAIT the remote access to webcams outside the NAIT system was not allowed.Two separate incidences occurred (malfunction of the autosampler and improper connections to the peristaltic pump), which would have been more readily apparent to the students at their remote location if they could have observed the instrument operation via a webcam.Third, the user's computer must install and run VMWare software to allow the website to access the computer directly connected to the laboratory instrument as a virtual machine.This is an additional step for the student.However, with the exception of printing files remotely using a Mac Notebook, the software appears to work well with a variety of devices.
Fourth, for the ICP-OES instrument the remote control connection provided excellent instrument stability.
Regardless of remote computer crash/closure or dropped wireless connection, once the network connection was re-established the instrument was found to be continuing its last task.

Academic Performance
Tables 1 and 2 present the academic performance (laboratory report grades) of student pairs that completed the in-person experiment first versus those who completed the remote access experiment first.b.Only one student in this group.

Student Perceptions
A total of 70 students were eligible to participate in this study.Of this number, 46 students completed consent forms and both surveys.Within the surveys responses to specific questions ranged from 42 to 46.
When completing the experiment by remote access 94% of the students completed the activity somewhere on NAIT campus.The majority of students were satisfied with the instructional materials as presented requesting only some minor revisions 11% (in-lab) and 33% (remote).Table 3 summarizes the Likert scale responses to nine of the survey questions.In addition, student comments were collected prompted by open-ended questions in the survey (see Appendix A).These general comments echoed information already collected from the responses to earlier specific questions, while also giving more anecdotal detail.In other cases, comments touched on areas not covered by set questions.Table 4 provides an overview summarizing (by comment type) some of the more common remarks.A majority of the students liked to see and hear the instrument at work in the laboratory.
They liked or were impressed by their ability to control the instrument from a remote location, and they felt that they learned more from their partner when they could not just ask the instructor for clarification.Half of the students felt that they understood the theory of the experiment better when there was an instructor present and they were getting hands-on experience.A small portion of the students felt that the remoteaccess experiment was more stressful.Similar numbers of students mentioned that not being able to the instrument working or having to trust that the instrument was working was stressful.

Table 4
Selected Summary of General Comments

General comment type Respondents (%)
Liked the ability to see the instrument at work inlaboratory 66 Seeing the instrument in the laboratory helped them learn more 30 Did not like inability to see the instrument in the remote access experiment 29 Wanted a webcam added to allow observation for remote operation 22 Disliked the crowded, noisy laboratory and having to wait to use the instrument 30 Liked being able to control the instrument from somewhere else 65 Found remote access experiment was more stressful 24 Understood the theory of the laboratory better with an instructor and hands-on experience access to the instrument.This is not surprising in hindsight as remote access would be expected to be more similar to handling the real instrument compared with a computer simulation.However, there were a couple of clear differences in student attitude.Statistically more of the students felt that the in-person experiment helped them to understand the theory of ICP-OES better than the remote access experiment (93% vs 65%) and more students felt that they learned more in the in-person experiment than in the remote access experiment (52% vs 17%).While performance and meeting learning outcomes did not differ between modes, these survey results underscore the high value placed on the role of human interaction.Direct inperson student-teacher interaction is reduced via the remote mode and the student perception is that learning was also reduced.In contrast, because students worked in pairs (in both modes), a substantive amount of learning was attributed to the partner through student-student interaction.This is significant since learning from a peer has been recognized as a major contributor to successful learning (Johnson & Johnson, 1996;Hooper & Hannafin, 1991).
Furthermore, some of the general comments indicate that the remote access mode encouraged better understanding of the instrument software and provided some appreciated flexibility and autonomy for the learner.The influence of mode of laboratory delivery on a student's perception of what they have achieved compared with what they were meant to achieve has been previously described by Lindsay and Good (2005).A student's take on a particular learning environment has potential to shape the learning.The authors go on to report an increased acceptance of alternate modes of laboratory experiments by students who had experienced those modes, but with a continued "substantial bias" towards in-person labs (Lindsay & Good, 2005).Even though there are some stated perceived pros and cons, there was no overall preferred mode in our study.
It would appear to be safe to conclude that the students found the experiments to be similar in value, but that what they specifically learned in each of the experiments was not the same.They felt that they learned more about the theory of the experiment and operation of the instrument when they were in the laboratory with an instructor present.They felt that they learned more from their partner and about the software running the instrument when they were on their own at a remote location.

Conclusions
The design and operation of both experimental modes were mostly equivalent with the exception of a couple of technology issues (lack of automated scheduling and inability to print files remotely using a Mac Notebook) and one policy issue (access to live webcam of the instrument).However, for the ICP-OES instrument, the remote control connection provided excellent instrument stability, which is essential for running the experiment successfully.It would appear to be safe to conclude that the students found the experiments to be similar in value.Students learn equal amounts in both in-person and remote access approaches to instruction in the laboratory, but they learn some things differently.The in-person activities rely more on direct interactions with the instructor and leave students with the perception of having learned more hands-on skills and a better understanding of the theory of the experiment.The remote access activities leave them with the perception that they learned more about the operation of the instrument software.Since all students were eventually exposed to both modes of this experiment, they had the opportunity to make use of both approaches.The order in which students did the in-person or remote laboratories made no significant performance difference in this course.

Experiment 12
We are requesting your assistance in providing us with your views on the remote and/or in-lab experiments you have carried out in [the courses].The results of the survey are being used both as a scholarly assessment of the effectiveness of these laboratory experiences and as feedback to further development and improve the experiments.We ask that you take a moment now to do the survey.It should take about ten to fifteen minutes to complete.Participation is completely voluntary, anonymous and confidential.You are free to discontinue participation at any time during the study.No one except the researchers and their supervisors will be allowed to see the answers to the questionnaires.There are no names on the questionnaires.Only group information will be summarized for any presentation or publication of results.The anonymous data will be stored on a password protected computer at XXXX for three years at which time it will be permanently erased.The results will be used to further develop this course and would benefit future students.Thank you in advance for your assistance.

Remote Lab Access Student Survey
Note: You should have completed both the [course] in-person experiment and the [course] remote access experiment before completing this survey.

Anonymous Student Research Code
Reminder: This is the first three letters of your mother's name followed by the first three letters of your month of birth (no spaces).For example, mother's name is Jane and month of birth is December -anonymous research code is: JanDec Where did you access the remote lab website?Home Friends NAIT Other -please specify ______________________ I would recommend the remote access lab experiment to others.

Strongly Disagree Disagree Neutral Agree Strongly Agree
The material covered and the degree of difficulty is appropriate for the level of the course.
Strongly Disagree Disagree Neutral Agree Strongly Agree
difference in average report grades (%); average in person minus average remote access.A positive result means the students scored better in their second experience.A negative result means the students scored better in their first experience.

Table 1
Laboratory Report Grades for Student Pairs That Completed the In-Person Activity First a.The difference in average report grades (%); average remote access minus average in person.A positive result means the students scored better in their second experience.A negative result means the students scored better in their first experience.

Table 2
Laboratory Report Grades for Student Pairs That Completed the Remote Access Activity First

Table 3
Percent Student Agreement to Survey Questions a