The ALFA (Activity Log Files Aggregation) Toolkit: A Method for Precise Observation of the Consultation

Background There is a lack of tools to evaluate and compare Electronic patient record (EPR) systems to inform a rational choice or development agenda. Objective To develop a tool kit to measure the impact of different EPR system features on the consultation. Methods We first developed a specification to overcome the limitations of existing methods. We divided this into work packages: (1) developing a method to display multichannel video of the consultation; (2) code and measure activities, including computer use and verbal interactions; (3) automate the capture of nonverbal interactions; (4) aggregate multiple observations into a single navigable output; and (5) produce an output interpretable by software developers. We piloted this method by filming live consultations (n = 22) by 4 general practitioners (GPs) using different EPR systems. We compared the time taken and variations during coded data entry, prescribing, and blood pressure (BP) recording. We used nonparametric tests to make statistical comparisons. We contrasted methods of BP recording using Unified Modeling Language (UML) sequence diagrams. Results We found that 4 channels of video were optimal. We identified an existing application for manual coding of video output. We developed in-house tools for capturing use of keyboard and mouse and to time stamp speech. The transcript is then typed within this time stamp. Although we managed to capture body language using pattern recognition software, we were unable to use this data quantitatively. We loaded these observational outputs into our aggregation tool, which allows simultaneous navigation and viewing of multiple files. This also creates a single exportable file in XML format, which we used to develop UML sequence diagrams. In our pilot, the GP using the EMIS LV (Egton Medical Information Systems Limited, Leeds, UK) system took the longest time to code data (mean 11.5 s, 95% CI 8.7-14.2). Nonparametric comparison of EMIS LV with the other systems showed a significant difference, with EMIS PCS (Egton Medical Information Systems Limited, Leeds, UK) (P = .007), iSoft Synergy (iSOFT, Banbury, UK) (P = .014), and INPS Vision (INPS, London, UK) (P = .006) facilitating faster coding. In contrast, prescribing was fastest with EMIS LV (mean 23.7 s, 95% CI 20.5-26.8), but nonparametric comparison showed no statistically significant difference. UML sequence diagrams showed that the simplest BP recording interface was not the easiest to use, as users spent longer navigating or looking up previous blood pressures separately. Complex interfaces with free-text boxes left clinicians unsure of what to add. Conclusions The ALFA method allows the precise observation of the clinical consultation. It enables rigorous comparison of core elements of EPR systems. Pilot data suggests its capacity to demonstrate differences between systems. Its outputs could provide the evidence base for making more objective choices between systems.


Observational Data Capture (ODC) Objective
Collection of observational data about doctor-patient and doctor-computer interactions

Setup and process
The multi-channel consultation video is imported into Obswin (an observational data capture tool) in order to measure timings and occurrences of various aspects of the consultation.
First the interactions that need to answer the research question should be identified. If the number of variables is difficult to be measured in a single recording run (n>5), then they should be categorised into groups. Separate recording runs should focus on each group of variables.
Each consultation is watched at least once before the actual recording run by each rater, to get familiar with the content of the consultation. When the observation is in progress, the corresponding key is pressed to indicate onset, and pressed again to indicate offset.
Raters receive training about analysing the videos by the use of a written training manual. This should give a more encompassing definition of the variables and sample screen shots to give further clarity to the variables. This reduces the need for an informal teaching process and standardises the training process for all raters. As well as a general training manual, system specific guides will help to familiarise raters with design features unique to the system. The system specific training included a crib sheet for each video which provided a brief summary of the conditions discussed in each consultation.
The results are then combined into one file representing the whole consultation, forming a dataset from which summary statistics and graphs can be produced. Numerical data includes the time interval that a variable occurred over, its percentage interval and the number of it's "on" and "off" sets. After the end of the recordings, the intra class correlation coefficient was then calculated for each variable across all videos. Data can be displayed as occurrence graphs, displaying various activities that occur in the consultation. This linear form of representation shows the proportionate times of specific activities within a consultation and how these relate to each other.

Hardware/software requirements
ObsWin or similar observational data analysis tool

User Action Recording (UAR) Objective
To record the clinician's use of computer keyboard and mouse during the consultation

Setup and process
User Action Recorder (UAR) is a data collection tool that has been used for analysis of the consultation process. When activated, this programme captures keystrokes and mouse movements. The value of the pressed key and the co-ordinates of the mouse pointer are written into two separate log files with time-stamps. The software is copied into the doctor's computer before the start of the consultation recording session. It is activated before the first consultation, and left running until the end of the recording session. The two files representing the keyboard (keyboard.txt) and mouse activities (mouse.txt) are then copied for analysis. Used in conjunction with multi-channel video, UAR can be used to accurately calculate the time taken to complete various computer activities. Although observer rating of computer use was unreliable, ObsWin software can be used to play back the consultation videos at variable speeds. UAR is then used to accurately calculate time taken for various computer activities, including coded data entry, free text, prescriptions and referrals. E.g: Calculating time taken for prescription of medication.
-The consultation video is imported into Obswin software and then scrolled through to the point where prescriptions occur.
-In the instance of prescription, the start of the process was taken as the mouse click / key stroke used to launch the prescription template, and the end of the process taken as the mouse click / keystroke used to exit the template / save the data.
-The time-stamp on the multi-channel video screen is then correlated to the UAR log files.
-The time taken for that particular sequence of mouse clicks and keystrokes can then be calculated from the log file time-stamp.
Use of an Excel Macro makes the calculation process and the identification of the interactions segments efficient. After series of analysis, a time gap of more than 3 seconds between two adjacent entries was identified as an indicator for a possible break in doctor-patient interaction.

Voice Activity Recording (VAR) & Consultation transcripts Objective
To create a time-stamped transcription of the consultation

Setup and process
The VAR tool monitors the verbal interactions and creates a log file by analyzing the sound level The VAR also enables us to identify who initiates and terminates silence. We have observed how the clinician sometimes makes purposeless use of the IT to initiate silence to control the consultation. The format of the VAR log is designed to be compatible with most of the transcribing tools, and it is easily customisable. It has header fields which details format of the data recording. We have successfully imported this to the 'Subtitle Worksop' application, which links the VAR entries directly into the video segments. The number of log files that can be imported are currently set to 10, but this is limited only by the amount of display area available for the aggregated output. Once the number of log files the LFA tool needs to combine is configured, it allows the user to select the data format for each file.

Hardware/software requirements
Currently this is set to import ObsWin, UAR (User Activity Recording), VAR (Voice Activity Recording), XML and PRS (Pattern Recognition Software) data files. Next is the combination stage, which produces an unified output in the background. User can then either opt to view the aggregated output using number of formats available or export it. Combined output could be saved as text, ObSwin, or XML file formats. The Java XML package facilitates the conversion of aggregated output into XML format. This single output represents the data from multiple log files of different formats as a single transferable data file. This also enables the communication

Occurrence graph with video segments Objective
To synthesise the consultation observations into an analysable process model of doctorcomputer-patient interactions, with features interpretable by clinical system developers

Setup and process
This analysis stage of ALFA is performed using the 'Sequence diagram' graphical notation of the Unified Modelling Language (UML). We use UML 2.0. This UML process model represents the principal activities within the consultation, complexity of the interactions and how the sequence of activities is progressively defined. We can analyse how the interaction patterns may be dependent on the design of the clinical system and their effect on patient centeredness. The sequence diagram creation is a three stage process. First the basic interaction framework is plotted using the LFA occurrence graph output. This process can be semi-automated with the use of its XML output. This marks the start and end times of doctor-patient and doctor-computer interactions. We plot these findings into the doctor and computer lifelines in the 'Sequence diagram'; this defines their focus of control boxes and the message passing arrows between them. Direction of the arrows indicates the originator of each interaction. We then review the UAR and VAR log files to add precise details about the nature, objectives and relationship of identified verbal interactions of doctor-patient and doctor's use of the keyboard. The times-tamps for these activities are extractable from the XML output. We could then annotate the sequence diagram to explain the data flows associated with each interaction. Further enhancements of the sequence diagram skeleton using the multi-channel video outputs are aimed at informing the interdependence of each interaction. We viewed the computer interface activity of the multichannel code to identify the characteristics of doctor and computer interactions associated with the clinical system brands..  Figure 16: An UML Sequence diagram of a consultation; three life lines represent the doctor, computer and the patient. Arrows passing between them represents the interactions. Figure 17: UML Sequence diagrams comparing the blood pressure data entry task in four different EPR systems.