Cross-platform digital assessment forms for evaluating surgical skills

A variety of structured assessment tools for use in surgical training have been reported, but extant assessment tools often employ paper-based rating forms. Digital assessment forms for evaluating surgical skills could potentially offer advantages over paper-based forms, especially in complex assessment situations. In this paper, we report on the development of cross-platform digital assessment forms for use with multiple raters in order to facilitate the automatic processing of surgical skills assessments that include structured ratings. The FileMaker 13 platform was used to create a database containing the digital assessment forms, because this software has cross-platform functionality on both desktop computers and handheld devices. The database is hosted online, and the rating forms can therefore also be accessed through most modern web browsers. Cross-platform digital assessment forms were developed for the rating of surgical skills. The database platform used in this study was reasonably priced, intuitive for the user, and flexible. The forms have been provided online as free downloads that may serve as the basis for further development or as inspiration for future efforts. In conclusion, digital assessment forms can be used for the structured rating of surgical skills and have the potential to be especially useful in complex assessment situations with multiple raters, repeated assessments in various times and locations, and situations requiring substantial subsequent data processing or complex score calculations.

Numerous structured assessment tools have been introduc ed for the evaluation of surgical skills, applying a multitude of different scales and rating techniques.The data collection forms used in such tools provide a structured framework for the eval uator and are commonly paperbased.The traditional paper based rating forms are tried and tested, allow fast assessment, and are intuitive for raters regardless of their digital readiness.However, paperbased rating forms can involve a timecon suming process of manual score calculation.This difficulty can be addressed by using imagescanning techniques such as optical mark recognition, although such techniques require dedicated equipment.
Only a few reports have described digital rating forms [13], with the exception of reports dealing with computerbased testing as such.Digital assessment forms for evaluating surgi cal performance might be considered more inconvenient, in secure, and vulnerable in comparison with paperbased forms, and moreover, the evaluator must be familiar with the neces sary electronic devices and platforms.Nevertheless, digital forms could have advantages over paperbased forms, espe cially in complex assessment situations where multiple perfor mances and raters need to be managed, assessments occur at a range of times and/or locations, or when substantial subse quent data processing is needed.Digital assessment forms of fer the possibility of connecting to an online database, which could support simultaneous assessment using multiple devic es and evaluators as well as providing immediate scoring.
A range of software platforms and technologies could po tentially be used for digital assessment forms, and the advan http://jeehp.orgJ Educ Eval Health Prof 2015, 12: 13 • http://dx.doi.org/10.3352/jeehp.2015.12.13 tages and disadvantages of currently available electronic form management tools have recently been described [1].In con trast to other reports on the development of digital assessment forms that required the integration of multiple types of tools and surveys for complex workplacebased or practicebased assessments [1,2], in this case, it was only necessary to assess saved finalproduct performances from a virtualreality otolo gy simulator using one assessment form that allowed multiple raters on different platforms.Therefore, we developed a digital assessment form for this use.This led to the development of several other examples of digital assessment forms for evaluat ing skills in the context of surgical training, including both technical and nontechnical skills.In the following sections, the development and testing of these forms will be described, and their application in the assessment of surgical skills will then be discussed.
First, it was necessary to choose a database platform for the digital assessment forms.We chose FileMaker Pro 13 (File Maker Inc., Santa Clara, CA, USA), because it supports differ ent desktop platforms (Windows 7/8, OSX 10.7 or newer) as well as the iPad and iPhone via the FileMaker Go app.The da tabase can either be stored on the device itself or be hosted on a local or external server, allowing for webbased access through most modern web browsers.The estimated costs (online pric es, June 2014) are US$329 for one license for FileMaker Pro 13 (a free trial is available), as well as US$132 or more annual ly for external database hosting or US$348 for the software necessary to establish a local server.The FileMaker Go app for iPhone/iPad is currently free.We found that the FileMaker platform provided excellent flexibility in designing the data base, displaying the fields, and designing the visual layout of the forms, and therefore it was easy to develop new forms.In our experience, developing a new basic rating form from scratch took from two to eight hours.Good online documentation exists for this software, as well as excellent beginner's guides and external consulting can be hired if more complex solutions are needed.These factors facilitate the development of digital assessment forms that meet local requirements.
Next, we developed working examples of several rating formsincluding a range of rating structures-for some of the assess ment tools that have been reported in the fields of surgery and otorhinolaryngology.An overview of the newly developed forms is given in Table 1, and an example of one of the devel oped forms can be seen in Fig. 1.The forms have been made public for free download [4] and they can be used as is, modi fied as needed, or serve as inspiration for novel work.In each form, the time of assessment and the evaluator identification (ID) is automatically saved with each assessment.The evalua tor needs to enter the participant ID and the case ID if relevant.The forms have checkboxes, dropdown menus, radio buttons, and/or free text fields for rating and feedback.Cumulative scores and subscores are automatically calculated and updated as the evaluator performs the assessment, and these scores are stored along with the entered data.The data can later be ex ported to a range of different formats, including Excel ® (Mi crosoft, Redmond, WA, USA) and commaseparated files, for further processing.
Finally, the digital assessment forms were tested with File Maker Pro 13 on Windows 7 and OSX 10.7, and FileMaker 13 Go on iPhone and iPad running iOS 7. The forms were tested using both devicestored and externally hosted databases.Us ing WebDirect access to the external server, testing was also performed with a range of common web browsers (Firefox, Chrome, Safari, and Internet Explorer) on the abovemention ed devices.Overall, the digital assessment forms proved stable and consistent across platforms and devices.However, if web based access is likely to be the primary mode of use, then it would be preferable to optimize the forms accordingly, as the layout can change slightly in the web browser compared to the desktop software and mobile app.As well, some problems and http://jeehp.orgJ Educ Eval Health Prof 2015, 12: 13 • http://dx.doi.org/10.3352/jeehp.2015.12.13 crashes occurred using webbased access from handheld de vices.Currently, the only handheld devices on which these forms are accessible are the iPhone and iPad, for which the FileMaker 13 Go app is available.Some general issues should be considered when adopting digital assessment forms.First, each rating form needs to be tailored to the specific assessment situation.Second, evalua tors need to be introduced to the digital rating forms and the device(s) that will be used.Moreover, combining data from more than one device or rater will need to be done manually unless a serverhosted database with online access is used.Fur thermore, regular backup and device management should also be considered.Finally, as technology develops and new devic es and platforms emerge, the digital assessment forms will need to be updated, resulting in continued development and main tenance costs.
The digital assessment forms described in this report have been developed and tested in a local, controlled, and nostakes setting for evaluating recorded performances.The next step would be applying these digital assessment forms to the live evaluation of surgical performance, in a reallife setting and in highstakes assessments.Doing so would require significant experience with the digital platform in the local institution, and it would be necessary for the local setup to have been thor oughly tested and proved stable; for example, if a serverbased deployment is chosen, Internet connectivity must be assured.We therefore recommend implementing digital assessment forms for performance rating after the consideration of insti tutional needs, equipment, expertise, and resources, and after the advantages and disadvantages have been weighed in com parison with paperbased rating forms.Further research into the use of digital assessment forms for evaluating surgical skills compared to paperbased rating forms will help determine whether digital assessment forms prove to be timesaving, reli able, and feasible.ORCID: Steven A. W. Andersen: http://orcid.org/0000000234919790

Fig. 1 .
Fig. 1.The digital Objective Structured Assessment of Ultrasound Skills (OSAUS) assessment form in FileMaker Go on iPad Mini.