Evaluation of a training program for device operators in the Australian Government ' s Point of Care Testing in General Practice Trial : issues and implications for rural and remote practices

Introduction: From September 2005 to February 2007 the Australian Government funded the Point of Care Testing (PoCT) in General Practice Trial, a multi-centre, cluster randomised controlled trial to determine the safety, clinical effectiveness, cost-effectiveness and satisfaction of PoCT in General Practice. In total, 53 practices (23 control and 30 intervention) based in urban, rural or remote locations across three states (South Australia [SA], New South Wales [NSW] and Victoria [VIC]) participated in the Trial. Control practices had pathology testing performed by their local laboratory, while intervention practices conducted pathology testing by PoCT. In total, 4968 patients (1958 control and 3010 intervention) participated in the Trial. The point-of-care (PoC) tests performed by intervention practices were: haemoglobin A1c (HbA1c) and urine albumin:creatinine ratio (ACR) on patients with diabetes, total cholesterol, triglyceride and high density lipoprotein (HDL) cholesterol on patients with hyperlipidaemia, and international normalised ratio (INR) on patients on anticoagulant therapy. Three PoCT devices measured Archived at Flinders University: dspace.flinders.edu.au © MD Shephard, BC Mazzachi, L Watkinson, AK Shephard, C Laurence, A Gialamas, T Bubner, 2009. A licence to publish this material has been given to ARHEN http://www.rrh.org.au 2 these tests: the Siemens DCA 2000 (Siemens HealthCare Diagnostics, Melbourne, VIC, Australia) for HbA1c and urine ACR; Point of Care Diagnostics Cholestech LDX analyser (Point of Care Diagnostics; Sydney, NSW, Australia) for lipids; and the Roche CoaguChek S (Roche Diagnostics; Sydney, NSW, Australia) for INR. Point-of-care testing in the General Practice Trial was underpinned by a quality management framework which included an on-going training and competency program for PoCT device operators. This article describes the design, implementation and results of the training and competency program. Methods: An education and training resource package was developed for the Trial consisting of a training manual, a set of A3 laminated posters and a CD ROM. Five initial training workshops were held for intervention practices from each geographic region between August and October 2005 at three centres – Adelaide (SA), Bendigo (VIC) and Dubbo (NSW). These workshops combined theoretical training in the principles and practice of PoCT with ‘hands on’ practical training delivered in interactive small group sessions. At the completion of training, practice staff undertook a written and practical competency assessment and received a certificate of competency as a qualified device operator. Following each initial training workshop, practice staff completed a short satisfaction survey. Five refresher training workshops covering all geographic regions were delivered during late August 2006, coinciding with the 12 month point of the live phase of the Trial. At the completion of the Trial in February 2007, device operators completed a further questionnaire. Results: Sixty device operators from 31 practices completed training and competency assessment as part of the Initial Training Workshop series. A further 20 device operators from 12 of the practices were trained in the 12 month period after the initial workshops; 19 of these staff were from rural or remote practices. In total 80 device operators comprising 74 practice staff and six GPs from 31 practices were trained and received competency certificates as part of Trial. In all, 19 device operators left the Trial either through personal resignation from an existing practice or because their practice withdrew from the Trial; the majority (84%) were from rural and remote practices. A total of 42 device operators from 25 practices attended refresher training in the second half of 2006. Results from the satisfaction questionnaire completed by device operators following the initial training workshops showed there was unanimous agreement that the posters were useful for the conduct of daily PoCT and practical training in small groups was satisfactory as a training method. The quality and appropriateness of the PoCT training resources and the workshop overall was rated as either good or excellent by all respondents (100% and 78%, respectively). The responses by device operators to the post-Trial satisfaction questionnaire found a high level of satisfaction with PoCT across all geographic regions. Device operators from remote practices had the highest satisfaction levels for quality of training, usefulness of the training manual, ease of use of devices, confidence in the accuracy of PoCT results and preference for PoCT over laboratory testing. The usefulness of the posters for conducting PoCT achieved the highest satisfaction rating among operators from all three geographic regions. However the highest staff turnover rates and the highest number of requests for training of additional staff were from rural and remote practices. Conclusion: The methods established for the implementation and delivery of training and competency assessment for the PoCT in General Practice Trial were appropriate and effective. Results of the evaluation showed rural and remote practices have a greater need for training and support compared to their urban counterparts and may require more flexible training options to cater for much higher rates of staff turnover.

Introduction: From September 2005 to February 2007 the Australian Government funded the Point of Care Testing (PoCT) in General Practice Trial, a multi-centre, cluster randomised controlled trial to determine the safety, clinical effectiveness, cost-effectiveness and satisfaction of PoCT in General Practice.In total, 53 practices (23 control and 30 intervention) based in urban, rural or remote locations across three states (South Australia [SA], New South Wales [NSW] and Victoria [VIC]) participated in the Trial.Control practices had pathology testing performed by their local laboratory, while intervention practices conducted pathology testing by PoCT.In total, 4968 patients (1958 control and 3010 intervention) participated in the Trial.The point-of-care (PoC) tests performed by intervention practices were: haemoglobin A1c (HbA1c) and urine albumin:creatinine ratio (ACR) on patients with diabetes, total cholesterol, triglyceride and high density lipoprotein (HDL) cholesterol on patients with hyperlipidaemia, and international normalised ratio (INR) on patients on anticoagulant therapy.Three PoCT devices measured Introduction Point-of-care testing (PoCT) is defined as a pathology test performed on-site by or on behalf the treating doctor at the time of patient consultation, allowing the test result to be used to make an immediate decision about patient treatment 1 .Technological advances in the manufacture and design of PoCT devices has resulted in the global uptake of PoCT, currently increasing at a rate of 12% per annum 2 .The scope and application of PoCT is now expanding rapidly from the traditional hospital base to community-based primary care settings where the care of patients, particularly with chronic diseases, is now often focussed 1 .
In Australia, PoCT for diabetes management is now firmly established in the Indigenous health sector through the national Quality Assurance for Aboriginal and Torres Strait Islander Medical Services (QAAMS) Program and it has proven robust, safe and clinically and culturally effective [3][4][5][6] .There is growing interest in PoCT in the Australian general practice setting and a review of the role and value of PoCT in this setting highlighted that rural and remote practices could be the main beneficiaries of PoCT 7 .
However the review concluded that further work was needed to determine the clinical and economic benefits of PoCT in general practice.As a result, the Australian Government recently funded the PoCT in General Practice Trial, one of the largest and most comprehensive studies of PoCT ever conducted in this primary care setting 8 .
The PoCT in General Practice Trial was a multi-centre, cluster randomised controlled trial to determine the safety, clinical effectiveness, cost-effectiveness and satisfaction of PoCT in general practice.Three lead organisations were contracted by the Australian Government to deliver the Trial, working collaboratively from an Adelaide base.They were the Discipline of General Practice from the University of Adelaide (who were responsible for overall Trial management and evaluation), the Community Point-of-Care Services (CPS) unit at Flinders University and the RCPA Quality Assurance Programs (QAP) Pty Ltd.
The Trial ran for 18 months in the period 2005-2007.In total, 53 practices based in urban, rural or remote locations across three states (South Australia, New South Wales and Victoria) participated in Trial.In all, 23 practices were randomised to the control group, which had pathology testing performed by their local laboratory; and 30 practices were randomised to the intervention group, which conducted pathology testing by PoCT.A total of 4968 patients (1958 control and 3010 intervention) participated in the Trial, all of whom had either diabetes, hyperlipidaemia and/or were taking anticoagulation therapy (warfarin).Full details on the Trial methodology, rationale, recruitment and baseline patient characteristics have been reported elsewhere 8 .
The following point-of-care (PoC) tests were performed by intervention practices: haemoglobin A1c (HbA1c) and urine albumin:creatinine ratio (ACR) on patients with diabetes; total cholesterol, triglyceride and high density lipoprotein (HDL) cholesterol on patients with hyperlipidaemia; and international normalised ratio (INR) on patients on warfarin therapy.

Ethics approval
The PoCT in General Practice Trial was approved by five relevant independent Australian Human Research Ethics Committees.The Trial is registered with the Australian Clinical Trial Registry, Number 12612605000272695.A spiral bound set of 12 laminated A3 posters was also prepared for the Trial.The posters were designed to provide device operators with a user-friendly, step-by-step guide on how to perform a patient test, a quality control test and a quality assurance test for each PoC test.An example of the posters is shown (Fig1).The poster concept had previously proven very popular among Aboriginal health professionals trained as device operators for the QAAMS PoCT program for diabetes management in Aboriginal medical services 3 .

Design of point-of-care testing training resource package
The CD ROM contained an electronic copy of the training manual and poster set and associated spreadsheets.

Point-of-care testing training workshops
Five Initial Training Workshops were held for intervention practices from each geographic region between August and October 2005 at three centres -Adelaide, South Australia (to cater mainly for urban practices), Bendigo, Victoria (for rural practices) and Dubbo, New South Wales (for remote practices).These workshops were each of two-day's duration.They combined theoretical training in the principles and practice of PoCT, the tests and devices used and quality management procedures with 'hands on' practical training delivered in interactive small group sessions, with one scientist from the PoCT Device and/or RCPA QAP groups working with a maximum of three general practice staff.The training workshops also included sessions on PoCT accreditation and Trial protocol implementation provided by the Trial management and evaluation team.
Archived at Flinders University: dspace.flinders.edu.auAt the completion of training, practice staff were required to correctly answer a set of written competency questions related to each test and to perform a PoC test in the presence of a scientist.The written questions related to the clinical utility of tests and issues of technical or procedural nature covered in the theory section of training.The PoC test performed by the operator as part of initial competency assessment was either a quality control or quality assurance sample both of which had a set target value and an acceptable limit of performance around that target.Device operators were required to obtain a value for the sample tested that was within the acceptable limits to pass their practical competency.
On successful completion of this written and practical assessment a competency certificate was presented to each practice staff member.
From January to August 2006, further training sessions were held for intervention practices that required new practice staff to be trained.These were generally conducted as 'one-on-one' sessions between the new operator and a CPS scientist and were often held on-site at the practice concerned.

Assessment of point-of-care testing training methods
At the completion of each initial training workshop, practice staff completed a satisfaction survey from the PoCT device group designed to assess the general appropriateness of the methods employed and the overall effectiveness of PoCT training provided by the device group.The survey contained six questions, with respondents rating their level of satisfaction with each question according to a Likert scale 9 .
At the completion of the Trial in February 2007, device operators completed a further questionnaire devised by the Trial evaluation team (from the Trial management group) in which they were asked to indicate how strongly they agreed/disagreed with eight statements concerning their satisfaction with aspects of PoCT using a visual analogue scale (VAS).A VAS is a horizontal line, 10 cm in length, with the left end labelled as 'strongly disagree' and the right end labelled as 'strongly agree'.Participants mark on the line the point they feel represents their level of satisfaction and the distance from the left end of the line to the mark is measured.A larger value indicates a higher level of agreement with the statement.The median score (and inter-quartile range) was calculated for each statement using Statistical Program for Social Sciences v 13.0 (SPSS Inc; Chicago, IL, USA).Scores were split by geographic region to ascertain whether there was any difference in satisfaction levels with PoCT between urban, rural and remote practices.

Practice staff trained as point-of-care testing device operators
At the conclusion of the initial training workshop series 60 device operators from 31 practices completed training and received competency certificates, having passed both written and practical assessments.
From January to August 2006, a number of further training requests from practices were received due to either: (i) new nursing staff needing to be trained because their previous device operator had left the practice; or (ii) the practice requested training for further nursing staff in addition to their existing operators.In total 20 staff (1 urban, 6 rural and 13 remote) from 12 practices (1 urban, 4 rural and 7 remote) underwent training and competency certification during this period.
Thus, in total, 80 device operators comprising 74 practice staff and six GPs from 31 practices were trained and received competency certificates as part of Trial.All 6 GPs who completed training as device operators were from rural (n = 2) and remote (4) locations.A summary of these practices and operators, split by geographic region, is provided (Table 1).

Satisfaction questionnaires and general feedback from point-of-care testing device operators
Initial training: In response to the question 'How would you describe the level of instruction on the theory of PoCT?' 56/57 respondents (95%) stated the level of instruction was appropriate.For the questions 'Do you feel the posters will be useful for you on a day to day basis?' and 'Was working in small groups with a supporting scientist for practical instructions satisfactory as a training method?',there was unanimous agreement among the 60 respondents that the posters were useful and the mode of practical training was satisfactory.The quality and appropriateness of the PoCT training resources was rated as either good or excellent by all 60 respondents (8 and 52, respectively).Figure 2 shows the responses of device operators to the questions 'How would you rate the support given to you during training by the scientific team?' and 'How would you rate the PoCT device training workshop overall?'The response rate to these questions was 85% and 78%, respectively.
Refresher training: From the feedback sessions conducted as part of refresher training, PoCT operators were in general agreement that: • the poster set provided for practices remained useful, its size and clarity was good • PoCT devices had generally proven robust during the Trial • patients were generally satisfied with the PoC testing process and they felt a greater sense of ownership of their pathology results.

End-of-trial satisfaction survey:
The responses by device operators to the Trial evaluation team's post-Trial satisfaction questionnaire are displayed (Table 3).In total, 90% (55/61) of the active operators at the end of the Trial completed the questionnaire.Overall, a high level of satisfaction with PoCT was reported by operators across all geographic regions.Device operators from remote practices had the highest satisfaction levels for quality of training, usefulness of the training manual, ease of use of devices, confidence in the accuracy of PoCT results and preference for PoCT over laboratory testing.Rural device operators showed lowest levels of satisfaction for quality of training, competency in using the devices and confidence in the accuracy of PoCT results.Urban device operators recorded the highest levels of satisfaction for usefulness of posters and competency in using the devices, but a lower degree of satisfaction with ease of device use and preference for PoCT over laboratory testing.The usefulness of the posters for conducting PoCT achieved the highest satisfaction rating among operators from all three geographic regions.Responses from device operators from all geographic regions to satisfaction surveys conducted by the PoCT Device Group (after the initial training workshop series) and by the Trial evaluation team (at the conclusion of the Trial) indicated that there was widespread acceptance of training methods and their effectiveness.The acceptance of the poster sets as a user-friendly and practical training resource was confirmed by the very high satisfaction levels reported among device operators.
Point-of-care testing has particular application for rural and especially remote health practices where access to laboratory services may be limited, turnaround time for receipt of results may be delayed and rate of patient return for follow up of laboratory results may be low 20 .The positive responses from remote operators relating to ease of use of PoCT devices, confidence in the accuracy of results and preference for PoCT over laboratory testing augurs well for the acceptance of PoCT in this geographic sector.However the challenges faced by the rural and remote health services, particularly in relation to sustaining both workforce capacity and the ability to conduct PoCT are considerable.High rates of staff turnover are a constant problem for rural and remote services.In this study, 16 (84%) of the 19 device operators who left the study were from rural (6) or remote (10) practices; 5 (63%) of the 8 device operators who resigned were from these geographic regions.Four (80%) of the 5 practice withdrawals were from rural and remote areas.During the live phase of the Trial, on-going device operator competency was assessed by the routine conduct of quality testing.The results of this testing have been reported elsewhere and indicate that device operators conducted PoCT to a generally acceptable standard 21 .
In conclusion, the methods established for the implementation and delivery of training and competency assessment were appropriate for the PoCT in General Practice Trial.However, findings from this study have emphasised the greater need for training and support for PoCT services in rural and remote practices and the necessity for more flexible training options to address much higher rates of staff turnover.
Archived at Flinders University: dspace.flinders.edu.au An education and training resource package was developed for the Trial consisting of a training manual, a set of A3 laminated posters and a CD ROM.This resource package was written by the PoCT Device Manager (MS), with input from scientists from the PoCT Device Group and the RCPA QAP.The 162 page colour training manual contained both an introduction section and a test-specific section.The introduction covered the theory of PoCT and discussed the importance of quality management of PoCT devices, in particular the principles behind internal quality control and external quality assurance testing.The test-specific section covered the practical side of performing each of the PoC tests measured in the trial (HbA1c, urine ACR, total cholesterol, triglyceride, HDL cholesterol and INR), and described each test systematically under the following common headings: • the clinical use of the PoC test • a description of the PoCT device used to measure the test • a description of the PoCT method • how to perform the PoC test on a patient sample • how to test a quality control sample • how to test a quality assurance sample • how to perform basic maintenance procedures on the PoCT device.The manual also contained an appendix which provided information on topics including: • selected papers on the clinical use of each of the PoC tests • selected papers on how the PoCT method compared with the laboratory method for each test • specifications of the PoCT devices used in the Trial • common error messages for each PoCT device.

Figure 1 :
Figure 1: Example of a poster showing how to conduct point-of-care testing for INR on a patient sample (with permission of the Australian Government Department of Health and Ageing).

Five refresher training workshops
covering all geographic regions were delivered during late August 2006, coinciding with the 12 month point of the live phase of the Trial, as a commitment to continuing education and training for device operators.Refresher training workshops included an overview of the Trial thus far from the PoCT device, trial management and RCPA QAP groups and a feedback session for device operators to discuss issues related to the use of the PoCT devices, quality management and trial management procedures.Only device operators who had undergone full training and received a competency certificate were able to conduct routine PoCT during the Trial.A competency register was maintained by the PoCT device group throughout the Trial.
device operators (3 urban, 6 rural and 10 remote) from 11 practices (3 urban, 2 rural and 6 remote) left during the life of the Trial.Of these, 8 device operators (3 urban, 2 rural and 3 remote) from 6 practices (3 urban, 1 rural and 2 remote) resigned for personal reasons.The remaining 11 device operators (4 rural and 7 remote, including one GP) left the Trial because their practice withdrew from the Trial.At the completion of the Trial, 61 device operators were still actively conducting PoCT at their practice.Of these active operators, 42 (69%) attended refresher training workshops held during the second half of 2006 (Table

1 Figure 2 :
Figure 2: Point-of-care testing device operator response to selected questions at the completion of initial training workshop series.
Maintaining and delivering staff training and the capacity to conduct quality-assessed PoCT can also be strained in the face of high staff turnover.It is noteworthy that 11 (92%) of the 12 practices that requested additional training in the 9-12 month period post the initial training workshop series were from rural (4) or remote(7) practices.Further 19 (95%) of the 20 new staff trained were from rural(6) or remote(13) practices.These observations highlight that not only do rural and remote practices have a greater need for training and support compared to their urban counterparts but also require more flexible training options to cater for much higher rates of staff turnover.Regular face-to-face training by a primary training team may be impractical and too expensive for a remote general practice experiencing high staff turnover.Access to electronic training resources such as DVD or web-based training through web-streamed video presentations (such as those being currently used in the QAAMS Program[5][6] ) may provide alternative training options for this geographic niche.

Table 2 : Practices and device operators undertaking refresher training by geographic region
Archived at Flinders University: dspace.flinders.edu.au

Table 3 : Point-of-care testing device operator responses to end-of-trial satisfaction questionnaire by geographic location Characteristic Location median (IQ range) Urban n = 13 Rural n = 19 Remote n = 23
PoCT, Point-of-care testing.