Cardiac Rhythm Device Identification Using Neural Networks

Objectives This paper reports the development, validation, and public availability of a new neural network-based system which attempts to identify the manufacturer and even the model group of a pacemaker or defibrillator from a chest radiograph. Background Medical staff often need to determine the model of a pacemaker or defibrillator (cardiac rhythm device) quickly and accurately. Current approaches involve comparing a device’s radiographic appearance with a manual flow chart. Methods In this study, radiographic images of 1,676 devices, comprising 45 models from 5 manufacturers were extracted. A convolutional neural network was developed to classify the images, using a training set of 1,451 images. The testing set contained an additional 225 images consisting of 5 examples of each model. The network’s ability to identify the manufacturer of a device was compared with that of cardiologists, using a published flowchart. Results The neural network was 99.6% (95% confidence interval [CI]: 97.5% to 100.0%) accurate in identifying the manufacturer of a device from a radiograph and 96.4% (95% CI: 93.1% to 98.5%) accurate in identifying the model group. Among 5 cardiologists who used the flowchart, median identification of manufacturer accuracy was 72.0% (range 62.2% to 88.9%), and model group identification was not possible. The network’s ability to identify the manufacturer of the devices was significantly superior to that of all the cardiologists (p < 0.0001 compared with the median human identification; p < 0.0001 compared with the best human identification). Conclusions A neural network can accurately identify the manufacturer and even model group of a cardiac rhythm device from a radiograph and exceeds human performance. This system may speed up the diagnosis and treatment of patients with cardiac rhythm devices, and it is publicly accessible online.

the corresponding communication equipment to the bedside.
Unless they have access to the records of the implanting hospital or the patient can tell them, staff must use a process of trial and error to identify the manufacturer, which causes uncertainty and leads to delays which can be medically harmful.
Experts can sometimes distinguish among devices from a chest radiograph, and algorithms are available to assist with this. However, expertise or confidence in using the algorithm are not widespread, and even with the best available algorithm, identification is not perfect. Indeed, up to 80% of physicians report having "frequently" had difficulties identifying devices (3).
The most recent algorithm for visual discrimination among devices shown on a chest radiograph is 8 years of age (3) and therefore does not include current devices. Even at that time, the study authors reported only 90% accuracy in identifying the manufacturer.
The present study reports the development, validation, and public availability of a new neural network-based system which attempts to identify the manufacturer and even the model group of a device by using a chest radiograph. The study was designed in 3 phases consisting of data collection, development of the neural network, and assessment of the network.

METHODS
Development of the neural network was divided into 2 stages. Stage 1 involved selecting the optimal network design. Stage 2 involved training the "final" model, which is then assessed using the unseen "test set", allowing a comparison with humans.  (4). From each radiographic image, a square region of interest slightly larger than the device was extracted. This region maximized the signalto-noise ratio for the network and guaranteed anonymization. These cropped images were then resized to 224 Â 224 pixels and normalized to yield pixel values between 0 and 1. It was noted during extraction that, in several cases, when a manufacturer introduced a new model, there was no detectible change on the radiograph. This may represent purely a change in software or an indistinguishable replacements of parts; therefore, models with identical appearance were placed in "model groups." The first step was to randomly allocate 5 images from each of the 45 classes to be kept aside as the final "test set." This would not be shown to the network at any stage in its training and would only be used once when reporting its final accuracy.
The remaining "training set" was used to train the network at 2 different stages. The first stage was to decide which underlying network to use (including structural features such as the number and size of layers) and details of how the training process would run (including the avidity with which synapses are adjusted, termed the "learning rate"). All tested neural networks were convolutional neural networks which contain neurons that learn to recognize specific features within their own "visual fields." These networks are organized in a hierarchical structure akin to the human optic cortex and excel at solving image classification problems (5)(6)(7)(8)(9). The second stage was the detailed process of adjusting the weights (akin to the synapses in a biological neural network) so that the job of classifying pacemakers could be performed. Both stages used the training set but in different ways.
For the first stage ("network design") ( Figure 1), each candidate neural network design was assessed by its ability to learn from 75% of the training set and correctly make predictions for the remaining 25% of the training set. This was done 4 times, so that all of the training set could participate in turn in both roles. This process is termed "4-fold cross-validation" (unrelated to the final testing which is performed using a completely separate test set).
The second stage ("final model training") begins with the neural network design chosen by the first stage. This starts with a fresh neural network with no prior exposure to device images. The network is then trained from the entire training set, resulting in the final trained network.
Finally, this final network is exposed for the first time to the "test set," which has been kept separate    the training set. The study flow chart is shown in Figure 1.   To demonstrate the utility of saliency mapping,  Figure 3A shows a Medtronic Advisa pacemaker. Readers are invited to identify which other panel ( Figure 3B, 3C, or 3D) is also an Advisa and to ask how they would teach others to differentiate between the 2 model groups on a radiograph.
Once they have done this, readers are invited to examine   In Figure 3, most humans and, indeed most expert cardiologists, have difficulty in differentiating between the 2 models of pacemaker. However, not only does the neural network accurately distinguish between them but the saliency map highlights the feature that distinguishes them most clearly. Moreover, once this salient feature is pointed out to humans (Figure 4), they now find it straightforward to make the distinction.
Online Appendix 3 shows saliency maps for every model group. Studying these may assist clinicians by using it to sharpen their eye for cardiac device identification.  Table 2 shows markedly different levels of performance across different neural network architectures. Of the neural network designs that launched machine learning into prominence, VGGNet is the only 1 still in common use because of its elegant simplicity yet relatively good performance. Surprisingly, however, its performance on this task was poor.
This may reflect the necessity for more advanced neural network components, such as "residual connections" and "dimensionality reduction" through "1 Â 1 convolutions." ResNet was the design that pioneered residual connections, which constitute a method that makes available the original image to all subsequent layers of the network rather than only the first layer. Separately, GoogLeNet Inception was the pioneer for condensing information between layers using 1 Â 1 convolutions so that the network's sophistication was less constrained by the handling of large numbers of parameters.
The design that performed best, however, was Xception, the 1 that made extensive use of both of these innovations, residual connections and 1 Â 1 convolutions.
STUDY LIMITATIONS. This neural network recognizes devices in common use in our region of the United Kingdom. However, it will not be able to identify devices not listed in Table 1.  Saliency plots from the neural network can help guide us where to look. The answer to the question in Figure 2 is C. Saliency plots reveal that the network is focusing on a feature present in the AT500s (red circles), which is absent in the Advisas. Having this pointed out by the network now makes it easy to return to Figure 3 and correctly categorize them. These example images also demonstrate the neural network's ability to deal with dramatic differences in image quality, radiography, penetration, and orientation.
Howard et al. Deployment from "bench to bedside" can be difficult with neural networks, because the large processing power needed is not always present at the point of care. This was mitigated by providing an online Web portal that anyone could use (19).

CONCLUSIONS
This study demonstrates a convolutional neural network is able to accurately identify the manufacturer and model of a cardiac rhythm device from a radiograph. Furthermore, its performance significantly exceeds that of a cardiologist using a flowchart approach. This neural network is free to try (19).