Can verbal autopsy improve the diagnostic accuracy of Physicians in Shanghai? Application of SmartVA

Accurate data on causes of death are essential for policy makers and public health experts to plan appropriate health policies and interventions to improve population health. Whereas approximately 30% deaths of Shanghai either occur at home or are not medically attended; the recorded cause of death in these cases may be less reliable than for a hospital death. Verbal Autopsy is a practical method that can help determine causes of death in regions where medical records are insucient or unavailable. In this research, the smart VA tool was adopted to assign the cause of death of home deaths and to validate the accuracy and eciency of the tool, the results were compared with routine practice to ascertain the value, if any, of incorporating VA into the diagnostic practices of physician in Shanghai certifying the cause of home deaths.


Results
Overall CSMF accuracy improved from 0.93, based on the initial diagnosis, to 0.96 after the application of SmartVA. The misclassi cation of the initial diagnosis compared to that from the MRR. 86.3% of the initial diagnoses assigned the correct CODs, after the VA investigation, 90.5% of the post-VA diagnosis assigned the correct CODs.

Conclusions
Although Shanghai has an established and well-functioning CRVS system, SmartVA for Physicians contributed to an improvement in the accuracy of death certi cation. In addition, SmartVA may be a useful tool for inferring some special causes of death, such as those CODs classi ed as undetermined.

Background
Accurate data on causes of death are essential for policy makers and public health experts to plan appropriate health policies and interventions to improve population health. In Shanghai, a mega-city with a population of 24 million, the vital statistics registration system registers almost all deaths of the resident (Hukou) population [1] . Deaths that occur in hospital are certi ed by the attending doctor. For deaths that are not medically attended, the family members of the deceased present to Community Health Centers (CHC), usually with available medical documentation such as discharge summaries, medical records and laboratory test results, and the CHC doctor on duty reviews the records and issues a death certi cate. Nonetheless, approximately 30% of all deaths still either occur at home or are not medically attended; the recorded cause of death in these cases may be less reliable than for a hospital death.
Verbal Autopsy (VA) is a practical method that can help determine causes of death in regions where most deaths occur at home or medical certi cation practices are unreliable because only outpatient medical records are available [2,3] . Automated VA tool involves a questionnaire to ascertain signs and symptoms preceding death and the application of a diagnostic algorithm to interpret the interview data in order to assign the most probable cause. To enable physicians to immediately review the outputs of a verbal autopsy, SmartVA for Physicians has been developed. This innovation produces a summary of all endorsed symptoms, as reported by family members, providing more information for the certifying physician to determine the cause of death (COD) for each decedent [4,5] .
In order to improve the quality of death certi cation in Shanghai (especially for deaths occurring outside health facilities) and to reduce uncertainty about data quality among policymakers, SmartVA for Physicians was applied to a sample of community deaths to inform the certi cation of COD. The ndings were compared with routine practice to ascertain the value, if any, of incorporating VA into the diagnostic practices of physician in Shanghai certifying the cause of home deaths.

Methods
SmartVA Auto-Analyze package SmartVA Auto-Analyze is a software package that builds on SmartVA Analyze [6][7][8] and includes the Population Health Metrics Research Consortium (PHMRC) shortened VA questionnaire, the Open Data Kit (ODK) suite for data collection, and the modi ed Tariff 2.0 algorithm for computer analysis of the verbal autopsy interview responses. SmartVA Auto-Analyse was developed to be used by physicians in real time, produces a list of up to three most likely causes of death at the individual level and is commonly referred to as Smart VA for Physicians (for brevity, we use the term Smart VA in this paper).
A local VA team with support from the University of Melbourne Technical Team trained 32 CHC doctors as VA interviewers. User manuals with detailed instructions and Standard Operating Procedures (SOP) were introduced during the training and were made available to Shanghai Municipal Center for Disease Control and Prevention (SCDC). The interviewers also received training on standard death certi cation practices as well as training on operating Android based tablets to conduct SmartVA interviews and troubleshooting. After the training, the interviewers underwent supervised eld practice to ensure they had the requisite skills and knowledge of the concepts.
A local IT technical/ data management staff member, with support from the University of Melbourne technical team, installed Open Data Kit Collect software, and the electronic SmartVA questionnaire and media le onto tablets, and SmartVA-Auto-Analyse onto computers, to prepare all devices for SmartVA data collection.

The intervention
This pilot study took place in 16 community health centers (CHC) from 3 districts selected to represent urban, suburb, and urban-suburb areas in Shanghai.
From December 2017 to June 2018, each home death was investigated by a trained CHC doctor on duty. Doctors identi ed an appropriate respondent (>18 years of age, cared for the deceased, or most familiar with the deceased) from among the family members reporting the death, requested their consent to participate in the pilot study and interviewed them. At the end of the interview, the CHC doctor assigned a provisional diagnosis with underlying cause of death (UCOD), and then ran the SmartVA-Auto-Analyse program for each death to receive up to 3 suggested UCOD. The doctor then reviewed the provisional UCOD, the cause(s) from SmartVA-Auto-Analyse and the endorsed symptoms and used this information to assign the nal UCOD (Fig 1).

Fig1
. The eld implementation procedures for this study. Legend: Initial Diagnosis represents the provisional diagnosis assigned by the CHC doctors after reviewing the deceased's medical history. Tariff1/2/3 represent 3 suggested UCODs assigned from the SmartVA tool. Final UCOD represents the UCOD of the CHC doctor combined all the messages and assigned a nal UCOD to ll out the death certi cate. The medical record of all deaths was also reviewed by a Medical Record Review (MRR) team who reviewed the medical records where available, the MRR UCOD was assigned by the team.

Monitoring and evaluation
Each CHC doctor was asked to complete a Microsoft Excel spreadsheet ("COD information form" box in Figure 1) with the data on demographics, provisional COD, Tariff CODs and the post-VA UCOD for each case. This spreadsheet was submitted to SCDC by the CHC doctor at the end of each month, for monitoring the progress and quality of the study implementation. After 6 months of data collection, a program manager from SCDC integrated the data from all 16 CHCs and performed further analysis.

Gold Standard (GS) UCOD and data analysis
The medical records for all deaths for which a VA was carried out in these 3 districts during same period were carefully evaluated an independent Medical Record Review (MRR) team. The MRR team assigned each death a 'gold standard' (GS) UCOD, based on the GS criteria for classi cation of COD developed by the PHMRC [4,9,10] . Under these criteria GS1 refers to the highest standard of diagnostic accuracy of UCOD and GS4, the lowest ( Figure 1). For example, the GS1 criteria of lung cancer is histological con rmation whereas the GS4 of that is unsupported clinical diagnosis.
Causes of death from both the SmartVA sample and the UCOD from the MRR were transformed to the SmartVA cause list for comparison, given this was an abbreviated list of causes as appropriate for VA. The statistical analysis was performed using R 3.6 software.

Results
Based on our rigorous respondent choosing criteria and the on-duty status of participating doctors in this research. A total of 619 (37.6%) out of 1648 community deaths were included in this study. Of the 619 deaths for which a SmartVA interview was conducted, 570 also had available medical records for further MRR.
There was no signi cant difference in the age and sex composition between the 570 deaths and the total number of CHC deaths in same area and time period (Table 1) (p=0.862 for sex) (p=0.135 for age). The majority of deaths were in the 70 years and above age-group. For all home deaths in the 16 selected areas in 2017, the causes of death were classi ed into causes from the Smart VA cause list (see from Additional File 1).
The cause speci c mortality fractions (CSMF) for all the home deaths in the 16 CHCs in 2017 and the CSMFs for the collected VA results from this study, conducted in the same 16 regions in 2018, showed a similar COD distribution (see Table 2).
From the MRR of the deaths analysed by SmartVA for Physicians, stroke was the leading cause of death, accounting for 17.8% of deaths, followed by other cancers (15.6%) and ischaemic heart disease, lung cancer and chronic respiratory diseases, accounting for 12.6%, 12.1% and 11.2% of deaths, respectively. All other causes accounted for less than 5% of the CODs. Broadly speaking, causes of death diagnosed by SmartVA were similar to those before implementation, with only slight changes in the ranking of causes of death (Table 2). The concordance between the initial diagnosis and MRR UCOD and post-VA diagnosis and MRR UCOD was assessed using Chance Corrected Concordance (CCC) ( Table 3). This metric evaluates the average sensitivity weighted by the number of diseases, which makes the study comparable with other studies. CSMF accuracy was also evaluated by measuring the absolute deviation of the initial diagnosis CSMFs and the SmartVA CSFMs from MRR UCOD. The closer this value is to 1, the higher the concordance of the results [7,11,12] .
The sensitivity and positive predictive value (PPV) were both high for the top six CODs. PPV was low for diabetes and other infectious diseases, which shows that a small number of original causes which were not diabetes or other infectious diseases were reallocated to other diseases after the VA investigation.
Although not dramatic, overall CSMF accuracy improved from 0.93, based on the initial diagnosis, to 0.96 after the application of SmartVA (see Table 3). As for speci c causes, the CCCs for the top six causes of death (stroke, other cancers, IHD, lung cancer, CRD, and stomach cancer, accounting for over 75% of deaths) all increased to higher than 0.90 after VA-assisted diagnosis. Detailed metrics are shown in Tables 3-5. Some of the CODs have noticeable increases in CCC, especially other non-communicable diseases and other infectious diseases, after Smart VA. Of interest is the change in CCC of other cardiovascular diseases and falls; both decreased after the VA investigation. This may be because many deaths may initially, mistakenly be attributed to cardiovascular diseases, a COD that may not be supported after systematic and comprehensive investigations.  Table 4 shows the misclassi cation matrix of the initial diagnosis compared to that from the MRR. The values in the yellow boxes are the CODs that were correctly classi ed after initial diagnosis; 86.3% (492/570) of the initial diagnoses assigned the correct CODs. Some of the misclassi cations were adjusted after the VA investigation (Tables 4, 5), leading to correct classi cation of 90.5% (516/570) of the post-VA diagnosis.
From the results of the initial diagnosis before VA investigation, other cardiovascular diseases and other infectious diseases were more likely to be misassigned to other causes. Among these, nearly one third of other cardiovascular diseases were misclassi ed as stroke (6/17) (Table 4).
In the CSMFs after SmartVA, misclassi cation was reduced, except for the categories of other cardiovascular diseases and falls ( Table 3). As before, other cardiovascular diseases were often misclassi ed as stroke, which accounted for 41% (7/17) of that diagnosis from MRR ( Table 5).
Analysis of the VA results with SmartVA Auto analyse resulted in the causes of 53 deaths, or just under 10% of the sample, being reclassi ed from their initially assigned causes. This was particularly the case for chronic kidney diseases (CKD), chronic respiratory diseases, and cirrhosis, as well as falls, ischemic heart diseases (IHD), other CVD (Cardiovascular Diseases), and undetermined causes. Among these corrected CODs, CKD (5), cirrhosis (4) and IHD (3) were most often correctly classi ed after VA.
Among the 53 changed CODs, only 22.6% (12/53) of causes were assigned correctly before VA (Table 6), whereas 67.9% (36/53) of the new CODs were assigned correctly according to MRR (Table 7). The number of misclassi ed conditions, compared to MRR, were also reduced. In the 53 changed cases, all the causes assigned before VA (Table 6) had a high degree of misclassi cation, except for cirrhosis and falls. After VA, misclassi cation was greatly reduced, except for falls and other cardiovascular diseases (Table 7). Four undetermined deaths were reallocated to other diagnoses. (The concordance and validation metrics of comparing initial/VA diagnosis with MRR results can be seen from additional le 2 and 3).
For the 53 deaths with changed UCOD after SmartVA, the initially assigned CODs were distributed randomly between the 15 diseases/causes. In the initially assigned CODs, no cases were assigned to leukemia/lymphoma or other cancers, however, according to the MRR results other cancers should be the 3 rd leading COD. Chronic kidney disease, undetermined, other injuries, pneumonia, cervical cancer and esophageal cancer were also not on the MRR disease list, while the CHC doctors assigned them as UCODs after initial diagnosis. This suggests a need for greater care when assigning these diseases as UCODs (see Table 8 for details).
With the assistance of SmartVA, the majority of deaths were assigned to other NCD (20.8%), chronic respiratory diseases (17.0%), and other cancers (17.0%).
Though a small degree of misclassi cation persisted, the post-VA diagnosis of the UCOD agreed more closely with the reference standard (MRR) than the initial diagnosis (see Table 8).

Discussion
The cases examined in this study are representative of the home deaths in Shanghai, as the age-sex distribution and cause-speci c mortality fraction had the same distribution as that seen in the home death database in Shanghai.
Although Shanghai has an established and well-functioning CRVS system, SmartVA for Physicians contributed to an improvement in the accuracy of death certi cation, as demonstrated by the increase in overall cause of death CSMF accuracy, from 0.93 to 0.96 after the introduction of SmartVA. In addition, SmartVA may be a useful tool for inferring some special causes of death, such as those CODs classi ed as undetermined, which while less of an issue for Shanghai, is a common problem in civil registration systems worldwide [13][14][15][16] . During this study, four undetermined CODs were revised after the SmartVA investigation. With the help of this tool, the Shanghai CRVS system could reduce the fraction of undetermined deaths.
In the 53 cases with changed COD after the VA investigation, of the top three causes of death, the most misclassi ed causes were chronic respiratory diseases (17%), other non-communicable disease (17%), as well as other cancers (13.2%), suggesting that for causes such as these a more careful examination of the available medical history may be needed by the certi er before assigning the UCOD.
The improvement in COD data following SmartVA in this study could be attributed to two factors. Firstly, the standardised operating procedures (SOPs) for COD assignment that were followed during the SmartVA investigation ensured a structured and consistent approach, leading to more accurate COD assignment. Secondly, the SmartVA procedure has systematic and comprehensive questions, which can ensure that all relevant medical information regarding the decedent's conditions are captured. Considering all of the diseases and conditions the individual may have suffered from together enabled the certi er to assign a more accurate UCOD.
As Shanghai is highly developed with a relatively advanced CRVS system, using SmartVA as a supplementary tool can work well as shown by this pilot study. However, as a routine procedure, the use of SmartVA may not result in a signi cant improvement in death registration data in the Shanghai system, nor be a cost-effective way to do so. Moreover, Shanghai CHC doctors' routine work already comprises checking and correcting MCCOD data, including re-interviewing the family of the deceased. In contrast, for other cities in China, especially in the remote areas in the west, that don't have a well-functioning death registration system, SmartVA may be more bene cial.
This SmartVA study has some limitations. Firstly, the SmartVA tool, especially the cause list, is not perfectly suited to the actual mortality fractions observed in Shanghai [17] . For example, liver cancer is not on the SmartVA cause list, and therefore the program does not assign it as a COD to any deaths, whereas liver cancer accounted for more than 2.6% of deaths in Shanghai in 2018. Secondly, while standard VA procedures are adapted to conditions where there is no medical history available for out of facility deaths [7,[18][19][20] ,in this study, the VA investigations were conducted after the certi er reviewed the previous outpatient medical histories of the deceased. This may cause some misconceptions based on the results of the outpatient records. Thirdly, in this study in Shanghai, the SmartVA procedure was implemented in 16 communities. Because the number of deaths in these communities was not very high and may not be representative of the whole population, further research should be done to determine the generalisability of SmartVA for Physicians, before it can be extended to all districts and counties.
In the future, SCDC will plan to adjust the work ow and operation speci cations of the application of SmartVA in Shanghai, to maximize the effectiveness of VA in improving cause of death inference and further reduce the proportion of undetermined causes of death, through selective application and integration into the existing CRVS system. In addition, SCDC shall also consider using Smart VA to identify the possible causes of death in cases with incomplete medical history information.

Conclusion
Although Shanghai has an established and well-functioning CRVS system, SmartVA for Physicians contributed to an improvement in the accuracy of death certi cation. In addition, SmartVA may be a useful tool for inferring some special causes of death, such as those CODs classi ed as undetermined.

Declarations
Ethics approval and consent to participate Ethics approval was obtained from Shanghai CDC (Ethics ID: 2016-28) and The University of Melbourne Ethics Committees (Ethics ID: 1647517.1.1). All participants were provided with a participant information sheet and consent forms in the local language.

Consent for publication
Written informed consent for publication was obtained.

Availability of data and materials
The data that support the ndings of this study are available from Shanghai Municipal Center for Disease Control and Prevention (Shanghai CDC) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Shanghai CDC.
Competing interests