Performance Assessment of Communicable Disease Surveillance in Disasters: A Systematic Review – PLOS Currents Disasters

Background: This study aimed to identify the indices and frameworks that have been used to assess the performance of communicable disease surveillance (CDS) in response to disasters and other emergencies, including infectious disease outbreaks. Method: In this systematic review, PubMed, Google Scholar, Scopus, ScienceDirect, ProQuest databases and grey literature were searched until the end of 2013. All retrieved titles were examined in accordance with inclusion criteria. Abstracts of the relevant titles were reviewed and eligible abstracts were included in a list for data abstraction. Finally, the study variables were extracted. Results: Sixteen articles and one book were found relevant to our study objectives. In these articles, 31 criteria and 35 indicators were used or suggested for the assessment/evaluation of the performance of surveillance systems in disasters. The Centers for Disease Control (CDC) updated guidelines for the evaluation of public health surveillance systems were the most widely used. Conclusion: Despite the importance of performance assessment in improving CDS in response to disasters, there is a lack of clear and accepted frameworks. There is also no agreement on the use of existing criteria and indices. The only relevant framework is the CDC guideline, which is a common framework for assessing public health surveillance systems as a whole. There is an urgent need to develop appropriate frameworks, criteria, and indices for specifically assessing the performance of CDS in response to disasters and other emergencies, including infectious diseases outbreaks.


Introduction
Disasters, whether natural or man-made, are common events worldwide 1 . These events kill and injure people, destroy health facilities, and disrupt health systems and lifelines 2 . Disasters displace populations and interrupt routine communicable disease management (CDM) programs, including surveillance systems and immunization programs 3 . In the conditions that develop after disasters, populations are very vulnerable to outbreaks of communicable diseases, and there are many examples of communicable disease outbreaks after disasters 4 . These include cholera in Haiti after the 2010 earthquake, malaria after floods in Brazil, dengue fever after floods in the Dominican Republic, and acute diarrhea after the 2005 Pakistan earthquake 5 . Accordingly, CDM has become one of the most important components of health care programs in disaster response management 4 . The most urgent task in CDM is the establishment of a surveillance system (SS) for timely detection of any increase in disease occurrence and the introduction of rapid control measures. Almost, all of health systems establish a SS in response to disasters 6 , 7 .
The establishment of an effective SS in a disaster or emergency setting is a complex and difficult process requiring a large number of resources including manpower, equipment, and administrative facilities. To determine whether CDS meets target goals, it is necessary to measure the performance of these SS 8 .
An appropriate and unique assessment system that is established according to disaster characteristics is required for the monitoring of SS. An effective assessment system should include appropriate indices and should be conducted in a correct Introduction Disasters, whether natural or man-made, are common events worldwide 1 . These events kill and injure people, destroy health facilities, and disrupt health systems and lifelines 2 . Disasters displace populations and interrupt routine communicable disease management (CDM) programs, including surveillance systems and immunization programs 3 . In the conditions that develop after disasters, populations are very vulnerable to outbreaks of communicable diseases, and there are many examples of communicable disease outbreaks after disasters 4 . These include cholera in Haiti after the 2010 earthquake, malaria after floods in Brazil, dengue fever after floods in the Dominican Republic, and acute diarrhea after the 2005 Pakistan earthquake 5 . Accordingly, CDM has become one of the most important components of health care programs in disaster response management 4 . The most urgent task in CDM is the establishment of a surveillance system (SS) for timely detection of any increase in disease occurrence and the introduction of rapid control measures. Almost, all of health systems establish a SS in response to disasters 6 , 7 .
The establishment of an effective SS in a disaster or emergency setting is a complex and difficult process requiring a large number of resources including manpower, equipment, and administrative facilities. To determine whether CDS meets target goals, it is necessary to measure the performance of these SS 8 .
An appropriate and unique assessment system that is established according to disaster characteristics is required for the monitoring of SS. An effective assessment system should include appropriate indices and should be conducted in a correct method. Such measures can improve health system response as well 9 .They can define shortcomings and provide a guide for future-response success.
Management experts believe that "what cannot be measured cannot be managed." Therefore, the measurement of performance is considered one of the most important components of efficacy and effectiveness 10 . In addition, according to current management knowledge, lack of an appropriate performance assessment is an important sign of weakness in a program or organization 9 .
With regard to the importance of performance assessment and its role in improving the performance of communicable diseases SS in response to disasters, there are 2 main questions: What kind of performance assessment frameworks, indices, and criteria currently exist for CDS systems in response to disasters and other emergencies, including infectious diseases outbreaks? What are the characteristics of these frameworks, indices, and criteria? The aim of this systematic literature review was to find the answers to these questions.

Materials and Methods
Before starting this systematic review, a written review manual was developed for searching strategy, study inclusion, and exclusion criteria. The study was then conducted according to this review process.We have reported our review according to the PRISMA guideline.

Research questions
The review aimed to answer the following questions: 1. What kind of performance assessment frameworks currently exist for CDS systems in disasters and other emergencies, including infectious diseases outbreaks? 2. What criteria and indices are used in the performance assessment of CDS systems in response to disasters and emergencies?
3. What are the characteristics of the articles on the performance assessment of CDS systems in terms of article type, study approach (qualitative or quantitative), study setting, results, hazard type, geographical location, and country or affiliation of first/corresponding authors?

Definitions
For the purpose of this study a "communicable disease" is "a disease caused by living agents as infectious agents, or their products, that can be transmitted from 1 patient to another" (Synonym: infectious disease) 11 .
In the literature, "surveillance" is "the ongoing, systematic collection, analysis, interpretation, and dissemination of data regarding communicable disease for use in developing preventive actions to reduce morbidity, mortality, and to improve health" 12 .
A "disaster" is "a man-made or natural event that disrupts the affected community functions and results in widespread losses that are greater than community resources" 13 . An "emergency" is "a condition that needs urgent attention and could become a disaster if not managed effectively." In this study, both natural and man-made disasters and emergencies were included.
Smith defines "performance assessment" as "a systematic process that seeks to monitor, evaluate, and communicate the extent to which various aspects of a system meet its key objectives" 14 .

Inclusion criteria
The following criteria were used to select relevant studies: Articles that were published in peer-reviewed journals and had addressed the performance assessment of CDS in response to disasters and emergencies (as defined above).
Articles in any format including editorials, case reports, reviews, and original research.

Exclusion criteria
All non-English articles, unless an English abstract was available.
Papers with abstracts that were not accessible or did not include enough information for extraction of the study variables.

Search strategy (Data sources and literature search)
We searched 5 electronic databases, including PubMed, Scopus, Google Scholar, ScienceDirect, and ProQuest. The databases method. Such measures can improve health system response as well 9 .They can define shortcomings and provide a guide for future-response success.
Management experts believe that "what cannot be measured cannot be managed." Therefore, the measurement of performance is considered one of the most important components of efficacy and effectiveness 10 . In addition, according to current management knowledge, lack of an appropriate performance assessment is an important sign of weakness in a program or organization 9 .
With regard to the importance of performance assessment and its role in improving the performance of communicable diseases SS in response to disasters, there are 2 main questions: What kind of performance assessment frameworks, indices, and criteria currently exist for CDS systems in response to disasters and other emergencies, including infectious diseases outbreaks? What are the characteristics of these frameworks, indices, and criteria? The aim of this systematic literature review was to find the answers to these questions.

Materials and Methods
Before starting this systematic review, a written review manual was developed for searching strategy, study inclusion, and exclusion criteria. The study was then conducted according to this review process.We have reported our review according to the PRISMA guideline.

Research questions
The review aimed to answer the following questions: 1. What kind of performance assessment frameworks currently exist for CDS systems in disasters and other emergencies, including infectious diseases outbreaks? 2. What criteria and indices are used in the performance assessment of CDS systems in response to disasters and emergencies?
3. What are the characteristics of the articles on the performance assessment of CDS systems in terms of article type, study approach (qualitative or quantitative), study setting, results, hazard type, geographical location, and country or affiliation of first/corresponding authors?

Definitions
For the purpose of this study a "communicable disease" is "a disease caused by living agents as infectious agents, or their products, that can be transmitted from 1 patient to another" (Synonym: infectious disease) 11 .
In the literature, "surveillance" is "the ongoing, systematic collection, analysis, interpretation, and dissemination of data regarding communicable disease for use in developing preventive actions to reduce morbidity, mortality, and to improve health" 12 .
A "disaster" is "a man-made or natural event that disrupts the affected community functions and results in widespread losses that are greater than community resources" 13 . An "emergency" is "a condition that needs urgent attention and could become a disaster if not managed effectively." In this study, both natural and man-made disasters and emergencies were included.
Smith defines "performance assessment" as "a systematic process that seeks to monitor, evaluate, and communicate the extent to which various aspects of a system meet its key objectives" 14 .

Inclusion criteria
The following criteria were used to select relevant studies: Articles that were published in peer-reviewed journals and had addressed the performance assessment of CDS in response to disasters and emergencies (as defined above).
Articles in any format including editorials, case reports, reviews, and original research.

Exclusion criteria
All non-English articles, unless an English abstract was available.
Papers with abstracts that were not accessible or did not include enough information for extraction of the study variables.

Search strategy (Data sources and literature search)
We searched 5 electronic databases, including PubMed, Scopus, Google Scholar, ScienceDirect, and ProQuest. The databases method. Such measures can improve health system response as well 9 .They can define shortcomings and provide a guide for future-response success.
Management experts believe that "what cannot be measured cannot be managed." Therefore, the measurement of performance is considered one of the most important components of efficacy and effectiveness 10 . In addition, according to current management knowledge, lack of an appropriate performance assessment is an important sign of weakness in a program or organization 9 .
With regard to the importance of performance assessment and its role in improving the performance of communicable diseases SS in response to disasters, there are 2 main questions: What kind of performance assessment frameworks, indices, and criteria currently exist for CDS systems in response to disasters and other emergencies, including infectious diseases outbreaks? What are the characteristics of these frameworks, indices, and criteria? The aim of this systematic literature review was to find the answers to these questions.

Materials and Methods
Before starting this systematic review, a written review manual was developed for searching strategy, study inclusion, and exclusion criteria. The study was then conducted according to this review process.We have reported our review according to the PRISMA guideline.

Research questions
The review aimed to answer the following questions: 1. What kind of performance assessment frameworks currently exist for CDS systems in disasters and other emergencies, including infectious diseases outbreaks? 2. What criteria and indices are used in the performance assessment of CDS systems in response to disasters and emergencies?
3. What are the characteristics of the articles on the performance assessment of CDS systems in terms of article type, study approach (qualitative or quantitative), study setting, results, hazard type, geographical location, and country or affiliation of first/corresponding authors?

Definitions
For the purpose of this study a "communicable disease" is "a disease caused by living agents as infectious agents, or their products, that can be transmitted from 1 patient to another" (Synonym: infectious disease) 11 .
In the literature, "surveillance" is "the ongoing, systematic collection, analysis, interpretation, and dissemination of data regarding communicable disease for use in developing preventive actions to reduce morbidity, mortality, and to improve health" 12 .
A "disaster" is "a man-made or natural event that disrupts the affected community functions and results in widespread losses that are greater than community resources" 13 . An "emergency" is "a condition that needs urgent attention and could become a disaster if not managed effectively." In this study, both natural and man-made disasters and emergencies were included.
Smith defines "performance assessment" as "a systematic process that seeks to monitor, evaluate, and communicate the extent to which various aspects of a system meet its key objectives" 14 .

Inclusion criteria
The following criteria were used to select relevant studies: Articles that were published in peer-reviewed journals and had addressed the performance assessment of CDS in response to disasters and emergencies (as defined above).
Articles in any format including editorials, case reports, reviews, and original research.

Exclusion criteria
All non-English articles, unless an English abstract was available.
Papers with abstracts that were not accessible or did not include enough information for extraction of the study variables.

Search strategy (Data sources and literature search)
We searched 5 electronic databases, including PubMed, Scopus, Google Scholar, ScienceDirect, and ProQuest. The databases were searched for articles published up to the end of 2013. In addition grey literature [1] was searched through the "New York academy of medicine grey literature reports" 15 . Websites of CDC and WHO searched for relevant guidelines. We also reviewed the reverences of retrieved studies to identify additional articles.
[1] -Grey literature definition: "That which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers." We chose key terms and developed a search strategy based on the National Library of Medicine "Medical Subjects Headings (MeSH)". To search the other databases, the PubMed search strategy was adopted. We limited our search to titles and abstracts of articles.

Study screening process
First, the selected key words were entered into the database search boxes, and the search was limited to abstracts and titles. The results of the key words search were reviewed by a member of the review team (JB). If the study met the inclusion criteria, it was included in the review. If there was any doubt about meeting the inclusion criteria a decision was made based on the consensus of the review team. Articles unrelated to the aim of the present study were excluded. The remaining titles were entered into an Excel spreadsheet and sorted. Duplicates were excluded. Next, the abstracts of the related titles were screened for their precise relevance to the aims of the present study. If an abstract met the inclusion criteria, it was included in the review. Abstracts that were not precisely relevant were excluded. The remaining papers were included in the review. The full texts of these articles were downloaded from the databases. If an article was not available free of charge, we paid for access. Two papers did not have full text that was accessible to us, and the study variables were not extractable from the abstracts, so they were excluded from the review.

Data analysis
The finally included papers were evaluated by a member of the review team (AA) using a data abstraction sheet developed by the research team. This data sheet included the study variables: name of the journal; name of the first author; number of authors; publication year; type of potential hazard; model/framework used for the CDS performance assessment; indices/criteria and tools used for the CDS performance assessment; the study approach; and the location of study. In the extraction of CDS performance assessment criteria, indices, and the study approach, our first priority was the authors' statement. If criteria, indices, and the study approach were mentioned in the article, the data were included in our data abstraction sheet. If not, the review team used a consensus approach to decide whether the data should be included.

Ethics and dissemination
Ethical approval was not required for this literature review.

Literature search
The initial search strategy resulted in a total of 3928 articles/documents (3902 resulted from database searching and 26 documents resulted from Grey literature and websites searching). Of these, 3698 titles did not fulfill the inclusion criteria and were excluded, leaving 230 articles/documents that were considered potentially relevant. These papers/documents were entered into an Excel spreadsheet and sorted alphabetically. Duplicates (93 titles) were discarded. In the second phase, the abstracts of the remaining articles/documents (137 titles) were examined. In this step, 114 irrelevant abstracts were excluded and 23 papers/documents were considered for analysis. Two articles full texts were not accessible and their abstracts were not informative enough thus excluded. Four potentially relevant documents were not accessible too. In total, 16 papers and one book were included in the final review list for data extraction. Figure 1 outlines the literature search and the study selection process.
were searched for articles published up to the end of 2013. In addition grey literature [1] was searched through the "New York academy of medicine grey literature reports" 15 . Websites of CDC and WHO searched for relevant guidelines. We also reviewed the reverences of retrieved studies to identify additional articles.
[1] -Grey literature definition: "That which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers." We chose key terms and developed a search strategy based on the National Library of Medicine "Medical Subjects Headings (MeSH)". To search the other databases, the PubMed search strategy was adopted. We limited our search to titles and abstracts of articles.

Study screening process
First, the selected key words were entered into the database search boxes, and the search was limited to abstracts and titles. The results of the key words search were reviewed by a member of the review team (JB). If the study met the inclusion criteria, it was included in the review. If there was any doubt about meeting the inclusion criteria a decision was made based on the consensus of the review team. Articles unrelated to the aim of the present study were excluded. The remaining titles were entered into an Excel spreadsheet and sorted. Duplicates were excluded. Next, the abstracts of the related titles were screened for their precise relevance to the aims of the present study. If an abstract met the inclusion criteria, it was included in the review. Abstracts that were not precisely relevant were excluded. The remaining papers were included in the review. The full texts of these articles were downloaded from the databases. If an article was not available free of charge, we paid for access. Two papers did not have full text that was accessible to us, and the study variables were not extractable from the abstracts, so they were excluded from the review.

Data analysis
The finally included papers were evaluated by a member of the review team (AA) using a data abstraction sheet developed by the research team. This data sheet included the study variables: name of the journal; name of the first author; number of authors; publication year; type of potential hazard; model/framework used for the CDS performance assessment; indices/criteria and tools used for the CDS performance assessment; the study approach; and the location of study. In the extraction of CDS performance assessment criteria, indices, and the study approach, our first priority was the authors' statement. If criteria, indices, and the study approach were mentioned in the article, the data were included in our data abstraction sheet. If not, the review team used a consensus approach to decide whether the data should be included.

Ethics and dissemination
Ethical approval was not required for this literature review.

Literature search
The initial search strategy resulted in a total of 3928 articles/documents (3902 resulted from database searching and 26 documents resulted from Grey literature and websites searching). Of these, 3698 titles did not fulfill the inclusion criteria and were excluded, leaving 230 articles/documents that were considered potentially relevant. These papers/documents were entered into an Excel spreadsheet and sorted alphabetically. Duplicates (93 titles) were discarded. In the second phase, the abstracts of the remaining articles/documents (137 titles) were examined. In this step, 114 irrelevant abstracts were excluded and 23 papers/documents were considered for analysis. Two articles full texts were not accessible and their abstracts were not informative enough thus excluded. Four potentially relevant documents were not accessible too. In total, 16 papers and one book were included in the final review list for data extraction. Figure 1 outlines the literature search and the study selection process.
were searched for articles published up to the end of 2013. In addition grey literature [1] was searched through the "New York academy of medicine grey literature reports" 15 . Websites of CDC and WHO searched for relevant guidelines. We also reviewed the reverences of retrieved studies to identify additional articles.
[1] -Grey literature definition: "That which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers." We chose key terms and developed a search strategy based on the National Library of Medicine "Medical Subjects Headings (MeSH)". To search the other databases, the PubMed search strategy was adopted. We limited our search to titles and abstracts of articles.

Study screening process
First, the selected key words were entered into the database search boxes, and the search was limited to abstracts and titles. The results of the key words search were reviewed by a member of the review team (JB). If the study met the inclusion criteria, it was included in the review. If there was any doubt about meeting the inclusion criteria a decision was made based on the consensus of the review team. Articles unrelated to the aim of the present study were excluded. The remaining titles were entered into an Excel spreadsheet and sorted. Duplicates were excluded. Next, the abstracts of the related titles were screened for their precise relevance to the aims of the present study. If an abstract met the inclusion criteria, it was included in the review. Abstracts that were not precisely relevant were excluded. The remaining papers were included in the review. The full texts of these articles were downloaded from the databases. If an article was not available free of charge, we paid for access. Two papers did not have full text that was accessible to us, and the study variables were not extractable from the abstracts, so they were excluded from the review.

Data analysis
The finally included papers were evaluated by a member of the review team (AA) using a data abstraction sheet developed by the research team. This data sheet included the study variables: name of the journal; name of the first author; number of authors; publication year; type of potential hazard; model/framework used for the CDS performance assessment; indices/criteria and tools used for the CDS performance assessment; the study approach; and the location of study. In the extraction of CDS performance assessment criteria, indices, and the study approach, our first priority was the authors' statement. If criteria, indices, and the study approach were mentioned in the article, the data were included in our data abstraction sheet. If not, the review team used a consensus approach to decide whether the data should be included.

Ethics and dissemination
Ethical approval was not required for this literature review.

Literature search
The initial search strategy resulted in a total of 3928 articles/documents (3902 resulted from database searching and 26 documents resulted from Grey literature and websites searching). Of these, 3698 titles did not fulfill the inclusion criteria and were excluded, leaving 230 articles/documents that were considered potentially relevant. These papers/documents were entered into an Excel spreadsheet and sorted alphabetically. Duplicates (93 titles) were discarded. In the second phase, the abstracts of the remaining articles/documents (137 titles) were examined. In this step, 114 irrelevant abstracts were excluded and 23 papers/documents were considered for analysis. Two articles full texts were not accessible and their abstracts were not informative enough thus excluded. Four potentially relevant documents were not accessible too. In total, 16 papers and one book were included in the final review list for data extraction. Figure 1 outlines the literature search and the study selection process.

Results of included studies
The 16 articles that were finally selected for review were divided into 5 groups according to the theme of the study. These 5 themes were: performance assessment of syndromic surveillance systems (31.2%); mortality/morbidity SS (25.0%); public health/disease surveillance (12.5%); the applications of cost analysis, efficacy, effectiveness, and usefulness in performance assessment of SS (25.0%); and the review of performance assessment indicators (6.2%).
The relevant book is about the communicable diseases control in emergencies, and it has a specific section for CDS in disasters.
In the reviewed articles and book, there was no specific performance assessment framework for SS in disasters. However, the CDC updated guidelines for public health surveillance system evaluation 12 was used exclusively in 3 studies. In the performance assessment of mortality, morbidity, and CDS systems 16 , 17 , 18 , 19 , the CDC guidelines were also used as part of the assessment 12 . The CDC guidelines are based on 9 criteria including simplicity, flexibility, data quality, sensitivity, positive predictive value (PPV), timeliness, acceptability, representativeness, and stability.
Of the CDC public health surveillance evaluation attributes, the most widely applied was timeliness, which was used in 7 studies. Flexibility was used in 5 studies, data quality in 4, simplicity in 3, stability in 3, and usefulness and representativeness in 2 studies. In all cases, timeliness, data quality, sensitivity, specificity, PPV, cost, and representativeness were calculated quantitatively. Flexibility, usefulness, simplicity, and acceptability were calculated in a qualitative manner. Stability was calculated both quantitatively and qualitatively.
In the reviewed book 20 10 indicators suggested for PA of SS including: zero reporting, completeness, timeliness, the number of cholera cases for which samples were confirmed by the laboratory, the number of malaria cases confirmed by blood smear date of onset of the first case, date of reporting using outbreak alert form, date of investigation, date of response. These findings are presented in Table 1.
Overall, in the 16 articles and one book that were included, 31 criteria/measures and 35 indicators were used or suggested for the assessment/evaluation of the performance of CDS systems in response to disasters and emergencies.

Results of included studies
The 16 articles that were finally selected for review were divided into 5 groups according to the theme of the study. These 5 themes were: performance assessment of syndromic surveillance systems (31.2%); mortality/morbidity SS (25.0%); public health/disease surveillance (12.5%); the applications of cost analysis, efficacy, effectiveness, and usefulness in performance assessment of SS (25.0%); and the review of performance assessment indicators (6.2%).
The relevant book is about the communicable diseases control in emergencies, and it has a specific section for CDS in disasters.
In the reviewed articles and book, there was no specific performance assessment framework for SS in disasters. However, the CDC updated guidelines for public health surveillance system evaluation 12 was used exclusively in 3 studies. In the performance assessment of mortality, morbidity, and CDS systems 16 , 17 , 18 , 19 , the CDC guidelines were also used as part of the assessment 12 . The CDC guidelines are based on 9 criteria including simplicity, flexibility, data quality, sensitivity, positive predictive value (PPV), timeliness, acceptability, representativeness, and stability.
Of the CDC public health surveillance evaluation attributes, the most widely applied was timeliness, which was used in 7 studies. Flexibility was used in 5 studies, data quality in 4, simplicity in 3, stability in 3, and usefulness and representativeness in 2 studies. In all cases, timeliness, data quality, sensitivity, specificity, PPV, cost, and representativeness were calculated quantitatively. Flexibility, usefulness, simplicity, and acceptability were calculated in a qualitative manner. Stability was calculated both quantitatively and qualitatively.
In the reviewed book 20 10 indicators suggested for PA of SS including: zero reporting, completeness, timeliness, the number of cholera cases for which samples were confirmed by the laboratory, the number of malaria cases confirmed by blood smear date of onset of the first case, date of reporting using outbreak alert form, date of investigation, date of response. These findings are presented in Table 1.
Overall, in the 16 articles and one book that were included, 31 criteria/measures and 35 indicators were used or suggested for the assessment/evaluation of the performance of CDS systems in response to disasters and emergencies.

Results of included studies
The 16 articles that were finally selected for review were divided into 5 groups according to the theme of the study. These 5 themes were: performance assessment of syndromic surveillance systems (31.2%); mortality/morbidity SS (25.0%); public health/disease surveillance (12.5%); the applications of cost analysis, efficacy, effectiveness, and usefulness in performance assessment of SS (25.0%); and the review of performance assessment indicators (6.2%).
The relevant book is about the communicable diseases control in emergencies, and it has a specific section for CDS in disasters.
In the reviewed articles and book, there was no specific performance assessment framework for SS in disasters. However, the CDC updated guidelines for public health surveillance system evaluation 12 was used exclusively in 3 studies. In the performance assessment of mortality, morbidity, and CDS systems 16 , 17 , 18 , 19 , the CDC guidelines were also used as part of the assessment 12 . The CDC guidelines are based on 9 criteria including simplicity, flexibility, data quality, sensitivity, positive predictive value (PPV), timeliness, acceptability, representativeness, and stability.
Of the CDC public health surveillance evaluation attributes, the most widely applied was timeliness, which was used in 7 studies. Flexibility was used in 5 studies, data quality in 4, simplicity in 3, stability in 3, and usefulness and representativeness in 2 studies. In all cases, timeliness, data quality, sensitivity, specificity, PPV, cost, and representativeness were calculated quantitatively. Flexibility, usefulness, simplicity, and acceptability were calculated in a qualitative manner. Stability was calculated both quantitatively and qualitatively.
In the reviewed book 20 10 indicators suggested for PA of SS including: zero reporting, completeness, timeliness, the number of cholera cases for which samples were confirmed by the laboratory, the number of malaria cases confirmed by blood smear date of onset of the first case, date of reporting using outbreak alert form, date of investigation, date of response. These findings are presented in Table 1.
Overall, in the 16 articles and one book that were included, 31 criteria/measures and 35 indicators were used or suggested for the assessment/evaluation of the performance of CDS systems in response to disasters and emergencies. LGC, Gillet JB, Vergeiner G, Krafft T, Brand H 24 CDC's updated guidelines for surveillance system evaluation(Incomplete) Validity, timeliness, sensitivity, specificity Pandemics Akgun D 36 Case report Water sanitation, immunization, organization of health services, public education Flood Conolly MA (Editor). 20 Zero reporting, completeness, timeliness, the number of cholera cases for which samples were confirmed by the laboratory, the number of ma All hazards Discussion The aims of this research were to review and extract the frameworks and indices that have been used for assessing the performance of CDS The first disaster-related article was published in 1945, but the first article related to the performance assessment of CDS in disasters was not Searching for the term "disaster" in PubMed resulted in more than 60,261 papers, but we could only find 16 papers about CDS systems perfor 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 . According to current management knowledge, lack of an appropriate performance assessment system is a symptom of weakness and disorde 9 . For this reason, there has been growing interest in the performance assessment of organizations and programs in recent years 21 . Although there are several performance assessment frameworks documented in the literature, in our review we were unable to find any esta 22 . This suggests that there is no consensus on the application of the CDC guidelines for the performance assessment of CDS in disasters. Furthermore, some economic assessment criteria have been suggested for use in the performance assessment of SS in disasters, but these c The selection of appropriate criteria and indexes for CDS assessment is essential to meaningful research. In the included studies, 31 criteria/in 17 , 22 , 23 . This is another indication that there is no clear agreement on the application of the CDC guidelines for the performance assessment of CDS Different definitions have been proposed for the CDC criteria, and different scientific methods (qualitative and quantitative) have been applied 23 . Choudhary defined timeliness as "the average time that a death is reported by a surveillance system" 17 , while Farag defined the same term as "the speed of data transmission between surveillance system steps" 18 . In the article by Josseran, sensitivity, specificity, and PPV were suggested as components of the effectiveness of surveillance systems 23 , while in another study the same indices were used for validity 24 . In the study by Farag, validity was based on data quality 18 . These discrepancies show that there is a lack of consistency in the assessment of this specific field. Despite the reasonably wide spread use of the CDC guidelines for systems performance assessment in response to disasters and emergencie 3 , 9 , 25 . Many scientists, universities, and research centers have identified this deficiency, and it has been a research priority for many years 1 , 2 , 26 . However, there is still an urgent need for the development of universal frameworks, methodologies, metrics, and criteria for the performance Limitations This review has some limitations. During the study period, the Web of Science (ISI) was not available, and was excluded from the search proc Another limitation was the identification of criteria and indices. We addressed this problem by using a consensus approach between reviewers Finally, there was limited access to the full text of some papers and four potentially relevant documents were not accessible. We were unable Conclusion Performance assessment is an integral component of the management of all organizations and the lack of performance assessment is consid Correspondence Ali Ardalan, MD, PhD. Department of Disaster Public Health, School of Public Health, Tehran University of Medical Science, Tehran, Iran. Email: aardalan@tums.ac.ir