Interactive Visualization Applications in Population Health and Health Services Research: Systematic Scoping Review

Background Simple visualizations in health research data, such as scatter plots, heat maps, and bar charts, typically present relationships between 2 variables. Interactive visualization methods allow for multiple related facets such as numerous risk factors to be studied simultaneously, leading to data insights through exploring trends and patterns from complex big health care data. The technique presents a powerful tool that can be used in combination with statistical analysis for knowledge discovery, hypothesis generation and testing, and decision support. Objective The primary objective of this scoping review is to describe and summarize the evidence of interactive visualization applications, methods, and tools being used in population health and health services research (HSR) and their subdomains in the last 15 years, from January 1, 2005, to March 30, 2019. Our secondary objective is to describe the use cases, metrics, frameworks used, settings, target audience, goals, and co-design of applications. Methods We adapted standard scoping review guidelines with a peer-reviewed search strategy: 2 independent researchers at each stage of screening and abstraction, with a third independent researcher to arbitrate conflicts and validate findings. A comprehensive abstraction platform was built to capture the data from diverse bodies of literature, primarily from the computer science and health care sectors. After screening 11,310 articles, we present findings from 56 applications from interrelated areas of population health and HSR, as well as their subdomains such as epidemiologic surveillance, health resource planning, access, and use and costs among diverse clinical and demographic populations. Results In this companion review to our earlier systematic synthesis of the literature on visual analytics applications, we present findings in 6 major themes of interactive visualization applications developed for 8 major problem categories. We found a wide application of interactive visualization methods, the major ones being epidemiologic surveillance for infectious disease, resource planning, health service monitoring and quality, and studying medication use patterns. The data sources included mostly secondary administrative and electronic medical record data. In addition, at least two-thirds of the applications involved participatory co-design approaches while introducing a distinct category, embedded research, within co-design initiatives. These applications were in response to an identified need for data-driven insights into knowledge generation and decision support. We further discuss the opportunities stemming from the use of interactive visualization methods in studying global health; inequities, including social determinants of health; and other related areas. We also allude to the challenges in the uptake of these methods. Conclusions Visualization in health has strong historical roots, with an upward trend in the use of these methods in population health and HSR. Such applications are being fast used by academic and health care agencies for knowledge discovery, hypotheses generation, and decision support. International Registered Report Identifier (IRRID) RR2-10.2196/14019


Background
As digital medicine advances, visualization applications in population health increasingly provide ways for researchers and practitioners to explore and communicate findings [1], supporting knowledge discovery from disparate large data sources [2]. Visual analytics (VA) has been defined as the "science of analytics reasoning facilitated by visual interfaces" [3], and it is an interdisciplinary field combining visualization, statistical analysis, and advanced analytics such as machine learning and cognitive sciences [4]. A specific approach within VA is the use of interactive visualization, which Ola and Sedig [2] define as computational tools that store, process, and visually represent data, to facilitate interactive exploration. Interactive visualization increases the potential for big data use in health care by supporting sense making, knowledge discovery, and hypothesis generation [2,5]. Simple visualizations such as scatter plots, heat maps, and bar charts typically present 2 facets of the data, displaying attributes and relationships between 2 variables such as a disease condition and risk factors. Interactive visualization methods allow for presentation of multiple related facets such as risk factors to be studied simultaneously, leading to insights through exploring trends and patterns [2,5].
Population health research involves the study of data related to health outcomes and determinants of population health [6,7], whereas health services research (HSR) studies the health system in relation to access, quality, costs, and patient outcomes [8,9]. Both fields involve the analysis of large secondary data sources such as clinical databases, administrative data sets, and electronic medical records (EMRs) [10][11][12]. In a prior review, we summarized evidence on VA applications in these interrelated fields of health care [13]; this review complements it by reviewing the evidence on interactive visualization applications in population health and HSR.
Recent systematic reviews have summarized visualization methods in varied areas of health care. Among the most cited reviews, the study by West et al [1] synthesized literature on the use of visualization approaches for exploratory analysis of electronic health records (EHRs). Similarly, another well-cited review by Carrol et al [14] summarized the literature on visualization and analytics tools used in infectious disease epidemiology, particularly in relation to geographic information systems (GIS), molecular epidemiology, and social network analysis methods. Islam et al [15] offered a comprehensive view on data mining and theoretical approaches in health care. Wu et al [16] summarized evidence on visualization and analytic technologies for characterizing evaluation methods in health informatics, an area primarily concerned with clinical care. The most recent related review by Chung et al [17] focused on visual approaches in mental health care policy and systems. To our knowledge, interactive visualization applications have not been studied as a body of literature separate from data visualization and VA; hence, this review is the first systematic synthesis on the subject.

Rationale for a Companion Review
This companion review is our second synthesis of literature on visualization and analytics tools, techniques, and approaches in population health and HSR. Our first publication focused on VA methods in these areas, where we offered an updated definition of VA in health care as "an approach, method, or application for analytic reasoning, exploration, knowledge discovery, and sense making of complex data, using one or more interactive visual interfaces, employing analytic and visual engines" [13]. As part of VA applications, analytic engines involve advanced machine learning, database querying, and manipulation.
Interactive visualization applications typically engage a front-end visual engine such as Tableau [18], Qlik [19], and PowerBI [20]. Although all VA methods carry a visualization component, which may or may not be interactive, interactive visualization applications typically do not involve or report an analytic component. Hence, this companion review on interactive visualization applications illustrates the state of evidence in population health and HSR, focusing on contemporary methods, approaches, tools, and co-design from real-world use cases. This review will be helpful for health care researchers, practitioners, and decision-makers to understand and adopt visualization-based data analysis.

Objectives
The primary objective of this scoping review is to describe and summarize the evidence on interactive visualization applications, methods, and tools being used in population health and HSR and their subdomains in the last 15 years, from January 1, 2005, to March 30, 2019. Our secondary objective is to describe the use cases, metrics, frameworks used, settings, target audience, goals, and co-design of applications.

Review Methodology and Protocol
Scoping reviews outline the size and scope of available literature and identify the quality and extent of research evidence [21]. We briefly describe the methodological processes relevant to the second part of the review in this section, whereas further details can be found in the published protocol [22]. We primarily followed the guidance provided by the Joanna Briggs Institute [23], as well as the framework for conducting scoping reviews described by Arksey and O'Malley [24], with improvements suggested by Levac et al [25] and Peters et al [26], while using the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) checklist provided by Tricco et al [27] for reporting. The major steps were as follows: determining the research question, identifying relevant studies, abstracting data, and summarizing and reporting the results. The operational concepts and definitions are presented in Textbox 1.

Concepts and definitions
• Population health, adapted from Kindig and Stoddart [6] and Kindig [7] • "The health outcomes of a group of individuals, including the distribution of such outcomes within the group," includes "health outcomes, patterns of health determinants, and policies and interventions that link these two" • Health services research, adapted from the Canadian Institutes of Health Research [8] and National Libraries of Medicine filters for health services research [28] • Research with the "goal of improving the efficiency and effectiveness of health professionals and the health care system" • Access to services • Encompasses the partnership of health workers, patients, and designers who aspire toward change, depending on shared knowledge to achieve "better outcomes or improved efficiency" • Whether any participatory approach toward co-design was reported by the authors

Eligibility Criteria
Eligible articles included peer-reviewed published journal and full conference papers in English related to use cases of interactive visualization in population health and HSR. We included articles on spatiotemporal visualization but excluded articles presenting cartographic methods and tools for GIS because these were outside the scope of the research objectives. Similarly, we did not include articles on human-computer interaction, user design, and articles without a use case. Non-peer-reviewed work such as editorials, conference abstracts, and short articles were excluded. The eligibility criteria are presented in Textboxes 2 and 3.

Sources of Evidence and Search Strategy
The search strategy, its conceptualization, and steps for operationalization are detailed in the review protocol [22]. The search was externally peer reviewed using the Peer Review of Electronic Search Strategies Guideline [33] and included an extensive list of search terms and their variants to cover all related concepts of population health, HSR, visualization, analytics, and interactivity [22]. The 6 databases searched, their platforms, and results are summarized in Table 1. We further hand searched 10 relevant journals, in addition to internet searches [22]. We used the Covidence (Veritas Health Innovation Ltd) platform for screening citations [34] and EndNote (Clarivate) for reference management [35].

Data Charting and Synthesis of Results
In all, 2 independent reviewers screened articles at each stage of the review, including title and abstract screening, full-text screening, and data abstraction. A third reviewer acted as an arbiter in case of conflicts and for validating the data abstracted for their content and level of detail.
The data abstraction encompassed the major concepts in 6 major themes: (1) study characteristics (country, problem category, settings, and target audience), (2) tools and techniques used, (3) data type and visualization methods, (4) domains of population health and HSR, (5) innovation of the application and its current availability and use, and (6) if the application was co-designed with the target audience.

Selection of Sources and Presentation of Results
We identified 14,099 articles from the 6 databases searched. Given the varied sources of the articles, we adapted the method described by Bramer et al [36] for removing duplicate references using EndNote X9 [35]. Among the 14,099 articles, we considered major citation details and identified, double-checked, and removed 2078 (14.74%) duplicates, comparing the title, identifiers, publication platforms, and abstracts. From the remaining 12,021 articles, another 711 (5.91%) duplicates were removed after importing into Covidence [34]. We excluded 96% (10,819/11,310) of the references during the title and abstract screening and 89% (435/491) of the articles during the full-text screening. We did not find additional articles from reference lists of recent systematic and narrative reviews, hand searches of individual journals, and internet searches. Hence, of the initially identified 14,099 articles, we have summarized 56 (0.39%) for reporting in this review. The reasons for exclusion during the full-text-screening are detailed in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram ( Figure 1), whereas the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews reporting checklist is presented in Multimedia Appendix 1.
We have also summarized our results in a visual format using a publicly accessible Tableau dashboard, a screenshot of which is presented in Figure 2 [37]. The abstracted data and complete workbook are available to support replication, adaptation, and further analysis. Operational concepts for each category and reported theme are detailed in the Methods section (Textbox 1).

Application's Analytic Capability, Goal, and Frameworks Used
There was overlap in the analytic capability of the tools with applications capable of descriptive analytics (53/56, 95%), predictive analytics (4/56, 7%), and visual exploration of complex data sets (37/56, 66%). Regarding the goal of the visualization application, there was overlap between knowledge discovery (56/56, 100%) and decision support (47/56, 84%). Of the 56 articles, 6 (11%) used a framework in their methods for developing the application. These frameworks are summarized in Table 4. Multimedia Appendix 3  lists the analytic capability and goals of each application. Table 4. Articles mentioning the use of methodological frameworks (N=6).

Methodological frameworks used in developing interactive visualization applications Author and year
Display principles for visual monitoring by Few et al [94] Alibrahim et al (2014) [38] Ten guidelines by Kelleher and Wagener [95] Bieh-Zimmert et al (2013) [45] Followed the Omaha System [96] Monsen et al (2015) [69] Visualization principles (overview, zoom and filter, and details on demand) based on theories from Shneiderman [97] and Chen [98] Ratwani et al (2015) [80] For developing the web app, the 5-stage user-centered design model described by Kinzie

Applications in Current Use and Public Availability
Most of the applications were mentioned as being currently available and in use (31/56, 55%). Related to public access, a third of the applications were available to the public (18/56, 32%). There were applications using free or open source tools (18/56, 32%) and those using proprietary tools (19/56, 34%), or the tools were not mentioned (18/56, 32%).

Significance of the Review
Data visualization in health has a lengthy history going back to the influential work of John Snow and Florence Nightingale in the 19th century. The field of interactive visualization has developed in parallel with computing power and the availability of large, complex health care data sets for diverse audiences such as clinicians, public health researchers, practitioners, and decision-makers [1,41], with considerable progress made in design methodologies [14]. Our review is a novel synthesis and summary of the literature from a vast body of research that had not previously been covered.
In this methodological review, we aim to capture the current state of knowledge and evidence on the topic of interactive visualization applications in population health and HSR, distinguishing them from conventional graphical presentations in health care and the related field of VA. We explored areas in population health and HSR to ascertain where these techniques have been used and identified trends and opportunities for the use of these applications.
As population health and health services researchers and practitioners, our perspective and interest in pursuing this research question were based on developing an in-depth understanding of the state of evidence on the use of visualization-based approaches for big health care data analyses. We anticipated that the review would help diverse audiences in population health and HSR learn from practical applications, inform future research endeavors, and help introduce the analytic method to researchers and students. We discuss our findings in this section with these overarching aims, contrasting the findings from previous reviews in other areas of health care using visualization approaches.

Gaps and Opportunities for Application Co-design
Data visualization aims to convey information at a glance, although it assumes that the audience has expertise and visual literacy on the subject matter [101]. In their review of visualization-based applications in infectious disease epidemiology, Carrol et al [14] summarize the audience's information needs and learning behavior and point to 3 important barriers to relaying information to target audiences: (1) time constraints, (2) prior knowledge, and (3) cognitive load [14]. Hence, the design process is imperative for an effective application that allows the user to successfully understand the presented data. Various methodologies outlining effective design requirements and experiences from stakeholders to create new products and solutions have been explored [102][103][104]. For our scoping review, we opted to use the term co-design, which is more commonly used in health care literature, as opposed to design thinking and other related terms [13].
In this review, we found that at least two-thirds of the applications involved co-design approaches, involving stakeholders for developing interactive visualization applications. This was in contrast to a smaller proportion of co-designed VA applications (18%), which were mostly prototypes developed by and for data scientists at academic centers [13]. In line with this finding, more than half of the interactive visualization applications were developed in-house within health care organizations. We termed these initiatives embedded research as part of co-designed applications to indicate that these were initiated within the organizations in response to an identified data-driven need for knowledge generation and decision support. We could not find such applications in the VA literature [13]. This indicates an important trend because participatory design and development in health has proven to be a key element in better viability and uptake in planning and implementation of services [105,106].
Notably, a third of the articles in this review did not mention a co-design method, which could be due to authors either opting to omit it or because these were covered elsewhere. We recommend that future research indicate whether the application used co-design approaches. It is important to describe the context in adequate detail to appreciate stakeholder needs, experience, and satisfaction. Furthermore, to map and present methods in sufficient detail, we suggest using established frameworks such as the Munzner Nested Process [107] or Design Thinking for Visualization [108] as reporting tools.

Contrasting Interactive Visualization and VA Applications
Through our recent work in studying visualization methods and applications in population health and HSR, we establish that the fields of interactive visualization and VA share communities of practice, methods, and approaches, but they are conceptually separate with important differences. We highlight the major ones here.
We found that interactive visualization applications were initiated by and targeted at researchers and practitioners within government health care organizations tasked with health services delivery, planning, and policy advice. In contrast, most of the VA applications were from and developed for data scientists [13]. In addition, most interactive visualization applications were developed using front-end engines, especially proprietary tools not requiring an advanced knowledge of coding [13]. Most VA methods and applications were prototypes developed using different combinations of tools, with a very small number using proprietary software [13]. Related to theoretical or conceptual frameworks, VA applications offered 13 different frameworks, whereas we could not identify any of these in this review of interactive visualization applications. However, the latter applications mentioned the use of frameworks at different stages of developing the applications. VA applications also expressly mentioned statistical and machine learning techniques as part of the analytic engine, whereas interactive visualization applications mostly used simple descriptive aggregative techniques. In another distinction, most VA applications were prototypes, whereas most interactive visualization applications were developed for knowledge generation and decision support [13].
As population health and HSR are overlapping concepts, many articles in both reviews overlapped with their foci, methods, and the metrics studied. Among the VA articles, almost all (54/55, 98%) had a population health focus, whereas a third (18/55, 33%) were on HSR. There was a smaller overlap among the interactive visualization applications, with approximately two-thirds (38/56, 68%) focusing on population health and approximately half (29/56, 52%) on HSR.
The categories of problems have important similarities and variations with epidemiologic surveillance for infectious disease being the major category that the applications targeted (38% for both VA and interactive visualization). The next problem categories for VA applications were medical record pattern identification (20/55, 36%), population health monitoring (9/55, 16%), and health system resource planning (2/55, 4%). For interactive visualization applications, these included resources and services monitoring or planning (12/56, 21%), health service monitoring or quality (5/56, 9%), and medication use patterns (5/56, 9%).
Comparing tools in current use, about a third (21/55, 38%) of the VA applications were in use at the time of publication, whereas others were either not available or were prototypes. Moreover, a few (7/56, 13%) applications were accessible for public use, while less than a third were developed using free open source tools (13/56, 24%). Among the interactive visualization applications, more than half (31/56, 55%) were mentioned as being in current use, whereas about a third (18/56, 32%) were available to the public, and the same proportion were developed using free or open source tools. There was a greater proportion of use of proprietary tools (19/56, 34%) for interactive visualization applications compared only a 10th of VA applications (5/55, 10%).
The trend for the use of visual presentations was toward the use of different maps in both applications. Choropleth maps were the most frequently used for interactive visualizations (13/56, 24%), followed by thematic maps (10/56, 18%), event timelines (7/56, 13%), and network maps (4/56, 7%). VA applications showed a similar trend with thematic maps (17/55, 31%), timelines (8/55, 15%), and heat or choropleth maps (6/55, 11%). This corresponds to the findings of the review by Chung et al [17] on visualization methods in the area of mental health systems, which indicated that the most common means of presenting data was through maps [17].
Because of the differences in the methods involved in developing the applications, software tools varied greatly. VA tools were a mix of software tools used for the analytic and visual engines, whereas interactive visualization applications reported visual engines alone. However, there were still similarities in the use of tools. Tableau was the most frequently reported tool for interactive visualization applications (7/56, 13%), followed by D3.JS (5/56, 9%); ArcGIS and Instant Atlas (3/56, 5% each); and R/R-Shiny, Open Street Map, Google Maps API, SQL, and Java-based applications (2/56, 4% each). The most common tools found for VA applications were R-based tools (7/55, 13%), followed by D3.JS (4/55, 7%); SQL (4/55, 7%), Java-based tools (3/55, 5%); and Python-based tools, HTML 5, or Google Maps API (2/55, 4% each). Front-end visual engines such as Tableau were used by only 1 VA application in combination with Weka as the analytic engine.
Finally, an issue that we identified in both our reviews was the lack of reporting detail in the articles, which is important for the replicability and adaptation of the methods used in developing applications. We suggest using part of the VA Reporting Checklist that we presented in our previous work on VA, particularly around the details on the visualization engine for the standard reporting of interactive visualization applications [13].

Recent Trends of Using Interactive Visualization Methods
Our results showed that thematic mapping, including choropleth maps, was the most common visual presentation across all problem categories of population health and HSR. This was particularly the case for epidemiologic monitoring and surveillance. The recently created COVID-19 dashboards fall into the same category of applications [109]. Mapping also surfaced as a popular method for health resource monitoring, particularly for the planning of health care services [41,56,62,64,67,73,77,78,80,87,110,111].
Among different conditions of interest, a significant number of applications were developed for studying trends in cancer [43,44,52,61,71,88]. Being a worldwide population health issue, the greater use of interactive visualization methods in cancer could be due to the availability of dedicated registries and secondary administrative data [88]. In global health, applications focused on surveillance of communicable diseases [47], outreach campaigns [64], methods to examine health inequalities [53], and effects on health from global climate change [61]. In HSR, 6 applications directly or indirectly highlighted inequities in health, particularly in regard to effective planning and advising policy [43,53,63,70,82,89]. There was 1 article examining social determinants of health in HIV [63].
As two-thirds of the applications were focused on the visual exploration of complex data sets, this indicated a clear trend toward the use of this technique for exploratory analyses. Although most applications were meant for descriptive analytics and visual exploration of complex data sets, of the 56 applications, 4 (7%) were also capable of predictive analytics [39,68,71,93]. The methodological frameworks that were applied to developing the applications pertained to visual monitoring [38], level of detail in visual presentation [80], use of scientific publication visualizations [45], information management [69], user-centered web-based applications [83], and user interaction [91].

Opportunities for Future Applications and Research
Experts highlight the preference of researchers for interactive graphics to facilitate data exploration and abstraction, and they suggest greater, varied learning opportunities from the use of interactive visualization tools [14].
In comparison with standard, traditional statistical analyses, interactive visualization techniques can play an important complementary role through knowledge generation as well as establishing associations and causality. Interactive visualization methods enable a data discourse, leading to in-depth data-driven insights, while having the advantage of improved perception with reducing cognitive load [2,5]. This interplay of direct data manipulation and analysis allows simultaneous study of trends and patterns in the analytic process, while formulating and testing hypotheses [13,112]. Furthermore, these methods are considered apt for studying correlations in high-dimensional data with a large number of time points [112]. This translates into a powerful technique for using big health care data, allowing a deep exploratory dive without an a priori hypothesis to identify data-driven trends and patterns.
In this review, although we observed various applications of interactive visualization, we found limited evidence of its use in global health. Given the massive open access data sets available from agencies such as the World Bank and the World Health Organization (WHO), research can focus on studying a plethora of population health and HSR indicators [113,114]. The WHO's Global Health Observatory provides population health-related data and statistics from 194 member states, particularly on nutrition, virological surveillance, workforce, and health systems, whereas the World Bank's open data repository features macroeconomic and social indicators such as gender and aid effectiveness. The methods can be helpful in ecologic studies, such as those comparing indicators across and within nations.
Related to this is another major opportunity for the use of interactive visualization in studying inequities, especially those rooted in social determinants of health. Although the social determinants of health have become a major focus for investigating structural inequities, we found only 1 article examining related aspects in the HIV sector [63]. Social determinants of health are defined by the WHO as "conditions in which people are born, grow, live, work and age...shaped by the distribution of money, power and resources at global, national and local levels" [115]. This is especially relevant for investigating structural inequities related to issues of access and use based on race, gender, disability, income distribution, and indigenous populations [116,117]. Taking Canada's example, investigating proximal factors for health among indigenous populations is one of the priority areas for improving health care [116]. Furthermore, high-quality Canadian data can be used to investigate inequities to better understand gaps in access and use of services by underserved populations. This can be done through national administrative data sources such as the Canadian National Ambulatory Care Reporting System, Discharge Abstract Database, and Hospital Morbidity Database, which store data for emergency and ambulatory care [118], as well as hospital inpatient discharges and day surgery [119].
Another major opportunity comes from the extension of using multiple data sources for studying patient journeys and care pathways. With the increasing use of EMR and EHR technologies, especially in primary care, there is an opportunity for researching patient populations along the continuum of care. Another such example is from the United Kingdom's Clinical Practice Research Datalink database, which forms the largest collection of anonymized primary care patient records [120]. In Ontario, Canada, the Electronic Medical Record Administrative Data Linked Database offers high-quality linked data for exploring trends and patterns in care and its provision with the advantage of capturing quality of care measures involving prescriptions and investigations [121].
In a recent review on visualization approaches for supporting mental health systems and policy research, Chung et al [17] indicate that there is a gap in studies that influence policy. Although policy was not our main area of focus for this review, the work indicates that there is an opportunity for informing and advising policy based on the use of big data, especially in the important area of mental health services.
Although the potential for the use of interactive visualization tools for bringing together disparate data sources is valued, there are related concerns for data interpretation, quality, accuracy, and handling [1,14]. Meeting the needs of diverse users and interdisciplinary teams as well as promoting the understanding of visual approaches are 2 related and important challenges to be cognizant of [1,14]. Researchers indicate that understanding the value of these techniques among health care organizations and public health agencies is key to realizing the potential of these methods regarding decision support [17].

Implications and Value-add From the Review
Our work is unique in several respects. Complementing our work on VA applications in population health and HSR [13], this review amalgamates the findings from studies on interactive visualization applications, while delineating the literature to construct a holistic picture on the use of visualization approaches in these areas of health care. Interactive visualization is an increasingly popular method, especially for embedded research within health care organizations. Although traditional statistical methods inform causality and associations of various conditions, interactive visualization presents a complementary opportunity for knowledge discovery, hypotheses generation, and decision support using big health care data.
As a novel method, we present findings from both our scoping reviews on VA and interactive visualization in a dynamic, interactive, and visual format using Tableau dashboards [37,122]. In the interest of greater transparency and replicability, we provide the abstraction database with relevant fields for adaptation and further analysis [37].
We highlight opportunities in areas of research that could benefit from visualization-based methods to promote the understanding and uptake of the methods among the communities of research and practice. This work would also prove useful in further developing visualization-related analytic methods.

Limitations
Although there are several important limitations that we are cognizant of in reporting this review, we made extensive efforts to identify relevant literature, delineate the body of literature on interactive visualization applications, incorporated rigor in our methods through all stages, and went through extensive steps toward validation to present our findings.
We cast a wide net in our literature search covering 6 databases, published the study protocol, and had our search strategy externally peer reviewed. However, we may have missed relevant literature residing in subject-specific databases such as those of digital art, mathematics, geography, and computer science. In addition, our review was limited to peer-reviewed literature from journal articles and full conference papers, and we focused on health care-related databases. We did not include CINAHL and ACM Digital Library because we could not find unique articles, separate from MEDLINE and IEEE Xplore, during the pilot searches.
In addition, in line with the first review on VA, this literature synthesis is limited to articles published between January 1, 2005, and March 30, 2019. We situate and report the review within the same period as the one on VA applications to complement and contrast findings. Many COVID-19-related visualization products that surfaced later are not included in this review for both reasons of feasibility and the subject being extremely specialized and falling under outbreak analytics. However, we plan a rapid analysis of COVID-19-related visual products later in the year. Although we describe interactive visualization applications, we allude only briefly to the challenges in the use of these methods because this was beyond the scope of this review.

Conclusions
Visualization in health has strong historical roots. This systematic literature synthesis informs the state of evidence and trends toward the use of interactive visualization methods in the important and interrelated areas of population health and HSR. We note a significant trend in the use of interactive visualization applications being used in health care organizations, which we term embedded research. Such applications are being used by academic and health care agencies for knowledge discovery and generation, as well as decision support. Many of these applications have been co-designed with relevant stakeholders. Although we found a wide array of applications in different subdomains of population health and health services, there are multiple opportunities for the use of these methods in investigating global-and national-level indicators and social determinants of health, as well as constructing patient journeys for a holistic picture of the continuum of care.