Different scientific approaches are needed to generate stronger evidence for population health improvement

In a Perspective, Martin White and Jean Adams discuss challenges in the evaluation of interventions intended to benefit population health.


The challenge of multiple possibilities
Two examples of evaluations of low-agency population interventions recently published in PLOS Medicine, both focussing on the same fiscal policy, illustrate some of the challenges of this approach. Ryota Nakamura [4] and Juan Carlos Caro [5] and their respective colleagues both evaluated a revision to the taxation of sugar-sweetened beverages (SSBs) in Chile, introduced in October 2014, using retrospective, natural experimental designs. The tax reform resulted in an increase in the taxation of SSBs containing at least 6.25 mg/ml of sugar (from 13% to 18%) and a reduction in taxation of SSBs containing less than 6.25 mg/ml of sugar (from 13% to 10%). Both studies used a similar approach to evaluation; both approached the PLOS Medicine | https://doi.org/10.1371/journal.pmed.1002639 August 28, 2018 1 / 5 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 evaluation from an economic perspective, underpinned by theory relating to price elasticities of demand. Both used the same main data source (from a consumer panel representing a stratified random sample of households in urban areas of Chile). But each made different choices in how the data were analysed. Key similarities and differences between the two studies are summarised in Table 1. The two studies came to similar conclusions concerning the overall effectiveness of the policy (weak) and its distributional impacts (more effective in higher socioeconomic groups), but the details of their findings differed widely in magnitude and, in some cases, direction. For example, Nakamura  strategies, interventions that appear similar may also vary in detail. For example, a wide range of different approaches to SSB taxation have now been implemented internationally [6,7]. In addition, contextual factors are likely to lead to similar interventions achieving different effects in different settings. In these situations, meta-analysis is likely to be challenging; only outcomes common to studies can be analysed and important factors that may influence outcomes will be overlooked. Other, more conceptual, approaches to synthesis may be more appropriate [8].
Disconnected efforts to generate evidence on public health challenges can also be inefficient. In the case of the Chilean SSB tax evaluations, two teams with funding from two different countries led studies in parallel. With greater coordination (and, without doubt, no small measure of imagination and political will), the funding could have been pooled, enabling a wider range of questions to be answered instead of the same set twice. Whilst pragmatic evidence on the effect of SSB taxes on price and sales is now beginning to accumulate [9][10][11][12][13][14], fewer attempts appear to have been made to investigate wider questions on, for example, reformulation, total diet, health outcomes over different time frames, and the political processes that facilitate and hinder SSB taxes being implemented. Greater coordination of efforts internationally could facilitate more efficient and more strategic approaches to accumulating evidence on SSB taxes, as well as a range of other low-agency population health interventions for NCD prevention.

The challenge of complexity
Both of the evaluations described above took a relatively simple approach to evaluating a somewhat complex problem. Neither paper presented a theory of change [15] for the intervention they were evaluating. However, it is clear that in both cases the implicit theory of how the intervention was thought to achieve its likely impacts was relatively simplistic: a linear pathway from price change to changes in consumption [16]. Yet both sets of authors also hinted at more complicated mechanisms to explain unexpected findings, such as the socio-economic patterning of outcomes. For example, they suggested that other purchasing behaviours (e.g., taking advantage of promotions, buying in bulk), commercial behaviours (e.g., flexing or cross-subsidising prices across products or brands), or the effects of co-occurring education, regulation of product labelling, advocacy campaigns, and media coverage may have influenced their observed outcomes [4,5].
Public health scientists have increasingly highlighted the importance of taking context into account in evaluating population interventions [17]. It has also been suggested that such interventions may be better considered as events in complex, adaptive systems, rather than within simpler linear causal models [18]. Systems thinking prompts us to ask different questions, which in turn leads to different kinds of evaluations. For example, in the case of the Chilean SSB tax, instead of asking 'What are its impacts on price and sales?' we would ask 'What are the diverse impacts of announcing and introducing the tax, how do these come about, and how do they interact over time?' While still considering impacts on price and sales, this would also prompt us to think more broadly about the wide-ranging potential intended and unanticipated consequences of such a tax at all levels of society, in all sectors, and for all stakeholders. It also prompts us to consider the potential for change in outcomes as a continuous process, rather than a single event, consequent upon the continuous adaptation that occurs within food systems in response to external stimuli. While such approaches to evaluation remain in their infancy, some valuable examples are emerging, such as a recent study evaluating the impacts of a voluntary scheme in which retailers were challenged to reduce availability of low cost, high strength beers and ciders [19,20]. In this study, the researchers engaged with the complexity of the context and the potential for wide-ranging consequences of the intervention, assessing these through multiple methods and diverse sampling strategies, as well as through analyses informed by complexity theory and systems thinking.

Meeting the challenges
Broader, more nuanced, and more informative research questions cannot be answered using traditional, more simplistic approaches to evaluation. The present evidence base is significantly limited by a failure to embrace different ways of thinking and working, including addressing questions of context and those related to complex adaptive systems [17,18]. However, a consequence of taking a broader approach to evaluation that embraces complexity will be that it challenges prevailing methodological orthodoxies. Researchers, funders, and journals may all be reluctant to relinquish existing hierarchies of evidence and 'traditional' methods of evidence synthesis. With so many intractable global public health challenges associated with NCDs in need of robustly researched solutions, this seems short-sighted. Greater ambition and leadership are needed among researchers, funders, and policymakers to enable smarter approaches to the development and evaluation of low-agency population interventions, including taxes on unhealthy commodities, regulation of marketing, and structural interventions to make active living easier. Ambition and leadership coupled with greater international collaboration to identify opportunities for-and to coordinate efforts to fund, implement, and build capacity for-quasi-natural and natural experimental evaluations of these interventions could more rapidly advance science on NCD prevention.