Development of an Intervention Setting Ontology for behaviour change: Specifying where interventions take place

Background: Contextual factors such as an intervention’s setting are key to understanding how interventions to change behaviour have their effects and patterns of generalisation across contexts. The intervention’s setting is not consistently reported in published reports of evaluations. Using ontologies to specify and classify intervention setting characteristics enables clear and reproducible reporting, thus aiding replication, implementation and evidence synthesis. This paper reports the development of a Setting Ontology for behaviour change interventions as part of a Behaviour Change Intervention Ontology, currently being developed in the Wellcome Trust funded Human Behaviour-Change Project. Methods: The Intervention Setting Ontology was developed following methods for ontology development used in the Human Behaviour-Change Project: 1) Defining the ontology’s scope, 2) Identifying key entities by reviewing existing classification systems (top-down) and 100 published behaviour change intervention reports (bottom-up), 3) Refining the preliminary ontology by literature annotation of 100 reports, 4) Stakeholder reviewing by 23 behavioural science and public health experts to refine the ontology, 5) Assessing inter-rater reliability of using the ontology by two annotators familiar with the ontology and two annotators unfamiliar with it, 6) Specifying ontological relationships between setting entities and 7) Making the Intervention Setting Ontology machine-readable using Web Ontology Language (OWL) and publishing online. Re sults: The Intervention Setting Ontology consists of 72 entities structured hierarchically with two upper-level classes: Physical setting including Geographic location, Attribute of location (including Area social and economic condition, Population and resource density sub-levels) and Intervention site (including Facility, Transportation and Outdoor environment sub-levels), as well as Social setting. Inter-rater reliability was found to be 0.73 (good) for those familiar with the ontology and 0.61 (acceptable) for those unfamiliar with it. Conclusion: The Intervention Setting Ontology can be used to code information from diverse sources, annotate the setting characteristics of existing intervention evaluation reports and guide future reporting.


Introduction
Effects of interventions to improve health vary considerably across contexts of settings and target populations. While this is widely acknowledged in the literature, the specific elements in the context and their mechanisms of action on outcomes are either assumed or obscure . In order to understand this variation arising from the different aspects of context, it is helpful to synthesise evidence about the ways in which these modifying variables influence intervention effectiveness. This requires detailed and consistent specification of study contexts. There are many different classification systems and ontologies describing interventions, including their settings and target populations; however, these have limitations such as incomplete coverage and relevance across the range of international contexts. In this paper, we consider intervention setting. A forthcoming paper will report the development of an Intervention Population Ontology (Finnerty et al., In preparation).
Intervention settings are not currently consistently reported with enough specificity or comprehensiveness to allow accurate replication. The CONsolidated Standards of Reporting Trials statement (CONSORT;Schulz et al., 2010) includes one item referring to setting (Item 4b -Settings and locations where the data were collected), with its extension for social and psychology interventions CONSORT-SPI (Montgomery et al., 2018) adding an additional item (Item 4b -Where applicable, eligibility criteria for settings and those delivering the intervention). The Template for Intervention Description and Replication checklist (TIDieR;Hoffman et al., 2014) includes one item for setting (Item 7 -Where: describe the type(s) of location(s) where the intervention occurred, including any necessary infrastructure or relevant features). The recent Typology of Interventions in Proximal Physical Micro-Environments typology (TIPPME: Hollands et al., 2017) allows specification of micro-level aspects of the physical environment related to behaviours. Although this was based on an exhaustive review of the literature, TIPPME is restricted to interventions in microenvironments or contexts aimed at changing selection, purchase and consumption of food, alcohol and tobacco. We currently lack a classification system to aid researchers in describing in detail, and using shared language, the variety of settings of behaviour change interventions (BCIs) or indeed behaviour more broadly.
What at first sight would seem to be a fairly straightforward task of describing intervention settings is actually very complex, given the diversity of entities, terms and definitions across academic disciplines, employment sectors and cultures. Ontologies are a tool for addressing this diversity by enabling 'semantic inter-operability' by associating computational data with unambiguous shared meaning (Hastings, 2017;Michal et al., 2012). Ontologies are data structures that enable precise specification of knowledge in a given domain (Arp et al., 2015). In information science, ontologies provide a set of: i) unique and unambiguous identifiers representing types of entity (such as objects, attributes or processes), ii) labels and definitions corresponding to these identifiers, and iii) specified relationships between the entities (Arp et al., 2015;Larsen et al., 2017;Norris et al., 2019). These labels, definitions and relationships comprise a 'controlled vocabulary' and formal specification for the given domain. Ontologies are dynamic representations that are maintained and updated according to new evidence about entities and relationships (He et al., 2018). Machine-readable ontologies provide an excellent structure for annotating scientific reports to allow evidence synthesis . As seen in other fields such as genetics (Ashburner et al., 2000), the availability and use of ontologies allows an active, iteratively developed basis for shared knowledge and understanding . As machinereadable artefacts, ontologies can be harnessed for annotation and evidence synthesis, such as the automation of literature searching, statistical analysis workflows and database searching and browsing, as well as in other computational applications (Hastings, 2017) (see glossary of italicised terms in Table 1).
As yet, no ontology exists to describe the complexity of behaviour change intervention settings (Norris et al., 2019). A comprehensive Behaviour Change Intervention Ontology (BCIO) is being developed as part of the Human Behaviour-Change Project . The BCIO consists of an upper level with 42 entities, one of which is Behaviour change intervention setting, specified as part of the Context in a given BCI scenario . Drawing on the methodology used to develop a taxonomy of behaviour change techniques (BCTTv1;  and other relevant ontologies (Norris et al., 2019), the current study aimed to develop an ontology for specifying and classifying characteristics of the settings in which interventions take place. These settings are generally applicable beyond the scope of behaviour change interventions. This paper reports the development and final version of the Intervention Setting Ontology.

Methods
The Intervention Setting Ontology was developed in an iterative process of seven steps (Wright et al., 2020).
Step 1 -Defining the scope of the Intervention Setting Ontology A definition and overall topic for the ontology was set by reviewing dictionaries and the reporting guidelines of CONSORT (Schulz et al., 2010), CONSORT-SPI (Montgomery et al., 2018), TIDieR (Hoffmann et al., 2014) and TIPPME (Hollands et al., 2017).
Step 2 -Identifying key entities and developing the preliminary Intervention Setting Ontology An initial prototype version of the ontology was developed using both a bottom-up and top-down approach. In the bottom-up approach, 100 published reports of BCIs were reviewed to develop an initial list of intervention setting characteristics. These reports were randomly selected from a larger dataset of BCI reports partially annotated for behaviour change techniques, mechanisms of action, and modes of delivery, covering a range of health behaviours (Carey et al., 2019;Michie et al., 2015).
In the top-down approach, existing classification systems of intervention setting characteristics were identified from: i) published ontologies containing terms related to behaviour change Annotation guidance manual Written guidance on how to identify and tag pieces of text from intervention evaluation reports with specific codes relating to entities in the ontology, using EPPI-Reviewer software.
Basic Formal Ontology (BFO) An upper level ontology consisting of continuants and occurrents developed to support integration, especially of data obtained through scientific research.

Entity
Anything that exists, that can be a continuant or an occurrent as defined in the Basic Formal Ontology.

EPPI-Reviewer
A web-based software program for managing and analysing data in all types of systematic review (meta-analysis, framework synthesis, thematic synthesis etc. It manages references, stores PDF files and facilitates qualitative and quantitative analyses such as meta-analysis and thematic synthesis. It also has a facilitate to annotate published papers.

https://github.com/
Inter-rater reliability Statistical assessment of similarity and dissimilarity of coding between two or more coders. If inter-rater reliability is high this suggests that ontology entity definitions and labels are being interpreted similarly by the coders. Gwet, 2014. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Gaithersburg, Advanced Analytics.

Interoperability
Ontology developers should collaborate with others wherever possible to re-use entities and limit duplication of work. Interoperability of ontologies sits within the OBO Foundry principle of Commitment to Collaboration. http://www.obofoundry.org/principles/fp-010-collaboration.html

Issue tracker
An online log for problems identified by users accessing and using an ontology.

OBO Foundry
The Open Biological and Biomedical Ontology (OBO) Foundry is a collective of ontology developers that are committed to collaboration and adherence to shared principles. The mission of the OBO Foundry is to develop a family of interoperable ontologies that are both logically well-formed and scientifically accurate. Smith et al., 2007;www.obofoundry.org/

Ontology
A standardised representational framework providing a set of terms for the consistent description (or "annotation" or "tagging") of data and information across disciplinary and research community boundaries.

Parent class
A subsuming class within an ontology that is related to one or more child (subsumed) classes. The preliminary ontology contained a label and definition for each entity representing an intervention setting characteristic. Definitions were developed using pre-specified guidance, with the standard format of definitions being: A is a B that C, or involves or relates to C in some way, where A is the class being defined, B is a parent class and C describes a set of properties of A that distinguish it from other members of B . It was piloted with published BCI reports focusing on smoking cessation and physical activity behaviours . BCI reports were annotated independently by two researchers in batches of 10, with each entity annotated as either present or absent. Two types of inter-rater reliability measures were used: i. percentage of agreement between coders and ii. Cohen's Kappa (Cohen, 1960). Kappa statistics are only reported in instances where the researchers allocated a code to at least five cases (Michie et al., 2015). Satisfactory interrater reliability was achieved by the time 55 papers had been coded. After this, no additional adjustments were made to the prototype version of the Intervention Setting Ontology.
Step 3 -Refinement of the ontology through literature annotation, discussion and revision The preliminary ontology was revised by the research team based on the results of the pilot annotations. Using EPPI-Reviewer 4 software (Thomas & Brunton, 2010), two researchers independently annotated 30 BCI reports on smoking cessation interventions using the revised Intervention Setting Ontology. An open alternative to this software used for annotation is PDFAnno (Shindo et al., 2018). Discrepancies were discussed and the ontology structure, definitions and annotation guidance manual were revised. A second set of annotators followed the same procedure for another set of 45 BCI reports of smoking cessation, and 40 BCI reports of physical activity. All reports were randomised controlled trials from one of three datasets: Cochrane Reviews, papers annotated for behaviour change techniques and papers from the IC-SMOKE project ( Step 4 -Expert stakeholder review Ninety-eight members of a panel of behavioural scientists and public health expert stakeholders were invited to give feedback on the Intervention Setting Ontology resulting from Step 3. These experts comprised i) 65 behavioural scientists who had provided feedback on previous projects at the Centre for Behaviour Change, ii) 16 experts from under-represented countries identified through the BCTTv1 database, and iii) 17 stakeholders who expressed interest in being involved in the Human Behaviour-Change Project stakeholder initiatives. Experts from both 'well-represented' countries (UK, USA, Canada, Australia, the Netherlands) and other 'less-represented' countries were randomly selected to provide feedback using Researcher Randomizer.
Feedback was collected through an online questionnaire, using Qualtrics TM software (Full survey https://osf.io/8audy/ (West et al., 2020)), with the task designed to take no longer than 45 minutes to complete. The task asked experts to: 1. identify the characteristics of intervention setting that were of interest to them when trying to understand variation in the effectiveness of BCIs (open-ended question). Experts were advised to consider a specific behaviour when answering this question e.g 'physical activity' 2. rate the importance of each of the setting entities on a 5-point Likert scale (1 = "not important", 2 = "slightly important", 3 = "moderately important", 4 = "important", 5 = "very important" or "don't know/not sure"). For example: "How important do you think each of the following Geographic location characteristics are to understand variation in the effectiveness of at least some behaviour change interventions?" (Country of intervention & Within country location), and 3. provide feedback on the completeness and comprehensiveness of the Setting Ontology.
Experts were also asked to indicate: i) if there were any entities missing (If yes, which should be added), ii) if there were any entities or definitions that should be changed (if yes, what changes should be considered), and iii) If there were any entities that should be placed in a different location in the classification hierarchy of the Intervention Setting Ontology.
A thematic analysis of the responses was conducted and means and standard deviations of ratings were calculated. The feedback from the expert consultation was discussed by the research team and the Intervention Setting Ontology and annotation guidance were revised.
Step 5 -Inter-rater Reliability of Annotations using the Intervention Setting Ontology Assessment of inter-rater reliability of the annotations by two researchers leading the development of the ontology was conducted using 50 papers from Cochrane reviews (30 for smoking cessation and 20 for physical activity). Inter-rater reliability was also assessed for annotations by two behaviour change experts unfamiliar with the ontology but with experience in annotating BCI reports. Annotation was of a random sample of 50 randomised controlled trials from a database of papers coded by Behaviour Change Techniques, with no restrictions on the outcome behaviour. Inter-rater reliability was assessed using Krippendorff's Alpha (Hayes & Krippendorff, 2007) using Python 3.6 (https://github.com/HumanBehaviourChangeProject/Automation-InterRater-Reliability) (Finnerty & Moore, 2020), as unlike Cohen's Kappa, Krippendorff factors in both agreement and disagreement within annotations.
Step 6 -Specifying the relationships between Intervention Setting Ontology entities The research team established relationships between ontology entities to formalise the knowledge present in the ontology. This process was conducted in line with Basic Formal Ontology principles which have been used extensively in biomedical ontologies (Arp et al., 2015). The suitability of common relationships from Basic Formal Ontology (Arp et al., 2015) and the Relation Ontology (Smith et al., 2005) were assessed, including the basic hierarchical relationship 'is_a' which holds between classes where one class is a subclass of another class, and 'located_in', which relates an entity to a spatial region demarcating a location.
Step 7 -Making the Intervention Setting Ontology machine-readable and available online The Intervention Setting Ontology was initially developed as a This OWL version of the Intervention Setting Ontology was then stored on the project GitHub repository, as GitHub has an issue tracker which allows feedback to be submitted by members of the community which can be responded to, and if necessary, addressed in subsequent releases. When the full Behaviour Change Intervention Ontology has been confirmed, it will be submitted to the OBO Foundry (Smith et al., 2007).

Results
Step 1 -Defining the scope of the Intervention Setting Ontology Given that 'setting' is defined in a general lexicon as 'the place or type of surroundings where something is positioned or where an event takes place', an intervention's setting was defined more precisely as 'An aggregate of entities that form the environment in which a BCI is provided.' Step 2 -Identifying key entities and developing the preliminary Intervention Setting Ontology The initial prototype version of the Intervention Setting Ontology encompassed a four-level hierarchical structure, containing 76 unique entities (https://osf.io/g8qfv/ (West et al., 2020)).
Inter-rater agreement for identifying the presence of a setting entity was low in terms of percentage, at 45.5%. Kappa statistics varied from 'perfect' for entities such as Accommodation to low agreement (κ=0.300) for entities such as Community setting.
Step 3 -Refinement of the Intervention Setting Ontology Based on the annotations from Step 2, changes were made to the ontology. Two terms, 'particular' and 'unclear/not reported', were deleted as they did not meet the ontological requirement of being unique discrete entities with corresponding definitions and attributes (Arp et al., 2015). Other changes were: 3) Outdoor environment was added to the ontology; 4) Attribute of location was added to the ontology, including new entities Area social and economic condition and Population and resource distribution (previously placed in Geographic location). Changes to labels and definitions were made to reflect the structural changes.
Step 4  Step 5 -Inter-rater reliability of annotations using the Intervention Setting Ontology Inter-rater reliability from the 50 papers annotated by those familiar with the ontology was found to be good (a=0.73). The random selection of 50 papers used for inter-rater reliability testing in those unfamiliar with the ontology resulted in papers with the following target behaviours: physical activity (k=16), dietary behaviours (k=9), sexual behaviours (k=8), alcohol (k=7) and other behaviours such as medication adherence (k=11). The inter-reliability for these annotations was acceptable (a=0.61) (Hayes & Krippendorff, 2007).
Step 6 -Specifying the relationships between Intervention Setting Ontology entities Relationships from the Relation Ontology (Smith et al., 2005) were used to connect classes, namely the basic hierarchical relationship 'is_a' which holds between classes where one class is a subclass of another class, and 'is_attribute_of' which holds between classes where one class is a quality or feature of the other.
Step 7 -Making the Intervention Setting Ontology machine-readable and available online A downloadable version of the final Intervention Setting Ontology is available from GitHub (Norris et al., 2020). The hierarchical structure, URIs, labels and definitions for all entities are described in Table 2. The ontology is accompanied by an annotation guidance manual that provides guidance on how to annotate for these entities in BCI reports (available at https://osf.io/76jty/) (West et al., 2020).
The final version of the Intervention Setting Ontology presents a six-level hierarchical structure comprising of 72 unique entities. There are two upper-level classes: Physical setting (BCIO: 026000: A physical environment in which a BCI is delivered) and Social setting (BCIO: 029000: An aggregate of people with whom a BCI population interacts). Physical setting includes Geographic location (GAZ:00000448: A reference to a place on the Earth, by its name or by its geographic location, used from the existing Gazetteer Ontology), Attribute of location (BCIO: 026003: Features of a given location, such as social and economic characteristics) and Site (BFO_0000029: A threedimensional immaterial entity that is (partially or wholly) bounded by a material entity or it is a three-dimensional immaterial part thereof).

Discussion
This study developed the Intervention Setting Ontology to specify formally the characteristics of the settings in which behaviour change interventions (BCIs) take place, as part of the Behaviour Change Intervention Ontology . Although developed primarily to specify settings of behaviour change interventions, the settings are generally applicable to other types of intervention or contexts. The ontology consists of 72 entities structured hierarchically with two upper-level classes: Physical setting (BCIO:026000: A physical environment in which a BCI is delivered) and Social setting (BCIO:029000: An aggregate of people with whom a BCI population interacts). Physical setting is further sub-divided by three upper-level classes: Geographic location, Attribute of location (including Area social and economic, Population and resource density sub-levels) and Site (including Facility, Transportation and Outdoor environment sub-levels). Inter-rater reliability was found to be 0.73 (good) for those familiar with the ontology and 0.61 (acceptable) for those unfamiliar with it, as assessed by Krippendorff's alpha. Together with 'population', it makes up Context which is part of a wider set of lower-level ontologies within the Behaviour Change Intervention Ontology (BCIO).
The ontologies within the BCIO are connected to each other by specified relationships. For example, the contextual entity of Intervention Setting is related to the contextual entity of Population: who receives an intervention (Finnerty et al., In preparation). In addition, entities within the Intervention Setting Ontology can be integrated or linked to ontologies beyond the BCIO, a key feature of OWL ontologies which encourages re-use and adoption (Hastings, 2017).
Ontologies should be dynamic representations that are maintained and updated according to new evidence about entities and relationships (Arp et al., 2015;He et al., 2018). The Intervention Setting Ontology and all other ontologies within the Human Behaviour-Change Project will be updated as they are informed by advances in behavioural science and by online feedback from ontology users via the GitHub portal.

Strengths and limitations
Domain experts are often not formally consulted when ontologies are developed (Norris et al., 2019), with the result that development may be restricted to the knowledge, thinking A three-dimensional immaterial entity that is (partially or wholly) bounded by a material entity or it is a three-dimensional immaterial part thereof Facility OMRSE:00000062 An architectural structure that bears some function.  (Wright et al., 2020). This process incorporates international expert stakeholder feedback, as has also occurred in other related projects e.g. BCTTv1, Michie et al., 2015; Linking BCTs and Mechanisms of Action, Carey et al., 2019;TIPPME, Hollands et al., 2017;MAGI framework, Borek et al., 2019. Another strength is the integration of existing terms from other ontologies where they exist, preventing duplication of entities within the wider ontology space (Norris et al., 2019). The use of entity IDs for each entity in the ontology provides a machinereadable identifier for integration in future systems and also allows interoperability between existing ontologies.

Upper-Level
The Intervention Setting Ontology has been found to be useful to manually annotate a large body of published intervention evaluation reports . These manual annotations are informing the development and testing of information extraction algorithms (Ganguly et al., 2018) to automate the process of identifying and organising knowledge about interventions within published reports . This corpus of manually and automatically extracted data on intervention setting characteristics is being made available as it is produced on the Human Behaviour-Change Project's GitHub page. As machine-readable representations of knowledge, these ontologies provide a framework for applying Artificial Intelligence to synthesising and interpreting evidence e.g. by identifying patterns of data organised by the BCIO. Reasoning algorithms allow real-time up-to-date evidence synthesis that can be used to answer variants of the "big question" of behaviour change: "What works, compared with what, for what behaviours, how well, for how long, with whom, in what setting, and why?", across a wide range of contexts . This body of work has the potential to have far-reaching use by and implications for policy-makers, practitioners and researchers, for example, by informing evidence-based guidelines, extrapolating knowledge to under-researched populations and settings, and identifying knowledge gaps.
A limitation of this work is that the intervention reports annotated within the ontology development mainly addressed two health-related behaviours, smoking cessation and physical activity. This was due to the ontology being developed within the Human Behaviour-Change Project, which is using smoking cessation and physical activity interventions as initial use cases . However, external inter-rater reliability was tested across diverse behaviours and found to be acceptable. Future application of the ontology to a wider collection of behaviours and contexts will help extend and improve it.

Conclusions
The Intervention Setting Ontology provides a classification system that can be used reliably to specify the characteristics of settings where interventions take place. It will contribute to the improvement of research reporting and replication, enabling easier evidence synthesis across studies. The ontology can be used within computational tools to speed up the accumulation, interpretation and application of knowledge, such as the Knowledge System being developed within the Human Behaviour-Change Project . The Intervention Setting Ontology is intended to act as a foundation from which future research can build, as an ongoing and collaborative process. The ontology will allow us to increase our understanding of the settings in which interventions are implemented and how effectiveness varies across settings.

Ethics
Ethical approval was granted by University College London's ethics committee (CEHP/2016/555). © 2020 Noone C. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Chris Noone
School of Psychology, National University of Ireland Galway, Galway, Ireland This article reports the development of an intervention setting ontology for behaviour change research. The rationale for this work is strong as the field would clearly benefit from the availability of a shared language for discussing where behaviour change interventions take place.
This paper reports this development process very clearly and the underlying data is easy to navigate and understand. I do agree with my fellow reviewer that the article would be easier to follow if the results for each stage were reported after the methods for that stage are detailed. This would be analogous to the structure of multi-study articles.
One important aspect of this project that could be explained in more detail is how the research community can contribute to the ongoing development of this ontology using the GitHub repository.
For example, the term "developing country" is contested and the apparent conflation (if I have interpreted the onotology correctly) of low income countries and low income areas within countries, might be issues that researchers would like to provide feedback on, but many are not familiar with GitHub. Perhaps a guide on providing feedback could be developed and placed in the OSFproject associated with this article?

Is the work clearly and accurately presented and does it cite the current literature? Yes
Is the study design appropriate and is the work technically sound? Yes

Are sufficient details of methods and analysis provided to allow replication by others? Yes
If applicable, is the statistical analysis and its interpretation appropriate?