Delivering Behaviour Change Interventions: Development of a Mode of Delivery Ontology [version 1; peer review: 1 approved, 1 approved with reservations]

Background: Investigating and improving the effects of behaviour change interventions requires detailed and consistent specification of all aspects of interventions. An important feature of interventions is the way in which these are delivered, i.e. their mode of delivery. This paper describes an ontology for specifying the mode of delivery of interventions, which forms part of the Behaviour Change Intervention Ontology, currently being developed in the Wellcome Trust funded Human Behaviour-Change Project. Methods: The Mode of Delivery Ontology was developed in an iterative process of annotating behaviour change interventions evaluation reports, and consulting with expert stakeholders. It consisted of seven steps: 1) annotation of 110 intervention reports to develop a preliminary classification of modes of delivery; 2) open review from international experts (n=25); 3) second round of annotations with 55 reports to test inter-rater reliability and identify limitations; 4) second round of expert review feedback (n=16); 5) final round of testing of the refined ontology by two annotators familiar and two annotators unfamiliar with the ontology; 6) specification of ontological relationships between entities; and 7) transformation into a machine-readable format using the Web Ontology Language (OWL) language and publishing online. Results: The resulting ontology is a four-level hierarchical structure comprising 65 unique modes of delivery, organised by 15 upper-level classes: Informational, Environmental change, Somatic, Somatic alteration, Individual-based/ Pair-based /Group-based, Unidirectional/Interactional, Synchronous/ Asynchronous, Push/ Pull, Open Peer Review Reviewer Status Invited Reviewers 1 2 version 1 10 Jun 2020 report report Lucie Byrne-Davis , University of Manchester, Manchester, UK 1. Ann DeSmet , Université Libre de Bruxelles, Brussels, Belgium 2. Any reports and responses or comments on the article can be found at the end of the article. Page 1 of 19 Wellcome Open Research 2020, 5:125 Last updated: 14 SEP 2020


Introduction
Patterns of human behaviour contribute significantly to the global disease burden, as well as to a wide range of environmental and social problems (e.g. Gakidou et al., 2017;Watts et al., 2017). The development of behaviour change interventions, defined as coordinated sets of activities designed to change specified behaviour patterns (Michie et al., 2011), can be an effective and cost-effective solution to such global problems. Research investigating the development, evaluation and implementation of behaviour change interventions, as well as evidence syntheses, demonstrate striking variability in effectiveness across different studies (see Cochrane database, e.g. Flodgren et al., 2017;Ussher et al., 2012). Understanding this variability is difficult given the complexity of interventions, with variations in content and delivery potentially interacting with each other and with the intervention setting, population and target behaviour.
Being able to specify intervention characteristics in a way that facilitates replication and evidence synthesis is an important step in building evidence efficiently and cumulatively. This requires conceptual frameworks that organise knowledge using clear, coherent, and shared terminology (Michie et al., 2017). Such frameworks promote communication and collaboration across disciplines and research groups, and can be helpful in advancing knowledge generation to inform intervention development, implementation, evaluation, and reporting (Craig et al., 2008;Hoffmann et al., 2014;Moher et al., 2001). Another benefit of using conceptual frameworks is that they can enhance researchers' ability to examine associations between specific intervention components and outcomes (Sheeran et al., 2017). This allows for a more thorough understanding of interventions and how they bring about their effects which, in turn, can inform the development of more effective interventions.
The BCTTv1 is a hierarchical taxonomy used to classify the potentially 'active ingredients' of behaviour change interventions, known as behaviour change techniques (BCTs) (Michie et al., 2019;Michie et al., 2013;Michie et al., 2015). It includes 93 discrete BCTs, each with a consensus-based label, definition and example(s). BCTTv1 has been used to identify and define BCTs in intervention research (Newbury-Birch et al., 2014;Paul et al., 2017;Young et al., 2014) and to categorise intervention content in evidence syntheses (Arnott et al., 2014;Jones et al., 2014). By providing a common language with which to describe interventions, BCTTv1 has facilitated a level of rigour and specificity in reporting intervention content that was not previously commonplace (Sheeran et al., 2017). While BCTTv1 has provided a shared language for specifying intervention content, there are other aspects of behaviour change interventions that have received comparatively little attention, including how such content is delivered (Dombrowski et al., 2016).

Ontologies
BCTTv1 is an example of a taxonomy, a knowledge representation structure in which a controlled vocabulary of agreed-upon terms is arranged hierarchically. An ontology is a more expressive structure for organising knowledge (see glossary of italicised terms, Table 1). It includes a controlled vocabulary, unambiguous identifiers for each entity, and additional information such as synonyms and examples of usage. It includes relationships between entities, usually beyond the hierarchical class-subclass relationship as well as a formal, logic-based encoding of domain knowledge where possible (Arp et al., 2015;Hastings, 2017;Larsen et al., 2017;Michie & Johnston, 2017;Norris et al., 2019). Ontologies enable entities to be compared and integrated across fields of study and allow large datasets to be synthesised efficiently using computational tools (e.g. in biology, the Gene Ontology (Ashburner et al., 2000).
The potential for ontologies to facilitate knowledge synthesis in behaviour change is being developed in the Human Behaviour-Change Project (Michie et al., 2018;Michie et al., 2020a;Michie et al., 2020b). This collaboration between behavioural scientists, computer scientists and systems architects is building a database and platform for researchers, practitioners and policy-makers to address variants of the 'big question' of behaviour change: "What works, compared with what, how well, with what exposure, with what behaviours (for how long), for whom, in what settings and why?" Answering this involves extending previous work to classify all entities of behaviour change interventions and the relationships between them, i.e. a Behaviour change intervention ontology (BCIO), specified by a controlled vocabulary that by the upper level of the BCIO (Michie et al., 2020b) contains 42 entities. The Behaviour change intervention delivery entity of the ontology (i.e. the means by which BCI content is provided), comprises (a) BCI Source (i.e., a role played by a person, population or organisation that provides a behaviour change intervention), (b) BCI Schedule of delivery (an attribute of a behaviour change intervention that involves its temporal organisation), (c) BCI Style of delivery (an attribute of a BCI delivery that encompasses the characteristics of how a behaviour change intervention is communicated), and (d) BCI Mode of delivery (an attribute of a BCI delivery that is the physical or informational medium through which a behaviour change intervention is provided).

Delivery of Behaviour Change Interventions
An important characteristic of behaviour change interventions is the method or methods by which the content (i.e. the techniques) is brought to its target population (i.e. its mode of delivery; MoD). MoDs can act synergistically or antagonistically with BCTs in influencing intervention outcomes and effects. An example of this is a meta-analysis of evidence about the effectiveness of smoking cessation interventions, which

Annotation guidance manual
Written guidance on how to identify and tag pieces of text from intervention evaluation reports with specific codes relating to entities in the ontology, using EPPI-Reviewer software.

Basic Formal Ontology (BFO)
An upper level ontology consisting of continuants and occurrents developed to support integration, especially of data obtained through scientific research.

Entity
Anything that exists, that can be a continuant or an occurrent as defined in the Basic Formal Ontology.

EPPI-Reviewer
A web-based software program for managing and analysing data in all types of systematic review (metaanalysis, framework synthesis, thematic synthesis etc. It manages references, stores PDF files and facilitates qualitative and quantitative analyses such as metaanalysis and thematic synthesis. It also has a facilitate to annotate published papers.

Inter-rater reliability
Statistical assessment of similarity and dissimilarity of coding between two or more coders. If inter-rater reliability is high this suggests that ontology entity definitions and labels are being interpreted similarly by the coders.
Gwet, 2014. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Gaithersburg, Advanced Analytics.

Interoperability
Ontology developers should collaborate with others wherever possible to re-use entities and limit duplication of work. Interoperability of ontologies sits within the OBO Foundry principle of Commitment to Collaboration. http://www.obofoundry.org/principles/fp-010collaboration.html

Issue tracker
An online log for problems identified by users accessing and using an ontology.

OBO Foundry
The Open Biological and Biomedical Ontology (OBO) Foundry is a collective of ontology developers that are committed to collaboration and adherence to shared principles. The mission of the OBO Foundry is to develop a family of interoperable ontologies that are both logically well-formed and scientifically accurate.

Ontology
A standardised representational framework providing a set of terms for the consistent description (or "annotation" or "tagging") of data and information across disciplinary and research community boundaries.

Parental class
A subsuming class within an ontology that is related to one or more child (subsumed) classes.

Reconciliation
The process of discussing differences between the annotations of two paired annotators on the same papers. Differences are discussed before a final reconciled version of coding for each paper is produced.

Unique resource identifier (URI)
A string of characters that unambiguously identifies an ontology or an individual entity within an ontology. Having URI identifiers is one of the OBO Foundry principles. http://www.obofoundry.org/principles/fp-003-uris.html

Web Ontology Language (OWL)
A formal language for describing ontologies. It provides methods to model classes of "things", how they relate to each other and the properties they have. OWL is designed to be interpreted by computer programs and is extensively used in the Semantic Web where rich knowledge about web documents and the relationships between them are represented using OWL syntax.
https://www.w3.org/TR/owl2-quick-reference/ found effectiveness to be higher with increasing numbers of intervention techniques but only if delivered in person and not when delivered in written form (Black et al., 2020).
Three systematic reviews have extracted information about MoDs (Bock et al., 2014;Genugten et al., 2016;MacDonald et al., 2016), and an annotation scheme for MoD within internetbased interventions has been developed (Webb et al., 2010). However, MoD has received comparatively little attention in intervention research (Dombrowski et al., 2016), and there is a lack of clarity and consensus across behavioural intervention research regarding how MoD is defined, what it includes, and how it should be reported. This is in contrast to the reporting of BCTs as the content of behaviour change interventions, for which there is now wide shared understanding, for example, featuring in the Encyclopaedia of Behavioural Medicine (Michie et al., 2019) and in many hundreds of publications. The various conceptualisations of MoD, and the lack of a shared language or framework with which to describe it has made the study of interactions between it and other intervention entities difficult to analyse systematically (Dombrowski et al., 2016). Here, we define MoD as the attribute of BCI delivery that is the informational or physical medium through which a behaviour change intervention is provided (Michie et al., 2020b). For example, providing someone with information about the health consequences of performing a particular behaviour could be conducted face-to-face (e.g. by a GP), through a poster or leaflet, or through a digital device (e.g. an app). 'Item 6: How' of the TIDieR framework highlights the need for researchers to clearly specify the MoD of BCTs; an ontology provides a mechanism for doing this. The development of a MoD ontology that can be linked to other ontologies relevant to behaviour change interventions would be an advance for developing scientific understanding, the development and evaluation of interventions and methods for evidence synthesis.

Aim
The aim of the MoD Ontology is to provide a clear, usable and reliable classification system to specify the MoDs of behaviour change interventions, including single BCTs. The development of an ontology with clear and unambiguously defined terms enables precision of reporting, which in turn promotes evidence synthesis, replication and analyses of associations between MoDs, other intervention characteristics and intervention outcomes.

Methods
The ontology was developed in seven iterative steps (detailed below), involving reviewing existing classification systems, annotation of behaviour change intervention reports (including testing of inter-rater reliability) and feedback from international expert stakeholders (outlined in Table 2).
Step 1: Development of the preliminary ontology and piloting Descriptions of MoD entities were extracted from 20 published behaviour change intervention evaluation reports, randomly selected using a random number generator from a larger database of reports annotated by behaviour change techniques and mechanisms of action (Michie et al., 2018), covering a range of health behaviours. Next, two researchers independently piloted the preliminary MoD ontology with another set of intervention reports, taken from the same database and using the same selection method. Guidance on how to annotate papers for MoD was developed by the research team, providing clear instructions on how to code each entity, including definitions and examples for each. Reports were annotated in batches of 10 until a satisfactory and stable criterion of inter-rater reliability was achieved. Inter-rater reliability of the extent to which researchers capture the same information from a report was measured in two ways. The first was percentage agreement of instances where both researchers had annotated an MoD. The second was the proportion of times annotators agreed on a code when both of them captured the same information from a report. This was calculated at every level of the hierarchy, and it was performed using Cohen's Kappa (Cohen, 1960), in Microsoft Excel 365. Kappa values >.61 were deemed as 'substantial' and values >.81 as 'strong' (Landis & Koch, 1977). The preliminary ontology was revised and updated iteratively throughout the annotation process. Where there were discrepancies between the two annotators, these were discussed, and amendments were made to the ontology if both annotators judged that these changes would improve clarity. In the case of disagreement, a senior member of the research team was consulted.
Step 2: Stakeholder review (Round 1) Nine international behavioural scientists with experience in behaviour change interventions, across a range of behavioural domains, were invited to provide feedback on the structure, content and terminology of the preliminary MoD Ontology. Following small adjustments based on this feedback, the MoD Ontology was published online, and a wider international research community was invited through mailing lists to submit feedback using an open Qualtrics form presenting the preliminary MoD structure, and entity labels and definitions (see https://osf. io/eyn3b/ (West et al., 2020)). Twenty-five behavioural scientists responded to indicate whether 1) there were any entities missing, 2) the structure was coherent, 3) there were changes needed in the terminology of the labels and definitions, and 4) there were additional suggestions for improvement.
Step 3: Inter-rater reliability testing (Round 2) The revised version was used to annotate MoD entities in a set of 55 published reports, randomly selected using a random number generator from the database mentioned in Step 1 (Michie et al., 2018). These papers covered the behavioural domains of physical activity, diet and smoking. Annotation of the reports was conducted independently by two researchers. The annotation process was carried out in batches of five papers. After every batch, annotations were compared, and discrepancies discussed. Inter-rater reliability was calculated using the same procedure as in Step 1. Where there were discrepancies, consensus was reached through discussion.
Step 4: Stakeholder review (Round 2) Experts who provided feedback in Step 2 were invited to submit feedback on the revised ontology. Experts were sent an email with a request to review the structure, labels and definitions of each entity, and indicate whether the structure was coherent and whether there was anything missing and provide suggestions for improved terminology. During this step, an ontology expert (JH) was consulted regarding the structure and definitions.
Step 5: Inter-rater reliability testing (Round 3) To test the range of applicability of this revised version of the MoD Ontology (as well as the annotation guidance manual), we conducted a final round of annotations as part of the annotations being conducted in the Human Behaviour-Change Project. First, two developers of the MoD ontology annotated reports that were selected from a database of reports used in the Human Behaviour-Change Project (Michie et al., 2017) (see https://osf.io/myje6/ (West et al., 2020)). These annotations were conducted using EPPI reviewer 4 software (Thomas et al., 2010). An open alternative to this software used for annotation is PDFAnno (Shindo et al., 2018). All reports were randomised controlled trials from one of three datasets: Cochrane Reviews, papers annotated for behaviour change techniques and papers from the IC-SMOKE project (De Bruin et al., 2016) (list of systematic reviews included as Extended data at https://osf. io/myje6/ (West et al., 2020)). There was a reconciliation process after the first batch of 10, followed by any necessary amendments to the annotation manual. These amendments mainly involved the inclusion of examples (e.g. illustrating when to code or not to code certain pieces of information as MoD).
To examine the usability of the MoD Ontology for researchers and intervention developers with no prior knowledge of the MoD Ontology, we conducted a final round of inter-rater reliability assessment by asking two researchers unfamiliar with the ontology and without specific expertise in modes of delivery to annotate a random sample of randomised controlled trials from a database of papers annotated by BCTs, with no restrictions on the outcome behaviour. Inter-rater reliability was assessed using Krippendorff's Alpha (Hayes & Krippendorff, 2007), using Python 3.6 (code available on GitHub (Finnerty & Moore, 2020)).
Step 6: Specifying relationships within the MoD Ontology The research team developed relationships between ontology entities to formally capture the types of knowledge that are present in the ontology. The relationships were specified following best practices from Basic Formal Ontology (BFO) described in Arp et al., (2015) and Relation Ontology (Smith et al., 2005). Relationships can be generic and shared across multiple ontologies (e.g the "is a" relationship between classes where one class is a subclass of another class, or the "part of" relationship which captures the relationship between wholes and their parts) or they can be domain specific, which are introduced when needed to formally capture relationships unique to a given domain.
Step 7: Making the MoD Ontology machine-readable and available online The MoD Ontology was initially developed as a table of entities, with separate rows for each entity annotated in columns for different types of annotation, including a primary label, definition, synonyms and relationships. When the MoD Ontology was at a stable level of development for initial release, it was converted into the Web Ontology Language (OWL) (Antoniou & van Harmelen, 2004) format, enabling it to be viewed and visualised using ontology software such as Protégé and to be compatible with other ontologies and software tools. The conversion to OWL used the ROBOT ontology toolkit library (Jackson et al., 2019), which provides a facility to create This OWL version of the MoD Ontology was then stored on the project GitHub repository (Finnerty & Moore, 2020), as GitHub has an issue tracker, which allows feedback to be submitted by members of the community that can be responded to, and if necessary, addressed in subsequent releases. When the full BCIO has been finalised, it will be submitted to the OBO Foundry (Smith et al., 2007).

Results
Step 1: Development of the preliminary ontology and piloting The data extracted from the behaviour change intervention reports led to the identification of 160 unique entities, which were represented in a four-level hierarchical structure, as well as two 'cross-cutting' entities (a description of the preliminary version is available as Extended data at https://osf.io/gu5ke/ (West et al., 2020)). A hundred reports were annotated, with adjustments made to the ontology as a result of the first 70; the ontology was stable for the final 30 reports. Average agreement between annotators for each batch of 10 reports varied between 72% and 95%. Inter-rater reliability was calculated for each level of the hierarchy separately and considered to be 'good' for all levels (% agreement 86.6 to 97.8; Kappa 0.68 to 0.97). Reliability was also calculated for each of the cross-cutting entities (Kappa = .55 and .75). Further details on the inter-rater reliability and changes made to the MoD Ontology in this step can be found as Extended data at: https://osf.io/r3wn2/ (West et al., 2020).
Step 2: Stakeholder review (Round 1) Feedback on the MoD ontology through the open review feedback form was received by 25 people, and these data were collated, synthesised, and discussed among the research team. This led to further amendments to the structure, content and terminology (full details on the feedback and corresponding changes made to the MoD Ontology are available as Extended data at https://osf.io/95n3a/ (West et al., 2020)).
Step 3: Inter-rater reliability testing (Round 2) For the 55 papers annotated in this round, agreement for whether a particular entity was considered an MoD was 61%; and agreement on the specific MoD code assigned was 87.9% (Kappa ≥ .857) (inter-rater reliability results are available as Extended data at https://osf.io/sw2jv/ (West et al., 2020)).
Step 4: Stakeholder review (Round 2) Feedback was received from 16 of the 25 experts invited. Based on this, the following changes were made: 1) the entities "other" and "unclear" were removed, as all entities represented in an ontology need to be fully specified; and (2) increased clarity was provided on how the cross-cutting entities related to the other upper-level classes (see https://osf.io/3zhbc/ (West et al., 2020) for more details").
For the revised version, definitions were developed using pre-specified guidance, with the standard format of definitions being: A is a B that C, or involves or relates to C in some way, where A is the class being defined, B is a parent class and C describes a set of properties of A that distinguish it from other members of B (Michie et al., 2019).
Step 6: Specifying relationships within the MoD Ontology Currently, the only relationship used in the ontology represent its hierarchical structure, i.e. "subclass of" (is_a) relationships (e.g. face to face MoD "is_a" human interactional MoD). Formal representations of knowledge using explicit logical relationships allow computational tools to perform additional checks and inferences to enhance the resulting consistency of reporting for complex interventions.
Step 7 -Making the MoD Ontology machine-readable and available online A downloadable version of the final MoD Ontology can be found on GitHub (Finnerty & Moore, 2020). The hierarchical structure, labels, uniform resource identifiers (URIs) and definitions for all entities are described in Table 3. The ontology is accompanied by an annotation manual that provides guidance on how to annotate for these entities in reports of behaviour change interventions (available as Extended data at https://osf.io/4j2xh/ (West et al., 2020)).  Printed material mode of delivery that involves information printed on a product or its packaging, or a label attached to or included with, a product or its packaging, and aims to convey information about that product.
Electronic mode of delivery BCIO:011010 Informational mode of delivery that involves electronic technology in the presentation of information to an intervention recipient.
Television mode of delivery BCIO:011011 Electronic mode of delivery that involves presentation of information that is broadcast and displayed by television.
Includes internet and satellite television. Electronic mode of delivery that involves presentation of information by an electronic screen positioned in a public location.

Upper-Level
Wearable electronic device mode of delivery BCIO:011015 Electronic mode of delivery that involves presentation of information by an electronic screen positioned in a public location.
Includes a watch, clip-on device, spectacles, in-ear devfice, vibrating device.
Electronic environmental object mode of delivery BCIO:011016 Electronic mode of delivery that involves an electronic device positioned in the environment of the intervention recipient that can gather information and respond to commands.
Includes robots, and 'internet of things'.

3-D projection mode of delivery BCIO:011017
Electronic mode of delivery that involves presentation of a 3-D image.
Includes hologram but does not include virtual reality headsets.
Virtual reality mode of delivery BCIO:011018 Electronic mode of delivery that involves use of virtual reality through a virtual reality headset and optionally body movement sensors.
Playable electronic storage mode of delivery BCIO:011019 Electronic mode of delivery that involves presentation of information stored on an object that is inserted into a playing device.
Radio broadcast mode of delivery BCIO:011020 Electronic mode of delivery that involves presentation of audio information that is broadcast and received by a radio receiver.
Call mode of delivery BCIO:011021 Electronic mode of delivery that involves a communication process in which a signal is sent by a caller to a recipient to alert them of the communication intent, giving the recipient the opportunity to engage with the communication.
Includes automated calls and audio messaging.
Audio call mode of delivery BCIO:011022 Call mode of delivery that involves only audio information in the communication.
Video call mode of delivery BCIO:011023 Call mode of delivery that involves video and audio information in the communication. Mode of delivery that involves changing the physical shape, size, structure or appearance of objects in the environment of the intervention recipient.

Upper-Level
This does not include use of textual or pictorial information. It includes lighting, speed humps, use of music, shape and size of containers of consumables.
Somatic mode of delivery BCIO:011034 Mode of delivery that involves devices or substances that alter bodily processes or structure.
Ingestion mode of delivery BCIO:011035 Somatic mode of delivery that involves ingestion of a chemical into the body.
Transdermal mode of delivery BCIO:011036 Ingestion mode of delivery that involves ingestion of a chemical through the skin.
Alimentary mode of delivery BCIO:011037 Somatic mode of delivery that involves ingestion of a chemical through the stomach or intestine.
Pill mode of delivery BCIO:011038 Alimentary mode of delivery that involves swallowing of a pill or oral capsule. Physical stimulus mode of delivery that involves application of physical pressure to the outside of the body.

Upper-Level
Includes massage. Physical stimulus mode of delivery that involves a device that is worn on the body.

Upper-Level
Includes surgery.
Individual-based mode of delivery *BCIO:011055 Mode of delivery that involves one recipient in the location where the intervention is delivered.
Pair-based mode of delivery *BCIO:011056 Mode of delivery that involves two recipients in the location where the intervention is delivered who have an interpersonal relationship.
Group-based mode of delivery *BCIO:011057 Mode of delivery that involves three or more people in the location where the intervention is delivered.
Uni-directional mode of delivery **BCIO:011058 Mode of delivery in which the only causal influence is from the intervention source to the recipient.
Interactional mode of delivery **BCIO:011059 Mode of delivery in which there is causal influence from the intervention source to the recipient and from the recipient to the source.
Synchronous mode of delivery ***BCIO:011060 Mode of delivery that involves delivery and receipt of the intervention or its components occurring at the same time or very close in time.
Asynchronous mode of delivery ***BCIO:011061 Mode of delivery that involves receipt of the intervention or its components taking place a significant period of time after delivery.
Push mode of delivery ****BCIO:011062 Mode of delivery that is not dependent on actions on the part of the intervention recipient.
Pull mode of delivery ****BCIO:011063 Mode of delivery that requires some action on the part of the recipient.
Gamification mode of delivery BCIO:011064 Mode of delivery that involves application of typical elements of game playing to other areas of activity, typically as an online marketing technique to encourage engagement with a product or service.
Includes point scoring, competition with others, and rules of play.
Arts feature mode of delivery BCIO:011065 Mode of delivery that involves application of creativity on the part of the intervention recipient.
Includes art therapy, music therapy, dance and acting.

Note.
Entity IDs correspond to Behaviour Change Intervention Ontology (BCIO);* Only one of individual-based, group-based or pair-based mode of delivery will apply; **only one of uni-directional or interactional mode of delivery will apply; ***only one of synchronous or asynchronous mode of delivery will apply; **** only one of push or pull mode of delivery will apply.
that can be present at the same time as at least one of the other MoD. For example, an intervention that is delivered through face to face (sub-class of Human interactional MoD), can also be classified as an Individual-based or Group-based MoD. It is worth noting that, given the exponential growth in new technologies, this MoD Ontology captures a specific moment in time, and will need updating as technologies and methods develop.

Discussion
The MoD Ontology provides a classification system for describing how behaviour change interventions and techniques are delivered. The ontology consists of 65 entities organised in 15 upper-level entities. Inter-rater reliability was found to be 0.80 (very good) for those familiar with the ontology and 0.58 (acceptable) for those unfamiliar with it, as assessed by Krippendorff's alpha. Together with Source, Schedule and Style it represents the characteristics of Delivery of a behaviour change intervention. Ontologies aim to be dynamic representations that are updated according to new evidence on entities and relationships. As with other lower level ontologies that form part of the BCIO (Michie et al., 2020b), the MoD Ontology will be improved upon and refined through application and feedback by users.
The MoD Ontology contributes to the growing number of methodological resources now freely available to intervention researchers (e.g. Bartholomew et al., 2011;Hoffmann et al., 2014;Hollands et al., 2017;Michie et al., 2013). For example, a Theory and Techniques Tool available free for online, provides an interactive dataset of links between BCTs and their mechanisms of action (i.e. the processes through which BCTs have their effects). The tool was informed by data from evidence synthesis (Carey et al., 2019) and expert consensus (Connell et al., 2019), which were triangulated (Johnston et al., 2018); all three sets of data are available in the tool.
The MoD Ontology contributes to a larger programme of work developing ontologies for other intervention components, the Human Behaviour-Change Project (Michie et al., 2018;Michie et al., 2020a). Within this project, lower level ontologies are being developed for intervention-related entities of content, delivery, tailoring, context, engagement, mechanism of action, and outcome behaviour within the BCIO (Michie et al., 2020b). These ontologies have been developed using an explicit, standardised, and tested method for ontology development created within the Human Behaviour-Change Project (Wright et al., 2020). As the development of the MoD ontology started prior to the development of the BCIO, the process of development was slightly different from the one described in this collection (Wright et al., 2020), containing more rounds of expert feedback and inter-rater reliability testing.

Strengths and limitations
These ontologies provide a framework for applying machine learning and reasoning algorithms to synthesise and interpret evidence, as well as predict outcome. This allows real-time up-to-date evidence to be interrogated by users such as policymakers, planners and intervention designers to answer variants of the "big question": "What works, compared with what, how well, with what exposure, with what behaviours (for how long), for whom, in what settings and why?", across a wide range of contexts. This body of work has the potential to have far-reaching use by and implications for policy-makers, practitioners and researchers, For example, by informing evidence-based guidelines and identifying knowledge gaps.
Further, the use of entity IDs for each entity in the ontology provides a machine-readable identifier for integration in future systems and also allows interoperability between existing ontologies.
Several limitations should be noted about the development process, and the resulting MoD Ontology. Given the rapid growth in new technologies and the fast-moving pace of behavioural science research, the MoD Ontology will need updating and refining as existing methods develop and new methods emerge. However, this is common to all ontologies and indeed considered 'best practice' in ontology development (Arp et al., 2015). Secondly, the intervention reports included in the annotation process were from two larger projects, the Theory and Techniques Project (Michie et al., 2018) and the Human Behaviour-Change Project (Michie et al., 2017). The intervention reports annotated within the ontology development mainly addressed two health-related behaviours, smoking cessation and physical activity; there is always the possibility that other literature may indicate modes of delivery not captured in our set of papers or by our group of experts. However, external inter-rater reliability was tested across diverse behaviours and found to be acceptable. Future applications of the ontologies to a wider collection of behaviours and contexts is likely to extend and improve the ontology. The inter-rater reliability of the annotations conducted by coders unfamiliar with the ontology was lower than that found in other ontologies of the BCIO such as the Intervention Setting Ontology (Norris et al., 2020b), a result that can be explained by the complexity of this ontology. Nonetheless, the coding guidelines were refined throughout the process and the level of reliability increased considerably between the first and second sets of 50 papers.

Conclusions
The MoD Ontology provides a foundation on which future research can build, and its development is intended to be an ongoing and collaborative process. By providing greater clarity about how an intervention and its components are delivered, researchers can add to knowledge as to how MoDs influence intervention effectiveness, both directly and in interaction with other intervention-related entities. This will inform the selection of appropriate MoDs for interventions.

Ethics
Ethical approval was granted by University College London's ethics committee (CEHP/2016/555). Participant consent was gained from the first page of the online Qualtrics survey. Could the authors elaborate more on the potential reasons for discrepancies in interrater reliability 'whether a particular entity was considered an MoD was 61%; and agreement on the specific MoD code assigned was 87.9%' in round 2?

○
Step 5: could it be that the lower agreement between raters was not related to the fact that they were less familiar with the ontology, but by the fact that there were was a wider variety in target behaviors in this selection of reports? Taxonomies are also mostly applied to diet, physical activity, addictive behaviours; could it be that the ontology does not fit as well with screening, infectious diseases etc?
○ Table 2 mentions inter-rater reliability twice for step 3: typo? ○ Table 3: definition of video game delivery seems to copy-pasted from the level above? ○ Table 3: Somatic alteration mode of delivery -also typo (copy-paste above)?
○ expertise to confirm that it is of an acceptable scientific standard.

Introduction
You introduce three classification systems but then move straight into the BCTTv1. It is not clear why you focus on that one and so this paragraph seems to come from nowhere. Could you make the reason you are moving from the three systems to the BCTTv1 more obvious? Also, you start a new paragraph after introducing the three systems but that is a very short paragraph, so I would suggest this needs to be one paragraph together. I also expected in the introduction to see more reference to previous taxonomies and problematising these to establish why this ontology was so important. You don't, for example, mention the EPOC taxonomy and I was not sure why.

Methods and results
Step 1. This step specifies health behaviours. Previously, you have not specified that this relates to health behaviours specifically, in fact you introduce this as including environmental and social problems, and some of the earlier work is related to health worker behaviours. It would be good to have some clarity about whether this is all human behaviour (which I think it is) and to what extent the methods relied on interventions related to health behaviours and whether this is a limitation of the methods. I know you do state this as a limitation but it would be good to see this up front. in the methods and a rationale for why the study was conducted in this way.
Step 2. Can you report the response rate (either in methods or results) and where the raters were from. I'm particularly interested in whether all were from a particular part of the world, what institutions were included. Much of the work rests on these individuals being experts so I think it would be appropriate to include some further information in the text that summarises their credentials and any potential biases they might introduce into the initial ontology.

Discussion
As per the introduction, it would be useful to see how this ontology fits with previous attempts at classifying modes of delivery. If there are none (if the EPOC taxonomy is not an example of this) then it would be good to state that as part of the reason for developing this anew.

Is the work clearly and accurately presented and does it cite the current literature? Partly
Is the study design appropriate and is the work technically sound? Yes

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility? Yes Are the conclusions drawn adequately supported by the results? Yes