Evaluation of Positive Choices, a National Initiative to Disseminate Evidence-Based Alcohol and Other Drug Prevention Strategies: Web-Based Survey Study

Background To prevent adolescents from initiating alcohol and other drug use and reduce the associated harms, effective strategies need to be implemented. Despite their availability, effective school-based programs and evidence-informed parental guidelines are not consistently implemented. The Positive Choices alcohol and other drug prevention initiative and website was launched to address this research and practice gap. The intended end users were school staff, parents, and school students. An 8-month postlaunch evaluation of the website showed that end users generally had positive feedback on the website’s usability, and following its use, most of them would consider the evidence base and effectiveness of drug education resources. This study extends this initial evaluation by examining the effectiveness and impact of the Positive Choices initiative over a 3-year period. Objective Guided by the five dimensions of the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework, the study assessed the impact of the Positive Choices initiative in increasing awareness and implementation of evidence-based drug prevention. Methods Data were collected between 2017 and 2019, using web-based evaluation and community awareness surveys. Data from the surveys were merged to examine reach, effectiveness, adoption, implementation, and maintenance using descriptive statistics. Google Analytics was used to further understand the reach of the website. The System Usability Scale was used to measure website usability. In addition, inductive analysis was used to assess the participants’ feedback about Positive Choices. Results A total of 5 years after launching, the Positive Choices website has reached 1.7 million users. A national Australian campaign increased awareness from 8% to 14% among school staff and from 15% to 22% among parents after the campaign. Following a brief interaction with the website, most participants, who were not already following the recommended strategies, reported an intention to shift toward evidence-based practices. The System Usability Scale score for the website was good for both user groups. The participants intended to maintain their use of the Positive Choices website in the future. Both user groups reported high level of confidence in communicating about topics related to alcohol and other drugs. Participants’ suggestions for improvement informed a recent website update. Conclusions The Positive Choices website has the capacity to be an effective strategy for disseminating evidence-based drug prevention information and resources widely. The findings highlight the importance of investing in ongoing maintenance and promotion to enhance awareness of health websites. With the increased use and acceptability of health education websites, teams should ensure that websites are easy to navigate, are engaging, use simple language, contain evidence-informed resources, and are supported by ongoing promotional activities.


Background
Adolescence is marked by considerable emotional, social, and physical changes, including increasing autonomy from parents, greater influence from peers, and increased likelihood to engage in risk-taking behaviors [1]. Corresponding to this period of experimentation, harms associated with risky behaviors such as alcohol and other drug (AOD) use peak during adolescence and early adulthood [2][3][4]. Globally, alcohol, tobacco, and cannabis are the most commonly used drugs among adolescents [4]. This also holds true for Australia, where a 2017 national survey among secondary school students aged 12 to 17 years showed that 46% had consumed a full serve of alcohol, 14% had used cannabis, and 13% had smoked cigarettes in the past year [5]. Early initiation of AOD use is associated with a range of negative outcomes including poor school performance, truancy, school dropout, juvenile offending, and increased risk of drug dependence and mental illness during adulthood [6][7][8][9]. To interrupt this trajectory and reduce the harms associated with AOD use, effective strategies are needed to prevent the onset and escalation of their use.
Studies have identified a number of modifiable individual risk factors that are consistently associated with AOD use among adolescents [10]. They include peer AOD use, low self-efficacy to refuse alcohol or other drugs, poor school engagement and connectedness, and mental health disorders such as depression and attention-deficit/hyperactivity disorder [10][11][12]. Studies also highlight the importance of parents in influencing and preventing adolescents' AOD use. Recent evidence shows an association between parental supply of alcohol and increased risk of alcohol-related harm during adolescence and early adulthood [13,14]. This has challenged the commonly held perception that allowing teenagers to drink alcohol under parental supervision protects them against alcohol-related harms. Similarly, less restrictive parental attitudes toward alcohol use are likely to lead to earlier and more frequent alcohol use and increased drunkenness among adolescents [15].
Prevention programs targeting individual and parental risk factors are effective in reducing AOD use among adolescents [16][17][18][19][20][21][22]. A number of prevention programs implemented during secondary school have consistently demonstrated effectiveness in reducing AOD use [18,21], and school-based delivery offers a number of advantages including tailoring of messages to students' developmental level and universal delivery to achieve wide reach [22,23]. Internet-and computer-based AOD prevention programs offer additional benefits in terms of engaging youth and increasing implementation fidelity and have been found to be effective in reducing adolescent AOD use [17]. Despite the growing evidence base, only a small proportion of schools implement effective AOD prevention strategies. A 2003 review of 3 decades of AOD education studies concluded that worldwide, effective AOD prevention is not widely implemented [24]. More recently, our 2017 survey of Australian schoolteachers found that <25% of teachers had implemented evidence-based AOD education programs. The study identified lack of confidence, resources, time, and support from school; attitude of parents and students; and difficulty in communicating as the main barriers to implementing programs [25].
Parenting strategies that are consistently associated with delayed initiation of alcohol use include parental monitoring, limited access to alcohol, parent-child relationship quality, parental involvement, and communication [20,26]. However, studies show that parents do not always act in accordance with evidence-based parenting strategies, which may be related to lack of clear guidance or confidence [27,28]. Although parents actively seek information about illicit drugs and parenting practices to prevent AOD use, primarily from friends or the internet, they often report low-to-moderate confidence in communicating and influencing their children's choices regarding AOD use [29]. Programs aimed at modifying parenting practices and promoting the use of effective prevention strategies have shown promise in reducing adolescent AOD use [30][31][32]. Honest communication between parents and adolescents, established on the basis of positive parent-adolescent relationship, has been associated with reduced alcohol use among adolescents [32][33][34]. This highlights the importance of engaging parents in efforts to reduce harms associated with adolescent AOD use.
It is critical that effective AOD prevention strategies are implemented consistently and widely to alleviate the considerable burden associated with AOD use. Therefore, teachers, school staff, and parents need to have easy access to evidence-based information, strategies, and programs that equip them to respond most effectively. The Positive Choices national AOD prevention initiative was funded by the Australian Government Department of Health to enhance access to and implementation of evidence-based AOD prevention strategies within school communities. The initiative responds to a call from school principals for support to implement evidence-based AOD prevention resources (Australian National Council on Drugs; 2013) and was developed iteratively with experts and end users. School staff, parents, and students provided input and feedback across two phases (for full details refer to the study by Stapinski et al [25]): (1) formative consultation to clarify scope and identify user needs and (2) review and feedback on a beta version of the website [35]. The final website was launched in December 2015 and provides web-based training, support, and access to a centralized database of evidence-based AOD prevention programs, recommendations, and resources [25]. The website emphasizes the importance of implementing resources that are supported by research evidence-only evidence-based resources are listed, and each resource page provides a rating to indicate the strength of the supporting evidence according to the Australian National Health and Medical Research Council's evidence hierarchy [36]. To facilitate implementation, resources are categorized based on purpose and intended audience in a searchable database, with brief factsheets to guide users about when and how to implement evidence-based prevention strategies. A survey was conducted 8 months after the launch to determine the initial impact of Positive Choices. Among teachers who accessed the website, most found it useful and reported that they would continue using it, would recommend it to others, felt more comfortable discussing AOD use prevention following website access, and felt that their students were more engaged with AOD education since using the website [25]. When compared with a general teacher sample, teachers who used Positive Choices were more likely to consider the evidence base and effectiveness of AOD education resources before using them in class [25]. Despite highlighting these benefits associated with the use of Positive Choices, this initial evaluation did not reveal the specific factors or features that facilitated the dissemination of evidence-based AOD prevention strategies.

Objectives
In this study, we extended this initial evaluation by conducting a more comprehensive examination of the effectiveness and impact of the Positive Choices initiative over a 3-year period, between 2017 and 2019. This is the first evaluation of a web-based health initiative that specifically promotes translation of evidence-based AOD prevention resources to school communities, parents, and youth. To guide this evaluation, we applied the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework, which was developed to facilitate comprehensive and rigorous evaluation of health promotion initiatives, spanning these 5 key dimensions. It has been widely applied to evaluate the implementation and real-world impact across a variety of settings, including educational settings [37][38][39][40]. Guided by this framework, this study assessed the impact of the Positive Choices initiative in increasing awareness and implementation of evidence-based AOD prevention strategies. The study aimed to evaluate the reach of the Positive Choices website, its effectiveness in improving access to and uptake of effective AOD prevention strategies, and its adoption in accessing evidence-based prevention strategies. Furthermore, implementation will be evaluated by users' capacity to interact with the website, and maintenance will be evaluated by users' intention to continue using the website for evidence-based strategies. The study also obtained feedback from end users to improve and optimize the Positive Choices website.

Methods
Data were collected between 2017 and 2019, from several sources to evaluate the dimensions aligned with the RE-AIM framework, as described in the following sections.

Overview
Reach was assessed via the measurement of access and awareness, using 2 data sources. Access was measured using site use analytics and operationalized as follows: How many unique users have accessed Positive Choices? Awareness was assessed via community awareness surveys and operationalized as follows: Have you heard of Positive Choices before?

Site Use Analytics
Google Analytics was used to obtain a detailed analysis of website traffic between January 2017 and March 2021. This included information on the number of unique website users and page views, pages viewed per session, and average time users spent on each page.

Community Awareness Survey: School Staff and Parents
A web-based survey was administered in July 2018 to assess the Australian community's awareness and use of Positive Choices resources. Using voluntary response sampling, school staff and parents were recruited via targeted advertisements on Positive Choices social media channels or mailing lists. Participants who completed the survey were offered the chance to enter a prize draw to win a laptop. Within this general sample, the proportion of the community that was aware of the website was identified by using a single question: Have you heard of Positive Choices before? Information was also collected to ascertain the number of participants who were using sources other than Positive Choices and how participants were accessing AOD information. The same survey was administered after 6 months (November 2018) and 12 months (May 2019) to determine whether awareness increased following a national social media campaign promoting Positive Choices.

Overview
Positive Choices aims to improve access to and uptake of effective AOD prevention strategies. Accordingly, effectiveness was assessed via an evaluation survey completed by Positive Choices end users (school staff and parents). It was measured through examination of whether engagement with Positive Choices was associated with increased intention to implement evidence-based AOD education and was operationalized as follows: Has the use of Positive Choices changed users' intentions in implementing evidence-based teaching and parenting practices?

Evaluation Survey: School Staff and Parents
To capture the effectiveness of Positive Choices, two anonymous evaluation surveys were administered: the first between August 2017 and September 2017 and the second between May 2019 and June 2019. Using voluntary random sampling, participants were recruited via the Positive Choices mailing list and social media campaigns. Eligible participants were school staff or parents or guardians of children or adolescents, were Australian residents, and had access to the internet and a device to complete the survey. Participants were reimbursed for their time with an Aus $40 (US $28) gift voucher (2017) or the opportunity to enter a prize draw to win a laptop (2019).
Participants were asked to spend time reading and interacting with the Positive Choices resources, after which they reported on their intentions to implement evidence-based prevention strategies. Data were collected via a web-based survey platform, Survey Monkey, and responses from the 2 evaluations were collated to provide an overview of the effectiveness spanning from 2017 to 2019.
For school staff, questions assessed whether they intended to (1) implement teaching resources that have been tested in schools and proven to prevent AOD use, (2) communicate with students about the risks and effects associated with AOD use, and (3) correct the misperception that AOD use is the norm. For parents, questions assessed whether they intended to (1) encourage open communication with their children about AOD, (2) have explicit conversation with their children about AOD use, (3) correct the misperception that AOD use is common, (4) clearly communicate about their expectations to their children, (5) change their own AOD use to model appropriate behavior, (6) avoid parental supply of alcohol, and (7) closely monitor their children's whereabouts. School staff and parents responded to these behavioral intention items by selecting whether they (1) already do or have done this, (2) plan to do this after viewing Positive Choices, or (3) do not plan to do this in the future.

Adoption and Implementation
Similar to the effectiveness dimension, the adoption and implementation dimensions were assessed via the web-based evaluation surveys. Adoption was assessed by identifying barriers to and enablers of access and uptake of evidence-based prevention strategies by school staff and parents. A single question assessed which of the following characteristics of web-based AOD prevention resources or information were most valued by school staff: (1) evidence-based information, (2) resources that had been tested in schools and proven to prevent AOD use, (3) engaging website, (4) interactive website, (5) website that is easy to navigate and use, and (6) simple and easy-to-understand language. Similarly, enablers for parents and guardians were identified through the same items with the addition of the following two items that pertained specifically to parents: (1) parental strategies that have been proven to be effective and (2) website with advice from other parents. Implementation was evaluated using the System Usability Scale (SUS) [41]. It is a standardized instrument used to measure the usability of products, software, apps, and websites. It provides participants with 10 usability-related statements on a 5-point Likert scale, ranging from "strongly disagree" to "strongly agree." SUS scores range from 0 to 100, with score ≥85 representing exceptional usability, score between 50 and 70 representing good usability, and score <50 representing unacceptable usability [41,42]. As such, participants' capacity to interact with the website to access and subsequently deliver evidence-based strategies was assessed using the SUS scores.

Maintenance
This dimension was assessed using the website evaluation surveys, as described previously. For the participating school staff and parents, the question assessed whether users intended to access Positive Choices in the future. It was conceptualized as follows: Did participants intend to maintain their use of the Positive Choices resources?

General Impression
To assess school staff and parents' general feedback about the Positive Choices website, the evaluation survey included the following open-text items: (1) Do you have any suggestions for improving the website? and (2) Do you have any final comments about the website?

Data Analysis
The evaluation surveys were administered via Survey Monkey, and the data were exported to Stata (version 14; StataCorp), which was used to generate descriptive data, including mean scores and response frequencies. Data from the 2017 and 2019 evaluation surveys were merged to examine reach, effectiveness, adoption, implementation, and maintenance, using descriptive statistics, separately for school staff and parents. The SUS score was calculated by converting raw individual scores for each question to a number; these scores were summed to obtain the total score, 5 was subtracted from the total score of all odd-numbered questions, and 25 was subtracted from the total score of all even-numbered questions. Then, the total score of the new values was multiplied by 2.5 [43].
Data on community awareness were also collected via Survey Monkey and exported to Stata to generate descriptive data. The data were examined separately for school staff and parents at baseline, 6 months, and 12 months.
Inductive analysis was used to assess participants' feedback about Positive Choices [44], using the qualitative data analysis software, NVivo (version 12; QSR International). Feedback from participants was examined by SN, who then developed a coding framework. Using the framework, common themes were identified independently by two coders (SN and TG).

Ethics Approval
The 2017 survey was approved by the Human Research Ethics Committee, University of New South Wales (project number HC12548), and the 2019 survey was approved by the Human Research Ethics Committee, University of Sydney (project number 2018/873).

School Staff
All school staff who completed the community awareness surveys were high school staff (719/719, 100%), most resided in NSW (259/719, 36%), and were based in major cities (381/719, 52.9%; Table 2). Most school staff (432/719, 60.1%) worked in public schools in a number of different roles (eg, head teacher, curriculum coordinator, language teacher, mathematics or science teacher, advanced skills teacher, student support or school counselor, and teacher assistant) and worked with Aboriginal or Torres Strait Islander students.

Reach: How Many Users Have Accessed Positive Choices?
Overview Between January 1, 2017, and March 31, 2021, Positive Choices had been visited by 1.7 million unique users, of which 1.53 million (90%) were first-time users. Figure 1 shows that the number of monthly users has consistently increased since January 2017. Similarly, the number of monthly page views have increased from 8367 in January 2017 to 123,699 in March 2021. In terms of geographical reach, most of the website's visitors are Australian residents (41%); however, the website is also frequently accessed by international users from countries such as the United States, India, the Philippines, and the United Kingdom.

Community Awareness
The spike in website visitors seen between July 2018 and November 2018 ( Figure 1) coincided with the community awareness campaign run from June 2018 to October 2018. During this period, there was an 84% increase in the average monthly users of the Positive Choices website from 20,115 to 37,047. There was also an increase in users' engagement on the Positive Choices Facebook page, with the number of likes increasing by 86%, from 670 to 1243. Results from the surveys showed that awareness about Positive Choices increased among school staff from 8% at baseline to 14% at 6 months (immediately after the awareness campaign) and 15% at 12 months. Among parents, awareness increased from 15% at baseline to 22% at 6 months (immediately after the awareness campaign) before returning to 15% at 12 months.
According to the 12-month survey, of the school staff who were not using Positive Choices for AOD education resources and information (191/269,71%), 45 Table 3 shows that approximately half of the school staff sample (35/73, 48%) were already implementing AOD education resources that were tested and found to be effective in schools; among those who were not implementing those resources, 89% (34/38) intended to after using Positive Choices. Most respondents were already communicating to their students about the risks and effects of AOD use (52/73, 71%), and of the remaining respondents, 90% (19/21) intended to do so after using Positive Choices. When it came to correcting students' misperceptions about AOD use, most respondents were already doing this (52/73, 71%) and of the remaining respondents, 81% (17/21) intended to commence this after viewing the Positive Choices website. Table 3. Effectiveness in changing intentions to use evidence-based strategies (school staff).

School Staff
Respondents who will not implement evidence-based strategies in the future, n (%) Among respondents who were not implementing evidence-based strategies, those who intend to after viewing Positive Choices, n (%) Respondents who are currently implementing evidence-based strategy (n=73), n (%) 4 (11) a 34 (89) a 35 (48) Implement teaching resources that were tested in schools and proven to prevent alcohol and drug use 2 (10) b 18 (90) b 53 (73) Communicate with students about the risks and effects of alcohol and drug use 4 (19) c 17 (81) c 52 (71) Correct the misperception that alcohol and other drug use is common or "the norm" a Sample size, n=38. b Sample size, n=20. c Sample size, n=21. Table 4 shows that most parents reported that they were already having open (100/127, 78.7%) and clear conversations (95/127, 74.8%) with their children about AOD use. Among those parents who were not already doing this, 85% (23/27) planned to have open and 78% (25/32) planned to have clear conversations with their children after viewing resources on Positive Choices. Most parents reported that they were already having explicit conversations about AOD with their children (88/127, 69.3%), and most of the remaining parents (36/39, 92%) planned to do so after using Positive Choices. Most parents in the survey were already avoiding supplying alcohol to their children (103/127, 81.1%). Of those who were not doing this, 38% (9/24) reported that they would avoid supplying alcohol after viewing resources on Positive Choices. Similar to the school staff, most parents reported that they were already correcting misperceptions about AOD use being the norm (80/127, 62.9%), and of the remaining parents, 83% (39/47) planned to implement this behavior after using Positive Choices. Most parents reported that they have already adapted their own AOD use to model appropriate behavior for their children (87/127, 68.5%), and of the remaining parents, 45% (18/40) planned to modify their behavior after using Positive Choices. After viewing resources on Positive Choices, most parents reported that they will monitor their children's whereabouts more closely (64/74, 86%). Table 4. Effectiveness in changing intentions (parents).

Parents
Respondents who will not implement evidence-based strategies in the future, n (%) Among respondents who were not implementing evidence-based strategies, those who intend to after viewing Positive Choices, n (%) Respondents who are currently implementing evidence-based strategy (n=127), n (%)

Implementation
Regarding usability of the Positive Choices website, the mean SUS score was 75 (SD 15.1; range 35-100), indicating good website usability.
Parents also reported high confidence in discussing AOD topics: 48.8% (62/127) were "very confident," 31.5% (40/127) were "confident," 17.3% (22/127) felt "somewhat confident," and 2.4% (3/127) felt "not very confident." These ratings suggest that confidence was not a significant barrier to evidence-based prevention in this sample. This is in contrast to the general community sample, from the community awareness survey, where lack of confidence was reported as the greatest barrier for parents to having conversations with their children about AOD use.

Implementation
The mean SUS score for the parent group was 74 (SD 12.9; range 40-100), indicating good website usability, similar to that found in the school staff group.

User Feedback on the Website
The main themes that emerged from the analysis of users' feedback on the website centered around the website's content and features and promotion and increasing the usability and diversity of the website. Most responses from participants were suggestions to improve the Positive Choices website's layout to make it less cluttered and more visually appealing and engaging (especially to the student user group) and to improve website navigation. The other subthemes that were identified included feedback about making the website more accessible to multicultural communities and students with additional learning needs, suggestions for new content and features, and the need to increase promotion. In response to this feedback, redesign of the Positive Choices website was conducted from December 2020 to January 2021, in consultation with a specialist user experience company. The themes and subthemes from the participants' feedback and actions taken in response to user suggestions are described in Multimedia Appendix 1.

Principal Findings
Adolescence is a time of increased susceptibility to engaging in risk-taking behaviors such as AOD use. Prevention strategies designed to target modifiable risk factors have been demonstrated to be effective in reducing AOD use and related harms among adolescents [16,18]. Despite this evidence, effective AOD prevention strategies are not widely implemented in schools or by parents. Positive Choices was developed to help overcome some of the barriers faced by school communities and parents and promote widespread implementation of evidence-based prevention. This study used the RE-AIM framework to evaluate the success of Positive Choices in increasing awareness and implementation of evidence-based AOD prevention practices among school staff and parents across the five domains: reach, efficacy, adoption, implementation, and maintenance.
A total of 5 years following its launch, this Australian AOD prevention website has reached >1.7 million users, and the page views have continued to grow. The user reach of Positive Choices has expanded beyond Australia, with 59% of users currently being international. Promotion via a national social media campaign proved to be an effective method for increasing awareness within the Australian community, with awareness increasing from 8% to 14% among school staff and from 15% to 22% among parents after the campaign. The findings suggest that sustained promotion efforts may be required to maintain high levels of awareness about Positive Choices. Even among participants who were not using Positive Choices, most reported using other web-based sources to obtain AOD education information and resources, thus supporting the value of the internet as a tool for disseminating health information and promoting evidence-based prevention strategies [45,46].
Although evidence shows that effective AOD prevention strategies are not commonly implemented by schools [24] or parents [27], most school staff and parents in this study reported that they were already implementing evidence-based AOD prevention strategies. Moreover, following a brief interaction with the website, most school staff and parents reported an intention to shift toward evidence-based practices in cases where they were not already following the recommended prevention strategies for young people. The exception was the recommendation that parents modify their own alcohol use to model appropriate behavior for their children, which may indicate that changing their own behavior is a significant challenge for parents, which requires additional attention and support. The sample intended to maintain their use of the Positive Choices website in the future to obtain evidence-based information and resources. The enablers that facilitated adoption and implementation were ease of availability and good website usability. Both user groups reported high level of confidence in communicating about AOD-related topics. Study participants' suggestions for improvement pertained to some of these enablers (eg, improving navigation and making the website more engaging), and thus, have informed a recent website update.
The widespread use of web-based health promotion tools and websites by school staff and parents to access information and resources highlights the need for quality control measures. This will ensure that they are evidence-based, up to date, and engaging and use simple language. In addition, the websites themselves should be accessible and easy to use and navigate. Findings from robust evaluations will allow critical assessment of the benefits of such tools and websites, inform content, and inform website updates and developments to optimize their usability. Although there have been evaluations of mental health information websites [47,48], to the best of our knowledge, this is the first evaluation of a web-based health initiative that specifically promotes the translation of evidence-based AOD prevention resources for school communities, parents, and youth.

Limitations
A limitation of the study is that the findings rely on users' self-reported intentions to implement evidence-based strategies, rather than actual assessment of their subsequent behavior. Furthermore, the study design assessed participants' feedback and behavioral intentions after they interacted with the website for a relatively short period. Future studies with a pre-post study design will enable a more comprehensive evaluation, including assessment of whether the website affected subsequent behaviors and implementation of evidence-based strategies by school staff and parents. Another limitation is that the targeted campaign used to recruit participants for the evaluation study may have resulted in a sample selection bias. Most of the surveyed school staff and parents in the current sample reported that they were already implementing evidence-based prevention strategies and were confident about discussing alcohol and drug use. These results contrast with previous evidence suggesting low confidence and implementation of evidence-based AOD prevention strategies among parents and in schools [24,27], and thus, may reflect that our recruitment strategies attracted a sample who were already interested in and aware of evidence-based AOD prevention approaches.

Conclusions
The findings from the evaluation of Positive Choices demonstrate that the website reached 1.7 million users, and it has the capacity to be an effective strategy for disseminating evidence-based AOD prevention information and resources. Furthermore, this evaluation highlights the importance of investing in ongoing promotion to maintain or enhance awareness of health websites. As the use and acceptability of health education websites increase, developers and health care and research teams should ensure that health websites developed in the future are easy to navigate, are engaging, use simple language, contain evidence-informed resources, and are supported by ongoing promotional activities. This study provides methodology and recommendations to guide future evaluations of web-based health tools to determine their effect on behavioral and health outcomes.