“Toxic Butts”: Key Performance Indicators from a California Statewide Social Media Campaign for Tobacco Control

This work was carried out in collaboration between all authors. Author JS designed the study, performed the statistical analysis, wrote the protocol, and wrote the first draft of the manuscript. Authors TN and TR managed the analyses of the study. Author SD managed data collection and assisted in the literature review. All authors read and approved the final manuscript. ABSTRACT Background: The “Toxic Butts” campaign was funded by the California Tobacco Control Program as part of a tobacco product waste toolkit. The campaign was to function as a case study, helping to establish best practices for social media use by tobacco control while at the same time serving as a training tool for local lead agencies interested in better leveraging Twitter and Facebook as primary sources, as well as with a third-party social media monitoring program. Interactions of social media users with the campaign’s content in Facebook and Twitter were examined using six key performance indicators. Two indicators offered novel approaches to quantify engagement, representing any action performed by a social media user with campaign content. Results: The six key performance indicators used to measure campaign performance indicated 1.1 million impressions of “Toxic Butts” campaign content by 340,200 individuals on Facebook, over the six-month period of the study. The largest proportion (42.96 percent) of the campaign’s Twitter followers (n=650) was between the ages of 35 and 44 years, whereas Facebook fans (n=1057) were primarily between the ages of 13 to 17 years (59.1 percent). Twitter followers were nearly evenly split between women and men (51 percent and 49 percent, respectively), whereas Facebook followers were mostly male (60.5 percent compared to 39.50 percent female). Health organizations (34.3 percent) represented the largest share of Twitter followers; Facebook followers were mostly individuals (92.8 percent). Engagement ratios clarified social media users’ interactions with campaign content month-by-month and were considered a valuable point of reference for the overall performance of campaign content. Discussion: The reported key performance indicatorsprovide a starting point of measures of engagement by social media users with a campaign, and the reach of a campaign’s content. Recommendations for future research are provided. use of evaluation metrics for social media in health promotion and in particular the use of Twitter. This case study provides an example of a health promotion social media campaign, detailing the use of several key performance indicators (KPI's) quantifying engagement by social media users with the campaign's Facebook page and Twitter account. The “Toxic Butts” social media campaign was funded by the California Tobacco Control Program (CTCP) of the Department of Public Health as part of a toolkit to mitigate tobacco product waste within the state. The social media campaign's primary goal was to increase public exposure to messaging regarding the environmental impacts of discarded cigarette butts. Previous laboratory research by the project team had established the toxicity of cigarette butts to aquatic organisms [7] and their ubiquitousness in the environment [8,9], but prior to the campaign, no large-scale effort had been made to disseminate this information to the public. The “Toxic Butts” campaign several organizations, including Surfrider This paper reports on the first six months of the campaign, from January 1, 2012 to June 30, 2012, the period funded by CTCP.


INTRODUCTION
Monitoring and evaluation (M & E) methods for health communication campaigns using social media are in their nascent stages, but there are many tools and techniques now available for use by managers and researchers. The US Centers for Disease Control and Prevention (CDC) offers broad guidance on developing evaluation plans for social media [1] and there have been efforts by other groups such as the Robert Wood Johnson Foundation (RJWF) to bring together thought leaders and to collect best practices in social media measurement [2]. Recent publications [3,4] provided guidance on the purpose and use of evaluation metrics for social media in health promotion and in particular the use of Twitter. Private industry is likewise adapting to widespread use of social media and its metrics. A 2012 survey by the University of Massachusetts Dartmouth of executives within the 500 fastest growing private U.S. companies (Inc. 500) reported that although 92 percent of those companies used social media to promote their brands, only 68 percent monitored brand or company performance in social media, while 22 percent had no social media M & E plan in place [5]. Neiger et al. [3] report that there are international organizations such as the #SMMStandards coalition that have formed to provide guidance on social media M & E, as well as to organize international conferences on the topic (e.g., the European Summit on Measurement held in June, 2014 in Amsterdam). Moorhead et al. [6] reviewed scientific literature for original research studies on the use of social media for health communication; these authors determined that the vast majority of studies were exploratory and/or descriptive. In order for the public health community to keep pace with industry standards and to generate a body of research based on social media, health communications professionals must first publish case studies of social media campaigns. These studies must ensure that a campaign's performance metrics are reported, with transparent and replicable methodologies.
This case study provides an example of a health promotion social media campaign, detailing the use of several key performance indicators (KPI's) quantifying engagement by social media users with the campaign's Facebook page and Twitter account. The "Toxic Butts" social media campaign was funded by the California Tobacco Control Program (CTCP) of the Department of Public Health as part of a toolkit to mitigate tobacco product waste within the state. The social media campaign's primary goal was to increase public exposure to messaging regarding the environmental impacts of discarded cigarette butts. Previous laboratory research by the project team had established the toxicity of cigarette butts to aquatic organisms [7] and their ubiquitousness in the environment [8,9], but prior to the campaign, no large-scale effort had been made to disseminate this information to the public. The "Toxic Butts" campaign collaborated with several organizations, including the Washington, DC-based American Legacy Foundation, the California Youth Advocacy Network (CYAN), and the Surfrider Foundation. This paper reports on the first six months of the campaign, from January 1, 2012 to June 30, 2012, the period funded by CTCP.

Key Performance Indicators
The phrase "key performance indicator," commonly referenced by its acronym ("KPI"), refers to a quantifiable performance measure and is standard terminology within private industry. This study presents six KPI's used by campaign staff to measure the "Toxic Butts" campaign's performance, in particularthose which assisted in understanding the growth of the campaign's following within each site, descriptive demographic variables of those followers, and how engaged followers were with campaign content. KPI's included: operating as health or environmental advocates, as well as news organizations, protobacco groups, individuals, and those categorized as "other". 4. Engagement by Twitter followers, measured as a ratio of interactions by followers to the total number of messages sent by the campaign. 5. Impressions of campaign content on Facebook, or the number of times users were exposed, as well as by percentage female and male. 6. Engagement by Facebook fans, measured as a ratio of interactions by fans to the total number of unique individuals who viewed any campaign content.
This paper provides one of the first real-world examples ofM & Emethods for a health promotion social media campaign. It may then inform future research on the use of social media by public health researchers.

Data Sources for Campaign Performance on Facebook and Twitter
Twitter's open API (application programming interface) allows software developers to design third-party programs that can access and collect data from the site. Facebook provides an internal tool to monitor and evaluate a Facebook page's performance, entitled "Facebook Insights." This tool provides general information such as the number of fans a page has acquired over a specific time period, as well as more detailed information such as where site traffic originates, performance metrics on individual posts, and aggregate demographic information of a page's fans, such as age, sex, and geographic location. Facebook has an open API as well, enabling the use of third-party programs to collect data from the site.
There are many third-party programs that access both Twitter and Facebook, gathering and visualizing each site's unstructured data in a multitude of ways. This study collected data from Facebook and Twitter primary sources when available, and used a third-party commercial program, Sprout Social (www.sproutsocial.com), for additional evaluation of campaign performance. Sprout Social accesses both Facebook and Twitter API and provides a variety of metrics on an account's performance over time.

Frequency, Duration and Variety of Campaign Content
Campaign staff published English-language content twice a day, seven days a week, for six months (January 1, 2012 to June 30, 2012). A mixture of paid and organic reach was used on Facebook (paid reach consisted of Facebook ads to increase page likes), whereas no paid advertisements were employed on Twitter. Aside from Facebook ads, the same campaign content was posted on Facebook and Twitter. Occasional exceptions were made to account for character limits on Twitter or when campaign staff engaged with users of a particular site, for instance when responding to a comment made by a follower of the campaign. Content consisted of images and video, text-only statements, and links to other online sources such as news stories. The vast majority of content related to the negative effects of tobacco product waste, and in particular cigarette butts, on the environment. As new content was posted each day, the use of content peripherally related to the campaign such as inspirational quotes having to do with environmentalism more broadly, or news stories about issues in tobacco control in general, were occasionally used.
Campaign staff used many metrics to evaluate performance. This pilot study reports a subset of these metrics, the six KPI's reported previously, focusing on those which gave an indication of the campaign's overall performance rather than its day-to-day operations.

Analysis for campaign performance on facebook and twitter
The majority of the analyses used in developing performance metrics for the "Toxic Butts" campaign were conducted automatically by Facebook Insights and Sprout Social. Some independent analyses were necessary to group Twitter followers and Facebook fans of the campaign into categories of users. To accomplish this, public profiles of each Twitter follower of the campaign were reviewed to determine whether an account explicitly stated that it represented a health organization, an environmental organization, an individual whose purpose was to operate as a health or environmental advocate, a news organization, a protobacco group, an individual with no known organizational affiliation or role as health or environmental advocate, and those which fell under the category "other." Profiles listed under "other" included either unrelated businesses or indecipherable profiles, for instance those containing no text. Facebook profiles were reviewed and placed in categories including health or environmental organization, "other," and individual.
The combined re-tweets and mentions of followers by month are divided by the total number of tweets sent by the campaign in that month to establish the engagement ratio for Twitter. A re-tweet is essentially equivalent to a "share" on Facebook. Twitter users indicate they want to re-tweet a tweet they have viewed, which broadcasts this tweet through their own account, to their own followers, with the addition of an "RT" in front of the text of the original tweet. A mention in Twitter, indicated by an "MT" in the text of a tweet, is employed when a user wants to call attention to another account. The "MT" is placed in front of the account name being mentioned.
For Facebook, the engagement ratio was generated by dividing the number of users who interacted in any way with the "Toxic Butts" Facebook page by the number of users who viewed any content associated with the page, whether fans of the page or not. Facebook engagement was specifically defined as: This method of measuring engagement on Facebook is recommended by the Sprout Social program."Unique users" is differentiated from "users" in that the former measure only counts a user once no matter how many actions she or he performs. In other words, if a user posts to a Facebook page twice that user is still only counted once.
Only the user profiles of Twitter followers and Facebook fans as of June 30, 2012, the last day of the study period, were reviewed. In other words profiles of users who un-liked the "Toxic Butts" page on Facebook or un-followed the "Toxic Butts" Twitter account at any time during the period January 1 to June 29, 2012, were not reviewed.

Campaign Performance on Facebook and Twitter
KPI #1: Number of Facebook fans and Twitter followers by age group.
KPI #2: Number of female and male fans and followers.
The largest proportion (42.86 percent) of the "Toxic Butts" campaign's Twitter followers were between the ages of 35 and 44 years, followed by those between the ages of 45 and 54 years (25 percent) ( Table 1). This contrasted with Facebook fans, who were primarily between the ages of 13 to 17 years (59.1 percent), followed by those aged 18 to 24 years (13.9 percent). Twitter followers were nearly evenly split between women and men (51 percent and 49 percent, respectively), whereas Facebook followers were mostly male (60.5 percent compared to 39.50 percent female).
KPI #3: Number of fans and followers representing health or environmental organizations, or operating as health or environmental advocates, as well as news organizations, pro-tobacco groups, individuals, and those categorized as "other".
When separated into user categories, the largest share of Twitter accounts were those representing health organizations (34.3 percent), followed by those of individuals with no known affiliation or expressed purpose of advocating for either health or environmental causes (20.8 percent) ( During the six-month campaign period there were a total of 236 re-tweets and mentions of the "Toxic Butts" campaign, out of a total 1,233 tweets sent by the campaign. The average engagement ratio during this six-month period was 19.1 (Table 3).
KPI #5: Impressions of campaign content on Facebook, or the number of times users were exposed, as well as by percentage female and male. KPI #6: Engagement by Facebook fans, measured as a ratio of interactions by fans to the total number of unique individuals who viewed any campaign content.
'Impressions' represent views by a Facebook user. From January through June, 2012, content posted on the "Toxic Butts" Facebook page was viewed by 340,200 unique Facebook users a total of 1.1 million times. These users were 52 percent female and 48 percent male. As mentioned previously, differentiating "unique users" from users allows for a more representative count. In this case, 340,200 people together viewed the "Toxic Butts" Facebook page 1.1 million times. In other words some people viewed the page multiple times.Not all impressions were due to fans of the "Toxic Butts" Facebook page. When a page's content is "liked" or "shared" by a fan of that page, that content (a picture, text, video, etc.) can appear in the "news feed," or timeline of activity of a user's friends and pages she or he follows, of that user's friends. In other words, content posted to a page can be viewed by far greater numbers of individuals than are fans of that page. The engagement score used to represent the level of interaction Facebook users had with campaign content was particularly high in January, the first month of the campaign, and then stabilized in the months following with an average of seven for the six-month period reported (Table 4).

CONCLUSION
The six KPI's used to measure campaign performance in this study help to illustrate the characteristics and actions of social media users who are exposed to a campaign. They, or indicators which provide similar data, are suggested as a basic means of evaluation for social media campaigns on Facebook and Twitter. These KPI's illustratedadifference between both the primary age group and sexof followers of the campaign's Facebook page and Twitter account, with Facebook fans younger and predominantly male, while Twitter followers were mostly older and evenly split between female and male. During the study period both Facebook and Twitter users as a whole were primarily between the ages of 18-29, with more women using Facebookand Twitter users nearly evenly split between women and men, according to a nationally representative survey by the Pew Research Center [10].
Since no effort was made to target Facebook or Twitters users of a particular sex, it was somewhat surprising to find that Facebook fans were predominately male. As impressions of campaign content on Facebook were evenly split between females and males, and in order to continue to post as much identical content as possible to both sites, staff chose not to tailor Facebook content to attract more females (or less males). Unscientific polling done by staff appeared to indicate that the slightly tongue-in-cheek name of the campaign, designed to be memorable, was more appealing to younger males on Facebook, which could account for this disparity.
Of categories of Facebook fans and Twitter followers, most were health organizations or advocates as opposed to environmental organizations or advocates, though the latter was represented on both Facebook and Twitter. The largest proportion of Twitter followers were health organizations, where as the largest proportion of Facebook fans were individuals. The average engagement score for Twitter (19.08) wasconsidered representative of the interaction between the campaign's followers and campaign content. Higher scores in the second half of the campaign were assumed to be the result of better brand awareness. After three months of outreach work and consistent messaging it appeared the "Toxic Butts" Twitter account had become a recognized source of information within its topic area.The engagement ratio used to determine interactions Facebook users had with campaign content was also considered representative of overall engagement, and showed a spike in such engagement during the first month of the campaign. This spike was believed to have been due to aggressive outreach efforts made on the site by campaign staff.Content associated with the "Toxic Butts" Facebook page received 1.1 million views by 340.2k unique users, which were nearly evenly split between females and males.

PERFORMANCE ON FACEBOOK AND TWITTER
Monitoring demographic variables of a Facebook page's fans and a Twitter account's followers allows for better tailoring of messages. For example, as most of "Toxic Butts" Facebook fans were between the ages of 13 and 17 years, posts made by campaign staff could potentially take a more conversational tone, translate any scientific research to account for lower levels of health literacy, and include more multimedia content to better engage users. More than two-thirds of Twitter followers were >35 years old, and approximately two-fifths represented a health or environmental organization. Tailored messages for these followers could account for higher levels of health literacy, perhaps sharing scientific studies and policy analyses in depth and more frequently. Additionally, there appears to be significant opportunity to employ Twitter as a means to expand the reach of a health campaign's content by making strategic partnerships. Twitter allows users to privately (through the use of a "Direct Message") contact accounts which follow their own. As stated previously, individuals who followed the "Toxic Butts" campaign represented health and environmental organizations and advocates, thereby opening up direct channels of communication with potential partners. Leveraging other organizations and individuals, who may possess influence both on the social media site and in a campaign's topic area more generally (such as a larger public health organization), could lead to a marked increase in resources and impact. Followers have already indicated an interest in a campaign by following its Twitter account, conceivably making reaching out to these potential collaborators easier than through more formalized, traditional channels of communication.  [11]; all public posts by accounts a Twitter user follows will appear in their feed, but this may matter little if the user is not re-tweeting or mentioning such posts (which indicates the user has actually seen the content). There are many metrics which can be used to measure engagement; the two KPI's included in this study represent one approach to quantifying a two-way flow of communication. For Twitter, the ratio of re-tweets and mentions to total tweets by the campaign was considered representative of the basic overall engagement by "Toxic Butts" followers. For Facebook, the Sprout Social recommended engagement ratio, detailed in the Methods section, was regarded as a good indication of the interactions fans were having with the "Toxic Butts" page. By including impressions, managers were provided with another metric to evaluate the reach and/or popularity of the page over time.

LIMITATIONS
Demographic data reported in this study are self-reported, and thus as limited as selfadministered survey-based research. All data presented were collected from publicly available sources, meaning that information from online accounts set to private was not included. This was not believed to have substantially impacted the performance metrics for Facebook and Twitter.
The use of impressions for social media evaluation has limitations similar to the use of Gross Rating Points (GRP's) in traditional marketing and advertising. There is no way to ensure a social media user saw and processed the content on the screen of their computer or mobile device, just as there is no way to ensure an individual sitting in their home actually viewed an advertisement on their television.

OPPORTUNITIES FOR FUTURE STUDY
Many companies offer software programs with monthly subscription fees to monitor the performance of a campaign's social media profiles, but public health researchers need examples of how these measurements can directly inform either the day-to-day operation of a campaign, its evaluation, or the evaluation of online health interventions. Given both the myriad online environments within social media and the purposes for which organizations and researchers employ social media interventions, KPI's may differ in order to reflect particulargoals and objectives. For best practices in health interventions to emerge there must be significantly more work done to quantify engagement, the resonance of particular content (such as video, images, links to news stories, open-ended questions, etc.) and other such measures and to relate these to outputs and outcomes of social media health campaigns. For instance, an outcome could be the number of fans or followers of a campaign, while an outcome could be the overall engagement of those fans and followers with the campaign. Studies on the return-on-investment (ROI) of social media use by public health communication campaigns would further assist in showcasing the utility of these sites, allowing for comparisons between campaigns using traditional and new media and placing the metrics reported here within a broader context. Lastly, the "Toxic Butts" campaign was designed to appeal, broadly, to individuals and organizations interested in environmental stewardship, tobacco control, or both. Despite the same content being shared on both sites, young individuals comprised the bulk of the campaign's fans on Facebook, whereas most Twitter followers were health or environmental organizations. There are numerous possible explanations for this difference, for instance perhaps health organizations are not fully utilizing Facebook for partnership building. Deciphering differences in receptivity to content by audience, topic, and social media sites employed, is recommended as an important area for further study.

CONSENT
Not applicable.

ETHICAL APPROVAL
Sprout Social is limited to capturing data that are publicly available. For posts made in social media, only those that are made by users who have indicated their posts may be made public are collected. All data collected from the Toxic Butt's Facebook and Twitter profiles are aggregated with no individual user's information reported. This research was granted a waiver from human subjects review from San Diego State University's Institutional Review Board.