Obstacles and Opportunities of Using a Mobile App for Marine Mammal Research

This study investigates the use of a mobile application, Whale mAPP, as a citizen science tool for collecting marine mammal sighting data. In just over three months, 1261 marine mammal sightings were observed and recorded by 39 citizen scientists in Southeast Alaska. The resulting data, along with a preliminary and post-Whale mAPP questionnaires, were used to evaluate the tool’s scientific, educational, and engagement feasibility. A comparison of Whale mAPP Steller sea lion distribution data to a scientific dataset were comparable (91% overlap) given a high enough sample size (n = 73) and dense spatial coverage. In addition, after using Whale mAPP for two weeks, citizen scientists improved their marine mammal identification skills and self-initiated further learning, representing preliminary steps in developing an engaging citizen science project. While the app experienced high initial enthusiasm, maintaining prolonged commitment represents one of the fundamental challenges for this project. Increasing participation with targeted recruitment and sustained communication will help combat the limitations of sample size and spatial coverage. Overall, this study emphasizes the importance of early evaluation of the educational and scientific outcomes of a citizen science project, so that limitations are recognized and reduced.


Introduction
An estimated 37% of marine mammals are at risk of extinction [1] due to long-term, broad-scale, and adverse geopolitical issues including pollution, habitat loss, shipping, and global climate change [2,3]. Monitoring the impacts of these pressures on the distribution patterns of wide-ranging marine mammals requires data across large temporal and spatial scales [4,5]. Current monitoring efforts lack the spatial extent and frequency required to detect abrupt and fast marine mammal declines [6], and therefore limit efforts to manage and protect marine mammals from various pressures. Furthermore, many applied and basic ecological questions are left unanswered as they occur at large geographic and temporal scales beyond the reach of traditional research methods [7].
One approach to combat the scarcity of such data is to embrace citizen science, a non-standardized alternative method of data collection [8]. Citizen science projects are often focused on collecting multi-year and regional-scale data [9,10] primarily to estimate species distributions [9,[11][12][13]. Citizen science volunteered geographic information projects gather a diverse set of data on both website and mobile application platforms [14][15][16] to accomplish various tasks, ranging from mapping wheel chair accessibility locations [17] to monitoring environmental health in a city [18]. Further, citizen science research represents a low cost approach for gathering large spatial scale data, enabling more data collection than what could feasibly be accomplished by a scientist with equivalent monetary and time budgets [9,11,13]. In addition, these projects often provide educational benefits including increased content knowledge and perspective on science [12,19,20]. One such example is the avian citizen science project eBird that contributed to over 150 peer-reviewed articles to date, increased scientific understanding of avian migration ecology, informed conservation policy, and provided an invaluable educational tool for users [21]. Similarly, these methods can be applied to marine mammal research as well. Examples of marine mammal citizen science research include: the American Cetacean Society surveys [22,23] annual volunteer cetacean counts in Hawaii [24], specific species projects [25,26], and analyses of historical whaling records [5,[27][28][29][30].
While citizen science projects can provide a vast array of scientific and educational benefits, the methods and data should be properly evaluated and interpreted with caution. Limitations including data fragmentation, uncertainty regarding data accuracy, and limited applicability for research, must be overcome to have a successful project [19,31]. In addition, unlike traditional scientific studies that use standardized transects and survey methods, citizen science projects often do not regulate the "survey area" or range covered by an individual data collector. This discrepancy can be seen in studies that show data over-reported in high use areas [32] and for uncommon species [33]. Sampling error can also occur when observers differ in their ability to detect, identify and quantify species or events [32], leading to variation in data accuracy [10,34,35]. Sample size, coupled with species' ecological and detection differences can alter the performance of distribution models [36]. Consequently, inadequate evaluation of these biases associated with citizen science data will lead to false results that fail to accurately describe the species distribution.
This study examined some of the opportunities and obstacles faced when using the citizen science Android-based mobile application (app) called Whale mAPP (www.whalemapp.org) to record opportunistic marine mammal sighting data in Southeast Alaska. In sum, the Whale mAPP project aims to apply tailored citizen science methods to monitoring marine mammals within the study area of Southeast Alaska. Overall, Southeast Alaska offers a potentially ideal location for collecting these data due to a high summer ship-based tourism [37] and marine mammal abundance [3,38,39]. Concurrently, long-term studies describing marine mammal distributions are resource intensive and therefore limited [38][39][40][41], making robust and reliable marine mammal data in this region valuable.
To test how well Whale mAPP could generate such a dataset, citizen scientists were recruited to use the app in the summer of 2014. Objectives included evaluating the potential educational and scientific benefits and limitations of Whale mAPP for the purpose of improving this and future citizen science projects. To achieve the educational objectives, citizen scientists completed a questionnaire before and after using the mobile app to assess participants' motivations, general experience, and educational outcomes of using the app. Technological glitches and participant retention added additional insight.
To evaluate data quality, spatial, user, and sample size biases were measured. A Steller sea lion case study was used to measure the robustness of the resulting species data by comparing the Whale mAPP data to data collected using traditional survey methods [41]. By examining the Whale mAPP citizen science project through both a scientific and educational lens, this study emphasizes the importance of evaluating all components of a citizen science project.
Overall, Whale mAPP is used as a case study to demonstrate the value of project assessment for improving citizen science recruitment, retention, and data quality. This study also describes common citizen science data limitations including user retention, spatial bias, sample size, and technology errors. Thus, evaluating these restrictions and designing a project to reduce biases is key for supporting a successful citizen science project.
Once activated, Whale mAPP automatically recorded the user's location every 30 s, resulting in a record of the survey track line (Figure 1). Upon sighting a marine mammal, users selected the binocular icon ( Figure 1) and were asked to identify the species using a drop-down menu. Additionally, users were asked to record the number of individuals (categorical options: 1, 2, 3, . . . to 10, then 11-15, 16-20, 21-30, etc.), distance and direction to the animal(s) (categorical options: N, NE, E, SE, S, SW, W, NW, and 0-500 m, 500 m-1 km, 1-2 km, 2-5 km, 5+ km), the animal's behavior (categorical options: feeding, logging, milling, socializing, thermal regulation, travelling, other, unknown behavior), weather conditions (cloud cover and wind), and a single five star confidence rating scoring the user's confidence in the accuracy of the data entered for each sighting (1 star being the lowest confidence, 5 stars being the highest confidence) ( Figure 1). Because volunteers could, but were not required to take a photo of the marine mammal, their species identification accuracy was assessed based on their confidence ranking. The assumption was made that a higher confidence rating coincided with greater accuracy. Upon completion, a recorded sighting was noted on the map with an icon on top of the black track line (Figure 1). Since many areas of Southeast Alaska do not support cellular reception, Whale mAPP stored data locally on a base map of Southeast Alaska that was accessible offline. Once cellular reception was available, the data were automatically transmitted to a geodatabase. A summary of the processes and topics evaluated in this study are provided as a flowchart ( Figure 2). Citizen scientists were recruited at the Alaska Whale Foundation's Center for Coastal Conservation in Warm Springs Bay, Southeast Alaska (57.09 • N, −134.84 • W). Every person who entered the center was told about the project, marine mammals in the area, and asked if they were interested in using Whale mAPP. Potential participants included guests and crew aboard nature tourism cruises, private vessel operators, and commercial fishermen. All participants were provided with a user manual and marine mammal identification guide with illustrations completed by Pieter Folken (full body drawings) and Courtney Hann (all other illustrations) ( Figure 3). While all participants were offered additional training, the length and duration varied based on their available time and interest. Additional training walked the citizen scientist through a mock scenario of how to use Whale mAPP, discussed scientific technique and rationale, and clarified data collection methods. Outside of this study, Whale mAPP citizen scientists rarely receive in-person training and instead rely on online instructions for using the app. Therefore, the in-person training was not required for participants in this study. To maximize user participation, citizen scientists were allowed to collect data following their preferred routes of travel [9,11]. A simple data entry platform with clear data protocols [43] and a standardized method [44] were implemented to minimize error and ensure the public could collect and submit accurate data.

Participant Surveys
Participant motivations, general experience, and educational outcomes were evaluated using two questionnaires: one completed prior to using Whale mAPP and the second completed after using the app for more than two weeks. Questions included marine mammal identification test, ranking knowledge of various topics, selecting why they chose to use Whale mAPP and why/if they stopped, noting the percent of time spent doing various activities while using Whale mAPP, noting actions taken outside of using Whale mAPP to learn about marine mammals, and ranking various statements regarding their experience using Whale mAPP. Questionnaires were designed using Qualitrics software and distributed in person and by email. All statistical analyses were performed with Microsoft Excel 14.2.0 and R [45]. Permission to collect data on volunteers over the age of 18 was granted by the Oregon State University 5234 Institutional Review Board, study number 6273.
Two informal learning goals were also considered in conjunction with this study: (1) participants' developing interest in science; and (2) participants' understanding of science knowledge [46]. The first goal, interpreting participants' developing interest in science, was assessed by identifying user interests and any actions they took to learn more about marine mammals. Consequently, to evaluate user motivation, participants responded to questions on why they chose to participate in the project, if they would continue to use Whale mAPP, and their enjoyment level from participating. From this, user attention and variability was determined by recording the time spent scanning the water and looking for marine mammals compared to focusing on other activities.
The second goal, evaluating participants' understanding of science knowledge, focused on interpreting the user's growth in marine mammal content knowledge. To assess content knowledge, participants responded to questions regarding marine mammal identification and knowledge. Then, responses from the post-use survey were compared to those of the preliminary survey to identify improvement in identification skills and content knowledge. Since the data were not normally distributed, paired Wilcoxon rank sum tests using the R package 'coin' were used to determine significance change between the preliminary and post questionnaire responses [47].

Mapping and Evaluating Whale mAPP Data
Marine mammal distribution maps were produced using a lattice-based density estimator [48]. Given the irregular boundaries and abundance of islands in Southeast Alaska, a lattice-based density estimator was used to generate estimates of core-(25% density), intermediate-(50% density), and broad-use (95% density) areas of species distribution. Similar methods were used to estimate marine species distributions in other areas with complex shorelines and islands [49,50]. The probability that the random walk stayed in the same location, M, was set in accordance with previous studies to be 0.5 [48,49]. The estimation of the optimal smoothing parameter, k, was determined using cross-validation in the package 'latticeDensity' for each species [51]. Node spacing of 50 m was used because it was sufficient for delineating the coastlines while still allowing computer computation of the complex study area. All analyses were conducted in R [45].
To demonstrate a potential use of Whale mAPP data for marine mammal research, the collected data on Steller sea lions was compared to results from Womble et al.'s (2005) standardized survey of haul out locations in Southeast Alaska. Haul out sites are areas where pinnipeds, in this case Steller sea lions, temporarily leave the water and 'haul out' on land. These are generally established locations and commonly used to describe Steller sea lion distribution [41]. To minimize inter-annual variation, we aggregated the scientifically collected data from March through May 2001 and 2002 [41] for comparison to Whale mAPP data between June and September 2014. This dataset and species were chosen as the case study because the dataset represents the most comprehensive published dataset that surveyed all of Southeast Alaska for Steller sea lions [41]. In addition, haul out sites are easily identifiable and comparable between datasets.
Whale mAPP Steller sea lion data accuracy were measured by counting the number of scientifically collected haul out locations [41] recorded by Whale mAPP users. To determine how sample size influenced the accuracy of Whale mAPP data, a discovery curve was generated that plots the percent of scientifically collected haul out sites [41] recorded by Whale mAPP users with every 5-unit sighting increase in Whale mAPP sample size. For each five-unit sample size increase, ten random Whale mAPP sub-samples were generated.
For the Steller sea lion case study, 'high survey effort' was differentiated to quantify how spatial bias affected the quality of citizen science data. To accomplish this, track line data of Whale mAPP user paths were mapped to examine spatial biases. Because 90% of total track line data were represented in the 75% lattice-based density contour of survey effort, the 75% lattice-based density contour was used to define 'high track line effort'. The 'high Steller sea lion sighting effort' was defined by the 50% lattice-based contour because these data included more than 90% of all Steller sea lion data. The final 'high survey effort' was the compilation of the 'high track line effort' and 'high Steller sea lion sighting effort' spatial layers. While, minimum survey effort was determined by combining all track line data with all Whale mAPP sighting data. Ultimately, this process created one layer representing all locations that Whale mAPP users traveled to at least once.

Results and Evaluations
Of the 216 people encountered at the recruitment center during the summer, 73.5% were interested in using Whale mAPP; however, of those only 44.7% possessed an Android device to download the app. Of the resulting 39 participants who followed through in using the app, 18 were private vessel owners, 18 were involved with nature tourism cruises, and three were commercial fishermen. From 20 June to 30 September 2014, these participants logged over 800 h to record 1261 marine mammal sightings and 10,892 km of track line data from the northern portion of Southeast Alaska to Seattle, Washington. In sum, 52.9% of sightings were of humpback whale (n = 665), 11.5% were sea otters (n = 146), 11.3% were harbor seals (n = 143), 9.2% were Steller sea lions (n = 117), 6.0% were killer whales (n = 76), 3.9% were Dall's porpoises (n = 50), 3.3% were harbor porpoises (n = 43), 1.0% were Pacific white-sided dolphins (n = 12), and <1% were California sea lions (n = 5), minke whales (n = 2), elephant seals (n = 2), fin whale (n = 1), and gray whales (n = 1).

Questionnaire Results
In sum, 29 (74%) Whale mAPP users completed the pre-use questionnaire, and of those, 24 (83%) followed up with the post-use questionnaire. A majority of volunteers, 78.3% reported they enjoyed using Whale mAPP, 73.9% found it easy to use and 82.6% would recommend it. When asked about their interests prior to using Whale mAPP, citizen scientists reported pre-existing interests in marine mammals, the ocean, using Whale mAPP, collecting marine mammal data, and to a lesser degree technology ( Figure 4). When asked whether they took any self-initiated actions to learn about marine mammals as a result of using the app, 56.5% reported talking to a peer, 47.8% read a book, 34.8% talked to a scientist, and 8.7% reported going on a wildlife tour, talking to family, or going to a museum. Volunteers were primarily motivated to use Whale mAPP due to their interest in marine mammals (n = 17), collecting data (n = 13), science (n = 13), citizen science (n = 11), their tourist company's association with Whale mAPP (n = 6), technology (n = 3), and/or for another reason (n = 2).
Marine mammal identification skills significantly improved after using Whale mAPP (Wilcoxon rank sum test, Z = 1.83, p-value = 0.035), with an increase in average test score rising from 76.3% ± 20.4% to 86.1% ± 19.3%. Yet, citizen scientists' content knowledge of marine mammal conservation topics did not change after using Whale mAPP for at least two weeks.
When the study ended in September 2014, 27.3% noted they were still using Whale mAPP. Of these, 83.3% were involved with tourism cruises as either a captain or naturalist staff. The remaining 16.7% were private vessel owners. Furthermore, all of these volunteers possessed very strong pre-existing interests for the ocean and marine mammals, and very strong to strong pre-existing interest in monitoring marine mammals and Whale mAPP.
The remaining 72.7% of citizen scientists reported that they stopped using the app because they either left Southeast Alaska (68.4%), felt it was too time consuming (15.8%), encountered technology problems (10.5%), and/or for another reason (5.3%). Of those that stopped using the app because they left Southeast Alaska, 64.3% were private vessel owners and 35.7% were involved with nature tourism cruises. Furthermore, around 26.1% of volunteers thought the app required too much data entry. Long-term commitment to data collection was limited as User effort also differed between individuals as the top five recorders collected 16.6%, 14.7%, 11%, 9.2%, and 6.6% of sighting data. The other 34 participants recorded the remaining 41.9% approximately evenly. In addition, nine users collected~73% of track line data, a value that may be due to user choice or app malfunctions as only 30% of track line data were correctly saved (participants did not know if their track line data was correctly uploaded or not). Around half of participants (52.2%) did not purposefully go out to use Whale mAPP; rather they used it when already travelling to a destination.
All participants spent the majority of their time scanning the water for marine mammals while using the app; however, users also reported simultaneously driving a vessel and talking for more than a third of the time ( Figure 5).

Applicability to Marine Mammal Research
For the lattice-based density method, the total water area used to estimate home ranges was 15,630 km 2 , with each node located 0.05 km apart. Only sightings north of latitude 54 • , where a majority (91.6%) of the sightings were located, were analyzed. Due to the structure of Whale mAPP, only presence data were recorded. It cannot be assumed that all marine mammals encountered on each track line were recorded. This is true for all marine mammal surveys, as even trained observers miss animals due to weather conditions, animal diving, or change. Data removed from the analysis included duplicate track line data (n = 10), data with unidentifiable users, data noted as a "mistake" by the recorder, and sightings that were revisions to previous recordings (n = 7). Additionally, due to low confidence rating (less than 3 rating) and poor visibility (<3 mile visibility), 47 sightings were removed, primarily for humpback whales (n = 22), but proportionally more for cryptic harbor (n = 4) and Dall's (n = 4) porpoises. An average of 77.4 ± 11.5% of removed sighting were due to poor visibility. Data with confidence ratings lower than a three were removed because this ranking indicated that citizen scientists had lower than 75% confidence in the accuracy of those data. Data with visibility of less than three miles were removed because Southeast Alaska can have sudden, thick fog that significantly reduces visibility to a few hundred meters. Therefore, visibility can greatly impact the number of marine mammals a citizen scientist can spot from a boat.
To illustrate the potential of Whale mAPP data to inform marine mammal distribution patterns, three lattice-based density maps were created for humpback whales (n = 665, k = 1), sea otters (n = 146, k = 3) and harbor seals (n = 143, k = 3) ( Figure 6). Humpback whale distributions were comparable to other scientific datasets along common travel routes, but data gaps were present in the southern part of Southeast Alaska (south of 56.5 • N) and the more remote northern sections of Glacier Bay and Icy Strait [38,39]. Comparison of Whale mAPP results regarding sea otter distribution to previous work [52] suggests their range has expanded from the outer western edges to throughout Southeast Alaska since early 1990. Harbor seal distribution was difficult to compare to other studies, because no detailed published studies were found outside of Glacier Bay or specific Southeast Alaskan inlets [53][54][55]. Thus, this lack of pre-existing data highlights the added value of citizen science data.

Steller Sea Lion Case Study
Altogether, Whale mAPP users recorded 54.2% of Steller sea lion haul out sites identified by the scientifically collected data [41], which improved to 72.2% accuracy when the comparison was limited to the areas surveyed at least once by Whale mAPP users (Figure 7). Accuracy increased further to 90.9% when the comparison was focused on high Whale mAPP survey effort areas, a common measurement of spatial bias (orange areas in Figure 7).
Furthermore, the discovery curve shows a linear increase in the percent of scientifically collected haul out sites [41] identified by Whale mAPP users with an increase in sample size of Whale mAPP Steller sea lion sightings ( Figure 8). This data demonstrates how sample size can impact citizen science data accuracy and needs to be considered when interpreting results.

Spatial Bias
Approximately 70% of the track line data collected by 11 citizen scientists were not recorded due to a technological glitch in which the track line data were not uploaded to the cloud geodatabase. The remaining 30% (n = 120 track lines) were used to estimate areas of high and low survey effort. Twenty-eight, or~71.8% of participants, contributed to the vessel track line data. On average, each user travelled 80 km and recorded 3.5 ± 4.0 marine mammal sightings per track line. Technological errors that led to the loss of track line data have subsequently been fixed in Whale mAPP, as these data are essential for interpreting spatial bias.   Higher survey effort (within the 75% density contour) occurred near the recruitment center (Warm Springs Bay), other towns, and along common travel routes such as Chatham Strait, Peril Strait, Fredrick Sound, and Stephens Passage (Figure 9). Reduced coverage occurred in remote areas, waterways far from the recruitment center, and/or restricted access areas, such as along offshore facing coastlines, Lynn Canal, Glacier Bay, and south by Wrangell. Variation in survey effort makes the data difficult to extrapolate for all of Southeast Alaska. Therefore, this caveat needs to be considered when determining Whale mAPP data quality and interpreting the results.

Discussion
With an increasing need for monitoring marine mammal populations in the face of broad scale environmental changes, citizen science research presents a low-cost solution to collect the vast spatial and temporal data required for robust regional scale population monitoring. This citizen science study cost less than $10,000 for the mobile application revisions, field work equipment, and graduate student time; and it covered a spatial area that had only been surveyed several times in Southeast Alaska, yet had never been surveyed for all cetaceans, pinnipeds, and sea otter species [38][39][40][41]. While the Whale mAPP citizen science project received immense positive feedback and can provide inexpensive and potentially useful scientific data, it still has biases and constraints. Thus, this study emphasizes the importance of acknowledging and measuring the limitations of each citizen science project, so that data collection methods and structure can adapt for improved education and scientific benefits.
This initial review demonstrates the potential of a marine mammal citizen science app to support both scientific and education objectives. For instance, Whale mAPP data identified 90.9% of Steller sea lion haul out sites identified by previous scientific surveys [41] in areas of high survey coverage.
In addition, short-term educational benefits, such as improved marine mammal identification skills and self-initiated learning, represent an added benefits of the application.
Citizen scientists' engagement level and retention play a vital role in the success of any citizen science project. Mobile application citizen science projects often receive funding for the initial app development and recruitment phase [15], but may find it challenging to acquire the long term funds to support continued app management, volunteer recruitment, and volunteer communication. This study stands testament to the effects of supplying resources for one field season, only to have the data collection cease without persistent recruitment and retention efforts. While, the initial excitement and data collection of Whale mAPP was promising, long-term participation beyond two weeks was limited to 27.3% of the citizen scientists. The remaining 72.7% of volunteers stopped using Whale mAPP because they left the area, believed the app required too much time, or encountered technological problems. This indicates that perhaps volunteers did not realize Whale mAPP works globally, were only interested in recording data for the Southeast Alaska project, or no longer had an opportunity to use the mobile app because they were no longer on the water. Furthermore, an even sharper decline in the number of marine mammal records in the years after recruitment indicate that perhaps this transient area is not an ideal location to recruit returning volunteers or that more effort is required to maintain communication and create a stimulating Whale mAPP community. Since the goal of many citizen science projects is to be self-sustaining, this study clearly demonstrated that this goal could not be accomplished with only one season of recruitment effort. For Whale mAPP, only one naturalist working on a nature tourism boat continued to use the mobile app two years after the initial recruitment summer. After three years, no originally recruited volunteers continued to use the app. Thus, to support and develop a long-term successful citizen science project, continual effort needs to be directed into recruiting and retaining volunteers, communicating with those volunteers, updating the mobile application, fixing technological problems, and funding the project so that resources are retained after one field season.
Targeted recruitment may also represent a viable option. Many crowdsourcing volunteers are actually not amateurs, but rather self-selected professionals and experts who elect to participate [56]. The five citizen scientists who contributed over 58% of the data likely represent these self-selecting experts. This trend is not uncommon, as there are often just a few citizen scientists who contribute a majority of the data [33]. Thus, finding these knowledgeable candidates will be key to collecting high quality data and retaining dedicated citizen science volunteers. Results from this study suggest Whale mAPP recruitment should focus on captain or naturalist staff on nature tourism vessels, especially those who express interest in marine mammals and the project. Since 83.3% of citizen scientists who continued using Whale mAPP were involved with nature tourism cruises and expressed very strong pre-existing interests in the ocean and marine mammals, and to a slightly lesser degree monitoring marine mammals and Whale mAPP, targeting this same audience would likely yield better results. This more specialized approach will likely connect to even more interested and dedicated citizen scientists with a stronger pre-existing interests [12,20,30,31] than recruiting volunteers from a nature outreach center, which likely resulted in a broadly targeted audience. Furthermore, if staff working on nature tourism vessels are targeted, the resulting participants would likely follow set travel routes, facilitating easier temporal data comparison. While a more standardized travel route might aid in data analyses, it may also lead to data gaps in narrow channels, marine protected areas, or locations not travelled by these vessels (Figure 9).
Previous citizen science studies also emphasize the importance of creating a community of volunteers who enjoy working together and identify with the projects' goals. This community can also play a role in driving the future direction and modifications of the citizen science project [30,57]. Putting more energy into the citizen scientists' experience through the Whale mAPP website, forum, or email notification may also help improve sustained volunteer commitment. Additionally, increasing data collection masks inevitable errors and improves data quality [7], a common concern for citizen science projects [10]. Like many things in life, having a supportive community can motivate people to continue contributing to a mutual project.
Another strategy to increase volunteer recruitment would be to provide an iOS version of Whale mAPP. Around half of the people encountered at the recruitment center did not own an Android, and therefore could not participate. By either creating an iOS version of Whale mAPP or providing Android tablets for dedicated volunteers, the project could reach this still untapped audience.
While still cheaper than running an entire field season to collect comparable data, this effort is not cost-free. Promoting Whale mAPP to the general public and target audiences, developing an iOS version, and providing tablets to dedicated users requires staff time and resources. Furthermore, sustaining new volunteers demands long-term communication, collaboration, and commitment. Securing long-term funding for such a project, although challenging, will likely by the only means to continue the quality of the citizen science project that collects this inclusive marine mammal data.
Results suggest this project is worth investing in because Whale mAPP provides an opportunity to learn more about many marine mammals by contributing to sustained marine mammal research and outreach. This outcome was one of the motivating reasons for why people chose to use Whale mAPP. Moreover, as a result of using Whale mAPP, participants, on average, improved their marine mammal identification skills, a positive learning outcome for many citizen science projects [12]. A more challenging, and often long-term, educational benefit is inspiring volunteers to engage in further learning outside of the citizen science project [19]. Few citizen science projects have reportedly accomplished this task [12]. The first steps to this goal are seen in the Whale mAPP volunteers who took action to learn more about marine mammals by talking to peers, reading about marine mammals, and talking to scientists. There are many methods for bolstering the educational components of a citizen science project. However, methods need to address various user types, from teaching marine mammal identification, to novice users to providing a more stimulating learning environment for the self-initiated participants with previous knowledge. Perhaps one way to address this gap could be for Whale mAPP to enhance citizen scientists' engagement in the scientific process and education on marine mammals by developing interactive maps that focus on connecting spatiotemporal processes (i.e., current, depth, time of year, etc.) with marine mammal distributions. Overall, providing more enhanced services via the application or website, or actively through ann onsite Whale mAPP steward, would likely enhance the reach of the project and prolong interests of Whale mAPP citizen scientists.
In addition to increasing science literacy and knowledge, citizen science projects also need to consider how researchers can use the data and what caveats are associated with the dataset. One common data limitation is spatial bias. Variability in citizen science survey effort is not uncommon, as a majority of the effort is often focused around common, human populated areas [11,[58][59][60]. This spatial bias toward populated areas is not necessarily a drawback, as urban areas frequently require regular monitoring because they can be more impacted by anthropogenic stressors [61,62]. Based on available track line data, these same trends of high survey effort near towns and common travel routes were present in the Whale mAPP data as well. Because citizen scientists often inadvertently collect more data in these regions, Whale mAPP could provide a unique and useful method to examine the impacts of anthropogenic stressors on marine mammals. An alternative method to combat spatial bias may be to highlight areas of low effort in the app's map display and encourage volunteers to travel to those low survey areas. Either way, survey effort and bias need to be considered when interpreting the citizen science data.
Results from comparing Steller sea lion citizen science data to traditionally collected data illustrate the importance of spatial coverage and sample size. For Steller sea lions, Whale mAPP data accuracy in co-locating the haul out sites identified by a previous scientific study [41] improved from~72% to 91% when the comparison was limited to high survey effort areas (Figure 7). Data quality also improved with an increased sample size (Figure 8). A Steller sea lion citizen science sample size of 73 recordings was adequate to predicting 72.2% of haul out sites identified with traditional research methods [41], an accuracy value higher than results comparing eBird and iNaturalist bird lists to National Park Service records [14]. These two results demonstrate that increased sample size and spatial coverage is crucial for scientific usefulness of data. Sample size is important for all research methods, including assessments of species distributions where low sample size effects both citizen science [14,35] and traditional scientific studies [63]. In sum, a large enough Whale mAPP data sample size and spatial coverage enable this citizen science method to become scientifically valuable for gathering broad scale and marine mammal data.
User training can also contribute to data quality. Sufficient training and accurate protocols are required to reinforce data consistency and accuracy [15]. Future work should specifically test the effectiveness of the Whale mAPP protocol and marine mammal identification guide by shadowing Whale mAPP citizen scientists while they use the app and associated materials, noting where error and/or frustration occur while simultaneously using Whale mAPP to compare the data recorded by the citizen scientists to that of a scientist. This study would provide a more thorough evaluation of the current Whale mAPP protocols, supporting future improvements to the user guidelines and the resulting data quality.

Conclusions
Overall, Whale mAPP received positive user feedback and produced valuable scientific data for the Steller sea lion case study. Whale mAPP data quality, like many citizen science data, improved with a larger sample size [21,33] and spatial coverage [64]. This will most likely be achieved by recruiting self-selecting experts, such as naturalists and captains working on nature tourism cruises, and developing the educational and engagement components of Whale mAPP beyond the level of improved marine mammal identification, and into something more beneficial for expert volunteers. Committing future funding to Whale mAPP user recruitment, especially of nature tourism staff, and building a Whale mAPP citizen science learning and networking community would contribute substantially to the success of the project. Subsequently, more consistent data could be collected, improving spatial coverage and sample size, thereby reducing limitations with the resulting Whale mAPP data. With these efforts, citizen scientists equipped with Whale mAPP could provide the broad scale, long term and continuous marine mammal monitoring data needed to identify at risk populations and fill many data gaps currently present in marine mammal conservation and management research.