Using citizen science to expand the global map of landslides: Introducing the Cooperative Open Online Landslide Repository (COOLR)

Robust inventories are vital for improving assessment of and response to deadly and costly landslide hazards. However, collecting landslide events in inventories is difficult at the global scale due to inconsistencies in or the absence of landslide reporting. Citizen science is a valuable opportunity for addressing some of these challenges. The new Cooperative Open Online Landslide Repository (COOLR) supplements data in a NASA-developed Global Landslide Catalog (GLC) with citizen science reports to build a more robust, publicly available global inventory. This manuscript introduces the COOLR project and its methods, evaluates the initial citizen science results from the first 13 months, and discusses future improvements to increase the global engagement with the project. The COOLR project (https://landslides.nasa.gov) contains Landslide Reporter, the first global citizen science project for landslides, and Landslide Viewer, a portal to visualize data from COOLR and other satellite and model products. From March 2018 to April 2019, 49 citizen scientists contributed 162 new landslide events to COOLR. These events spanned 37 countries in five continents. The initial results demonstrated that both expert and novice participants are contributing via Landslide Reporter. Citizen scientists are filling in data gaps through news sources in 11 different languages, in-person observations, and new landslide events occurring hundreds and thousands of kilometers away from any existing GLC data. The data is of sufficient accuracy to use in NASA susceptibility and hazard models. COOLR continues to expand as an open platform of landslide inventories with new data from citizen scientists, NASA scientists, and other landslide groups. Future work on the COOLR project will seek to increase participation and functionality of the platform as well as move towards collective post-disaster mapping.

Introduction Landslides, or mass movements, cause thousands of deaths and billions of dollars in infrastructural damage worldwide each year, warranting the need to understand their triggers and mitigate future losses [1,2]. Landslides are triggered by a variety of natural and human causes, most commonly rainfall but also earthquakes, freeze-thaw cycles, mining, and other causes. These triggers loosen slope materials, resulting in their downward and outward movement by gravity as landslides, debris flows, mudslides, rock falls, earthflows, and other mass movements-all referred to in this paper as landslides [3,4]. Knowledge of past landslides is the most important element for hazard and risk assessment because landslides are likely to occur in areas that have previously experienced a failure [5][6][7]. Therefore, the collection of past landslide information helps to predict future risk. To date, there are few global, publicly available inventories providing landslide data.

The value and limitations of landslide inventories, and the Global Landslide Catalog (GLC)
Robust and complete landslide inventories are crucial for understanding past landslide mechanisms and forecasting future events. Landslide inventories, defined as spatial, temporal or spatiotemporal datasets of landslide events, are routinely used for many subject areas from characterizing landscape evolution, to calculating susceptibility and hazard, to supporting emergency response and planning efforts. There are different ways to compile a landslide inventory: collecting landslide distributions for a single triggering event; mapping landslides identified from satellite imagery or aerial photos, in situ mapping, or cataloging reports from news media [6,[8][9][10][11]. Inventory production is the necessary first step for landslide risk management, but it must be followed by further work. NASA data and products help to support multiple steps in the process towards characterizing landslide risk (Fig 1), beginning with the landslide inventory.
The Global Landslide Catalog (GLC) exemplifies one type of landslide inventory that provides the estimated location and date of occurrence. The authors and supporting team members have been cataloging rainfall-triggered landslide events from 2007 to the present, collected primarily from online news reports found using a Google Alerts search for relevant keywords. A landslide event was defined as a single landslide or multiple landslides tied to a location associated with a single triggering event [12]. The GLC currently contains more than 11,500 reports, including 230 landslides from the Durham Fatal Landslide Database (DFLD) [12][13][14]. Gathering events from news reports was favorable to the authors' scientific goals because news reports can give an accurate event date, tying the events to probable triggering conditions that can be used for model development [14]. In contrast, the dates of landslides cataloged from satellite or aerial imagery are uncertain due to typically long periods between image acquisitions. Information about landslides can be combined empirically or deterministically with data on precipitation, slope, lithology, surface cover, etc. to characterize or model patterns and trends in landslide behavior over short to long time scales and local to global areas [5,9,10,15,16]. On a global scale, this data can help infer patterns; for example, characterizing the seasonality of landslides in different regions of the world based on climate drivers [16], comparing areas of the world based on their vulnerability to landslides [2], and evaluating the distribution and cost of fatal landslides [13,17]. The GLC has been corroborated as a template for landslide inventory design [13,18] and as a validation tool for worldwide landslide hazard and susceptibility modeling [15,16,19,20].
Despite the need for comprehensive landslide catalogs, routine mapping of landslides is time-consuming and resource-intensive [8,21,22]. Unlike hurricanes or earthquakes that have global monitoring systems (e.g. Joint Typhoon Warning Center, http://www.metoc.navy.mil/ jtwc/jtwc.html; World Meteorological Organization Tropical Cyclone Program, https://severe. worldweather.wmo.int/; Global Seismographic Network, https://www.iris.edu/hq/programs/ gsn), or wildfires and floods that often can be globally mapped from space due to their size and characteristics [23,24], landslide activity is not monitored at the global scale. Landslides are frequently triggered by other natural hazards like earthquakes, floods, or hurricanes, making the reporting of individual landslides difficult and their effects often indistinguishable from the triggering hazard [2]. Furthermore, landslides are usually small, widespread, and frequent, making it impossible with current technologies and funding to monitor and map landslides globally every day. A single storm can generate tens of thousands of landslides [7,25]. Using the GLC methodology, recording a single landslide event takes 20 minutes on average, which comprises of the time it takes to approximate the location of the landslide from the news report and input all relevant details from the report. The addition of all 11,500 landslides to the GLC has taken the GLC team a cumulative 1.6 years of 8-hour workdays to report.
Another challenge is identifying landslides in areas with limited imagery or data, limited newspaper reporting, and/or in remote areas, creating spatial and temporal bias to the landslide inventory data [6][7][8][9]. The GLC is affected by reporting biases that impact the spatial distribution of data in addition to the challenges listed for all landslide inventories [14]. These problems include reliance on reports written in English, improved reporting closer to populated areas or transportation networks, increased likelihood of reporting when human life and property is involved, whether reporting networks exist, and bias from political or economic differences among regions [12,14]. During validation of the Landslide Hazard Assessment for Situational Awareness (LHASA) model, Kirschbaum et al. (2018) found that many areas of modeled high landslide potential had few reports of past landslide events. The authors concluded that a spatially consistent landslide inventory is "of foremost importance" to advance the LHASA model [16]. Citizen science was used in this project to extend data collection beyond the limitations of current methods.

Enhancing the GLC with citizen science
Citizen science-or "crowd science", "community science", among other terms-is an increasingly popular methodology that is enabling the public to participate in the massive collection or analysis of data and solutions to scientific problems [26,27]. In the last two decades the number of publications involving citizen science projects increased by a factor of one hundred, attributed to the increasing use of citizen science data by professional researchers, the availability of technical tools, and the emphasis on better science outreach and communication [28,29]. The wide use of and access to the Internet, the Global Positioning System (GPS), and geospatial/mapping programs like Google MyMaps (https://www.google.com/mymaps) or Esri Arc-GIS (https://www.esri.com/) have enabled the public to crowdsource geographic information online for purposes including citizen science [30]. To obtain data of sufficient quantity and quality, citizen science projects must raise the public's interest and provide standard methods for data documentation and access. However, there are many examples of successful citizen science projects that have addressed these concerns with targeted outreach, proper project management, improving attitudes towards citizen science, and implementing standard training protocols and data quality checks [31,32]. Overall, the citizen scientist public gives scientists an opportunity to explore vast amounts of data and analysis tied to location and timebased information to advance research. Results from eBird (https://ebird.org/), a bird-sighting citizen science project, have been robust enough to visualize country-wide patterns in bird species distribution with more data than could be collected by traditional research [33].
Like eBird's influence on ornithology research, citizen scientists can help collect greater quantities of landslide data while specifically addressing some of the limitations pervasive in existing landslide cataloging efforts and the GLC. First, the sheer number of potential citizen scientists can capture a much greater quantity of data contributed, which can enable a much faster and less resource-intense mapping of large numbers of landslides. More people to collect landslides minimizes the burden on the NASA landslide team, allowing for a more up-to-date global catalog and for rapid response during major triggering events. In situ reports made by citizen scientists can add information about locations that cannot be visited by researchers to strengthen the detail and accuracy of submitted data and helping to remove event uncertainty, such as recording the state of the weather, approximate date, the surrounding environment, the movement and possible cause of the landslide, and the photographing of the slide [34]. Finally, citizen scientists may also capture reports from non-English speaking media, helping to address some regional biases in the GLC.
Landslide citizen science has already been applied across a variety of regions and methods. Table 1 lists all landslide citizen science projects that could be identified through internet searches. The scope and duration of these efforts vary widely and exemplify the diversity of citizen science approaches for landslide studies. The projects outlined in Table 1 are generally led by scientists and use simple submission systems. Fourteen of the 20 projects identified were government-led, university groups ran five projects, and a non-governmental organization (NGO) led one project. The top-down approach of landslide citizen science suggests that institutions are important for organizing (and funding) a project at the start and constructing the research question(s) and infrastructure. By doing so, institutions with professional knowledge of landslide hazard and risk assessment can define what types of data are needed to address their scientific questions and create a platform suitable for their target community. Projects vary in scale but they tend to ask citizen scientists for the submission of landslide event information through a document or via email, because reducing the level of professional expertise needed for data submission allows for the widest amount of participation [27,35].
In this paper, we present the methods and initial findings of the new Cooperative Open Online Landslide Repository (COOLR) project with a focus on its component, the citizen science platform Landslide Reporter. There are three main objectives to this manuscript: introduce the COOLR project and its methods as a tool for building and sharing a global landslide The list was compiled by performing an exhaustive search for relevant pages and publications from combinations of the keywords "landslide", "mudslide", "debris flow", "citizen science", "crowd science", "crowdsourcing", "report a landslide", "volunteer mapping", and "participatory mapping" through Google (https://www. google.com/), Google Scholar (https://scholar.google.com), Web of Science (http://www.webofknowledge.com/), and Twitter (https://twitter.com). Searches for a landslide citizen science page or "Report a Landslide" pages were also conducted on the websites of the geological surveys of all 50 states of the United States. Projects were considered relevant to this study if the public was contributing to a landslide inventory for the project to be used for scientific research. b The project is currently active.
https://doi.org/10.1371/journal.pone.0218657.t001 inventory using citizen science; present the preliminary results of the Landslide Reporter project as a proof-of-concept; and discuss future improvements to the project to advance landslide data gathering with citizen science.

The COOLR project
The Cooperative Open Online Landslide Repository (COOLR) was launched on March 22, 2018 and includes Landslide Reporter (https://landslides.nasa.gov/reporter) and Landslide Viewer (https://landslides.nasa.gov/viewer) applications. Collectively, COOLR and its components are designed for the reporting and sharing of worldwide landslide data. COOLR is an open repository for landslide events, including data from NASA's GLC, citizen scientist-contributed reports, and other publicly available or shared landslide inventories. Landslide Reporter is the citizen science web application used to report landslide events to COOLR, which are stored and visualized in a separate web application, Landslide Viewer, along with other landslide data and remote sensing products.

Data storage and web application design
COOLR data are stored at the NASA Center for Climate Simulation (NCCS), and served through in-house ArcGIS servers, at https://maps.nccs.nasa.gov/server/rest/services/ in the folder "global_landslide_catalog". The landslide repository can be downloaded from Landslide Viewer, which is automatically updated two times a week. Alternatively, the data are also available on NASA's Open Data Portal (https://data.nasa.gov) under "Global Landslide Catalog," which is updated less frequently. Landslide Viewer has the most updated version of COOLR and the GLC. Landslide Viewer was built using the Foldable Theme on Esri Web AppBuilder. Landslide Reporter was customized from the Esri Crowdsource Reporter application (https://github. com/Esri/crowdsource-reporter). COOLR data are available for download in a file geodatabase (.gdb) format, comma-separated values (.csv) format, and shapefile (.shp) format. An additional data reference file is included to provide attribution for inventories that have been directly added to the catalog and provided from outside groups.

Repository design
The GLC served as the model for structuring the data fields during the creation of the repository. A modified version of the GLC data fields presented in Kirschbaum et al. (2015) is used as the input fields for COOLR. The fields included in COOLR are presented in Table 2. More information about the data fields can be found in Kirschbaum et al. (2010Kirschbaum et al. ( , 2015. Between the publication of Kirschbaum et al. (2015) and the launch of the COOLR project, the "very large" landslide size category was separated into "very large" and "catastrophic" landslides. At the launch of the COOLR project, a new definition to quantify the landslide size categories was appended to each of the size categories to standardize the category and reduce uncertainty when a value for the volume is given. The size category quantifications were informed by observing the reported volumes or reported number of trucks needed to haul away debris and size classifications in a random sample of past landslide events in the GLC. To establish a minimum volume quantity, we observed that small landslides typically only took a few hours to clear with one standard commercial dump truck, which can hold about 8 to 11 m 3 of material.
To establish a maximum volume, we consulted a list of well-known historical landslides available and observed volumes for catastrophic landslides exceeded one million m 3 [45,46]. The logarithmic scale of volumes was determined to be the best method to encompass all possible landslide sizes reported to the GLC.
The most important fields within each report within the GLC and COOLR are the date and location of the landslide, which are important for hazard model validation. Other information  Table 2

Category Information on Category
Event ID Unique ID for each reported landslide event.
Source name Source of report information, includes news source, field observation (in-person observation), disaster database, personal communication, etc.
Source link URL link to news report or other online source that has a listing of the report.

Date
Reported year, month, and day that the landslide event occurred. Recorded as M:D:YYYY.

Time
Reported hour and minute of the failure, recorded as HH:MM (24-hour clock, local time).
Event title Common name given to the landslide event or a descriptive title given by the user submitting the event.
Event description Includes event information such as dimensions of the landslide, characteristics, impacts, timing or situation resulting in the slope failure and other relevant information.
Location description Location information such as the nearest geographic location (e.g. village, city, region, landmark) if known.
Location accuracy This field assigns a qualitative uncertainty in the landslide location based on the estimated circular area over which the landslide realistically occurred, described as a radius from the event coordinates to a given radius (in kilometers Landslide size This category is to identify the relative size of the landslide in an attempt to differentiate small landslides occurring in backyards and along roads from larger landslides that have caused catastrophic damage and cover wide areas. The "Size Classification" values are from Kirschbaum et al. (2010), which describe the landslide cataloging methodology. A quantified scale was developed for cases in which volume is reported, and has been used since March 2018 b .
• Small: Small landslide affecting one hillslope or small area. � Possible damage: Minimal impacts to infrastructure and roads, no fatalities or few fatalities. � Volume: <10 cubic meters • Medium: Moderately sized landslide that could be either a single event or multiple landslides within an area, and involves a large volume of material.
� Possible damage: Moderate impact to infrastructure and roads, no fatalities or few fatalities. � Volume: 10 to <1000 cubic meters • Large: Large landslide or series of landslides that occur in one general area but cover a wide area.
� Possible damage: Substantial impacts to infrastructure and roads, likely moderate to high number of fatalities, tens to hundreds of people displaced.
� Volume: 1000 to <100,000 cubic meters • Very large: Very large landslide or multiple events that affect an entire region (often encompassing an entire village or larger area) � Possible damage: Substantial impacts to infrastructure and roads, high numbers of fatalities, thousands of people may be displaced.
� Volume: 100,000 to <1 million cubic meters • Catastrophic: Catastrophic landslide or multiple events that affect multiple villages, towns, and regions � Possible damage: Irreversible damage to infrastructure and roads; catastrophic numbers of fatalities, tens of thousands of people may be displaced.
� Volume: 1 million cubic meters or greater • Unknown

Landslide setting
The surrounding environment on which the landslide occurred. The most common settings are included: above road, below road, above river, above coast, burned area, deforested slope, urban, mine, retaining wall, natural slope, engineered slope, bluff, other, or unknown.
Number of fatalities Number of reported fatalities as a result of the event.
Number of injuries Number of reported injuries as a result of the event.
Storm name Includes the name or number of a tropical cyclone if identified (e.g. Hurricane Sandy, Typhoon Mangkhut, Tropical Depression No. 12).

Photo link
The image address to a photo from the source of the event information, in the form of a URL.

Comments
Comments about the report directed towards end-users about the quality of the report, quality of source information, accuracy of location, etc.
(Continued ) such as the trigger and time of the event are helpful to identify a possible triggering event for the landslide [12]. COOLR is separated into two geodatabases, one for landslide event points and one for landslide event polygons. There are two main sources for data in COOLR-the GLC and citizen science data from Landslide Reporter (Fig 3). The different contributors of the data are specified within the "Event import source" field. GLC data contributed from within NASA are labeled as "GLC", and Landslide Reporter data contributed by citizen scientists are labeled as "LRC". After the GLC was incorporated into COOLR, the database was cleaned to remove duplicate landslide events and events where no source was verified. Both the GLC and LRC data are added to the same point or polygon geodatabase. The database is checked prior to a Landslide Reporter submission to ensure that there are not any duplicate reports in the GLC and LRC; however, a single event may be reported separately to the point geodatabase and the polygon geodatabase.
A third component of COOLR involves importing landslide inventories from landslide experts and citizen science projects to create a framework for open landslide data sharing and access (Fig 3). These additional inventories are included through the ArcGIS REST API or through manual import. Landslide inventories hosted on servers external to NCCS are added via their Esri REST API directly to Landslide Viewer. Landslide inventories that must be imported manually and stored with NCCS are fitted to COOLR's fields ( Table 2) and added to COOLR's geodatabase file. Each manually added landslide inventory is modified into COOLR's fields on a case-by-case basis and validated with the author of the data prior to entry. A .csv file on the same page as the downloadable repository file lists the citation information for manually added inventories.

COOLR citizen science and administration process
We target all citizen science audiences to participate (landslide experts and non-experts) that have an education level of middle school and above. Researchers working with landslide data may be motivated to improve the catalog quality for themselves and others since the entire database is easily downloaded in several formats. There are also university professors that have expressed an interest in having their classes participate in this project as part of a lab for the class to teach about using and analyzing geospatial data while contributing to a global cataloging effort. Landslide experts may also be interested in this project because Landslide Reporter may help them keep track of landslides during or after field research. Non-expert citizen scientists have also become interested in participating. They are motivated to contribute in order to learn more about landslides from the introductory materials provided at https://landslides. nasa.gov, address landslide issues within their communities, and/or help scientific research. There are additional ways to incentivize citizen scientists to participate, which we discuss below in "Future improvements to COOLR." We seek to attract a diverse public who can fill in gaps in the inventory and alleviate current limitations, and therefore we are not targeting any specific region. We hope to attract citizen scientists, whether living in the area or reading international news reports, who can submit data for regions that currently have a dearth of reports, such as in the Southern Andes, the East African Rift Zone and Turkey and Iran [16]. In contrast, we also welcome landslide reporting for data-rich areas like the United States because we hope regular users will keep the repository up-to-date.
The ancillary goals of using citizen science in COOLR are to educate the public, serve as a teaching tool for educators about landslides, and encourage further scientific exploration. By using Landslide Reporter and Landslide Viewer, citizen scientists can learn how to differentiate the types of mass movements (e.g. mudslide, rock fall, translational slide, etc.) by following the training materials on the website. Additionally, Landslide Viewer provides both the open landslide catalog as well as satellite and model products that can highlight the distribution of landslide hazard and impacts around the world or be downloaded in several different formats. Fig 4 highlights the current submission and validation process for Landslide Reporter. To begin contributing, citizen scientists must log in to access the reporting form on Landslide Reporter. Citizen scientists must accept COOLR's Landslide System Contribution Policy, Landslide Contributor License Agreement, and the Take Down Policy before using the web application. Upon login with a Google account or Facebook account, each citizen scientist is automatically assigned a unique user ID, a random string of numbers and characters that is associated with their account. The ID is stored and used to find all of a citizen scientist's previously contributed landslides in the inventory of reports held for review. No personally identifiable information is stored in any part of the system, and the unique user IDs are not included with the downloadable version of the repository on Landslide Viewer.
There are several guides for how to fill in the fields within the application. Training materials are an important asset to help citizen scientists learn how to use the web applications and submit data that are valuable for scientific research [27]. There are five how-to guides currently available on https://landslides.nasa.gov, downloadable as .pdf files: • "Primer and Landslide Identification" teaches readers about why it is important to study landslides and provides information to help fill out the landslide category and landslide size fields.
• "Add a Landslide Event to COOLR" explains each field in the report form with guiding images following the contents of Table 2 and how to navigate the web application.
• "Landslide Identification Training" is a training protocol for citizen scientists with little to no professional experience. The identification guide gives examples of landslides types (landslide, mudslide, debris flow, and rock fall) and their qualitative sizes (small, medium, large, very large) following the information in Table 2.
• "Tips to Pinpoint Landslide Location from a News Article" helps citizen scientists use context clues in a news source to find a reasonable latitude and longitude for the location of a landslide using Google Maps or Google Earth.
• "Download and Export COOLR" guide has instructions on how to access the download page on Landslide Viewer, how to cite the data, how to open the .gdb file in ArcMap or QGIS, and convert the file to a .csv. These guides will be updated as needed. For visual learners, the website also contains videos with an overview for the project, how to navigate the web applications, how to export the data, and how to use the ArcGIS REST API. The website is also a resource to learn more about how the data are used by linking visitors related publications and projects.
Once the citizen scientist completes and submits their report, the data are held for review on the NCCS server. After submission, citizen scientists can still access the report on the server to delete or make edits to the report until the report is validated by the team at NASA. During the validation process, the NASA reviewer will check the entire submission and make any modifications necessary to improve the accuracy or detail of the report (Fig 4). All landslide events in need of validation are stored in an ArcMap database and submissions are held for review. These data are overlaid with a map of previously recorded events in COOLR. Each submitted report is first checked for duplicates (S1 File). If the date of the reported event matches or is very close in date to an event already in COOLR, the descriptions of the events are compared. If it is determined to be a duplicate event, then any new information from the new reported event will be merged into the older event report. Once all the information is extracted, the newer event data will be deleted from the inventory of submissions held for review. For new landslide events, the report is scrutinized for sufficient accuracy and detail (S1 File). This is performed either by comparing the reported information to the source link in the "Source name" and "Source link" fields or by checking the information to the extent possible if the landslide event was an in-person observation. Then, the report will either be added to COOLR or deleted, and updated in Landslide Viewer accordingly.

Communication and outreach
Landslide Reporter achieved a small following from outreach through social media. At launch in March 2018, the CitizenScience.gov and SciStarter blogs were the first to publicize the COOLR project [49,50]. SciStarter, in turn, publicized their post to the PLOS (Public Library of Science) Blog network's CitizenSci blog and to Discover's Citizen Science Salon blog. We also advertised the project on Twitter and Facebook, using hashtags like #citizenscience or #citsci. Subsequent spikes in interest (more than 100 "reblogs" and "likes") are attributed to the attention received through blog posts written by the American Geophysical Union's (AGU) The Landslide Blog and the NASA Earth Observatory [51,52]. The NASA Earth Observatory blog post was then shared through Earthsky [53] and their community reblogged this at least 50 more times.
Our existing communication strategy is to use Twitter and Facebook to share Landslide Reporter with the public, send updates about citizen scientist-submitted data, link to educational and relevant articles, and highlight project achievements. We also include our project on open citizen science project databases to help increase interest of and drive traffic towards our project. Currently, Landslide Reporter is part of the Federal Crowdsourcing and Citizen Science Catalog (https://www.citizenscience.gov/catalog/), the SciStarter Project Finder (https://scistarter.com/finder), and the Science by Bike project database (http://sciencebybike. com/explore-projects/). Through these methods, we have attracted the attention of and submissions from landslide professionals, academics, students, and the public, evidence that we are reaching the target audience of both landslide experts and non-experts. Commercial, academic, community volunteer, and government-led landslide groups have expressed interest in our project, and four of these groups have submitted landslide inventories now included in COOLR. One such group, the SERVIR-Mekong team (https://servir.adpc.net/) based in Huntsville, AL, conducted a landslide mapathon to fill in the gaps in Myanmar data in the GLC. The team collected over 1000 points from Google Earth imagery on locations of landslides in the region, transformed the information into the COOLR format, and provided the data for our team to manually include in the repository (Table 1).
We led a successful media blitz in August 2018 to engage more citizen scientists to submit to the project. A Tumblr post and social media stories on the main NASA Snapchat, Facebook, and Instagram shared the importance of studying landslides, connecting this work with other NASA missions such as the Global Precipitation Measurement (GPM) mission. The project linked back to the Landslides @ NASA website to encourage interested viewers to become citizen scientists. The event ultimately brought~5700 likes and~1600 shares across Twitter and Snapchat,~1100 notes (likes and shares) on Tumblr, and 80,000 views on the Snapchat story. These views translated into~7,400 page views on landslides.nasa.gov and almost 100 new followers across our project's social media platforms. As a result, there were~30 new landslide reports made to Landslide Reporter in the first 24 hours of the social media campaign. On the day of the campaign, an extreme rainfall even triggering widespread flooding and landslides occurred in Kerala, India [54]. Landslide Reporter received several reports of this event, highlighting the motivation of citizen scientists to contribute to disaster response.

Analysis for proof-of-concept
In the results section, we present the preliminary results on landslide events in the LRC from the end of March 2018 (project launch) to the end of November 2018 as a proof-of-concept. Though this is a newly launched system with limited time for data collection, we evaluate the current entries to determine if the citizen scientist-contributed landslide data is of scientific quality and if the data provided can help fill in areas that have a dearth of data. We use metrics of temporal distribution, spatial distribution (distances between events, density of events), the count of source information language and type, location accuracy, and associated landslide susceptibility from Kirschbaum and Stanley (2017) to analyze the data.
The analysis was carried out using the Esri ArcMap and Microsoft Excel. The data analyzed was downloaded from the public repository on Landslide Viewer with the exception of information about the number of citizen scientists. The number of citizen scientists were counted by counting the number of unique user IDs stored in the private version of the repository on the server. Analysis in ArcMap was performed using primarily the Spatial Analyst Package.

Results and discussion
Over the 13-month study period, citizen scientists have contributed 162 landslide events to COOLR using Landslide Reporter. These events have all been validated and added to COOLR. Additional reports were submitted, but due to incomplete information, duplicates, or the submission of an inventory as one event, they were not added to the repository. Forty-nine participants submitted the 162 landslide events. A bulk of reports were submitted by one "super user" citizen scientist, which corroborates the importance of these individuals within citizen science communities [55,56]. Most other citizen scientists contributed one or two reports. The reported landslides are distributed around the world, suggesting the potential reach this type of platform can have given more time and increased discoverability by citizen science networks. In the following analyses, the new Landslide Reporter Catalog (LRC) data were compared to the NASA Global Landslide Catalog (GLC), containing 11,666 landslide events.

Filling in gaps in the repository
We evaluate the spatial and temporal distribution of the 162 reports we received since March 2018, to determine whether our data are filling in gaps that have been observed in the Global Landslide Catalog. The reported LRC landslides are from 37 countries on five continents ( Fig 5A). A time series of these submissions are graphed in Fig 5(B). A landslide that took place in 1897 was excluded from the x-axis so the other yearly data can be seen clearly, and a second landslide was excluded for having no date. Predictably, citizen scientists submitted the greatest number of landslides in 2018 and second-greatest number in 2017. The number submitted in 2019 is steadily rising as the year progresses. The reported landslide events that occurred in previous years indicate that citizen scientists may also seek to fill in gaps in historical records. For example, three citizen scientists submitted information about the Mud Creek Landslide that occurred in Big Sur, California, USA, in May 2017. Though we had to delete the duplicate submissions because they were already recorded in the GLC in COOLR, we moved new details about the slide to the entry. One submission helped correct the location of the slide. These initial findings provide evidence for how citizen scientists are able to enrich the existing repository in space and time.
There is a large number of landslide points submitted to the LRC from Germany and Switzerland, mostly by a single super user. Despite the disproportionality, this fills a gap in the landslide repository within this area. We quantified the spatial density of points from the LRC and the GLC across Europe and subtracted the layers to see where each dataset had a more comprehensive distribution. The resulting map in Fig 6(A) exhibits blue areas where GLC data are present and red areas where LRC data are more prevalent. If the reports are compared with a global susceptibility map provided by Kirschbaum et al. (2016) [57] in Fig 6(B), the LRC reports are superimposed on the Alps and the hilly regions of Germany where landslide susceptibility is high but GLC reports are missing. Both maps visually demonstrate how citizen scientists can help fill in spatial gaps that exist in the GLC.
To further analyze how gaps may be filled on a global scale, we measured the distance of each LRC landslide event to the nearest GLC landslide event. Fig 7 depicts a histogram of all the distances calculated. The greatest number of landslides submitted by citizen scientists occurred within 10 km of any landslide event compiled by our team at NASA. The GLC is robust enough to begin to characterize global patterns in where and when landslides occur, so many LRC events will be located near GLC event data in highly susceptible areas [14]. However, there is a large proportion of events in the LRC that do not neighbour any GLC event, with 20 events extending further than 150 km from the nearest GLC event. The farthest LRC landslide event reported was a cliff collapse in Santa Clara del Mar near Buenos Aires, Argentina, on January 2018, located 1215 km away from any GLC event. The results further suggest that this citizen science method helps to identify new landslide reports in landslide-susceptible areas not currently covered by the GLC.
Lastly, we examined how the challenges of language bias and reporter choice bias may be addressed by citizen scientists . Fig 8 illustrates that more than 60% of reported events in the LRC come from non-English sources. Citizen scientists were able to submit or translate news articles and database entries from Catalan, Czech, German, Persian, Polish, Portuguese, Romanian, Spanish, Turkish, and Ukrainian sources and enter the information into our English web application. Some of the reports are in-person observations, listed with a person's name or a permutation of "in-person observation" as recommended in our how-to guide. Some of the eyewitness accounts were from landslide experts, submitted with expert details about the slide and sometimes a link to a publication. The initial findings are promising for demonstrating how the known biases in the GLC could be reduced with the help of landslide expert and non-expert citizen scientists who possess language capabilities and/or in situ observations to make reports in landslide-prone locations previously overlooked by the GLC methodology.

Spatial characteristics of submitted landslides
In addition to filling in regional or temporal gaps in COOLR, we want to know if citizen scientists are contributing spatially accurate and viable data for use with landslide maps and models. Fig 9 presents the location accuracy of landslide events in the GLC and in the LRC, a quantitative measure of the citizen scientist's uncertainty in the latitude and longitude of an event collected at the time of submission to Landslide Reporter ( Table 2). The greatest number of reports in the LRC had an "exact" location accuracy, while the greatest proportion of reports in the GLC had a 5 km location accuracy. These initial results suggest that citizen scientists ascertain they can accurately pinpoint landslide locations, an indication that the quality of the reporting in the LRC may enhance the COOLR database. During the data validation process, we noticed that some citizen scientists specified much larger location accuracy values in their reports (25 km and 50 km) but that we could verify these locations from the source article and correct them to a value within 5 km. In the future, new application features such as using built-in measurement tools and changes to the how-to guides may help to improve the characterization of report location accuracy. Nevertheless, validated results indicate that citizen scientists are submitting quantifiably accurate data to the LRC, and the improved spatial accuracy with respect to the GLC could help scientists discern the environmental factors affecting landslides for their scientific models.
Lastly, we tested how the new LRC data might help improve landslide hazard modeling by superimposing the LRC and GLC data with the landslide susceptibility map produced by Kirschbaum et al. (2016) [15,57]. Data from COOLR will be used to validate the susceptibility map, the LHASA model, and other products [15], so it is valuable to see whether landslides submitted are occurring in predicted landslide-prone areas. In Fig 10, the greatest proportion of LRC events are located in areas with high landslide susceptibility, while the smallest proportion of events are centered in areas of low landslide susceptibility. The results confirm the landslide susceptibility distribution of citizen scientist-contributed events is comparable to that of the GLC, demonstrating that the LRC may help to improve future versions of the landslide susceptibility map, LHASA model, and other modeling efforts within the community.
Despite initial indications that the LRC can help to address gaps in the GLC repository, it is too early in the project to conclude that the citizen science contributions have reduced spatial biases in the repository, and in some cases may have added additional bias due to the citizen scientist's geographic location (e.g. Germany) and proximity to populated areas. Further, the number of landslide reports in this sample are too small to draw conclusions about other spatial biases like political or economic differences among regions or draw patterns of landslide hotspots. However, the number of in-person observations and reports from non-English sources, the appearance of citizen scientist-submitted reports far from any GLC landslide events, and the density of LRC points within areas not covered by the GLC (Fig 6) suggest that Landslide Reporter may be an important and effective tool to enhance and improve global landslide catalogs as the citizen science database grows over time.

Future improvements to COOLR
While early results of COOLR-including Landslide Reporter and Landslide Viewer-are promising, many improvements could increase the success of this system and fulfill the project goal of serving as an open repository for global landslide data sharing. First, the COOLR project has begun including other landslide inventories, so a procedure for systematically adding these inventories into COOLR must be created. Inclusion through the ArcGIS REST API is straightforward because the inventory will be hosted elsewhere and then referenced as a new layer on Landslide Viewer. However, an improved manual import strategy is needed to include other landslide inventories directly into COOLR. The landslide report fields prioritized are heavily dependent on the research goals of an institution and may not include some of the key information the GLC and LRC requires, such as a date of the landslide event. Data fields not prioritized by COOLR may end up as free text in the description field. We also foresee that the process of integrating inventories will bring new challenges of spatial bias, temporal bias, and duplicate landslide events. Future work should follow best practices from recent research that analyzes other integrated landslide inventories created from heterogeneous datasets [11,58,59].
To appeal to broader and more diverse audiences, particularly in areas where there is a current dearth of landslide data available, the User Interface (UI) of Landslide Reporter can be improved for a better citizen science experience. The current design of the application is accessible in both web and mobile layouts and has a simple submission form; however, Landslide Reporter requires an internet connection to interact with the form. One solution is to develop a standalone application that can be downloaded onto a mobile phone or tablet to be used in situ or in areas of limited connectivity to increase use for fieldwork or disaster responses as well as encourage accurate submissions.
Another future improvement is to expand the interface to be accessible in other languages other than English, whether by human translation or through machine translation services like Google Translate (https://translate.google.com/). Machine translations are not perfect, but they provide a low-cost service when volunteer or paid translators are not available. If the application remains in English, the COOLR project website should provide translations of the how-to guides in other languages. At the time of this publication, we are working with volunteer translators through the newly-formed "Sciencing with NASA" NASA Facebook group (https://www.facebook.com/groups/Sciencing/) to translate a two-page introduction to Landslide Reporter. One volunteer has expressed interest in translating more, so we have asked her to translate the "Add a Landslide Event to COOLR" how-to guide so citizen scientists have access to the full instructions in a native language. We will make these materials available on our website to bring larger international audiences to the project.
Although a simple form has been used by many citizen science projects in Table 1, there may be better ways of collecting landslide data. A future iteration, supported by user feedback, could test different interfaces or data collection methods, such as asking yes/no questions to populate each field (e.g. "Did the landslide occur by a road?"), and having the option to submit or upload photos directly (an important element that is not included within the current architecture). Improving accessibility, transparency, and the usability of Landslide Reporter, along with other user experience changes, can reach greater numbers of potential citizen scientists for this project.
Next, the current setup for approving reports prevents citizen scientists from tracking all of their previously contributed landslide events and it does not send feedback on the status of newly contributed reports. Feedback is critical to the success of the project, because it lets citizen scientists know that their submissions are accurate and contributing to science, a significant motivator for participation [60]. A future version of Landslide Reporter should include an engaging User Interface where citizen scientists can see the total number of reports they contribute to COOLR and receive feedback on why their previously contributed reports were accepted, rejected, or modified during the quality control/data validation step in the process. "Gamifying"-or adding in elements of a game-the entire process may also encourage greater participation or more frequent participation from those who like to score points, compete with their peers, or collect prizes while doing citizen science [61]. Citizen scientists could be motivated to submit the greatest number of landslides if they see other users' total number of approved landslides on a leaderboard. Another potential motivation and feedback method is to reward citizen scientists with tangible prizes like badges or certificates. Such user-facing elements would improve the quality of reports received, further motivate citizen scientists who are driven by a desire to contribute to science, and show appreciation for citizen scientists' time and effort.
An exciting future avenue in which we hope to move is using Landslide Reporter for disaster response and mapping efforts, leveraging available remote sensing imagery and other sources to dynamically map landslides that can be immediately validated and seen by emergency response teams. Following a disaster, we envision that Landslide Reporter's polygon submission form could be used to systematically and collectively map landslides triggered by a major earthquake or storm, bringing in disaster-specific remote sensing data. It could also allow participants to validate other citizen scientist submissions so that data approval is timely. One example where this tool could have improved coordination and data sharing and dissemination following a disaster was the 2015 Nepal earthquake in which a 7.8-magnitude earthquake and its aftershocks resulted in widespread landsliding. A citizen science team of over fifty experts from around the world formed an Induced Hazards Team (Table 1) and used high-resolution imagery to map landslides that may affect disaster response and recovery activities [1]. We can also look to the successes of disaster mapping projects like Tomnod (http://www.tomnod.com/) and Humanitarian OpenStreetMap Team (https://www.hotosm. org/) for their best practices. Leveraging the power of citizen scientists, it could be possible to overcome difficulties in post-disaster landslide inventory collection like the mapping of hundreds of thousands of landslides triggered in a single event [62] or inaccuracies in landslide locations of the events in the GLC.
While increasing the functionality and accessibility of the User Interface is vital to expanding the reach and success of COOLR, the discoverability of this system is also fundamental. Therefore, we will develop new strategies to locate and attract potential citizen scientists for COOLR. We will continue to examine the successes of other citizen science data collection projects like eBird, NASA's GLOBE Observer (https://observer.globe.gov/), and the Community Collaborative Rain, Hail & Snow Network (CoCoRaHS) (https://www.cocorahs.org/) to learn more about how they are reaching out to new citizen scientists. We may also be able to leverage their and other citizen science networks to promote this project. Future outreach should focus on how to involve more community groups and educational groups beyond social media who could potentially benefit from collecting and using the data. High school and university professors are also an important group to target as they could incorporate COOLR into their Earth science course curriculums and labs to have students contribute to the repository as well as teach data analysis techniques. Furthermore, it has been studied that citizen scientists are driven to contribute by both intrinsic (desire to learn, do a hobby, meet people, impart knowledge, and help others) and extrinsic (desire to gain experience for their career) motivations. Thus, additional social research could focus on the specific motivations compelling citizen scientists to contribute to Landslide Reporter [63].

Conclusions
The COOLR project represents a unique approach to provide openly available landslide inventories to the community as well as encourage citizen science participation to expand and improve landslide data, given that it is the first landslide citizen science project on a global scale. The three main objectives for this manuscript were to (1) introduce the COOLR project and its methods as a tool for building and sharing a global landslide inventory using citizen science; (2) present the preliminary results of the Landslide Reporter project as proof-of-concept; and (3) discuss future improvements to the project to advance landslide data gathering with citizen science. To fulfill the first objective, this paper documented the setup of the Landslide Reporter and Landslide Viewer web applications, the repository design, and the citizen science methods. Project outreach is performed using the widely-used social media sites Facebook and Twitter with the help of news posts, NASA social media accounts, and citizen science project catalogs.
Next, preliminary results from the 162 new landslide events collected by 49 participants in the first thirteen months of operation demonstrated Landslide Reporter's ability as a successful citizen science project for spatially and temporally diverse data that fills in data gaps and challenges known data biases. The collected events exhibited a wide distribution with events that occurred in 37 countries and from 1897 to 2019. Duplicate events submitted by multiple citizen scientists still helped enrich the repository with new details about an event. A closer look at the density of events that occurred in Europe, submitted mostly by one enthusiastic citizen scientist, revealed that citizen scientist-contributed data is already filling in spatial gaps in the NASA Global Landslide Catalog (GLC) data. It also revealed the important role of citizen science "super users" working in tandem with citizen scientists who submit one or two reports. Furthermore, some of these Landslide Reporter Catalog (LRC) events were located hundreds to thousands of kilometers away from any existing GLC landslide events. Observations from social media and from the source information submitted with landslide events indicated we are reaching target audiences of landslide experts and amateurs around the world who brought news sources and personal observations in 11 languages into COOLR. More than 60% of reported events contributed by citizen scientists came from non-English sources and roughly 15% came from in-person sources, showing promise that language and newspaper reporting biases to the GLC may be reduced with the help of citizen scientists. An analysis of data location accuracy, a metric collected at the time of submission, revealed citizen scientists submitted proportionally more spatially accurate landslides in comparison to the GLC. Lastly, superimposing LRC and GLC events on the landslide susceptibility map produced by Stanley and Kirschbaum [15,57] depicted both catalogs have similar proportions of high, moderate, and low susceptibility events, corroborating the spatial accuracy of the citizen scientist data. This study analyzed the results of the first year since project launch, so it is too early to draw conclusions on how citizen science is handling other biases noted by Kirschbaum et al. [14]. However, these initial results show proof-of-concept for how this platform may help to address data gaps created by collecting reports through the GLC methodology through in-person observations, non-English sources, and the appearance of gap-filling LRC reports far from any GLC landslide event.
The global and open nature of COOLR means that the project could serve as a centralized place for the collection and dissemination of landslide data on local to global scales. We envision that researchers could use Landslide Viewer to locate existing landslide inventories and landslide data for an area of interest. Landslide Viewer has already begun collecting regional and state-wide landslide inventories by referencing their Esri ArcGIS REST API or by including them directly into COOLR. At the time of this publication, four state landslide inventories are referenced and four landslide inventories from academic and community volunteer groups are imported into COOLR, all available on Landslide Viewer. In parallel with Landslide Viewer, Landslide Reporter could be used as a central location for contributing landslide data, which can be especially useful to groups who do not have such a system in place locally for submitting such information.
At this time, the COOLR project's reach is constrained by the English language, and the web application could use other features to make collecting landslides in Landslide Reporter more engaging considering the limited number of submissions received over the first year. Therefore, the third objective of this manuscript was to discuss ideas to expand and evolve COOLR in directions that will improve the types of data collected, how it is collected, the quality of the submissions, and the number of submissions. Presently, we are bringing in volunteer translators to create a two-page document in other languages describing Landslide Reporter to reduce language barriers to use the application. Future versions of this project should strategize integrating other landslide inventories into the repository that have different fields in their databases. We also foresee improving the user experience by allowing users to access their past submissions, incentivizing citizen scientists through game elements in the application, inviting expert citizen scientists to participate in data validation, and adding in machine or human translation. Finally, we hope to include collective post-disaster mapping in Landslide Reporter, allowing citizen scientists to contribute to post-disaster response, recovery, and mitigation.
Given that COOLR is a research project and subject to the same funding processes as other research grants, the evolution of future improvements to COOLR outlined above are still unknown. Nevertheless, we are encouraged by the preliminary results and feel that citizen science is an optimal path for improving the GLC and opening up more landslide data to the global community. We hope to continue the project and will work with national and international partners and stakeholders to examine the different ways in which COOLR may evolve. We hope COOLR will become a highly useful tool for collecting and sharing landslide data on all geospatial scales and substantially improve access to and accuracy of landslide information that can be used for risk assessment, research, and decision-making to save lives and property.
Supporting information S1 File. Data validation protocol. The detailed protocol used for checking all citizen science data contributed to the Cooperative Open Online Landslide Repository (COOLR) through the Landslide Reporter application as of February 2019. (PDF) the members of the NASA GSFC Citizen Science Group for their continued support of the COOLR project through many iterations of feedback given on how to improve the landslides. nasa.gov website, the web applications, and the citizen science process before and after project launch. Thank you to Amy Kaminski (NASA HQ) and Karen James (NASA HQ/InuTeq) for their guidance and support for our project on an agency level. Thank you to our friends and family, colleagues, and early supporters who have offered guidance on project branding and logo design, beta tested our website and applications, shared our project, and gave valuable feedback. Thank you to Sean McCartney (NASA GSFC/SSAI) for reviewing an early draft of this manuscript. Thank you also to the PLoS ONE editor and the anonymous reviewers for providing important feedback on the manuscript to improve its structure and content. Thank you to all of the networks that have shared our project to the public including but not limited to NASA Earth Observatory, NASA Solve, NASA Education EXPRESS newsletter, the Federal Community of Practice for Crowdsourcing and Citizen Science (FCPCCS), SciStarter, the American Geophysical Union's (AGU) The Landslide Blog, EarthSky, and the United Nations Platform for Space-based Information for Disaster Management and Emergency Response (UN-SPIDER). Last but not least, thank you to our citizen scientists and friends who have contributed landslides to our project, shared our project on their social media feeds, and gave feedback. We are sincerely grateful for all of the minutes and hours of your time contributed to making the COOLR project possible.