Building de novo cryo-electron microscopy structures collaboratively with citizen scientists

With the rapid improvement of cryo-electron microscopy (cryo-EM) resolution, new computational tools are needed to assist and improve upon atomic model building and refinement options. This communication demonstrates that microscopists can now collaborate with the players of the computer game Foldit to generate high-quality de novo structural models. This development could greatly speed the generation of excellent cryo-EM structures when used in addition to current methods.

structure. Spurred on by the improved resolution of newly obtained maps, the growing cryo-EM community has generated hundreds of excellent-but also some error-containing and energetically unfavorable-atomic models [1,13,14]. Such errors not only jeopardize the cryo-EM field itself but also misguide downstream research that relies on accurate molecular models, such as mutational analysis and structure-based drug design.
Although rigorous structure and model validation tools tailored for cryo-EM are currently under intense development [14,15], improving the quality of cryo-EM model building remains an important area of research. The recent introduction of computational model-building tools geared toward cryo-EM offer the possibility of automated model building [16][17][18][19]. However, building accurate models into near-atomic resolution cryo-EM maps remains a substantial challenge, because atom positions at this resolution are not unambiguous and must be inferred with aid from molecular mechanics models.
Citizen scientists have been able to contribute to challenging problems in fields such as RNA design [20], neuroscience [21], sequence alignment [22], and quantum physics [23]. Thus, one possible model-building option is Foldit (https://fold.it), a citizen science computer game that challenges players to solve complex biochemistry puzzles [24]. Recent improvements to Foldit enable players to build protein structures into crystallographic, high-resolution maps more accurately than expert crystallographers or automated model-building algorithms [25]. Unlike crystallographic maps, which often rely on phase data inferred from model coordinates, cryo-EM maps are more suitable targets for Foldit because averaged EM data are directly interpretable and are independent of the model. Here, we show that crowd-powered model building by Foldit players can indeed substantially help cryo-EM scientists.
To assess the usefulness of Foldit for cryo-EM, the players were provided with cryo-EM densities corresponding to 4 segmented subunits of the antefeeding prophage (AFP, from a soil bacterium Serratia entomophila)-Afp1, Afp5, Afp7, and Afp9 [34]. For ease of comparison, the maps were filtered to 3.2-Å resolution to avoid local quality variation and contain information up to the same resolution of 3.2 Å, which is currently considered fairly high by the cryo-EM community but still arduously low for fully automated model-building algorithms. The players tried to achieve the highest possible Foldit score, which combines the Rosetta force field with map fitting [26,27]. The structures generated by players were compared with those produced by a cryo-EM expert who created models using the manual model-building and real-space refinement software Coot [28], followed by additional real-space refinement in Phenix [29]. Structures generated by the state-of-the-art automated model-building algorithms Rosetta "denovo_density," Phenix Map-to-Model, ARP/wARP, and Buccaneer [16][17][18][19] were included in the comparison. Standard EM validation tools and crystallographic statistics were used to evaluate the 4 approaches. Table 1 compares the results of the various methods, using multiple criteria to evaluate both the model fit to the map and physical plausibility. Rosetta, Phenix, and Buccaneer struggled to correctly place certain chains in the appropriate density (Fig 1, S1-S11 Figs). This difficulty likely stems from errors in side-chain assignment, because the map resolution is often too poor for unambiguous side-chain identification, and these approaches fit regions of the map with incorrect sequences (Fig 1). All 4 automated methods had difficulty generating plausible geometry ( Table 1, S1 Text). The Foldit structures and those generated by the microscopist produced accurate structures that were geometrically plausible and fit the maps well. Examining the models more closely shows that in most cases, the Foldit players placed slightly greater importance on bond geometry and steric clashes than the microscopist, who sacrificed these aspects for better fitting to the map (Table 1, Fig 1, S1 Text, S1 Table, and S1-S3 Figs). Although close in quality, at this resolution, we suggest it is appropriate to prioritize model grantome.com/grant/NIH/UH2-CA203780-02, SC and FK). This material is based upon work supported by the National Science Foundation under grant no. 1629879 (https://www.nsf.gov/ awardsearch/showAward?AWD_ID=1629879, SC). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. geometry over map fit (Fig 1D-1G). Indeed, above 3-Å resolution, outliers are unlikely to be sufficiently supported by experimental data [14].
The Foldit score function appears to correctly reflect model quality. As expected, because of the absence of the phase problem, and unlike previous Foldit collaborations with crystallographic data [25,30], the structures from Foldit players that were the best in each puzzle as determined by Phenix validation [14] were also the best according to the Foldit score function, which is based on the Rosetta score function [31], with terms that model properties such as electrostatics, hydrogen bonds, solvation, and torsion angles, with an additional parameter that accounts for electron potential map fit. This observation suggests that by collaborating with Foldit, only minimal work will be required by microscopists to obtain an accurate, highquality model.
Building models of large molecules into low-resolution data can be a time-consuming process for microscopists building structures by hand. However, in the 4 datasets presented here, For each of the 4 different proteins, any method that outperformed the other 5 for a particular metric is shaded in green, with any method outperformed by the other 5 shaded red. CC mask , CC box , CC peaks , and CC volume are correlation coefficients calculated between the model and the map. The differences between these correlation coefficients arise from whether the entire map is used (CC box ), only the map around the atomic centers (CC mask ), the molecular envelope defined by the model (CC volume ), or the strongest peaks in the model and map (CC peaks ) [14]. CaBLAM uses the geometry of Cα atoms to evaluate low-resolution structures [35]. Clashscore reports on the number and severity of steric clashes in a model, and Molprobity score combines the Clashscore with other geometric factors to provide an overall evaluation of model quality [35]. Foldit players had arrived at finished structures in less than 48 hours (S16 Fig). Examining the workflow of Foldit players revealed that different players used distinct strategies in their model building. In the case of Afp9, the winning players chose to prioritize map fitting first and waited until the end to optimize the geometry of the structure (S1 Movie). Alternatively, in the case of Afp5, the winning players instead performed geometry optimization intermittently over the course of map fitting (S2 Movie). The general consensus-among the winning players who generated these 4 Foldit solutions-was to fold the protein "by hand" in the early stages of the puzzle and then run "recipes" (in-game algorithms written by the players) toward the end of the puzzle. Detailed accounts from all of the Foldit players who produced these 4 models are described in the "Foldit Player Testimonials" section in S2 Text.
These results indicate that there are multiple routes toward cryo-EM model building and that Foldit players could greatly speed the arduous model-building process for many cryo-EM projects. Although collaborating with Foldit players currently requires contacting the Foldit developers, future developments will include the ability for cryo-EM researchers the ability to communicate with Foldit players easily.
The strategy described here takes advantage of the collective ability of nonprofessional citizen scientists; however, the Foldit modeling tools are also available for individuals. Foldit Standalone runs offline on a single workstation and can be used by researchers to build and refine their structures with the Foldit scoring function [32]. Alternatively, Foldit Custom Contests can now be administered by researchers to allow online, collaborative model building and refinement among a research group or department or even a class of students [33]. Although we anticipate that for best results, researchers should draw on the collective expertise of the Foldit players, these other options may be attractive in the very competitive cryo-EM field.
To conclude, with the rapid improvement of cryo-EM map quality, it is now paramount for our building and refinement skills and tools to improve commensurably. Enlisting the help of citizen scientists, such as Foldit players, is one option to do so.

Materials and methods
To generate puzzles for Foldit players, the cryo-EM map (EMD-4782) sharpened with an overall b-factor of 105 Å 2 [34] was segmented around each fitted monomer of Afp1, Afp5, Afp7, and Afp9 (PDB 6rao, [34]), with a radius of 3 Å around fitted atoms. For a detailed description of the Foldit puzzle setup and order, please see S1 Text. To calculate the FSC between models and map, a single version of an unfiltered, unsharpened segmented map was generated for each target by keeping a zone enclosing all fitted models (Microscopist, Rosetta, Phenix, Foldit) with a radius of 3 Å around fitted atoms. The FSC was then calculated between the

Fig 1. Comparison of model building for Afp7 in (A) an overall view and (B and C) views to compare side-chain fitting.
The Foldit structure is rendered in green, the microscopist structure in gray, the Phenix model in magenta, and Rosetta model in yellow. Because of the large deviations from the other structures, the Rosetta model is omitted in the zoomed-in views in parts B and C. The electron potential map is contoured at 2 σ. (D, E, and F) Comparison of key geometric and map fit parameters for each of the cases displayed here. (D) Comparison of Ramachandran outlier and allowed backbone conformations. (E) Comparison of Molprobity Clashscore-in both cases, lower is better. (F) Comparison of 3 different map-to-model correlation coefficients, in which higher values are better. More complete statistical analysis can be found in Table 1 and S8 Fig, with the underlying data provided in S1 Data. (G) Map-to-model FSC curves for Microscopist (gray), Foldit (green), Phenix (pink), ARP w/ARP (orange), and Buccaneer (blue) models. CC mask , CC box , CC peaks , and CC volume are correlation coefficients calculated between the model and the map. The differences between these correlation coefficients arise from whether the entire map is used (CC box ), only the map around the atomic centers (CC mask ), the molecular envelope defined by the model (CC volume ), or the strongest peaks in the model and map (CC peaks ) [14]. Afp7, antefeeding prophage 7; FSC, Fourier shell correlation; RMSD, root mean square deviation.
https://doi.org/10.1371/journal.pbio.3000472.g001 segmented map and a simulated map (up to Nyquist resolution with same pixel spacing) from each fitted model.

Ethics statement
Foldit has received IRB approval, and Foldit players provided informed consent to participate in research (University of Washington IRB STUDY00001238, titled: "Scientific Discovery Games").