Cynthia Dwork
Jon Kleinberg
ABSTRACT
We study the impact of content moderation policies in online communities. In our theoretical model, a platform chooses a content moderation policy and individuals choose whether or not to participate in the community according to the fraction of user content that aligns with their preferences. The effects of content moderation, at first blush, might seem obvious: platform speech is restricted. However, when user participation decisions are taken into account, its effects can be more subtle --- and counter-intuitive. For example, our model can straightforwardly demonstrate how moderation policies mayincrease participation and/ordiversify content available on the platform. In our analysis, we explore a rich set of interconnected phenomena related to content moderation in online communities. We first characterize the effectiveness of a natural class of moderation policies for creating and sustaining communities. Building on this, we explore how resource-limited or ideological platforms might set policies, how communities are affected by differing levels of personalization, and competition between platforms. Our model provides a vocabulary and mathematically tractable framework for analyzing platform decisions about content moderation.
Supplemental Material
- Shiza Ali, Mohammad Hammas Saeed, Esraa Aldreabi, Jeremy Blackburn, Emiliano De Cristofaro, Savvas Zannettou, and Gianluca Stringhini. 2021. Understanding the effect of deplatforming on social networks. In 13th ACMWeb Science Conference 2021. Association for Computing Machinery, New York, 187--195.Google Scholar
Digital Library
- Cynthia Dwork, Chris Hays, Jon Kleinberg, and Manish Raghavan. 2023. Content Moderation and the Formation of Online Communities: A Theoretical Framework. arXiv preprint arXiv:2310.10573 (2023).Google Scholar
- Anna Fang, Wenjie Yang, and Haiyi Zhu. 2023. Shaping Online Dialogue: Examining How Community Rules Affect Discussion Structures on Reddit. arXiv preprint arXiv:2308.01257 (2023).Google Scholar
- Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press.Google Scholar
- Carroll J Glynn, Ronald E Ostman, and Daniel G McDonald. 1995. Public Opinion and the Communication of Consent. Guilford Press, Guilford, Chapter Opinions, perception and social reality.Google Scholar
- Manoel Horta Ribeiro, Shagun Jhaver, Savvas Zannettou, Jeremy Blackburn, Gianluca Stringhini, Emiliano De Cristofaro, and Robert West. 2021. Do platform migrations compromise content moderation? Evidence from r/The_Donald and r/Incels. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1--24.Google Scholar
- Kate Klonick. 2017. The new governors: The people, rules, and processes governing online speech. Harv. L. Rev. 131 (2017), 1598.Google Scholar
- Yi Liu, Pinar Yildirim, and Z. John Zhang. 2022. Implications of Revenue Models and Technology for Content Moderation Strategies. Marketing Science 41, 4 (2022), 831--847. https://doi.org/10.1287/mksc.2022.1361 arXiv:https://doi.org/10.1287/mksc.2022.1361Google ScholarDigital Library
- Leonardo Madio and Martin Quinn. 2023. Content moderation and advertising in social media platforms. Available at SSRN 3551103 (2023).Google Scholar
- Mohamed Mostagir and James Siderius. 2022. Naive and Bayesian Learning with Misinformation Policies. Technical Report.Working paper, University of Michigan and Massachusetts Institute of Technology.Google Scholar
- Mohamed Mostagir and James Siderius. 2023. When Should Platforms Break Echo Chambers? (2023).Google Scholar
- Karl Popper. 1963. The open society and its enemies. Princeton University Press, Princeton.Google Scholar
- David G Post. 1995. Anarchy State and the Internet. Journal of Online Law, Article 3 (1995), 11 pages.Google Scholar
- Adrian Rauchfleisch and Jonas Kaiser. 2021. Deplatforming the far-right: An analysis of YouTube and BitChute. Available at SSRN (2021), 28 pages.Google Scholar
- Thomas C. Schelling. 1971. Dynamic models of segregation. The Journal of Mathematical Sociology 1, 2 (1971), 143--186. https://doi.org/10.1080/0022250X. 1971.9989794 arXiv:https://doi.org/10.1080/0022250X.1971.9989794Google ScholarCross Ref
- Craig Silverman, Craig Timberg, Jeff Kao, and Jeremy B. Merrill. 2022. Facebook Hosted Surge of Misinformation and Insurrection Threats in Months Leading Up to Jan. 6 Attack, Records Show. ProPublica (2022). https: //www.propublica.org/article/facebook-hosted-surge-of-misinformation-andinsurrection- threats-in-months-leading-up-to-jan-6-attack-records-showGoogle Scholar
- U.S. Supreme Court. 1927. Whitney v. California (247 U.S. 357) (L. Brandeis, concurring).Google Scholar
- Isaac Waller and Ashton Anderson. 2021. Quantifying social organization and political polarization in online platforms. Nature 600, 7888 (Dec. 2021), 264--268.Google ScholarCross Ref
Index Terms
- Content Moderation and the Formation of Online Communities: A Theoretical Framework
Recommendations
A Trade-off-centered Framework of Content Moderation
Content moderation research typically prioritizes representing and addressing challenges for one group of stakeholders or communities in one type of context. While taking a focused approach is reasonable or even favorable for empirical case studies, it ...
Identifying and Addressing Design and Policy Challenges in Online Content Moderation
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing SystemsMy doctoral research identifies and addresses the design and policy challenges of content moderation on the levels of smaller communities and larger platforms, using a combination of interviews and surveys. My prior work studies voice-based communities ...
Moderation Visibility: Mapping the Strategies of Volunteer Moderators in Live Streaming Micro Communities
IMX '21: Proceedings of the 2021 ACM International Conference on Interactive Media ExperiencesVolunteer moderators actively engage in online content management, such as removing toxic content and sanctioning anti-normative behaviors in user-governed communities. The synchronicity and ephemerality of live-streaming communities pose unique ...
Comments