PROTOCOL: Mapping the scientific knowledge and approaches to defining and measuring hate crime, hate speech, and hate incidents

Abstract The overallaim of the review is to map the definitions and measurement tools used to capture the whole spectrum of hate motivated behaviors, including hate crime, hate speech and hate incidents. This will benefit the field of hate studies by providing a baseline that can inform the building of cumulative knowledge and comparative research. The first review objective is to map definitions of hate crime, hate incidents, hate speech, and surrogate terms. Specific research questions underpinning this objective are: (a) How are hate crimes, hate speech and hate incidents defined in the academic, legal, policy, and programming literature?; (b) What are the concepts, parameters and criteria that qualify a behavior as being hate crime, hate incident or hate speech?; and (c) What are the most common concepts, parameters and criteria found across definitions? What are the differences between definitions and the elements they contain? The second review objective is to map the tools used to measure the prevalence of hate crime, hate incidents, hate speech, and surrogate terms. Specific research questions underpinning this objective are: (a) How are definitions operationalised to measure hate crimes, hate speech, and hate incidents?; and (b) How valid and reliable are these measures?


| The problem, condition, or issue
There is limited international consensus for how to define behaviors motivated by hate or containing a hate element, which include hate speech, hate incidents and hate crime (Schweppe, 2021 (Chakraborti & Garland, 2015;Hardy, 2019). This definition overlaps with what practitioners often define as hate incidents, that is, all malicious behaviors motivated by hate that fall below the threshold of criminality (Anti-Defamation League, 2019). Some use the term "hate incident" to capture all malicious behavior motivated by bias, including both criminal and noncriminal acts (Sadique et al., 2018). The definitions of hate crime and hate incidents overlap with the concept of hate Campbell Systematic Reviews. 2022;18:e1228. wileyonlinelibrary.com/journal/cl2 speech, which includes verbal or non-verbal manifestations of hatred, such as gestures, words or symbols like cross-burnings, bestial depictions of members of minorities, hate symbols, among others (Strossen, 2018). Some of these behaviors-for example, incitement to hatred, Holocaust denial-might be regulated by criminal law in certain jurisdictions (thus overlapping with some definitions of hate crime), by civil law, or not regulated at all (thus overlapping with some definitions of hate incidents). Determining whether a crime is motivated by hate is a well-known challenge in the literature, which led to the adoption of different definitional models: the "animus model" and the "discriminatory selection model" (Lawrence, 1999). The animus model requires that the hate element (i.e., a form of bias, prejudice or hostility) is present and visible in the crime. For example, the offender might be seen as yelling a racial slur while attacking the victim. Conversely, under the discriminatory selection model, a crime is defined as hate motivated by reason of the victim's characteristics and perceived identity. For example, selecting a victim from a minority group is sufficient to define a crime as a hate crime (Lawrence, 1999).
Cross-cultural research found that, at an international level, hate crime and hate speech statutes are strongly influenced by the different social, technical, historical and cultural contexts across nations (Sheppard et al., 2021). For example, Italy has banned the display of ideas and symbols of fascism with the Law 205/1993 known as Mancino law (Campani, 2016). In the German context, the legacy of the Holocaust is mainly responsible for the criminalization of public expressions of hate that could engender or promote violence to protected groups such as Holocaust denial and trivialization (Bleich, 2011;Kahn, 2005). On the other hand, in the United States freedom of speech is constitutionally protected and has been a central tenet of individual liberty that has prevented the country from passing stringent laws. But, some forms of speech are criminalized in the United States, such as speech that incites imminent threat of violence (Heyman, 2009 anti-Semitism or homophobia (among others). In the literature, these community-specific terms are used to capture attitudes and behaviors interchangeably. In some jurisdictions, there is a considerable overlap between the concept of hate crime and neighboring concepts like "extremism" and "terrorism." For example, several European countries criminalize membership of extremist groups, and include these acts within their national concepts of hate crime (Perry, 2016).
In the United States, domestic terrorism is often defined as hate crime (Taylor, 2019 (Green et al., 2001;Vergani et al., 2021). Some argue that watch groups might have a vested interest in inflating the perception of hate against the communities they represent (Kaplan, 1997). While acknowledging this critique, we believe that, in a context where hate crime data quality is often suboptimal because of all the limitations outlined above (Gerstenfeld & Grant, 2004;Saucier et al., 2006), community registers are a key source of data that can be used to compare to other sources (Mason et al., 2017).
Both on-and offline media are an important source of primary and secondary hate data. Web searches can be used to retrieve media coverage, law enforcement reports, and non-profit reports to create a database of bias and hate crimes incidents, as in the case of ProPublica. On-and offline media can also be the vehicle of hate speech, and there is a growing scholarship focusing on creating effective automated detection tools to capture and measure it (see Poletto et al., 2021;Williams, 2021

| Why it is important to do the review
Hate has been a persistent problem across human history (Sternberg, 2003), and it has become an ever more pressing issue in the wake of VERGANI ET AL.  (Schweppe, 2021;Sheppard et al., 2021). This mapping will help government and nongovernment stakeholders in North America, Europe, Australia and New Zealand inform the next generation of policies, programs, and research, as well as advocacy for improving legislation. The reasons behind the choice of these jurisdictions is explained in the "Population" section.
This project's outcomes (i.e., the final report and other publications) will provide a comprehensive mapping of the current portfolio of definitions and measurement tools available in North America, Europe, Australia and New Zealand. This will support the critical appraisal of strengths and weaknesses of different countryapproaches, and the strategic planning and development of the next wave of research, policy and legislative efforts and shape the next generation of anti-hate efforts globally.

| How this review might inform or supplement what is already known in this area
Many scholars have discussed the problems associated with the lack of consistent definitions and measurement of hate crime, hate speech and hate incidents both within federal countries like the United States and across different countries in North America and Europe (see Schweppe, 2021; see also Sheppard et al., 2021). They outlined the main issues (as discussed in the first section of this protocol) and highlighted the tendency for researchers, policy makers and practitioners to work in silos, each developing their own definitions and measurement of hate crime with little (or no) dialogue across sectors (Chakraborti & Garland, 2015;Perry, 2016 community-specific terms (e.g., anti-Semitism); 7) different types of measurement tools, including third party reporting systems and victimization surveys.
The review will allow us to unpack strengths and weaknesses of definitions and their operationalization for measurement, and discuss legislation, program and policy gaps overall, which will allow us to draw conclusions relevant for North America, Europe, Australia and New Zealand.
We conducted a search of the literature using the following terms to identify existing reviews: hate crime* OR hate speech* OR hate incident*. Searches of the following locations did not identify any existing systematic reviews (completed or ongoing) on the specific topic proposed in this proposal (i.e., definitions and measurements of hate crime, hate incidents and hate speech): Some criteria apply to both objectives (see "population," "phenomenon of interest," and "context" sub-sections).

| Types of studies
For both objectives, this review will include the following types of documents published after 1990: (1) academic literature in the social and psychological sciences (including criminology, sociology, political science, law and psychology, computer science and sub-fields like regional studies, religious studies and peace and conflict studies); (2) gray literature by watch-groups and other governmental and nongovernmental organizations.
Additionally, this review will include current legislation, including hate crime, hate speech, and hate incidents found in federal and state statutes, including criminal and civil law statutes, and international agreements (e.g., UN; Council of Europe; American Conventions, etc.). Reported case law will not be reviewed because it is a large field that would warrant a separate project with a different approach.

Review Objective 1: Mapping definitions of hate crime, hate incidents and hate speech
To achieve objective 1, we will include documents that (1) focus specifically on defining and conceptualizing "hate crime," "hate speech," "hate incidents" or any surrogate terms that are used to capture these concepts OR (2)  The documents might explicitly state that they are "defining" the term (e.g., "we define hate crime as … "), or directly describe what is meant by the term (e.g., "hate crime is …"). Definitions of key multinational organizations (such as the United Nations (and specific committees such as the UN Committee on the Elimination of Racial Discrimination) and the Organization for Security and Co-operation in Europe) will also be included in the review.
In relation to objective 1, this review will include: Academic literature: • We will include studies that might assess reliability and/or validity of the original measurement tool. Empirical studies-aside from those focusing on the development of a tool-will be included. Please find a full list of databases in The search terms within each column will be connected with "OR" operators; attribute-specific and behavior-specific search terms will be connected with a proximity operator where available, such as "W/n" in Scopus, or otherwise with an "AND" operator; and behavior-specific and country-specific search terms will be connected with an "AND" operator. We have chosen to use a proximity operator, where available, to connect attribute-and behavior-specific search terms to improve the precision of our searches. We will conduct one round of forward citation searching in Google Scholar of the academic studies that will be included in the review.
For each electronic search, we will document the following details to ensure reproducibility of our search strategy: • Date of the search (dd/mm/yyyy) • Name of the database • Search string used (copied and pasted from the search field) • Number of search results

| Searching other resources
To complement electronic searches, we will use complementary search strategies to minimize the risk of bias. These search strategies include: (1) Hand-searching of reference lists of any systematic reviews on hate crimes, hate incidents, hate speech, or a surrogate of these terms identified throughout the review process.
(2) Hand-searching of reference lists of any documents included in the review, including systematic reviews.  Table 2.
To ensure reproducibility of our search strategy, for each search carried out through means other than electronic searches, we will record: • Date the search (dd/mm/yyyy) To identify current federal and state statutes, we will draw on key gray literature providing the full list of current legislation regulating hate crime, hate speech and hate incidents in a given country. An example is the report of the Royal Commission of Inquiry into the Terrorist Attack on Christchurch Mosques on March 15, 2019, which identifies and lists all "Hate speech and hate crime related legislation." For each country, we will develop a list with relevant current legislation based on this strategy. Prior to coding, we will send these lists to academic experts of hate crime and hate speech legislation and legislation practitioners' from each country of focus for review.

Review Objective 1: Definitions
What follows are exemplars of the types of papers to be included in the review and their uses.
In Fuentes Osorio (2017) This definition of "hate crime" outlines three key criteria: first, hate crime is a "classic crime," that is, the behavior in question is defined as criminal under the law; second, hate crime is identifiable by the perpetrator's motivation to engage in criminal behavior; third, hate crime is identifiable based on the discriminatory nature of the choice of the victim. This study meets our inclusion criteria (2) and (3) in relation to objective 1, as it focuses specifically on hate crimes, hate incidents, hate speech or a surrogate for these terms and offers a specific definition of hate crime, hate speech, hate incidents or a surrogate for these terms.

Review Objective 2: Measurement tools
In Benier et al. (2016), the authors explore the applicability of ecological theories of hate crime developed in the United States for the Australian context. The authors draw on established definitions of hate crimes by Green et al. (2001), Victorian Equal Opportunity and Human Rights Commission (2010) and the FBI (2013) as […] unlawful, violent, destructive or threatening behavior in which the perpetrator is motivated in whole or in part by prejudice towards the victim's perceived race, ethnicity, religion, sexual orientation, gender identity, age, impairment or homelessness (Benier et al., 2016, p. 479).
The authors operationalize this definition in the form of survey questions on direct and indirect victimization experiences for a range of crimes, including break-and-enter of private residences, various forms of property damage as well as muggings, assaults and sexual assaults. The authors provide the following as an example of such a survey question (Benier et al., 2016, p. 485): While you have lived in this community, has anyone ever used violence such as in a mugging, fight or sexual assault against you or any member of your household anywhere in the community?
Survey respondents who answered this question with "yes" were then asked whether they believed the incident had occurred because of the victim's skin color, ethnicity, race or religion; only participants who answered both questions with "yes" were classified as reporting hate crime victimization, regardless of whether they reported direct or indirect victimization (Benier et al., 2016, pp. 485-486). Using multi-level logistic regression models, the authors estimated the effects of household-and neighborhood-level variables on the likelihood of hate crime victimization in Brisbane neighborhoods.
This study meets our inclusion criteria (1), (2), and (3) in relation to objective 2 because it presents empirical research, identifies how hate crime, hate speech, hate incidents or a surrogate for these terms are operationalized and describes the type of data employed or collected to study these terms and the method used to analyze the data.

| Selection of studies
Within our research team, we have divided responsibilities for the title/abstract as well as full-text screening and coding of potentially relevant documents based on expertise, anticipated volume of potentially relevant documents and language knowledge. The groups are as follows: Germany and USA; Canada, France, Great Britain and Ireland; Spain, Italy, Australia, and New Zealand. This is relevant insofar as it relates to the way we map out our screening and coding processes, as detailed below.
We will use EndNote and Zotero to manage all documents retrieved throughout the search process. The use of Zotero, however, will be limited to researchers who may not have access to EndNote, for the manual entry of legislation and gray literature documents. All academic documents will be imported into Endnote.
Legislation will undergo an expert review instead of title/abstract screening process. Upon completion of the search process, each country team will compile a list of legislation that it has retrieved and send this list to a legal scholar or practitioner with expertise in the respective country for review. We will provide the expert with a summary of our project and its objectives and ask them to remove any legislation from our list that is outdated and replace it with most up-to-date legislation. Subsequently, we will import all legislation into EPPI Reviewer for full-text screening.
All academic and gray literature citations will be imported from Endnote and Zotero into EPPI Reviewer to conduct title/abstract and full-text screening. Prior to screening, all duplicates will be removed in EPPI Reviewer.
We will use EPPI Reviewer's machine learning functionality "Priority Screening" to support title/abstract screening. Because we include documents in English, French, German, Italian and Spanish, we will progress through title/abstract screening in stages. In a first step, we will import all citations in English into EPPI Reviewer and remove all duplicates. In a second step, we will set up "Priority Screening" in EPPI Reviewer and "train" the machine learning algorithm based on a random sample of citations. To do so, each title/abstract in the random sample will be screened independently by the whole team researchers, with any disagreements resolved during team meetings. As we seek to combine the "training" of the machine learning algorithm with achieving intercoder reliability of 90% or higher. Once we have achieved the aforementioned level of intercoder reliability and the algorithm is considered to be working effectively, we will proceed with title/abstract screening of all citations in English. To complete the title/abstract screening for citations in English, each citation will be screened by one researcher assisted by "Priority Screening." Upon completing title/abstract screening for citations in English, we will conduct the title/abstract screening for citations in French, German, Italian and Spanish in a second stage. Our team has at least one speaker for each of the languages other than English. One person will train EPPI Reviewer's ML algorithm and then use the "Priority Screening" in these languages when the minimum requirements for ML training are met. Effectively, the ML functionality will act as a second screener for the corpus in languages other than English. To check for accuracy, when Title & Abstract screening will be completed, we will undergo a random checking of 5% of the documents in languages other than English.

| Review Objective 1: Definitions
Inclusion and exclusion criteria for objectives 1 and 2 will be identical for title/abstract screening. Specific additional criteria will be added for full-text screening. The inclusion criteria for title/abstract screening are provided in Table 3.
Documents to be included in our review in relation to objective 1 need to meet the following inclusion criteria for full-text screening (Table 4).
To avoid confusion about whether any document is included or excluded in relation to objective 1, coders will be asked to assign a code upon making their decision that will label whether a document was included or excluded in relation to objective 1, objective 2, or both.

| Review Objective 2: Measurement tools
The procedure for title/abstract screening for documents in relation to objective 2 will be identical to that outlined above for documents in relation to objective 1. However, consistent with the inclusion criteria outlined above, the inclusion criteria for full-text screening will differ. These criteria are provided in Table 5.
To avoid confusion about whether any document is included or excluded in relation to Aim objective, coders will be asked to assign a code upon making their decision that will label whether a document was included or excluded in relation to objective 1, objective 2, or both.
Our team will include documents in English, French, Italian, Spanish, and German. Because these are the languages spoken in the countries that we focus on, and will therefore capture virtually all relevant documents. Documents in other languages will be excluded. training to all coders and before starting the full coding we will test intercoder reliability on a selected sample to ensure the reliability of the coding. As with title/abstract and full-text screening, we will establish intercoder reliability based on a random sample of documents in English.
Each document in the random sample will be coded independently by all researchers who participate in data extraction; disagreements will be resolved in team discussions. Further coding to achieve intercoder reliability of 90% or higher will be conducted as required and until at least 90% intercoder reliability are achieved.

| Review Objective 1: Definitions
To achieve objective 1, for each document we will extract information about the components of the definition of hate crime, hate speech, hate incidents and surrogate terms. Components include: (1) how the motivation is named (e.g., racist, Anti-Semitic, hate, etc.); (2) how the nature of the behavior is named (e.g., crime, incident, speech, act, etc.); (3) how the target is described (e.g., a person, a property, a person and property, etc.); (4) whether perception is a bias indicator; and (5) the protected characteristics (if any). Additionally, we will code categorical variables identifying whether: • the definition is adopted by any governmental or nongovernmental organization or linked to any statute or legislation; • the definition allows for capturing different degrees of bias motivation; • the victim of the hate behavior is assumed to be interchangeable; • the definition reflects one of the two models described by Lawrence (1999) "discriminatory selection model" or "animus model." 3.5.2 | Review Objective 2: Measurement tools To achieve objective 2, for each document we will extract information about: the type of measurement tool, defined as a vehicle or an aid to collect information and data (e.g., an online module to collect data about hate incidents, or an automated text analysis algorithm, or a survey); the metrics used to measure hate, defined as parameters (measures) or indices used for measurement, comparison or tracking performance (e.g., bias indicators); and the methods used to measure hate, defined as the process and approach involved in a systematic inquiry of hate crimes, hate incidents, hate speech or surrogate terms, and generally refer to study design or application of an analytical method to this topic.
We will code categorical variables looking at information about the target identities that the measurement tool encompasses, the bias indicators used (if any), whether it is adopted by any government or nongovernment organizations, the variables collected about the incident, the victims, and the offender, and the indicators used to assess the severity of the crime. If there is any analysis of quantitative data collected using the measurement tool, we will collect-when present-information about context and sample, data collection methods, data analysis, and any reporting about feasibility, efficacy, reliability or validity of the instrument. We will also collect information about whether the instrument is at concept development, pilot or implementation stage.
Based on the OMERACT tool to assess quality of measurement tools (Beaton et al., 2019), we will collect information relative to the domains of truth (i.e., does the instrument cover all the protected characteristics present in the relevant legislation), feasibility (i.e., is the instrument accessible? Is it available in multiple languages?) and discrimination (is there stability in situations of no change? Does the instrument detect score change in situation of real change? How well does the instrument distinguish between groups?). We expect that, while information about truth and feasibility might be available for most instruments, information about discrimination might not.

| Assessing the methodological limitations of included studies
We propose the following criteria that draw mostly on assessments of theories and concepts (for objective 1) and gap maps of tools, metrics and methods (for objective 2). 3.6.1 | Review Objective 1: Definitions To assess the quality of definitions, we will conduct post-hoc analyses of the definitions against the following criteria: 1) Does the definition contain unambiguous concepts (Ritzer, 1991)?
For example, we will look at whether the hate motivation is clearly defined (e.g., is the threshold between hateful and non-hateful 3) Is the definition adopted by governmental or nongovernmental organizations, or current statutes? We will look for this information inside the document.
We will operationalize these criteria as three separate dimensions to assess the quality of definitions by attributing a score from 1 (strongly disagree), 2 (disagree), 3 (agree), 4 (strongly agree) to each.

| Review Objective 2: Measurement tools
To assess the quality of measurement tools, we use the approach described by Sparling et al. (2019) in their gap map of tools, metrics and methods in the field of food systems and agriculture-nutrition.
First, we will look at whether instruments are at different stages of development, which include: (1) concept development; (2) pilot; and (3) wide-spread application. Second, we will look at whether the documents report any information about (1) feasibility, efficacy or internal validity of the instrument, and (2) validity or reliability of the instrument. Third, we will consider how much information about the hate incident is included in the instrument (e.g., about the offender, victim, incident, bystanders, etc.) and whether the instrument is adopted by any governmental or nongovernmental organization.
Finally, we will collect information relative to the domains of truth, feasibility and discrimination (Beaton et al., 2019). Where present, we will report reliability and validity of measurement tools included in this review, using the template in Wells et al. (2009). 3.7 | Description of data to be mapped 3.7.1 | Review Objective 1: Definitions Given the qualitative textual nature of the information extracted (see Data extraction and management section) we will attribute categorical codes to the text extracted. The unit of analysis will be the definitions. For each definition, we will code components (e.g., attribute, behavior, protected characteristics, etc.) and other additional qualitative categories (e.g., whether the definition is adopted by any government or nongovernment organization, whether it allows for degrees of motivation, etc.).
Specifically, to answer each research question, the following information will be examined:

| Review Objective 2: Measurement tools
The majority of the information that we will collect about the measurement tools will be qualitative and textual (see Data extraction and management section), and we will attribute categorical codes to the text extracted. For each instrument, we will code tools, metrics and methods, stage of development and additional qualitative information (e.g., whether the instrument is adopted by any government or nongovernment organization).
Additionally, when reported, we will extract additional information about sample, context, and any available information about feasibility, efficacy, reliability and validity of the instrument.
Specifically, to answer each research question of the review we will look at how are definitions operationalized to measure hate crimes, hate speech, and hate incidents? We will code tools, metrics and methods. We will also look at how valid and reliable are these measures? We will look at stage of development and additional qualitative information, as well as any available information about feasibility, efficacy, reliability and validity of the instrument, when reported.

| Data mapping
Final tables and figures will be decided based on the findings. The report will include a section discussing under-researched areas and gaps. Tables and figures will include a PRISMA diagram, and might VERGANI ET AL.
| 13 of 16 include a visual representation of trends in definitions and measurement tools, and a visual and numeric summary of definitions and measurement tools highlighting the most and least common components and instruments. The report will include a narrative section where we will interpret and contextualize the existing literature within paradigms and historical and political contexts to develop new understandings of definitions and measurement tools. If possible, we will aim to develop typologies of definitions and measurement tools.
3.8.1 | Review Objective 1: Definitions The unit of analysis will be the definitions. We plan to present univariate analysis of the categorical data, as well as bivariate analysis (e.g., cross-tabulations, bar charts) of definition components and additional qualitative categories with document type, academic field, country, language, year, and authors' affiliation (whether a university, a government or nongovernment organization). We will also present a narrative report describing definition components and additional qualitative categories, which will be used to provide an assessment of the definitions (see section Assessing the methodological limitations of included studies).
The full list of the analyses that will be performed is available in Supporting Information Annex 1.

| Review Objective 2: Measurement tools
The unit of analysis will be the instruments to measure hate speech, hate incidents, hate crimes or surrogate terms. We plan to present univariate analysis of the data that will be collected and coded as part of this review, which includes categorical data, as well as bivariate analysis (e.g., cross-tabulations, bar charts) of instruments, metrics, tools, methods and additional qualitative categories with document type, academic field, country, language, year, and authors' affiliation (whether a university, a government or nongovernment organization).
We will also present a narrative report describing instrument's characteristics, stage and additional qualitative categories, which will be used to provide an assessment of the definitions, together with any available information about the instrument's feasibility, efficacy, reliability and validity (see section Assessing the methodological limitations of included studies).
The full list of the analyses that will be performed is available in Supporting Information Annex 1.

| Assessment and investigation of heterogeneity
This study will map the heterogeneity of the field under investigation, highlighting trends and differences between definitions and instruments adopted across geographical contexts, research and policy domains. This will be done by displaying visually, using different types of charts (e.g., heat maps, bar charts, line charts, see Sparling et al., 2019), the different types of definitions and measurement tools components that will be coded as described in the "Data extraction and management" section.
3.9.1 | Review author reflexivity The team has a variety of disciplinary backgrounds that reflect different positions and approaches in the field of hate crime studies, including qualitative and theoretical sociology (Perry), quantitative criminology (Chermak and Freilich), mixed methods political psychology (Vergani) and mixed methods online research (Scrivens). The team is well versed in relevant theory and in the study of subfields relevant to this review, such as Islamophobia, anti-Semitism, terrorism, and violent extremism. This review's aim is to map-not summarize or meta-analyze-the existing definitions and measurement tools across different disciplines. The review team has been maintaining a reflexive position throughout all the stages of the review process, and decisions have been discussed critically and regularly among the team members with regular debriefing sessions to support with decision-making and coding.
Based on the team's collective and individual experiences, it is anticipated that the findings will reveal a combination of approaches and disciplinary contributions across the whole spectrum of social and psychological sciences relevant to the study of hate behaviors. The team will remain mindful of conscious and unconscious presuppositions and support each other to minimize the risk of these skewing our analysis or the interpretation of our findings. The research team is composed of researchers who greatly contributed to the literature under investigation in this review. For this reason, we will make sure that the authors of a document will not review and assess it.

DECLARATIONS OF INTEREST
None.

PRELIMINARY TIMEFRAME
The final report is due December 9, 2022.

PLANS FOR UPDATING THE REVIEW
None.

SOURCES OF SUPPORT
This review is funded by a Campbell Collaboration grant awarded to Vergani, Perry, Freilich, Chermak, Scrivens via Public Safety Canada.