Must social performance ratings be idiosyncratic? An exploration of social performance ratings with predictive validity

2.1 Background

We refer to traditional ESG ratings as those that are aggregations of the information in accounting data, i.e. sustainability reporting, and, as such, claim to measure CSR or CSP. There are a multitude of such metrics, and the screening methods investors refer to when considering ESG in portfolio composition are a spectrum of overlapping approaches (Latinovic and Obradovic, 2013), making conceptual clarification both important and problematic. Despite the considerable biases that may be the effect of financial incentives, most CSR or CSP metrics are produced by commercial companies, not by financially independent institutions (Dahlsrud, 2008). Such ratings aspire to represent a holistic view of a company’s sustainable performance, defined as CSR or CSP. The holistic view is achieved through the aggregation of many feature-specific indicators, which involves gauging the importance of one against the other because all features and behaviors of a company cannot be equally important for the holistic assessment. The ESG industry seems not to have derived the ratings from a theoretical concept but has jumped immediately to measurement as if the validity of the ingredients would guarantee the validity of the compound metric. In our view, problems with traditional ESG ratings discussed in section 2.2 are caused by a contradiction between the idea of CSR activities or CSP as voluntary acts of “doing good” and the concept of “responsibility.” We therefore outline conceptual nuances as they have gradually emerged in the development of CSR and CSP since the 1950s before discussing validity problems with traditional ESG ratings.

Commercial raters propose the ESG ratings as measures of the extent to which companies act or perform CSR. The ratings are the offspring of a long conceptual evolution since Bowen published his seminal book Social Responsibilities of the Business Man (Bowen, 1953). Back then, three CSR themes dominated: the manager as public trustee, balancing competing claims to corporate resources and corporate philanthropy (Frederick, 1994). Formal definitions of CSR appeared in the 1970s with an emphasis on performance (Carroll, 1999). Accepting responsibility was not enough, and responding to responsibility and showing responsiveness became the keys. CSP was an attempt to relate the nuances of responsibilities and responsiveness, the CSR₁ and CSR₂, and represented a desire to emphasize the outcome of socially responsible initiatives (Carroll, 1979; Wartick and Cochran, 1985; Wood, 1991). The link between CSR and corporate financial performance (CFP) became popular in the 80s (Lee, 2008) and has since continued to be the dominant research issue and the main concern for industry practitioners.

Before we discuss the CSR–CFP link, there are a few conceptual distinctions that need to be clarified because they have implications for measurement. A conventional way of thinking about CSR (and CSP) is to assume that it includes certain features and behaviors of companies. The definition then becomes an issue of defining the features and acts by naming them in categories that represent types of CSR. A definition of CSR developed by researchers for research and used for longer than three decades is provided by Carroll’ s (1979) four categories of normative expectations: economic, legal, ethical and discretionary. The glue that provides internal consistency to this definition is that companies need to comply with societal responsibilities.

Two of the categories in Carroll’s concept do not, however, constitute normative expectations, or norms in a strict conceptual sense. In jurisprudence, which is a subject that specializes in the specification of the meaning of norms, the ‘ought to’ refers to a statement’s legal validity (Dworkin, 1985; Kelsen and Knight, 1967; Summers, 1963), which means that a statement is uttered as part of the legal system and obtains its normative meaning from its legal-institutional context. Similarly, formalization in organizations has been described in the sociological literature as the outcome of a process that associates behavioral expectations with organizational membership (Luhmann, 1995), which constructs and maintains the meaning difference between individual and organizational perception. These differences in meaning refer to Galtung’s sociological explanation of the difference between normative and cognitive expectations (Galtung, 1959). The sociological explanation of normative expectations is that they are formed by social decision-making processes associated with a social entity, e.g. a group or organization, establishing expectations that become criteria for membership in the entity. Normative expectations therefore differ from cognitive expectations in the way they can be changed in the face of non-compliance: individuals can become disappointed or upset when observing non-compliance with a norm (which they cannot themselves adjust to facts) but they can easily adjust their cognitive expectations if they are not consistent with reality.

None of these theoretical distinctions are noticeable in Carroll’s CSR concept. On the contrary, it mixes cognitive and normative expectations as if there were no difference between them. Carroll’s inclusion of economic responsibilities is inconsistent with how financial performance is viewed in practice – a company’s profit is expected by stakeholders as a fact, not as a norm. CSR defines the responsibilities companies need to comply with to earn the right to prosper financially, but society does not demand that companies or their owners prosper (Kang and Wood, 1995). This interplay between the responsibilities, rights and voluntary aspects of sustainable performances suggests a difference between CSR, which semantically refers to responsibilities and has a tradition of emphasizing compliance, and CSP, which emphasizes performance often in terms of discretionarily selected acts of “doing good” (Wood, 2010). The differences in focus and emphasis have important implications for operationalization. In Carroll’s terms, there is a difference in the degree of precision or determinacy between, on the one end of the spectrum, legal responsibilities, ethical or moral obligations and, on the other end, discretionary responsibilities. The most precisely determined responsibilities are the legal, for which there may be financial or even punitive repercussions for non-compliance. The moral obligations are less distinct than the law and are not institutionally clarified, e.g. through judicial procedure. There is arguably a moral responsibility for businesses to reduce CO₂ emissions, but it would be difficult for any business to know whether they have done enough or not. Carroll’s discretionary responsibilities express the expectation that any company should invest efforts in doing good, but neither the nature of the act nor its object are specified. The problem with this notion is that a responsibility to act in a manner the company finds “good” cannot be the norm because the difference between whether the company’s acts are compliant or not must be decided by the company. No one else knows what the company discretionarily regards as a good deed. Non-compliance is therefore undeterminable, and a CSR concept defined as compliance can therefore neither include financial performance nor the discretionary aspect of “doing good.”

For these reasons, the aspect of CSP that targets “doing good” is less suitable for operationalization than CSR, defined as the societal normative requirements on businesses. While “doing good” makes sustainability a subjective, highly observer-dependent construct, the compliance-oriented construct is anchored on societal preferences regarding business externalities. Furthermore, the doing good perspective is difficult to reconcile with the idea of a social contract and legitimacy theory because breaches of a social contract cannot be determined unless the terms of the contracts are fairly clear, and loss of legitimacy cannot be argued unless a company is at fault. The idea of companies deciding for themselves what they find to be sustainable business is, in our view, logically inconsistent with legitimacy theory, which emphasizes that companies earn their right to do business by complying with CSR (Deegan, 2002). In the following, we consider these conceptual differences, but we follow the practice of Christensen et al. (2022, 2021) and use CSR and CSP interchangeably, both referring to the extent that companies comply with societally defined environmental and social responsibilities. We treat ESG ratings as a practice-focused operationalization of various meanings of CSR and CSP, often vaguely defined and having come to include the governance aspect of a business. ESG ratings are provided by commercial firms that sell both underlying data and assessments to banks, insurance companies and funds. Before turning to the research that specifically addresses the validity of ESG ratings we briefly discuss findings about the relationship between companies’ CSR activities and their financial performance because it is considered evidence of the validity of CSR metrics.

In the traditional view, managers should only engage in activities that increase shareholder value (Friedman, 1970), but CSR can present companies and their owners with financial incentives. In contrast, it is conceivable that CSR may be carried out in the interest of shareholders even if they have a negative net present value. Activities could be in the interest of shareholders if they have preferences for CSR (Fama and French, 2007; Friedman and Heinle, 2016; Hart and Zingales, 2017), and the company’s management would then maximize shareholder welfare but not shareholder financial value with their CSR investments (Hart and Zingales, 2017). Other non-financial motives to engage in CSR activities are that it satisfies the preferences of broader stakeholder groups than the investors or because managers pursue personal goals (Adams and McNicholas, 2007) at the expense of the owners (Masulis and Reza, 2015), providing managers with a “warm glove effect” (Barnea and Rubin, 2010). In doing so, firms may sacrifice profits (Roberts, 1992; Benabou and Tirole, 2010).

Studies find significant associations for various CSR activities or policy measures regarding the relationship between firm value and CSR. However, the sign of the association differs between results. There is no consensus on whether there is a positive or negative relationship between CSR and financial performance or the firm’s value (Christensen et al., 2021). Because much CSR is measured as voluntary activities, there is a selection problem involved – there is often a positive association between voluntary CSR activities and firm value, but when companies are forced to implement CSR, the valuation effect can be negative (Manchiraju and Rajgopal, 2017). The literature also identifies several mediating factors that alter the sign and strength of the relationship between CSR and firm value. The value of companies is affected if CSR is related to positive media coverage (Cahan et al., 2015), to good products that cause satisfied customers and innovation (Luo and Bhattacharya, 2006), to customer awareness of CSR (Servaes and Tamayo, 2013) and to investors’ affective reactions to CSR performance (Elliott et al., 2014). The presence and absence of such mediators can cause conflicting results.

The conflicting results regarding the relationship between ESG and firm value are reflected in research findings about the results of investing in high/low ESG portfolios and individual high/low ESG stocks. A good example of the many contingencies that appear to interfere with the prospects of drawing straightforward conclusions is the findings of Zhang et al. (2022), who examined ESG using Bloomberg’s ratings of Chinese companies. For portfolios, they found a non-linear relationship between ESG and portfolio returns, with the highest and lowest ESG-ranking portfolios earning significantly positive abnormal returns compared with the other portfolios, and for individual stocks, they found pillar differences: good governance plays a distinguishingly positive role in increasing stock returns, while responsible social behavior has a negative impact and environmental performance has an ambiguous effect. They concluded that “investors cannot obtain excess returns by simply holding high-ESG profile stocks” (Zhang et al., 2022). They also found that ESG valuation is sector-sensitive. This study overall reflects the complexity involved in the CSR–CFP relationship and the selection issue discussed by Christensen et al. (2021), which means that the weak CSR–CFP or return on investment relationships are sensitive to sample composition because so many aspects of a business potentially impact returns that they cannot be controlled for.

The effects of CSR should appear in the companies’ accounts, but as with firm value, there are many inconsistent conclusions about whether it pays off for a company to invest in CSR. Furthermore, the evidence is normally not causal and is based on voluntary firm choices. Firms with strong performance may be prone to invest in CSR, and the relationship could therefore go from financial performance to CSR (Margolis et al., 2009). Combining studies in a meta-analysis does not address this underlying selection problem in CSR activities. A positive relationship between CSR and profitability has been found by Simpson and Kohers (2002), Flammer (2015), Cornett et al. (2016). A literature review by Kitzmueller and Shimshack (2012) finds weak support for a positive effect of CSR on firm profitability. Likewise, Margolis et al. (2009) perform a meta-analysis of 251 studies and find a small magnitude positive correlation between CSR and financial performance, but other meta-studies find a more robust positive relation between CSR and financial performance (Orlitzky et al., 2003; Busch and Friede, 2018; Atz et al., 2023). The results may be less reliable than they seem, however, because they aggregate results across studies of different aspects of CSR. Moreover, several studies suggest that the relation between CSR and financial performance is mediated by firm-level innovation and industry-level differentiation (Hull and Rothenberg, 2008), a firm’s intangible resources (Surroca et al., 2010), reputation and competitive advantage (Saeidi et al., 2015; Busch and Friede, 2018) and how a firm implements its CSR strategy (Tang et al., 2012). Finally, the relationship could even be U-shaped (Brammer and Millington, 2008).

With the many contingencies that can impact the relationship between CSR activities and companies’ financial performance, and thus the returns of investing in the companies, it is reasonable to have doubts about causes and effects. The literature appears to conclude that there might be a positive financial outcome from CSR activities, but that the evidence is inconsistent, either because of the complex nature of the relationship or because of the difficulties with measuring the extent to which companies engage in CSR. Some recent findings indicate that markets consider CSR-disclosures relevant as they value companies; for example, Grewal et al. (2020) found that sustainability reporting according to the materiality standards developed by the Sustainability Accounting Standards Board resulted in greater price informativeness for some companies. The financial effects of CSR activities on the reporting companies are supposed to be measurable through markets’ reactions to CSR disclosures. Such reactions would indicate financial materiality in the sense provided by referring to a US Supreme Court decision [1] that defined information as financially material if its disclosure would impact a reasonable investor’s decisions. The recent finding of Christensen et al. (2022) that ESG ratings, which focus on capturing ESG primarily in the financial materiality sense, differ as much as 30% in the assessment of an average CSR activity company supports the interpretation that the inconsistency among the cited findings can be because of differences in CSR metrics. In Section 2.2, we discuss the weaknesses of the commonly used ESG ratings as CSR metrics, and in Section 2.3, we pose our research question and outline how we address it by proposing a rating methodology that has features that should mitigate the problems we identify with traditional ESG ratings.

2.2 Problems with traditional environment, social and governance ratings

The inconsistent evidence regarding the CSR–CFP relationship suggests that ESG ratings may have validity problems. The standard method with which to assess ESG in research and practice is to adopt an off-the-shelf social-component ESG rating or to reconstruct such a rating from a selection of indicators (Chen and Delmas, 2011; Drempetic et al., 2020; Nitsche and Schröder, 2018; Oikonomou et al., 2018). Traditional ESG ratings compute the arithmetic average of ESG indicators (Chen and Delmas, 2011). The use of such simplistic models to estimate the arguably complex ESG concept has several constraints that contribute to the absence of convergent validity of such operationalizations found in previous research (Berg et al., 2022; Chatterji et al., 2016; Christensen et al., 2022). ESG ratings with zero or low validity are a great problem for the finance industry, researchers and corporate managers because the ratings are the most popular navigation instrument for assessing companies’ CSR, or ESG, among practitioners. We discuss the weaknesses of the traditional ESG rating methodology that, in our view, are the most likely to cause the low validity of traditional ESG ratings.

The first problem is the predominant use of discretionary indicator weights. The rating industry has no answer to questions such as: How should we weigh the indicator “Total Injury Rate Employees” against, for example, the indicator “Gender Pay Gap Percentage”? Neither theory nor empirical findings offer an objective answer to this type of question (Chatterji and Levine, 2006; Hillman and Keim, 2001), but weighing the relative importance of these two indicators against each other is necessary for an aggregated, holistic ESG rating – and entirely determines how companies are rated. For example, the rating industry does not know whether 20 workplace accidents for a particular corporation should be viewed as good or bad (Kotsantonis and Serafeim, 2019). Traditional ESG ratings therefore aggregate indicators discretionarily.

This core problem may be seen as though raters assume the meaning of ESG to be inherent in the indicators. Determining the weights of indicators, however, affects the measure’s ability to be a valid representation of ESG as a concept, be it CSR or CSP. What is actually measured with ESG ratings can be questioned because the inconsistency between leading raters is so great that they disagree about which should be the three most significant ESG categories (Berg et al., 2022). The inconsistency between raters’ assessments of the same company may be partly explained by the voluntary disclosure of ESG data by reporting companies (Orlitzky, 2013) and inconsistent collection of ESG feature data by raters (Berg et al., 2022), but Christensen et al. (2022) find that data availability and quality do not drive rater inconsistency as much as does the use of inconsistent aggregation methodologies. As data availability increases from the 25 percentiles of companies reporting the least ESG data to the top 25% of ESG reporting companies, the difference between ratings of the same company increases on average by more than 30%. Their finding is clear evidence that the main constraint of the current ESG rating methodology is the weighting schemes and associated (manual) procedures with which ESG ratings are computed from raw data. The use of arbitrary weighting schemes per definition produces arbitrary ratings, and it is not surprising that those different agencies produce inconsistent results compared with other raters.

Arbitrary, discretionary weighting postulates that the relative importance of features to the holistic construct is known by the rater, which is obviously not true (Callan and Thomas, 2009). For example, ESG ratings may overestimate the meaning of programs and policies for reducing a business’ impact on society relative to substantial outcomes (Delmas and Burbano, 2011). Discretionary weighting may dilute and distort the impact of the most substantial outcome indicators because it does not rationalize the actual weight differences. Dilution of the impact of the arguably most important indicators would be a hazard to construct validity because there is no guarantee that neglect of substantial outcomes would be consistent with society’s conception of acceptable ESG business practice. It follows from logic that subjective weighting schemes can explain the lack of validity of leading raters’ ESG ratings found in previous research.

In addition, the use of linear models with manually determined weights creates issues when it comes to representing nonlinearity in the underlying construct. It is likely that ESG indicators have nonlinear relationships with the holistic ESG construct because a policy to increase workplace safety must be more important the more injuries occur in a corporation. This feature interaction is inevitable for a broad range of ESG features. An arithmetic average of indicators-type rating models does not capture nonlinearity (Ding et al., 2020). There is potential nonlinearity in the relationships between individual ESG features and total ESG that is most likely caused by interaction between ESG features. For example, the importance of “Employee Health and Safety Training Hours” depends on the number of employee accidents and the injury rate in the company. The significance is near zero if the company has no issues with accidents or injuries. Furthermore, if CSR or ESG is defined as compliance with the responsibilities defined by society, the construct should have the form of a step function (sharp non-linearity) because companies are either compliant or non-compliant on each topic. Traditional rating methodology does not represent this high-degree of non-linearity.

In conclusion, the literature identifies serious deficiencies in traditional ESG ratings. In relation to this research, Barnett and Solomon (2006) found that the relationship between ESG and investment returns depends on the number of ESG screens the investor uses for screening. Their finding indicates that the screens may not measure ESG or measure it with substantial variation and error. Given these findings, responsible investors are appropriately suspicious of the information the ratings contain. Wong and Petroy (2020) find in their annual survey of major institutional investors’ perception and use of ESG ratings that asset managers are more and more dissatisfied with ESG ratings the more they analyze the ratings’ inner logic. Furthermore, the discretionary weighting scheme for ESG ratings appears not to be suitable for institutional investors’ ESG information preferences (Nofsinger et al., 2019). Institutional investors are indifferent to whether companies have ESG features that are not responsibilities, i.e. features not required of companies to have, but institutional investors avoid investing in companies with ESG features that suggest non-compliance with CSR in the responsibilities sense. Traditional ESG ratings do not satisfy such information needs because there is no evidence that these ratings would measure whether companies comply with environmental, social or governance responsibilities. Nofsinger et al. (2019) found that this asymmetric information preference is driven by the fact that, on the one hand, the potential benefits of companies’ CSR performance on topics not required of companies are offset by the costs of the CSR activities, while, on the other hand, the financial damages caused by companies’ non-compliance with ESG responsibilities are higher than the cost savings on non-compliance. Controversies caused by non-compliance, e.g. customer boycotts and strikes, trigger unsurmountable costs of non-compliance, and institutional investors therefore pay close attention to this aspect of CSR. There is no indication in the literature that traditional ESG ratings would satisfy the core needs of long-term institutional investors. On the contrary, Utz (2019) found that the Refinitiv ESG ratings do not predict relevant controversies and thus cannot help institutional investors avoid investment in non-compliant companies.

2.3 Social controversies as proxies for holistic social performance

The main problem with the traditional approach to ESG ratings appears to be that the relative importance of the many features that need to be addressed by the ratings cannot be determined without a considerable amount of subjective discretion. This is a problem that, for obvious reasons, calls for urgent exploration of potential solutions. We observe that the leading ESG raters, e.g. MSCI and Sustainalytics, predominantly focus on the financial materiality approach to ESG assessment, thus emphasizing the effects of sustainability issues on the reporting company and its owners rather than the effects of sustainability issues on society (cf. Grewal et al., 2020; Pizzi et al., 2022). This should not be seen as an expression of homogeneity, although some findings suggest that raters are affected by a global homogeneity trend (Saadaoui and Soobaroyen, 2018) and ESG practices can be affected by political changes (Aboud and Diab, 2019), because there is evidence of substantial differences between raters because of theorizing idiosyncrasies (Chatterji et al., 2016). Our research question addresses one potential solution by exploring whether social controversies are predicted by SP indicators, i.e. whether companies’ SP can be characterized by patterns of SP indicators typical of non-compliance with social responsibilities? We emphasize the relevance of using the societal critique of companies’ SP because social responsibilities are norms placed on companies by society, not by any specific stakeholder groups such as investors or individuals. The issues addressed by such responsibilities are of general concern and relevance, e.g. justice at work, basic human rights, gender and race discrimination etc. We agree with the approaches of certain sustainability information providers, e.g. Ravenpack (www.ravenpack.com) and RepRisk (www.reprisk.com), and the use of controversies in the Refinitiv ESG controversy score (www.refinitiv.com), in that controversies are a relevant label of social responsibility non-compliance. These three use controversies as a characterization of companies based on the actual track records of controversies in the manner of a wrongdoing index (Fiaschi et al., 2020). Wrongdoing indices are descriptive in the sense that they describe the amount and character of the critique each company has been given, but a wrongdoing index says nothing of a company that has received little or no media scrutiny. It is therefore not a rating of companies but rather a record of past misdemeanors. Our approach is a fundamentally different method of using the information contained in controversies. We learn to predict the likelihood that any company complies with social responsibilities based on their SP indicator pattern, regardless of the past record of controversies for a specific company. We use machine learning (ML) to build a model that characterizes companies based on how their SP indicator patterns are typically the target of media scrutiny. Our ML models learn to predict the likelihood of social controversies by looking only at the SP indicators, and the controversies are only used in our models to learn the typical SP features for a company that is criticized for social responsibility non-compliance. Thus, we use controversies not simply to label the companies on their historical track records, but to learn a model that discerns which SP feature patterns are likely predictors of such track records, whether or not a specific company has had controversies. The relevance of characterizing companies’ compliance with or non-compliance with social responsibilities from patterns of SP indicators is that the model that learns to recognize companies that have a high risk of non-compliance can function as a rating model with ratings anchored on the likelihood of compliance with social responsibilities.

The main advantage of a model that estimates the relative importance of SP features in the process of estimating the likelihood of companies’ compliance with social responsibilities is that this type of model relies less on the rater’s subjective discretion than traditional ESG ratings. The predictive methodology would anchor its assessments of the relative impact of ESG indicators on the estimation of compliance likelihood and would reflect society’s preferences for acceptable ESG practice. Because ESG includes a diverse and disparate set of topics (Oikonomou et al., 2018), we simplify the task by focusing on what raters and institutional investors refer to as the social category of ESG. A reason for our choice to focus on the social aspect of CSR and exclude environmental and governance issues is that the social aspect is much more frequently assessed by the media than the two other aspects. The number of controversies available from e.g. the Refinitiv database is more than 10 times as large for the social controversies compared to environmental and governance controversies. The societal feedback mechanism provided by social controversies is therefore a much richer source of information than the other types of controversies.

In addition to its consistency with the information needs of institutional investors (Nofsinger et al., 2019), this approach is also supported by the findings of Nirino et al. (2021) that corporations involved in controversies have lower returns on assets than other corporations. Moreover, Aouadi and Marsat (2018) found an effect of controversies on financial performance for high-attention corporations. In addition, corporations receiving recent “bad press” tend to suffer from high volatility in their stock price and market value, often experiencing steep stock price declines (Cui and Docherty, 2020; Muller and Kräussl, 2011). Such effects may, however, depend on the extent of corporate ESG disclosures prior to the negative exposure (Beelitz et al., 2021). However, these findings suggest that compliance is of interest to investors and that an ESG rating methodology anchored on compliance would provide ESG information consistent with institutional investors’ preferences. Our focus is therefore on the compliance aspect of CSR, while we ignore the voluntary “doing good” perspective. This distinction has also been discussed as the difference between CSP as “doing good” (Kothari et al., 2009; Mackey et al., 2007) versus social irresponsibility (Arora and Dharwadkar, 2011), with the “doing good” side receiving almost all the attention.

Based on these observations reported in previous literature, we investigate the possibility of developing a rating methodology based on a measure of comprehensive compliance with social responsibilities. Compliance can be measured indirectly by the absence of non-compliance, which is represented by the situation in which a company has not been criticized for non-compliance with important social responsibilities. Social controversies are indicators of non-compliance with social topic responsibilities because controversies occur when companies are perceived to be breaching such responsibilities (Nieri and Giuliani, 2018). We emphasize the distinction between describing social features (i.e. behavior and structures), which is the basis of traditional ESG ratings, and using social feature indicators for assessing a company’s compliance with social responsibilities, which is the goal of our investigated rating methodology. In our view, which contrasts with the view of traditional ESG rating methodology, a company that performs within the boundaries of its responsibilities should not be punished for being at fault because the company is in fact not at fault. The traditional ESG ratings, which do not use the responsibilities laid out by society for companies to follow as their performance standard and reference point for ratings, may conclude that a company that is acting consistent with societally established preferences is given a low rating because of the subjective preference methodology used in traditional ratings. All ratings that do not aspire to reflect societal standards are, in our view, inconsistent with the idea of CSR or ESG as a societal-level navigation system to guide the practice of socially responsible investment and the efforts of all business managers in their efforts to improve the SP of their businesses. We therefore view social controversies as signals that a company’s pattern of SP indicators is considered socially illegitimate from either a moral, soft law (international standards) or legal point of view.

The importance of investigating a rating methodology that can mirror societal responsibilities motivates our examination of the extent to which social controversies can be predicted with a company’s social feature indicators. From the compliance perspective, controversies provide information different from that provided by SP indicators. Social controversies are society’s reactions to social category behavior in the sense that behaviors are labeled inappropriate. Controversies thus label a particular company, which would, without the controversy, be represented only by a pattern of SP indicator values. Social controversies contribute information about the performance standards set by society that companies need to live up to (the social responsibilities). The controversies indicate that there may be problems with the transgressor’s structures and processes because, as signaled by controversy, those have not been sufficient to prevent the critique (Nieri and Giuliani, 2018).

Whereas traditional ESG ratings are anchored on the subjective preferences of the rater, the legitimacy of which must be questionable in the eyes of institutional investors, the non-compliance accusation communicated by controversy is put forth by a democratic institution, i.e. the media, but the critique is possible only if it argues that the critiqued company has violated an important responsibility toward society or significant stakeholders. Defining SP as compliance with social responsibilities therefore makes the development of rater-indifferent SP ratings possible. The role of social controversies in the development of such a rating methodology is to label companies, or rather their individual patterns of SP indicators, as compliant or non-compliant, referring to each company’s historical record of controversies. Social controversies concerning, for example, product harm (Cai et al., 2012; Klein and Dawar, 2004), human rights violations and breaches of labor law (Aouadi and Marsat, 2018) evoke detailed criticism of instances when corporations are non-compliant with their social responsibilities. In fact, controversies operate as a labeling process (Faulkner, 2011), through which non-compliant corporations risk legitimacy loss (Deegan, 2002).

Controversies obviously differ in origin from the self-reported SP indicators that originate from annual reports. This difference in the origin of the data makes the controversies “out-of-sample” data in relation to the SP indicators. This extra-indicator information assigns compliance relevance, or weights, to individual SP indicators and to indicator patterns based on whether the indicators and patterns are typical of companies exhibiting non-compliance. Our use of controversies to label patterns of SP indicators for each company enables assessment of the relative importance of individual indicators to a holistic SP construct because the relevance of each indicator can be defined as its contribution to the likelihood that a corporation is (non-) compliant with its social topic responsibilities. According to our argument, this is the most compelling reason for investigating social topic controversies. Controversies provide the constitutive information for the definition and measurement of a compliance-oriented and rater-indifferent SP rating.

Social controversies are, in addition to being the decisive component in a rating methodology anchored on societal social responsibilities, events of interest in themselves, and the critique they express is of such a nature and significance that institutional investors typically strive to avoid investing in companies that attract controversy in their effort to protect their reputation as responsible investors (Krueger et al., 2020; Nofsinger et al., 2019) because investment in controversial corporations may cause the withdrawal of funds (Grappi et al., 2013).

Controversies also reflect a lack of policies, structures and routines necessary for compliance with society’s expectations (Nieri and Giuliani, 2018). Controversies signal systematic breaches of moral or legal norms in corporations striving to earn profits and gain market share (Fiaschi et al., 2017, 2020; Giuliani et al., 2015; Surroca et al., 2013). A controversy indicates that the misbehaving corporation lacks governance structures or processes that would, if present and operational, have prevented the corporation from engaging in reprehensible behavior (Nieri and Giuliani, 2018), for example, inappropriate management-team or supply-chain procedures (Chiu and Sharfman, 2018). For SP, there is an abundance of such potential structures and processes that may, if failing, cause social controversies relating to anti-competitive behavior, business ethics in general, patents and intellectual property, a lack of respect for human rights, tax fraud, child labor, inappropriate executive management salary schemes, etc. If social controversies can be predicted by using the abundantly available SP indicators, this would suggest that the inappropriate behaviors that have become normal in misbehaving corporations (Earle et al., 2010) are present in the information provided by these corporations’ accounting disclosures. Our research may therefore, in addition to examining a precondition for controversy-anchored SP ratings, be understood as investigating the possibility of detecting such weaknesses in the at-fault corporations’ accounting.

3.1 Research design and measures

We adopted a cross-sectional design with predictive modeling performed through ML experiments. To develop an SP rating, it is appropriate to sacrifice the theoretical accuracy of explanatory modeling for the considerably higher empirical precision associated with predictive modeling (Collopy et al., 1994). A rating that cannot distinguish between corporations with high precision is of little use to an investor, as discussed by Shmueli (2010) and Bzdok et al. (2018). The differences between methodologies indicate that predictive modeling is better than explanatory modeling for developing a diagnostic method, such as an SP rating.

We used eight ML algorithms that learn to predict the social controversies of corporations by examining their SP indicators over a 10-year window. The five performance measures used in evaluating the predictive performance were precision, recall, F-measure, area under the receiver operating characteristic (ROC) curve (hereafter, AUC) and precision recall curve (PRC).

Data were obtained from Refinitiv Eikon, which compiles more than 400 ESG indicators, of which 129 are SP indicators enumerated in Appendix 1, and a summary description of the inputs and outputs of the ML models is presented in Figure 1 (regarding the algorithms in the figure, see Sections 3.2 and 3.3). SP indicators can be divided into substantial outcomes, for example, salary gap and employee satisfaction and policies to improve them (Delmas and Blass, 2010). We are not primarily interested in understanding the individual features that drive SP in general but in predicting SP using the collective information contained in the large number of SP indicators. Therefore, we use the full set of SP indicators provided by Refinitiv Eikon, together with their definitions of the indicators. The relative importance of the indicators is determined by the data, not the researcher. In addition to the SP indicators, we included seven corporate descriptors that do not target SP: total assets, net assets, market capital, return on assets, return on equity, the Global Industry Classification Standard sector and country of headquarters.

The data set covers 2,517 corporations over the 2009–2018 period, during which about 80% of the corporations experienced a social controversy. Table 1 describes the sample, which included all available corporations with an overall ESG rating from Refinitiv Eikon for all studied years. We chose this sample because we wanted to make sure that we selected corporations for which SP indicators were richly represented. Even in this ESG reporting-oriented sample, there were many corporations for which data on certain indicators are lacking. Regarding the country of headquarters, our sample includes corporations based in 61 countries. Most corporations were based in the USA (707); four other countries, i.e. Japan, the UK, Canada and Australia, had more than 100 corporations in the sample.

The corporations were classified into disjoint categories, one with high SP and one with low SP. Social controversies were frequent, with the average number of controversies per corporation during the 10-year window being 5.17. We defined the disjoint classes as corporations having fewer (high SP) and more (low SP) than the average number of controversies. We called the corporations “negative cases” in the former class and “positive cases” in the latter. The number of positive cases, each having six or more social controversies, was 561.

The goal was not to model the risk of future controversy based on a corporation’s past SP indicators but to assess the likelihood that an indicator pattern is associated with a corporation’s risk of having a controversy in a cross-sectional sense. The longitudinal aspect of the data was captured by using a simple method in which the indicators are averaged if numerical or encoded with dummy variables if binary.

3.2 Machine learning algorithms

The models for assessing SP were developed in ML experiments in which one algorithm at a time extracted information from SP indicators by associating patterns of indicators with social controversies. The eight ML algorithms have different functions and therefore capture different aspects of this learning task. The idea is not to optimize the prediction or find the best method, but to demonstrate whether prediction is possible with a library of algorithms. The broad set of algorithms illustrates how various aspects of the data affect predictability; for example, the impact of nonlinearity can be understood by looking at the difference between a linear and a nonlinear algorithm. We summarize the algorithms in alphabetical order below. All experiments are implemented in Python using scikit-learn and xgboost for fitting estimators and using Pandas for data management and manipulation.

Gradient boosting (GB) is an ML technique for regression and classification problems that produces a model comprising a prediction-model ensemble typically organized as decision trees. The model was built stage-wise to minimize a loss function. Multiple weak models were combined to form a strong ensemble model by reweighting the training data of the SP indicators to focus the learning on those corporations that the algorithm cannot predict correctly. GB defined a loss function and used the gradient of the loss function to reweight the firms to focus on misclassifications using the logistic loss. It has been shown to be a strong classifier for a wide range of tasks (Babajide Mustapha and Saeed, 2016; Sigrist and Hirnschall, 2019).

Linear support vector machine (linear SVM) is a multidimensional linear method that separates two classes of data by drawing an (n – 1)-dimensional plane, where n is the number of indicators in the feature space, and separating instances of data lying on one side of the plane from instances on the other side. Introduced by Vapnik (1995), this method is said to be an effective approach to pattern recognition and prediction. For example, prediction of stock price movements and bankruptcy can be addressed if the data are linearly separable (Xu et al., 2009).

Logistic regression (LR) estimates the coefficients of a regression equation and classifies data according to the output of this equation (Menard, 2011). With LR, the independent variables in the present case were the SP indicators, and the dependent variable was social controversies. Each feature value was multiplied by a weight and then summed. LR related controversies to the indicators with the goal of finding the best-fit set of a linear combination of indicators to distinguish between positive and negative cases. Each feature was multiplied by a weight, and then all were summed. The result was transformed by a sigmoid function, producing the binary output, and the supervised learning generated the coefficients predicting a logit transformation of the probability.

Given the class, naïve Bayes (NB) classifiers build on the assumption that all feature values are independent. These methods are probabilistic classifiers developed from Bayes’ theorem. Here, the NB classifier computed the conditional probability of a controversy from the pattern of SP indicators. Under the naïve assumption that these indicators are independent, the classifier was the conditional probability of a controversy multiplied by the product of the conditional probability of each SP indicator given a controversy (Gulo et al., 2015).

Neural networks (NN) are multiple layers of functions, called artificial neurons, with an output layer consisting of a logistic, softmax or linear regression model. When the layers relate to one another via an activation function, NNs can reproduce any linear or nonlinear function. With their artificial neurons, NNs mimic the functioning of the human brain’s interconnected neurons. The links between neurons have numeric weights that form the long-term memory of the NN. NNs have many good features, such as generalization capability, the ability to learn highly nonlinear relationships and not requiring any assumptions about data distribution. They are, however, of limited use in applications requiring interpretation because their operation is opaque. NNs have better forecasting abilities than statistical regression models and work well on fuzzy or complex data (Ghritlahre and Prasad, 2018). They are nonlinear and nonparametric models that capture unknown interactions (DeTienne et al., 2003; Safa and Samarasinghe, 2011; Sözen, 2009).

Radial basis function support vector machine (RBF SVM) is a version of linear SVM that uses a radial basis function to potentially improve data classification when the data are not linearly separable with an (n – 1)-dimensional hyperplane (Ring and Eskofier, 2016). The RBF SVM is called a kernel, which is a window for mapping the nonlinearity in the original space to a higher order space in which the data are linearly separable.

Random forest (RF) has become a popular ML algorithm and is considered state of the art in many applications. Random samples of training data are used to create decision trees, which are then combined to form an ensemble model (Breiman, 2001). At each node, a limited number of SP indicators are sampled to construct each tree separately, which increases variability. A majority vote is taken among the trees, here representing the social controversy prediction. RF works well with outliers and noise in the training set (Yeh et al., 2014), which can be expected with SP data because of “greenwashing” (i.e. information disseminated to create a false appearance of environmental friendliness) and the absence of binding standards for CSR disclosure, both of which are prone to causing data errors. Overfitting is avoided, and precise forecasts are obtained by using the majority vote among trees (Breiman, 2001).

Quadratic discriminant analysis (QDA) is a method that can estimate nonlinear dependencies between complex indicator patterns, such as those in the SP indicators and social controversies. Unlike linear discriminant analysis, it can capture such dependencies by not assuming that the covariance of each class of data is identical (Anagnostopoulos et al., 2012; Ou and Wang, 2009; Yuan et al., 2017). QDA generates a model developed from conditional data densities by constructing a quadratic decision boundary.

3.3 Experiments

Hyper-parameters for the algorithms are presented in Table 2 and in full in Appendix 2. These parameters ensure that our study can be reproduced and provide additional background to its interpretation because variations in settings affect the performance of algorithms. For simplicity, we used the default settings for scikit-learn, version 0.22.

A partition method that achieves reliable estimations of the generalization performance of the ML algorithms and economizes on scarce data is k-fold cross-validation. The cross-validation partitions the data set into k disjoint folds and allows training to be conducted iteratively on k – 1 fold, with one fold spared for testing. Repeating this procedure in a circular manner uses the entire data set for both training and testing, ensuring that the predictive performance is evaluated from k different viewpoints. The result of this procedure is k performance measures, the mean of which represents a more reliable estimate of the generalization performance than if the predictive performance were evaluated using one simple partition. In this study, we used stratified 10-fold cross-validation to ensure an equal number of positive and negative cases in each test set.

The performance of our ML models was estimated using several measures, each measuring different aspects of learning ability (Alpaydin, 2010). The measures are summarized in Table 3.

Precision measures the sensitivity of the classifier, i.e. its accuracy in predicting the controversy, and recall is considered the ability to find as large a fraction as possible of the controversy-affected corporations in the whole data set. Precision [equation (4.1)] and recall [equation (4.2)] are conflicting measures, while the F-measure [equation (4.3)] captures the trade-off between the two measures.

The AUC measure is the area under the curve defined as a plot of True_positive [equation (4.4)] versus False_positive [equation (4.5)]. It estimates the probability of a classifier ranking a true-positive case ahead of a false-positive case and is therefore a measure of the model’s ranking performance. Similarly, the PRC estimates the mean precision for multiple thresholds of recall and is used to measure the trade-off between precision and recall. It is defined as the area under the plot of precision versus recall. The main benefit of both AUC and PRC is that they are insensitive to the class distribution of the training and testing data, as opposed to the accuracy.

4.1 Predictive performance of different learning approaches

We first investigate whether the social component of the ESG rating from Refinitiv Eikon would provide appropriate data for classifying corporations with and without controversy. Figure 2 displays a histogram where the grey distribution shows corporations with controversy and the black distribution shows corporations without controversy. Visual inspection reveals that the distributions are similar, with no social-component ESG rating being a suitable point on the x-axis for defining the two classes. The social-component ESG rating is particularly insensitive to the likelihood of controversies in the 50–70 range, where most of the corporations are found. In this range, the two distributions do not provide any support for a classification. This evidence is consistent with the findings of Utz (2019), who found that Refinitiv ESG ratings do not predict controversies at all.

Analyzing the predictive models, we first discuss the ability of the ML algorithms to learn the SP estimate. Table 4 depicts the performance of the ML algorithms. The static measures precision, recall and F-measure primarily reveal the predictive performance of each algorithm at a set certainty threshold in predicting a positive case, whereas the dynamic measures AUC and PRC address performance trade-offs. While balanced performance is preferable, an emphasis on precision is inevitable because high precision is a key to portfolio composition. The precision column reveals that RF is the most precise (84.2%), followed by GB (79.4%), NN (78.4%) and QDA (77.5%). The bottom four clearly lag in performance.

RF’s high precision can be linked to its learning capabilities because it produces models that have uncalibrated probability estimates, which require the model to have high confidence in predictions. Table 4 also shows that GB has the highest levels in three of the five categories (i.e. F-measure, AUC and PRC), but the difference from RF in these three scores is small. It can also be seen that the top performers are not the same in all measures, suggesting that the choice of algorithm depends on which performance measure is the most important. Logistic regression has high recall and good AUC and PRC, suggesting that a linear model performs well in these aspects of the prediction despite the arguments that the underlying construct is complex and nonlinear.

In evaluating the ranking and predictive performance of the algorithms, we also compute the AUC and PRC in Figures 3 and 4. The AUC measures discrimination, equivalent to the probability of a randomly chosen positive instance being ranked higher than a randomly chosen negative instance, i.e. it is equivalent to the two-sample Wilcoxon rank-sum statistic. The AUC should be as close to the top left-hand corner as possible, and a curve below the dashed line is no better than a random guess. The plot in Figure 3 is consistent with Table 4, i.e. RF, GB, NN and logistic regression perform more strongly than the other algorithms. This is partly contrary to our expectation that logistic regression would be less suitable as a predictor of controversies because of the nonlinearity of the underlying construct.

The PRC is a dynamic measure of the predictive performance of the algorithms, and Figure 4 shows how much precision needs to be sacrificed to obtain a certain level of recall. Ideally, a curve should reside in the top-right corner, as far as possible from the bottom-left corner. As long as the algorithms need to predict with absolute certainty which corporations have controversies, they maintain high precision; however, because they are required to find a larger proportion of the total number of corporations with controversies, the algorithms are, as the graphs show, forced to sacrifice certainty to identify more corporations with controversies. Visual inspection confirms the calculations in Table 4. The black (i.e. prediction of non-controversy) and grey (i.e. prediction of controversy) curves are situated in the top right-hand corner for the same group of algorithms that performed well in terms of AUC. RF, GB, NNs and logistic regression can better accomplish the trade-off between precision and recall than can the other algorithms. The grey area at the bottom of the graphs represents the class distribution. Several algorithms offer high precision of around 0.8 at a recall well above 0.5.

4.2 Controversy prediction as social performance

An SP rating anchored to social controversies should be negatively correlated with the number of controversies, but the ML ratings do not automatically assign low ratings to all corporations that have been involved in many controversies. This is because the rating is estimated solely from SP indicators once the rating model is learned. When the ratings are calculated, the predictors do not know which corporations have had controversies and how many. Despite enduring more controversies than average, some corporations deviate from having social-component behavioral indicator patterns indicative of compliance with social norms, i.e. they look good from the rating model’s point of view. Such corporations are awarded high ratings. Other corporations have had no or fewer than an average number of controversies, although they exhibit the typical pattern of traits of corporations that are controversy prone according to our best-performing models. They are therefore awarded low SP ratings. Nevertheless, on average, corporations involved in more controversies should have lower ratings than corporations involved in fewer than average or no social controversies.

Figure 5 reveals a negative correlation between all ML ratings and the number of controversies corporations have. In this graph, the x-axis shows the number of controversies per corporation and the y-axis shows the ratings. The methods with the highest predictive performance all have smooth distributions. As can be seen, the complex models tend to use what could be called more discretion than do the linear models in identifying exceptions to the many-controversies/low-rating pattern. This would not be the case if the models were overtrained.

As expected, there is a difference between logistic regression and, for example, RF in how the ratings can depart from strict adherence to assigning a low rating to all corporations with many controversies. There is a considerably wider distribution in the ratings for corporations in the range of 20–40 controversies for RF than for logistic regression. This suggests that RF may give weight to more complex SP indicator patterns than does logistic regression, allowing the former model to detect exceptions to the many-controversies/low-rating rule when determining the SP rating. The ability to balance between such exceptions and high performance in predicting controversies is a virtue of a good SP rating, because some corporations that have had many controversies may have traits and behaviors typical of high SP performers.