Automated ESG Report Analysis by Joint Entity and Relation Extraction

Abstract

The banking industry has lately been under pressure, notably from regulators and NGOs, to report various Environmental, Societal and Governance (ESG) metrics (e.g., the carbon footprint of loans). For years at Crédit Agricole, a specialized division examined ESG and Corporate Social Responsibility (CSR) reports to ensure, e.g., the bank’s commitment to de-fund coal activities, and companies with social or environmental issues. With both an intensification of the aforementioned exterior pressure, and of the number of companies making such reports publicly available, the tedious process of going through each report has become unsustainable.

In this work, we present two adaptations of previously published models for joint entity and relation extraction. We train them on a private dataset consisting in ESG and CSR reports annotated internally at Crédit Agricole. We show that we are able to effectively detect entities such as coal activities and environmental or social issues, as well as relations between these entities, thus enabling the financial industry to quickly grasp the creditworthiness of clients and prospects w.r.t. ESG criteria. The resulting model is provided at https://github.com/adimajo/renard_joint.

Supported by Groupe Crédit Agricole; analyses and opinions of the authors expressed in this work are their own. The authors wish to thank the ESG team at CACIB for the document annotations and their valuable comments.

Appendices

A Evolution of Test F1 for the IBM Model

The annotation of 372 paragraphs (see Sect. 1) was deemed sufficient as the F1 scores for NER and Joint NER & RE stopped improving using the IBM proprietary model, as can be seen in Fig. 7.

B Named Entity Recognition Representation

A popular output representation of NER is BIO (Begin, In, Out) embedding, where each word is marked as the beginning, inside, or outside of an entity (see e.g. [19, 22]); however, this representation does not allow overlapping entities. On the other hand, span-based methods [5], which classify spans of words, can extract the spans of these overlapping entities. Figure 8 gives examples of BIO and span-based entity representations.

Fig. 7.

Evolution of F1 scores w.r.t. the number of paragraphs.

Fig. 8.

Examples of BIO (above) and span-based (below) representations.

In the ClimLL dataset, even though the entities are presented in the span-based format in the dataset, there is no overlapping entity. Thus, it is also possible to convert to BIO format. Multiple relations can exist in the same sentence but relations cannot span across sentences. This facilitates splitting the paragraphs by sentence.

C Single versus Multiple Relation Extraction

Relation extraction algorithms are divided into two categories: Single Relation Extraction [1] (SRE) algorithms which expect only one relation per input sentence and multiple relation extraction [7, 21] (MRE) where multiple relations may exist in a single input sentence (Fig. 9).

Fig. 9.

Multiple relations example.

In this work, multiple relations are considered.

D SpERT

1.1 D.1 Address a Shortcoming in Evaluation

While re-implementing the model, we noticed that, in the evaluation process, SpERT considers an incorrectly predicted entity span or relation as two negative observations. An example is presented in Fig. 10, where the model returns a set of predicted entities with “SpaceX” incorrectly classified as a person. In this case, the original evaluation process would iterate through the union of the true entity and the predicted entity sets. If an entity (including its span and type) is only presented in one of the sets, then it is considered to be classified as non-entity in the other. With this approach, “SpaceX” is considered to be incorrectly classified twice.

Fig. 10.

Illustration of a better evaluation process for SpERT.

Thus, instead of iterating through the union of the true entity and predicted entity sets that include both entity spans and types, we only consider the union of the true entity spans with the predicted entity spans. Similarly for relations, we only take the union of the true and predicted spans of the source and target entity pair. As a result, we obtain a more accurate evaluation step.

1.2 D.2 Proposed Improvements

Furthermore, we also proposed two improvements to the prediction stages. Because SpERT classifies spans into entities, when dealing with datasets in BIO representation, it has to discard overlapping entities. In the original implementation, predicted entity spans are looped through in no specific order and any span that overlaps with previous spans is discarded. We suggest, instead, prioritizing discarding spans with low classification confidence.

Fig. 11.

Entity loss (left), relation loss (center) and F1-score on ClimLL w.r.t. training epochs (right).

Secondly, we noticed that the true pairs of entity types are not considered in the relation prediction stage of the original SpERT: For example, the model can only predict that an entity pair has a “live in” relation if the source entity is a person and the target entity is a location, irrespective of the probability given by the relation prediction stage. Thus, we modified the model so that it only predicts a relation if this relation fits the types of the source and target entities.

E Evolution of Loss Functions

The entity and relation losses as well as the F1 score on the validation set throughout the training process (30 epochs) of SpERT on ClimLL are displayed on Fig. 11. Both entity and relation losses reached their minimum after only a few epochs while the validation F1 score kept improving.