Which will be your firm's next technology? Comparison between machine learning and network-based algorithms

Here we compare the relatedness assessments of the co-occurrences based networks with the machine learning algorithms, showing how the latter are able to give better prediction results. The results are shown in figure 2.

Zoom In Zoom Out Reset image size

In order to compare the various prediction methods from various viewpoints, we adopted different metrics to quantify the goodness of a prediction (these metrics are discussed in detail in Methods section):

• Area under the PR curve: the area under the curve in the precision-recall plane. The latter is obtained by varying the threshold that identifies the value above which the scores are associated with positive predictions;
• Precision@100: the fraction of the largest 100 elements of the score matrix S²⁰¹¹ that are actually activated;
• mPrecision@10: for each firm, we consider the largest 10 scores and we compute the fraction of realized activations; then we average over the firms.

In figure 2 we report the scores of the previous metrics for different values of the threshold parameter T; the results are consistent even if one varies such threshold (or uses the RCA to binarize).

We start noticing that the random benchmark is surpassed by all the different approaches, showing that all are able to compute a measure of similarity that is able to grasp links between the technology codes.

Also the autocorrelation benchmark is outperformed but in the mPrecision@10 case. In particular, it performs better when T increases, because the number of zeros in both the training and the test matrices increases (that is, the number of potential activations, the green elements in figure 1, that are not realized).

In the area under the PR curve and Precision@100, the only network-based algorithm that manages to overcome the CTS is the Taxonomy. In particular, it is interesting to observe how this network exceeds the Technology Space. We can argue that for the technology codes, a network based on the taxonomy principle, i.e. how firms move from low-complexity to high-complexity technologies only after developing the necessary skills (Zaccaria et al (2014)) shows a better prediction performance that a proximity-based one, i.e. a network where two technologies have a high link if they need the same capabilities (Hidalgo et al 2007).

The micro-partial approach does not show a competitive performance despite being quite popular in both academic and corporate applications (Smith and Linden 2017).

In any case, the superiority of the RF_CV and of the RF with respect to both benchmarks and density-based approaches (networks and CTS) is evident. Although the other algorithms are able to give prediction scores able to overcome the benchmark models (especially for T = 0), clearly these are not able to fully highlight the non-linear relationships among the technological portfolios of firms and the technological sectors they will move to.

In figure 3 we compare the frequency distributions of the scores of both the realized and the not realized activations for the RF (y-axis) and Technology Space (x-axis). In order to make them comparable, both scores are rescaled using the respective maxima and minima. The red line is the bisector, shown for further reference. From the left figure, it emerges that the RF assigns, on average, higher scores to those potential activations which will be actually realized in two years. On the contrary, the possible but not realized activations show similar distributions; this is due to the much greater number of true negatives which is present in both approaches. Note that, as expected, the scores given to the not realized activations are lower than the realized ones.

Zoom In Zoom Out Reset image size

Finally, it is also important to point out that in the present study, true positives are more significant than true negatives. This has a twofold rationale:

• The high class imbalance implies that a performance measure such as accuracy is not adequate for the problem. The majority of the elements of our matrices are equal to zero and therefore an accuracy measure would consider only the overwhelming number of true negatives. Even if we made a prediction in which we assume that all the elements of the matrix will be zeros we would get an accuracy higher than 99%. More in detail, the percentage ratio between elements equal to 1 and 0 is about 0.2% for T = 0 and 0.08% for T = 1.
• For a firm, it is more interesting to know which technologies are close to it than those it is already active in (i.e. which technologies it can successfully activate in the future), rather than knowing which ones it will not do in the future (which is often trivial information, due to a totally different scope, for instance).

Even if the prediction performance of the RF algorithm vastly outperforms the other approaches, its practical feasibility in policymaking could be limited by its low interpretability. From a policy perspective, indeed, it is not easy to justify a strategic decision such as investing or not in a technological sector on the basis of a quasi-black box algorithm. In order to provide a visual tool to inform and justify strategic decisions, we introduced the continuous Projection Space (Tacchella et al 2021), that uses the scores obtained from the machine learning algorithms to build a two-dimensional and, as such, easy interpretable space to visualize and describe the temporal evolution of bipartite networks. The key idea is to interpret the scores matrix obtained with the RF as a matrix of coordinates of technology codes in a high-dimensional space. These embeddings are then made representable by applying suitable dimensionality reduction techniques; in this case, t-SNE (Van der Maaten and Hinton (2008)). Note that here we are using the term 'interpretable' in a policy perspective: in this sense, machine learning algorithms are a black box in the sense that, for a policymaker, the origin of our results is not clear a priori (for instance, a company being close to a technology code). However, by applying t-SNE to the RF scores one can obtain a visual representation of the relative position of the codes in this new space we define. Now the motivation behind our relatedness assessment, and the consequent forecast, become (hopefully) clear: the company is close to a given technology if it is patenting in the close technologies. This is visible as a diffusion process only in a low-dimensional representation. We point out that this is not an explanation of how the RF works, which is beyond the scope of the paper, it is just an a posteriori justification of our results that, being representable in a two-dimensional plane, can provide insights about companies' innovation strategy (e.g., exploration vs exploitation).

Here we apply this methodology—which is fully described in the Methods section—to the firm-technology network; the result is a plane in which each technology sector is a point, and similar sectors are close. In figure 4(a) the different colors correspond to IPC macro-categories, i.e. the first of the 4 digits that define the classification codes. We point out that, differently in network-based representations, here the similarities are simply represented by the spatial proximity between technology codes. The use of Euclidean distances instead of topological ones permits to use of a wider range of tools, for instance, clustering and anomaly detection algorithms. A visual inspection of the CTS permits to obtain a number of insights: in figure 4(a) one can observe that technology codes tend to cluster following the macro-categories; this is a first hint that the positions in the plane present a certain degree of significance. However, also the departures from the classification reveal meaningful relationships. In particular, on the upper left one can observe the presence of a dense area where it is possible to find veterinary medicine close to farm. In the motor vehicles area on the left, we find motor vehicle technology codes; in particular, there is a red color technology code (A47C) that corresponds to chairs and seats specially adapted for vehicles, black technology codes colors, corresponding to B60 (considering the first 3 digits), that represent vehicles, light blue technology codes corresponding to the first 3 digits F01 and F02, i.e. machines and engines, and combustion engineering, and one brown technology code color (G05G), physics of command systems. Weapons area is associated with weapons technologies: we find principally (considering the first 3 digits) codes B63 and B64, i.e. ships and aircrafts, C06 associated with explosive chemistry, and F41 and F42, i.e. weapons and ammunition.

Zoom In Zoom Out Reset image size

The example discussed above can be generalized by comparing two frequency distributions of distances (see figure 4(b)). The first distribution (cyan line) is relative to the distances between the newly activated codes and the patenting company (that is, the distance from the closest sector the company patented in); the second (orange line) is the density distribution of all distances between the points in the CTS. We can see how the first distribution has a lower mean, evidencing that, on average, firms tend to patent in codes that are relatively close to what they already do. Obviously, also high distances are present, indicating strategic choices which lead the firm far from its usual scope.

In order to show a concrete application of the CTS, we show in figure 5 the portion of this space relative to an American nanotechnology company, Nanotek Instruments Inc., as an example. In 2002 Nanotek patented three inventions, two based on batteries (https://patentimages.storage. googleapis.com/f4/d8/3d/d663e43fe48e2b/US6773842.pdf and https://patentimages.storage.googleapis.com/ 66/b3/7f/6fa873ae402fbf/US6864018.pdf) and the third is the nano-scaled graphene plates (https://patent images.storage.googleapis.com/e5/3d/0d/1c25e5f68a77ab/US7071258.pdf). The first two are associated with the code H01M, while the third with the codes C08K, C04B, C01B and C22C, which correspond to the gold points. The red points are the technology codes which Nanotek patented in 2000 and 2001, while, as mentioned, the gold ones are those activated in 2002. The black arrows underline the non-random position of the new technologies, that are close to the ones already present in the patenting activity of the company. This is because we find that technology codes that have a high similarity are represented close to each other, and therefore a sort of 'technological diffusion' is expected starting from the codes that firms already have in their portfolios (as shown in figure 4(b)).

Zoom In Zoom Out Reset image size

The bipartite firm-technology network is obtained by matching two databases: AMADEUS for firms and PATSTAT for the technology codes.

4.1.1. Firms

4.1.2. Technology codes

The starting database can be represented using the following structure: 12 matrices, one for each year from 2000 to 2011, that link 426983 firms f (rows) to 7456 (six-digits) technology codes t (columns). We chose to work at a higher aggregation level, and so to compress the technology codes from 6 to 4 digits, summing the columns corresponding to the 6-digit codes with the same first 4-digits. From the 6 to the 4 digit level the number of technologies goes from 7456 to 643. This operation leads to both better quantitative results and shorter computation times (from a qualitative point of view, instead, the results are unchanged).

A key element of both the machine learning and the network-based approaches is to provide an assessment of the similarity between technology codes; this information can be extracted from the co-occurrences of technological sectors in the same firms. So we consider only firms that, in the years from 2000 to 2009, make at least 2 technology codes; these firms are 197944.

This leads to the data mentioned in the main text: 12 V yearly matrices that link 197944 firms and 643 technology codes.

In order to compute the relatedness measures, in the economic complexity literature (Hidalgo et al 2007, Zaccaria et al 2014, Pugliese et al 2019a) one usually computes the revealed comparative advantage or RCA (Balassa 1965), and then these matrices are binarized using a threshold equal to 1. As far as exports are concerned this choice of threshold has a natural economic meaning, traceable to the works of Ricardo and Balassa himself: considering the bipartite country-product network, RCA_c,p ⩾ 1 means that country c is significantly competitive in the export of the product p (Hidalgo et al 2007). So the country's share of that product in its market is equal to or greater than the product's share on the world market. However, the economic meaning of patent submission is different, so the choice of RCA is not straightforward. In this work, we binarize the matrices V with different values of threshold T, without computing the RCA; in this way, the matrices V are better interpretable as how much a technology code t is present in the patenting activity of a firm f. We have in any case check the robustness of our results for different threshold values and the use of RCA.

In this and in the next sections we discuss how to obtain a prediction score matrix S for 2011 from each method starting from the same training data V and M, relative to the years 2000–2009. The score matrix gives the model's estimation of the likelihood that a firm will patent in the given technology sector, and the comparison between the scores and the actual M²⁰¹¹ using the performance metrics will give an assessment of the models' performance.

The basic idea of network-based approaches is to compute similarity of technology codes from their co-occurrences in companies. Introduced by Teece et al (1994), and popularized in the network/complexity community by Hidalgo et al (2007), the basic quantity is the number of firms that have patented inventions relating to both codes:

$$\begin{equation}{B}_{t,{t}^{\prime }}^{\mathrm{C}\mathrm{O}}=\sum\limits _{f}{M}_{f,t}{M}_{f,{t}^{\prime }}.\end{equation} \tag{ 1 }$$

The idea is that if many firms are active in two technology sectors t and t' at the same time, this means that the capabilities, the techniques and, in general, the necessary means to patent in these sectors, are roughly the same, and so these sectors are, in this sense, similar, or related.

Different scholars presented various ways to normalize the co-occurrences, on the basis of different theoretical frameworks or interpretations. In general, we can write:

$$\begin{equation*}{B}_{t,{t}^{\prime }}=\frac{1}{A}\sum\limits _{f}\frac{{M}_{f,t}{M}_{f,{t}^{\prime }}}{C}\end{equation*}$$

and discuss the various options for the quantities A and C:

• Simple co-occurrences (Teece et al 1994): for A = 1 and C = 1 one simply counts the number of companies that are active in both sectors. This case corresponds to ${B}_{t,{t}^{\prime }}^{\mathrm{C}\mathrm{O}}$ of equation (1);
• Technology Space (same normalization of the Product Space (Hidalgo et al 2007)): A = max(u_t , u_t') and C = 1, where u_t = ∑_f M_f,t is the ubiquity of technology code t, that is, the number of firms active in that technology sector. Using this type of normalization we give a lower connection weight to those technology codes done by many firms, which we can consider basic.
• Taxonomy (Zaccaria et al 2014): A = max(u_t , u_t') and C = d_f , where d_f = ∑_t M_f,t is the diversification of firm f. The Technology Space, for how it is built, gives a higher score for high complexity technology codes (i.e. codes done by few firms) and, as a result, bias towards them. Consequently, it is not possible to justify the evolution of low-complexity technology codes toward high-complexity ones. Normalizing also for the diversification we avoid this problem as we penalize low ubiquity scores and low complexity technology codes are weighted more.
• Micro partial (Teece et al 1994): we compute
  $$\begin{equation*}{B}_{t,{t}^{\prime }}^{\mathrm{M}\mathrm{P}}=\frac{{B}_{t{t}^{\prime }}^{\mathrm{C}\mathrm{O}}-{\mu }_{t{t}^{\prime }}}{{\sigma }_{t{t}^{\prime }}}\end{equation*}$$
  with
  $$\begin{equation*}{\mu }_{t{t}^{\prime }}=\frac{{u}_{t}{u}_{{t}^{\prime }}}{N},\end{equation*}$$
  and
  $$\begin{equation*}{\sigma }_{t{t}^{\prime }}^{2}={\mu }_{t{t}^{\prime }}\frac{(N-{u}_{t})(N-{u}_{{t}^{\prime }})}{N(N-1)},\end{equation*}$$
  where N is the number of companies. Here we use a null model in which the ubiquities of the technologies are kept fixed and everything else is randomized. This case can be analytically solved: the resulting distribution for the co-occurrences is hypergeometric with mean μ_tt' and variance ${\sigma }_{t{t}^{\prime }}^{2}$. We call this network micro partial following the notation used by Cimini et al (2022): this null model is microcanonical in the sense that the degree sequence is exactly fixed and partial because only one layer is constrained. So the idea is that, if the weight of the link between two technology codes t and t' exceeds the expected value μ_tt', this means that t and t' are highly related with respect to this random case. Furthermore, as a t-statistic, ${B}_{t,{t}^{\prime }}^{\mathrm{M}\mathrm{P}}$ measures how much the observed link between the two technology codes exceeds what would be expected if the companies were randomly assigned.

For the latest formulas, we obtain one matrix B^Net for each network. In order to consider all years available in the training data, we used as M matrix in the previous formulas a total matrix obtained by summing the V matrices from the years 2000 to 2009, using all the 197944 firms, and then binarizing this sum.

Based on the network used, we get a B^Net which we use in the coherence equation from Pugliese et al (2019b):

$$\begin{equation}{S}_{f,t}^{2011}=\sum\limits _{{t}^{\prime }}{M}_{f,{t}^{\prime }}^{2009}{B}_{{t}^{\prime }t}^{\mathrm{N}\mathrm{e}\mathrm{t}},\end{equation} \tag{ 2 }$$

where ${M}_{f,{t}^{\prime }}^{2009}$ is the M matrix obtained by binarizing the V²⁰⁰⁹ matrix. In practice, t is highly coherent with the patenting activity of firm f if f is active in many sectors highly connected with t. On the contrary, if a sector is far from what a firm actually does, we will assign it a lower activation score. Note that this equation differs from the density equation of the Product Space (Hidalgo et al 2007); we use coherence instead of density since we have found a better predictive performance.

RF (Breiman 2001) is a tree-based machine learning algorithm that we use to better capture the non-linear links between technology codes. In particular, we use this binary classification algorithm to determine whether or not a technology code will appear in the patenting portfolio of a particular company in the future starting from the knowledge of the technology codes in which the firms patented in the last training year.

In general, during the training of a supervised machine learning algorithm, an input data X matrix is passed. Because our problem is a supervised one, each vector (row) of the matrix is associated with a label presented in a different input y. To give an example, X can be the matrix where each row is a flattened handwritten digit, and each element of the row is the intensity of a pixel; in this case, y will be the label corresponding to the digit, and that must be associated, in order to be recognized, to all those present in X. Once the model is trained, one gives new samples X_test and the model is able to make associate a prediction y_test (in this case, a digit), to each sample.

In our case, we train one RF for each technology code: we want the RF to learn which typologies of portfolios are associated with each code after two years. So, as samples matrix X we use the matrix obtained by concatenating, or stacking vertically, the V matrices from the year 2000 to 2007, and as y we use one column at a time (and therefore one technology code at a time) of the matrix obtained by concatenating the matrices M from the year 2002 to 2009. In this way, each row is a firm in a year from 2000 to 2007, that has 643 features. We associate this sample to the respective label in y, that is, if after 2 years the technology code associated with the element in y is active, or not. In such a way, we associate the codes of each portfolio with the possible presence of the target code in the future.

From a practical viewpoint, we use the 'RandomForestClassifier' from the 'sklearn.ensemble' python library (Pedregosa et al 2011), called in this way:

$$\begin{equation*}\text{RandomForestClassifier.fit}({\mathbf{V}}_{2000}^{2007},{\vec{\mathbf{M}}}_{2002}^{2009}),\end{equation*}$$

where ${\mathbf{V}}_{2000}^{2007}$ are the vertically stacked matrices and with the vector symbol over M we indicate that one column is used at a time, that is, we train one RF for each technology code. The delay of 2 years is used to insert a dependence on time, as we want to produce forecasts about the innovative development of firms. We optimized the RF parameters as described in the supplementary information; the results shown here refer to: number of trees = 50, min_samples_leaf = 4; max_depth = 40 and method = 'entropy'. The use of all available companies in the training is computationally demanding, so we used only the top 10 000 most diversified firms (10KHD firms). If we use more firms for the training we get a saturation of the forecast performances (see the supplementary information). The fact that firms with higher diversification should be used is due to the fact that these provide better coverage of the possible technologies and the possible combinations among them.

After fitting the data, that is, training the machine learning model, we obtain the S²⁰¹¹ scores by using the V²⁰⁰⁹ matrix as X_test in a predicting phase. The command line reads

$$\begin{equation*}{\mathbf{S}}^{2011}=\text{RandomForestClassifier.predict_proba}({\mathbf{V}}^{2009})\end{equation*}$$

and this associates a probability to activate the target technology to each firm in 2009. In figure 6 we schematically represent how S²⁰¹¹ is obtained from the RF, the latter being represented by a set of differently trained decision trees.

Zoom In Zoom Out Reset image size

In this paper, we also compare the approach described above with a cross-validated RF, for which we use the same optimized parameters and, in order to provide a consistent comparison of the results, the same training and test sets. In this respect, note that the ${\mathbf{\text{X}}}_{\text{test}}$ should always produce a prediction for all the 197944 firms (including the 10KHD firms used in the training). In the cross-validation framework, we train k = 4 different RFs, using the technique called k-fold cross validation, that is we separate into k = 4 groups both the 10KHDs used for the training and the 197944-10KHDs used for the test. Then, in training each RF we remove one of the 4 groups of 10KHD companies. The test is instead performed on the removed group from the 10KHD companies along with 1/4 of the 197944-10KHD firms used for the test. This is performed 4 times, each time removing a different set of firms. In the end, we will have prediction scores for all companies by merging the scores produced by the four RFs.

The idea behind the use of cross validation is the following. During the training, the RF basically learns two pieces of information: to recognize the portfolio of a company and the similarity among technologies. Even if we are more interested in the latter, the learning of the two cannot be avoided. However, we can try to force the algorithm to use the similarities in the test phase: if we give a new company in the X_test, the RF cannot recognize it and so it is forced to use the similarities to produce its predictions. This procedure, even if computationally more demanding, leads to better results, as shown in figure 2.

RF shares with most the machine learning algorithms an intrinsic difficulty of interpretation, i.e. the rationale behind how the input is connected to the output is not evident. In this respect, network approaches (note: if made sparse by a suitable filter) are more clear, since the coherence or density-based approach are clearly visualizable: a technology is coherent with a firm's portfolio if has a lot of heavy connections with what the firm already does. In order to restore the interpretability of networks and keep the predictive performance of machine learning, Tacchella et al (2021) propose the Continuous Projection Space, which here we reformulate, with suitable modifications, as the CTS.

To compute the CTS we start from the RF CV method but, in X, only the first 2K HD firms are used, because we have a saturation of the scores: using more firms does not change the scores and increases the computational time.

Another difference with the RF CV is that the predictions are obtained using as X_test the same 2K HD firms used for the training (i.e., in the CTS X = X_test; however, we use k-fold CV to avoid overfitting problems). At the end we obtain a scores matrix of shape [N × years] × [#t], where N is the number of companies (N = 2000), years = 10 and #t = number of technology codes = 643; in total, this scores matrix has shape 20 000 × 643.

Each column of the score matrix represents the likelihood that each company (rows) will patent each technology code (columns). We can then argue that two sectors are similar if the RF predicts that the same companies will (or will not) produce patents in these sectors. In this sense, the columns of the score matrix can be seen as the coordinates in a high-dimensional space for each technology code, where the number of dimensions is given by the number of companies multiplied by the number of training years (in this case, 20 000). In order to provide better interpretability to the relatedness assessment, one should find a low-dimensional visualization of this high-dimensional representation. Obviously, it is impossible to visualize this continuous space of technologies in such a high dimensionality; so we project these points in a lower dimension by combining a variational—autoencoder neural network (Kingma and Welling 2013), to reduce the dimension from 20 000 → 150, and then t-SNE (Van der Maaten and Hinton 2008), to reduce the embedding space from 150 → 2 dimensions, finally obtaining the CTS, that we show in figure 4(a). Now the similarity between technology codes is simply given by the relative distance in this 2 − D space, and it is easy to understand and visualize how firms move from the codes already present in their portfolios to the ones that are immediately close, as shown in figure 5.

Now we want to use the idea of a coherent diffusion in this low dimensional space to produce forecasts; in practice, to obtain a score matrix S²⁰¹¹ to compare with the possible activations of 2011. We start by computing a similarity matrix for the CTS, which for the sake of simplicity we keep calling B. We use the distances between technology codes on the CTS and Gaussian kernels:

$$\begin{equation*}{B}_{i,j}=\frac{{e}^{-{\Vert}{\mathbf{y}}_{i}-{\mathbf{y}}_{j}{{\Vert}}^{2}/2{\sigma }_{i}^{2}}}{{\sum }_{k}\;{e}^{-{\Vert}{\mathbf{y}}_{i}-{\mathbf{y}}_{k}{{\Vert}}^{2}/2{\sigma }_{i}^{2}}},\end{equation*}$$

where y_i is the coordinate of the ith technology code in the CTS (i.e. in the 2 − D space). The σ_i is the standard deviation of the Gaussian kernel related to the technology code ith; this parameter can be set differently for each i code, through a binary search process in which the number of first neighbors is fixed. As we can see in figure 4(a), there are codes in dense areas and codes in less dense areas, so the idea is to assign a high sigma value to the codes in less dense areas and low sigma values in more dense areas in order to keep the interaction with the number of first neighbors constant. The binary search process is described in the supplementary information where we also show that the best optimal value of nearest neighbors is 75.

After the similarity matrix B is obtained, one can compute the score matrix S²⁰¹¹ from the coherence equation equation (2):

$$\begin{equation*}{S}_{f,t}^{2011}=\sum\limits _{{t}^{\prime }}{M}_{f,{t}^{\prime }}^{2009}{B}_{{t}^{\prime },t}.\end{equation*}$$

The number of nearest neighbors is calculated out of sample using the four-fold cross validation as in the case of the RF CV: we use 3/4 of the companies to determine the number of nearest neighbors that maximize the area under the PR curve and then we calculate the scores using the equation (2) for the remaining companies.

Note that the CTS is, as the network approaches, density-based: the more a firm surrounds a technology sector, the more likely it will be part of its patenting activity in the near future.

In order to understand the effective goodness of our forecast results, a comparison with some relatively trivial benchmark models is required. We used two benchmark models:

• The first consists of simple randomization of the technology codes. In practice, we shuffle the columns of the M²⁰⁰⁹ matrix in the calculation of equation (2). The B used is that calculated with the Technology Space network starting from the not randomized M²⁰⁰⁹ (using the other networks there is no significant change in the metric scores). In this way, the diversification of firms is preserved.
• The second benchmark model checks the hypothesis that the simple temporal autocorrelation of the bipartite networks can explain the observed dynamics. In this case, we use the mean of the Vs matrices from 2000 to 2009, $\bar{\mathbf{V}}$ (all the years used in the training) of the test firms as score matrix S²⁰¹¹, that is, element-wise:
  $$\begin{equation*}{\mathbf{S}}^{2011}=\bar{\mathbf{V}}.\end{equation*}$$
  In this way, we check if the number of patents done in the past can forecast the number of patents done in the future by the same company in the same technology sector. As shown in figure 2, this benchmark model can outperform some of the density-based approaches.

In order to compare the goodness of the predictions of the different approaches, we use standard evaluation metrics, widely used for classification tasks in supervised machine learning (Hossin and Sulaiman 2015). As different metrics capture different aspects of the prediction problem, only the comparison between various measures of performance can provide a global view of the effectiveness of a forecasting approach.

The elements that we want to predict are the possible activations, that is, those elements of M²⁰¹¹ that were always zero in 2000–2009. The 0s are called negatives, and the 1s are called positives. The elements equal to 1 that are correctly predicted are called true positives (TP); and similarly one can define the false positives (FP), the true negatives (TN) and the false negatives (FN) as, respectively, the 0s predicted as 1s, the correctly predicted 0s, and the 1s predicted as 0s.

To evaluate the predictions done with the different approaches, we have used three performance metrics:

• Area under the PR curve: this indicator is equal to the area in the precision-recall plane. Precision is defined as $\frac{\mathrm{T}\mathrm{P}(\tau )}{\mathrm{T}\mathrm{P}(\tau )+\mathrm{F}\mathrm{P}(\tau )}$, while recall is equal to $\frac{\mathrm{T}\mathrm{P}(\tau )}{\mathrm{T}\mathrm{P}(\tau )+\mathrm{F}\mathrm{N}(\tau )}$. These quantities are close to 1 if FP and FN are respectively minimized. Note that in order to compute precision and recall one has to specify the scores' binarization threshold τ, that is, the number above which the score is associated with a positive prediction (1). The PR curve is defined by varying the τ parameter because for different values of τ we obtain different precision and recall values. The last step is the computation of the area under this curve, which is independent of the threshold τ.
• Precision@100: to compute this indicator we focus on the top 100 score elements in S²⁰¹¹: if the model is correct, many of these possible activations should become realized activations. The Precision@100 is the ratio between the number of these 100 that are true positives (that is, correctly predicted realized activations), and 100, i.e. the number of elements that we are considering. This represents a global assessment, that considers the score matrix as a whole.
• mPrecision@10: while the Precision@100 provides a global measure of the precision of the approach, we would like to have a measure of our average predictive performance for each firm. To do this, we evaluate the mPrecision@10. We consider the 10 highest scores for each row, i.e. for each firm, and compute the fraction of true positives. Then we average over the firms. Since most of the firms do not show at least 10 realized activations, the global number is far from 1. We have computed the mPrecision also restricting ourselves only to the firms with 10 or more realized activations, finding similar qualitative results.