Assessing Machine Learning Techniques for Predicting Banking Crises in India

Abstract

The historical prevalence of banking crises and their profound impact on global economies underscores the imperative for policy makers to refine their crisis forecasting frameworks. Against this backdrop, the present study endeavors to predict potential banking crises in India by leveraging a spectrum of artificial intelligence and machine learning techniques (AI-ML). These techniques encompass logistic regression, random forest, naïve Bayes, gradient boosting, support vector machine, neural networks, K-nearest neighbors, and decision trees. Initially, a banking fragility index was constructed utilizing monthly banking data spanning 2002 to 2023, demarcating the periods of crisis and stability. Subsequently, an extensive array of early warning indicators (EWIs) encompassing asset prices, macroeconomic factors, external influences, and credit-related variables were employed to forecast crisis periods. Our findings reveal that AI-ML models exhibit reasonable accuracy in predicting banking crises. Moreover, advanced model performance metrics highlight neural networks and random forest models as particularly effective in crisis prediction, surpassing other methodologies. Notably, among the EWIs, variables related to credit, interest rates, and liquidity emerge as possessing relatively higher information value in discerning fragilities within the Indian banking system. Importantly, the methodological framework presented herein can be extrapolated for banking crisis prediction in other economies.

1. Introduction

According to the comprehensive datasets provided by Laeven and Valencia (2020) and Nguyen et al. (2022), 151 systemic banking crises occurred across 201 countries from 1970 to 2019, emphasizing the frequency and global impact of such crises over the period. The studies further indicate that banking crises often precede or coincide with currency and debt crises, leading to significant financial downturns. While financial crises more frequently originate in emerging countries, banking crises are also observed in advanced economies. Furthermore, historically, financial crises have been found to be very costly. On average, financial crises following World War II led to an output loss of approximately 9 percent and they lasted about 1.7 years from peak to trough, while the unemployment rates post-crisis averaged at around 7 percent for a duration of 4.8 years (Reinhart and Rogoff 2009).

The distinctive characteristic of the Indian banking system is its dominant presence within the overall financial landscape. The majority ownership is held by the Indian government, with private entities and foreign investors also holding significant stakes. As an emerging economy, India has undergone various economic and financial reforms since the 1990s to establish internationally competitive banks with enhanced productivity. However, it has not been immune to challenges; it experienced a systemic banking crisis in 1993 and has since grappled with escalating levels of non-performing assets, irrespective of ownership. With a population exceeding 1.4 billion and inherent structural issues, coupled with aspirations to become a USD 5 trillion economy, even a minor banking crisis can have disastrous consequences. Therefore, there is an urgent need to proactively address and contain potential systemic banking crises before they escalate.

The genesis of the financial crisis can be tracked by observing the divergence in the trend movements of certain key macroeconomic and financial variables, which may signal the buildup of vulnerabilities. Also, as banks lie at the center for risk propagation in a crisis, specifically, post-global financial crisis (GFC), the variables pertaining to the banking sector are tracked closely to detect the incipient stress and buildup of vulnerabilities. Therefore, due attention has been paid to building models to identify the incipient stress in the banking system and identify the stress signals at an early stage. This analytical framework led to the generation of early warning indicators (EWIs), with information content to signal the formation of vulnerabilities that may lead or indicate an impending crisis (Aldasoro et al. 2018). Policy makers track these EWIs, viz., the credit-to-GDP ratio, credit-to-GDP gap, credit growth rates, household debt levels, etc., to gauge the status of the economy and the banking and financial systems (Gersl and Seidler 2010).

Traditional banking supervision models of regulators relied on using confidential entity-level data and supervisory inspections to gauge the health of the banks and banking system, viz., the CAMEL ratios and other risk-based supervision measures (RBI 2023). Alternatively, using publicly available data, other models were also developed to track the performance of the banking system (Gersl and Seidler 2010; Serwa 2013). These models were developed to measure or indicate the fragility in the banking system (Kibritcioglu 2003; Allen and Gale 2004) using broad measures, viz., the bank deposit, credit, foreign currency assets, bank reserves to indicate the presence or absence of fragility in the banking system.

While gauging the fragility of the banking system at a point in time is the first step, it is more pertinent or useful if the emergence of fragility can be signaled at an early stage for initiating preventive action by the policy makers. Studies explored the relationship between banking fragility and EWIs using traditional models, viz., probit and logit models, multinomial logit, etc. (Babecký et al. 2014; Caggiano et al. 2014; Candelon et al. 2014). These studies deployed EWIs comprising macroeconomic, banking, real, and external sector variables which could signal the buildup of fragilities in the banking sector. Recently, tracking the advancements in the use of artificial intelligence and machine learning (AI-ML) models, some studies have applied advanced AI-ML models like neural network, random forest, support vector machine models for predicting fragility and financial crisis using EWIs (Akkoc 2012; Martinez 2016; Sevim et al. 2014). Such studies have displayed an advantage over the traditional parametric and non-parametric models, as they can handle large volumes of data and indicate the direction of fragility with greater accuracy.

Hence, this study intended to extend the AI-ML framework for predicting banking fragility/crisis using EWIs for the Indian economy. Following Kibritcioglu (2003), we first constructed two sets of banking fragility index (BFI) values to identify the crisis periods by reflecting on the inherent business models of the Indian banks between 2002 to 2023. Subsequently, we applied the AI-ML models to a curated set of EWIs to forecast potential crises in the Indian banking system. This study offers two significant contributions. Firstly, it expands the literature on banking crisis prediction using AI-ML models from an emerging market perspective. Secondly, it provides policy makers with insights into key EWIs that carry higher information value for signaling impending stress within the banking system.

The rest of the paper is divided into four sections. Relevant literature dealing with the prediction of banking crises and the use of AI-ML models is presented in Section 2. This is followed by Section 3 which presents the data and methodology used for the study. The results and discussion are given in Section 4, while Section 5 provides the concluding observations and guidance for future work.

2. Literature Review

Bankruptcy prediction spans various sectors, including banking. It encompasses a rich body of literature leveraging diverse methodologies, from traditional ratio analysis to cutting-edge AI-ML techniques. This review focuses exclusively on studies employing machine learning techniques within the banking sector, presenting them chronologically.

Nur Ozkan-Gunay and Ozkan (2007) analyzed data from the Banks Association of Turkey (1990–2000), revealing the effectiveness of artificial neural networks (ANN) in predicting bank defaults in Turkey, albeit with exceptions. Gutiérrez et al. (2010) introduced hybrid logistic regression with product unit and radial basis function networks, showing strong prediction rates for banking crises across 79 countries during the sample period of 1981–1999. Roy and Kemme (2011) identified asset bubbles as common precursors to banking crises on the basis of severity and systemic crises in the US, UK, Spain, and Ireland during 2007–2008. In India, Ghosh (2011) found moderate banking sector stability during 1997–2007, with a decline in 2006 attributed to reduced loan loss provisions. Zaghdoudi (2013) highlighted decreased profitability and debt repayment ability as key indicators of Tunisian bank failures.

Drehmann and Juselius (2014) evaluated (EWIs) for banking crises, emphasizing the effectiveness of the credit-to-GDP gap and debt service ratio over non-core liabilities. Caggiano et al. (2014) identified economic decline, banking system illiquidity, and currency mismatches as primary predictors of systemic banking crises in Sub-Saharan African Low-Income Countries (SSA-LICs). Iturriaga and Sanz (2015) developed multilayer perceptrons and self-organizing maps to predict bank distress using US commercial bank data (2002–2012), highlighting the concentration of failed banks in real estate loans. Caggiano et al. (2016) concluded that the multinomial logit model outperformed the binomial logit model in predicting systemic banking crises across 92 economies for the sample period of 1982–2010.

Ristolainen (2016) demonstrated the superiority of artificial neural networks over logit regression in predicting banking crises across 18 countries. Tanaka et al. (2016) favored a random forest-based early warning system for predicting bank failures over conventional methods among OECD member countries. Dabrowski et al. (2016) advocated for dynamic Bayesian networks for systemic banking crisis early warning systems across eleven developed European economies. Sohn and Park (2016) prioritized credit growth as a more informative indicator of banking sector crises than the credit-to-GDP gap. Antunes et al. (2018) identified equity prices, house price growth, and debt service ratio among the most useful indicators for signaling banking crises in European countries. Papanikolaou (2018) developed a dual early warning model for bank distress using US commercial and savings bank data, emphasizing its strong out-of-sample predictive power.

Aldasoro et al. (2018) highlighted the utility of household debt service ratio and cross-border claims in assessing banking system vulnerabilities. Carmona et al. (2019) found that low retained earnings compared to average equity, the pre-tax return on assets, and total risk-based capital ratio were associated with a higher risk of bank failure among US national commercial banks during the sample period of 2001–2015. Climent et al. (2019) found superior predictive accuracy of the two-year before failure model in assessing bank distress in the Eurozone, emphasizing the importance of total assets, non-operating income, capital, and liquidity. Beutel et al. (2019) compared machine learning models’ predictive performance with the logit approach in predicting banking crises across advanced countries using a longer (1970–2016) sample period of data, highlighting an overfitting problem. Petropoulos et al. (2020) identified random forests as superior in out-of-sample and out-of-time predictive performance among US-based banks.

Shrivastav and Ramudu (2020) achieved 93 percent accuracy in predicting Indian bank defaults using support vector machine methodology on a dataset spanning 2000 to 2017. de Haan et al. (2020) emphasized the role of bank balance sheet information in predicting future banking crises, highlighting the significance of low bank liquid assets and high financial leverage. Filippopoulou et al. (2020) evaluated risk indicators from the European Central Bank and European Systemic Risk Board, emphasizing the importance of specific banking variables over macroeconomic variables. Hartini et al. (2021) achieved 90 percent accuracy in predicting banking crises using the random forest method across 79 countries with a dataset spanning 1981 to 1999. Using the banking sector fragility index data from February 2002 to March 2017, it was concluded that predicting the Indian banking crisis is achievable with artificial neural network techniques (Gupta and Kumar 2022). Liu (2023) automated variable selection using the LASSO method, identifying credit growth, domestic and global credit gaps, and real house price growth as a crucial EWI of systemic banking crises. Citterio and King (2023) advocated for Environmental, Social, and Governance (ESG) indicators to enhance predictive power and reduce the misclassification of distressed banks.

In summary, these studies showcase the evolution and effectiveness of machine learning techniques in predicting banking crises, underscoring the importance of accurate early warning systems for financial stability. Also, as noted in the earlier section, there is very scant literature that deals with the use of AI-ML models in the context of the emerging economy, viz., India. The study aims to fill this gap in the literature.

3. Data and Methodology

3.1. Construction of the Crisis Variable—Banking Fragility Index

The aim of the current study is to predict the banking crisis period using a host of AI-ML techniques on a set of explanatory variables indicating the risks faced by the banks. However, a priori classification of the crisis period is to be carried out for training the AI-ML models. Hence, as the first step, we used the monthly data on banking variables, and leveraging the banking fragility index (BFI) developed by Kibritcioglu (2003), to identify the crisis period in the banking sector.1 Methodologically, identifying banking fragility contrasts with analyzing banking crisis based on events, i.e., based on specific trigger events as agreed by financial experts or announced by the governments (Caprio and Klingebiel 2002). While tracking crises based on specific events is relatively easier, it only serves as a reference point but does not indicate the inception of a crisis. Any forward-looking assessment methodology should enable the policy maker to smell stress in advance (RBI 2023). As opposed to tracking specific thresholds for variables, the BFI methodology enables identification of the buildup of stress as a deviation of current values of banking variables form their trend values. Thus, the BFI enables assessing the buildup of vulnerabilities at an early stage. Further, the BFI is based on more frequent monthly or quarterly data, resulting in better signaling regarding underlying trend movements.

Literature identifies three main sources of banking crisis emanating from assets, liabilities, and foreign exposures, which are captured by loans and advances, deposits, and net foreign assets, respectively (Kaminsky and Reinhart 1999; Goldstein et al. 2000). Further, an excessive risk taking on any of these channels results in losses, leading to the depletion of capital and bank reserves (Gourinchas et al. 2001). The loss of reserves not only impacts current businesses, but also the ability of the bank to explore new businesses, thus further accelerating the stress on the balance sheet, resulting in insolvencies and precipitating crises (Jagtiani et al. 2000).

BFI attempts to capture the excessive risk-taking in the banking system by quantifying the monthly deviations of growth in banking variables. Specifically, in its original formulation, the BFI is made up of three constituents reflecting the deviations in growth rates at any given time ‘t’ in deposits (DEP), claims on domestic private sector (CPS), and foreign liabilities (FL) of banks. As standardized values are used, the index avoids the possibility of domination by one of the components (Kibritcioglu 2003). Also, the growth rates are computed with a 12-month window, avoiding any seasonality issues impacting the interpretation of the results.

$$BSF{3}_t=\left\{\frac{({CPS}_t-{\mu }_{cps})}{{\sigma }_{cps}}+\frac{({FL}_t-{\mu }_{FL})}{{\sigma }_{FL}}+\frac{({DEP}_t-{\mu }_{dep})}{{\sigma }_{dep}}\right\}/3$$

(1)

Although, BFI is predominantly deployed for assessing banking fragility cross-country (Cevik et al. 2013), some studies have applied it for single countries also (Ahmad and Mazlan 2015; Ari and Cergibozan 2016; Wafula et al. 2022).

Accordingly, the above framework is extended to the Indian context by constructing two types of banking fragility indices, BFI_4 and BFI_6, each with four and six components, respectively. The BFI_4 has four components, viz., bank deposits, bank credit, net foreign currency assets, and net bank reserves. Further, banks in India are mandated to park a specified portion of their deposits in approved government securities under Statutory Liquidity Ratio (SLR) regulations. Consequently, a substantial portion of banks’ assets (investments) are directed toward the sovereign2 Furthermore, the commercial banks provide food credit for procurement of food grains by the food corporation of India. Also, with the increasing integration of the Indian economy with the global economy, the banking system has also witnessed higher foreign currency inflows both in terms of assets and borrowings. Hence, an expanded version of the BFI is computed to capture the broader range of risks faced by the banks in India.

The BFI_6 has six components, viz., bank deposits, non-food credit, net foreign currency assets, net foreign currency borrowings, other investments,3 and net bank reserves. Both BFI_4 and BFI_6 are computed using the following specifications:

$$\mathrm{B}\mathrm{F}\mathrm{I}\text{\_}4=\left[\right\{{D_t-{\mu }_d\}/{\sigma }_d+\{C}_t-{\mu }_c\}/{\sigma }_c+\{F_t-{\mu }_f\}/{\sigma }_f+\{R_t-{\mu }_r\}/{\sigma }_r]/4$$

(2)

$$\mathrm{B}\mathrm{F}\mathrm{I}\text{\_}6=\left[\right\{{D_t-{\mu }_d\}/{\sigma }_d+\{NC}_t-{\mu }_{Nc}\}/{\sigma }_{Nc}+\{{FA}_t-{\mu }_{FA}\}/{\sigma }_{FA}+\{{FB}_t-{\mu }_{FB}\}/{\sigma }_{FB}+\{R_t-{\mu }_r\}/{\sigma }_r+\{{OI}_t-{\mu }_{OI}\}/{\sigma }_{OI}]/6$$

(3)

where D, C, F, and R represent the year-on-year (YoY) change in deposits, bank credit, net foreign currency assets, and net bank reserves, respectively, at time t. Additionally NC, FA, FB, and OI represent the YoY change in non-food credit, net foreign currency assets, net foreign currency borrowings, and other investments, respectively, at time t, while μ and σ represent the mean and standard deviation of the variables over the sample period.

The data for the above variables are sourced from the commercial banking survey data published by the Reserve Bank of India4 at a monthly frequency from March 2002 to March 2023, i.e., 253 monthly observations. As the variables are in nominal values, they are deflated using the Wholesale Price Index (WPI: 2015 = 100) series. The real values are then used to compute the YoY change for the variables.5 Based on YoY change values, the BFI_4 and BFI_6 indexes are computed using the specifications mentioned in Equations (2) and (3).

According to the literature (Kibritcioglu 2003; Gupta and Kumar 2022), the BFI values can be used to mark the periods of banking crisis. If the values of the BFI are non-negative, the banking system is said not to be in crisis, and negative BFI values indicate a crisis. Accordingly, the crisis period for the Indian banking system is then marked as a binary variable (0 or 1) based on the BFI_4 and BFI_6 values. The crisis periods can be treated as being in a continuum alternating between crisis and no-crisis zones, and accordingly, the target variables are assigned values 0 and 1.

BFI ≥ 0 (No − Crisis:0)

BFI < 0 (Crisis:1)

While the threshold-based classification of crisis periods can be performed, for the purpose of this study, attention is limited to binary classification by the AI-ML models, viz., the prediction of crisis and no-crisis period.

3.2. Selection of Early Warning Indicators—Explanatory Variables

The next step was to determine the set of explanatory variables that capture the broad risks faced by the banks and can therefore act as EWIs. The literature provides a wide gamut of variables that capture the risks to the banking channels. Kauko (2014) reviewed the empirical literature on EWIs for predicting a banking crisis. The study identifies three waves of literature concerning banking crisis prediction and discusses the key variables used as EWIs across studies. While the first wave of literature focused on the descriptive analysis of a crisis, the third wave focused on the varied effects of GFC on real economic variables through the financial sector. However, in the second wave, empirical methods took center stage following two approaches, viz., the binary regression and signal method for predicting the banking crisis. Further, studies in the domain of the signal method (Kaminsky and Reinhart 1999; Borio and Lowe 2002; Drehmann et al. 2011) correlated financial stability or instability with the movement of macroeconomic variables beyond certain threshold values. The non-linear nature dependence between crisis variables and explanatory variables is also captured by the signal method (Alessi and Detken 2011). Most EWIs used in such studies can be categorized broadly as macroeconomic factors, external sector factors, credit related factors, asset prices, etc., with varying degrees of influence on the risks to the banking sector (Kauko 2014).

Asset Prices: The theoretical role of volatility in asset prices in fueling financial crisis is outlined in Minsky (1977) and Kindleberger (1978). Many studies indicate that banking crises are predated by an increase in asset prices, often fueled by debt-driven asset purchases particularly in the real estate sector (Connor et al. 2012). The Global Financial Crisis (GFC) is a case in point indicating the interplay between falling asset prices and the banking crisis. Several individual countries have also witnessed similar incidents of asset bubbles impacting the financial and banking sectors (Reinhart and Rogoff 2009). While most studies focused on the trends in housing prices to a signal crisis, few studies have also explored the role of stock indexes as an EWI of the crisis (Roy and Kemme 2011). To capture the broad trends in asset prices in India, and considering the data availability, the stock index NIFTY 50 is included in the set of EWIs for this study.

Macroeconomic Variables: Another major risk factor faced by the banks is owing to the stress build up in the assets due to adverse movement of macroeconomic variables like industrial production, inflation, interest rates, etc. Real interest rates are found to be a significant EWI both in the case of advanced and other economies (Roy and Kemme 2011; Jordà et al. 2011). Further, monetary aggregates relative to GDP or forex reserves are also found to have information content to serve as EWIs (Büyükkarabacak and Valev 2010). Similarly, inflation is found to be an important EWI in the case of emerging economies (Lo Duca and Peltonen 2013). By creating stress for individual borrowers across sectors, such real variables can amplify the risks to the banking system (Drehmann et al. 2011; Rajan 2005). To capture these risks, consumer price inflation, wholesale price inflation, 91-day T-bill (yield to maturity), and the index of industrial production are included in the set of EWIs.

External Sector Variables: During the post-liberalization period, the influence of the external factors on domestic risk factors affecting the banking sector has increased. The ‘capital bonanza’ offered by the developing countries attracts inflows, raising asset prices and domestic credit (Reinhart and Reinhart 2009). Also, studies have found that most banking crises are preceded by a currency crisis (Kauko 2014). Besides the current account deficit, variables like foreign debt, flows of foreign direct investments, and other balance of payment statistics are found to signal an impending crisis (Obstfeld 2012). Also, the capital flows and exchange rate movements have a bearing on the bank fund costs and borrowers’ performance (Hahm et al. 2013). Further, given that the Indian economy is import-dependent for its energy reliance, the movements in crude oil prices have an adverse impact on domestic economic sectors (Burkart and Coudert 2002). Hence, to comprehensively capture such risks due to external factors, the following variables are included in the set of EWIs: crude prices, real effective exchange rates, openness, terms of trade, and current account deficit. Furthermore, to capture the emerging risks of short-term debt to GDP, foreign direct investment to GDP, gross fiscal deficit to GDP, and money supply to forex reserves are also included in the set of EWIs.

Credit-Related Factors: Notwithstanding the above factors, the banking sector can face risks due to inherent exuberance or unsustainable lending. Funding long-term assets with short-term liabilities, leveraging favorable liquidity conditions or interest rate scenarios, the banks may expose themselves to risks if the interest rate or liquidity scenarios turn adverse (Sufi and Taylor 2022). Credit aggregates are found to be good EWIs of a banking crisis (Drehmann and Juselius 2014). The Bank of International Settlements (BIS) regularly monitors the nature of financial booms through tracking credit ratios like the credit-to-deposit ratio, credit-to-GDP, credit-to-GDP gap, and bank assets-to-GDP of the economies as EWIs. Aldasoro et al. (2018) found that a family of EWIs comprising credit aggregates tracked by the BIS has discriminatory power to identify crisis and non-crisis periods. Hence, these credit-related parameters are included in the set of EWIs.

Therefore, after a systematic review of literature, 21 variables were chosen as a qualifying set of EWIs for this study which are most apt for the Indian context. The data for these variables are collected at a monthly frequency, and for the variables with a lower frequency, monthly values are derived using the cubic spline interpolation technique. The variables are incorporated either as actual values, computed ratios, and YoY growth rates. The following

Table 1. Early warning indicators—data types and source.

Sl.	Name of the Variable	Data Type	Source *
1	Treasury bill Yield to Maturity	Actual Values	RBI
2	Terms of Trade	Computed Value	IMF
3	Openness	Computed Value	OECD
4	Consumer Price inflation	YoY Growth	OECD
5	Wholesale Inflation	YoY Growth	OECD
6	Index of Industrial Production	YoY Growth	IMF
7	Crude Price	Actual USD	OPEC
8	Stock Prices	YoY Growth	NSE
9	Reserve Money	YoY Growth	RBI
10	Reserve Money to Forex Reserves	Computed Ratio	RBI
11	M2 to Forex Reserves	Computed Ratio	RBI
12	M3 to Forex Reserves	Computed Ratio	RBI
13	Real Effective Exchange Rate	YoY Growth	BIS
14	Current Account Deficit to GDP	Actual Values	OECD
15	Short-Term Debt to GDP	Actual Values	OECD
16	Foreign Direct Investment to GDP	Actual Values	RBI
17	Gross Fiscal Deficit to GDP	Actual Values	RBI
18	Credit–Deposit Ratio	Actual Values	RBI
19	Credit-to-GDP Gap	Actual Values	BIS
20	Credit-to-GDP Ratio	Actual Values	BIS
21	Bank Assets-to-GDP Ratio	Actual Values	RBI

* Notes: RBI—Reserve Bank of India, BIS—Bank of International Settlements, OECD—Organization for Economic Cooperation and Development, IMF—International Monetary Fund, OPEC—Organization of the Petroleum Exporting Countries, NSE—National Stock Exchange, India.

depicts the variable and data sources.

The monthly crisis periods marked using BFI_4 and BFI_6 values as “0” or “1”. These labels are then predicted using AI-ML models on the above set of EWIs from 2002 to 2023. The performance of the AI-ML models is gauged using performance metrics, as discussed in the next section.

3.3. AI-ML Models and Performance Metrics

The following AI-ML techniques are used for predicting the crisis variable6:

• Logistic regression;
• Random forest;
• Naïve Bayes;
• Gradient boost;
• Support vector machine;
• Neural net;
• K-nearest neighbors;
• Decision tree.

The data are split into a 70:30 ratio for generating training and testing datasets, respectively. The model objective is predicting the crisis class, i.e., 0 or 1. The results from the model give a prediction for each period, i.e., in this case every month is labelled as “1” or “0” being predicted as being in crisis or not, respectively. The prediction is then compared with the actual classification based on the BFI_4 and BFI_6 values. Thus, in effect, the AI-ML algorithms used in this study are addressing a classification problem of labelling the “crisis” and “non-crisis” months.

3.4. Performance Metrics

The literature establishes that the performance of classification models is evaluated through the construction of confusion matrices (Kuhn and Johnson 2013). These matrices are a cross tabulation of the number of actual cases and predicted cases as given below. In general, the positive class refers to the variable of interest. In this case the “crisis” period is a positive class, with the “non-crisis” period being a negative class. The confusion matrices (

Table 2. Confusion matrix.

Number of Instances		Actual
		Positive	Negative
Predicted	Positive	True Positives (TP)	False Positives (FP)
	Negative	False Negatives (FN)	True Negatives (TN)

) are then used for computing metrics that enable comparison of the model performance.

The accuracy rate is the foremost metric to assess the performance of AI-ML models in a classification problem. The accuracy rate indicates the number of correct predictions, i.e., both positive and negative instances out of total instances in the dataset. While this is an intuitive and straight forward measure, it does not account for the error rates in the misclassification of the positive and negative instances. To account for the error rates, measures like precision, sensitivity, and specificity are developed.

Precision captures the rate at which the model is predicting the actual true positive instances out of the total positive predictions (true positive plus false positives). This is often associated with the inclusion error, i.e., labeling non-crisis periods as crisis periods. Higher precision rates give confidence that the model can classify the variable of interest accurately, without higher false alarms. Alternatively, recall or sensitivity captures the rate at which the model can capture the true positive class out of the total positive instances (true positives plus false negatives). This is often associated with the exclusion error, i.e., missing out labeling the actual crisis periods. Higher recall or sensitivity indicates that the model is able to label most crisis periods, leaving out fewer positive instances. Further, the F1-score is computed as a harmonic mean of the precision and recall to balance both the inclusion and exclusion errors. Furthermore, specificity captures the rate at which the model is able predict the true negative class out of total negative instances (true negatives and false positives). In this sense, specificity is an analogous measure of sensitivity, except that it is used for the true negative class. The computation of these measures is indicated in

Table 3. Model performance metrics.

Test Metric	Specification
Accuracy	TP + TN/(TP + FP + FN + TN) Total correct predictions/ Total instances in the dataset
Precision	TP/(TP + FP) Correct positive predictions/ Total positive predictions
Recall (Sensitivity)	TP/(TP + FN) Correct positive predictions/ Total positive instances
Specificity	TN/(TN + FP) Correct negative predictions/ Total negative instances
F1-Score	2 × (Precision × Recall)/(Precision + Recall) Harmonic mean of precision and recall

Source: Bekkar et al. (2013).

, which is used for comparing the performance of various AI-ML models.

While the general performance metrics give a fair idea of the discriminating power of the model, it must be noted that for a given level of accuracy, all AI-ML models pose a tradeoff between the inclusion (precision) and exclusion (recall or sensitivity) error of the models (Lüdtke et al. 2011). Further, the general performance metrics discussed above majorly focus on the prediction of the positive class (true positives). They fail to provide a balanced view of model performance based on both positive and negative classes, and error rates. Hence, using the specificity and sensitivity ratios, the following advanced metrics are computed to provide a balanced comparison of the model performance (Bekkar et al. 2013). The advanced metrics are described below.

Geometric mean (GM) of the specificity and sensitivity, i.e., √(sensitivity*specificity).

This ratio penalizes low performance on positive or negative classification. Higher values indicate better model performance.

Likelihood ratio (LR), more specifically the negative likelihood ratio, is computed as follows. LR = (1 – Sensitivity)/Specificity. Intuitively, this captures the ratio of probabilities to classify false negatives to true negatives. Lower values are better.

Balanced accuracy (BA): This measure is a simple average of the specificity and sensitivity. Like the geometric mean, this also penalizes poor performance on positive or negative class identification, computed as BA = (sensitivity + specificity)/2.

Youden’s γ: Youden’s index is a linear transformation of the sensitivity and specificity given as γ = Sensitivty − (1 − Specificity).

Intuitively this indicates penalizes the probability of correct positive classification (true positives) with the probability of incorrect negative classification (false positives). Higher values indicate better model performance.

Area under receiver operating characteristic curve (AUROC): This represents the accuracy rate at which the positive and negative classes are distinguished by the models, i.e., it plots the true positive rate (sensitivity) against the false positive rate (1-specificity) at various thresholds. The models with the highest discriminatory power (higher area under the curve) are better at predicting the true positive class.

Using the above advanced performance metrics, the better performing models are identified. In addition to the above, the information value (IV) of the explanatory variables for each model is computed. A conjunction of better models and IV values of the explanatory variables is used as the basis for identifying the relative importance of EWIs in predicting the banking crisis. The results are provided in the next section.

4. Results and Discussion

4.1. Model Performance Metrics

Using the BFI_4 and BFI_6 values, the crisis period is marked as a binary variable. The dataset with a binary classification of the crisis period with the value “1” and no-crisis period with the value of “0” is used for training and testing the models in a 70:30 ratio. The model performance metrics for prediction based on BFI_4 values are given in

Table 4. Performance metrics of the models for identifying a bank crisis period using BFI_4 values.

Model	Accuracy	Precision	F1-Score	Sensitivity	Specificity
Logistic Regression	0.92	0.92	0.92	0.91	0.93
Random Forest	0.91	0.91	0.91	0.89	0.93
Gradient Boosting	0.91	0.91	0.91	0.91	0.90
Support Vector Machine	0.92	0.93	0.92	0.89	0.97
K-Nearest Neighbors	0.89	0.89	0.89	0.93	0.83
Decision Tree	0.83	0.83	0.83	0.87	0.77
naïve Bayes	0.87	0.87	0.87	0.89	0.83
Neural Network	0.92	0.92	0.92	0.93	0.90

Note: Shaded portions indicate top two best performing models for the given metric.

. The accuracy of the model is around 90 percent across models, which compares well with the findings from similar studies (Petropoulos et al. 2020; Nazareth and Reddy 2023). Also, the other model performance metrics indicate the predictive power of the models, with precision,, F1-scores, sensitivity, and specificity being around 90 percent for most of the models. The F1 score, which is a balanced measure, is higher for logistic regression, and support vector machines is at 0.92, indicating that these models have better predictive power than the other models.

Similarly, the performance metrics for the models using the BFI_6 values are given in

Table 5. Performance metrics of the models for identifying a bank crisis period using BFI_6 values.

Model	Accuracy	Precision	F1-Score	Sensitivity	Specificity
Logistic Regression	0.89	0.89	0.89	0.90	0.89
Random Forest	0.92	0.92	0.92	0.90	0.94
Gradient Boosting	0.87	0.87	0.87	0.83	0.91
Support Vector Machine	0.87	0.87	0.87	0.88	0.86
K-Nearest Neighbors	0.88	0.89	0.88	0.83	0.94
Decision Tree	0.83	0.84	0.83	0.76	0.91
Naive Bayes	0.84	0.84	0.84	0.85	0.83
Neural Network	0.95	0.95	0.95	0.95	0.94

Note: Shaded portions indicate top two best performing models for the given metric.

. It can be observed that the overall performance metrics for BFI_6 correspond to that of BFI_4 values.

In the case of crisis variables based on BFI_6 values, the performance metrics indicate that random forest and neural network models are performing better than other models. The F1- score of the random forest model is at 0.92, while that of the neural network models is 0.95. This indicates that for models based on BFI_6 values, neural networks give better predictions. Therefore, the model performance metrics indicate that for both BFI_4 and BFI_6 values, neural network models have better predictive power than the other models used in this study.

Notwithstanding the above, as mentioned in the Section 3, the study uses advanced model performance evaluation metrics, viz., the geometric ratio, likelihood ratio, balanced average and Youden’s index, and AUROC (area under receiver operating characteristic) curve to benchmark the models. The advanced model performance metrics for the models for both BFI_4 and BFI_6 values are given in

Table 6. Advanced performance metrics of AI-ML models for predicting a bank crisis.

	Geometric Mean		Likelihood Ratio		Balanced Average		Youden’s Index		AUROC
Model	BFI_4	BFI_6	BFI_4	BFI_6	BFI_4	BFI_6	BFI_4	BFI_6	BFI_4	BFI_6
Logistic Regression	0.92	0.89	0.09	0.11	0.92	0.89	0.85	0.79	0.96	0.95
Random Forest	0.91	0.92	0.12	0.10	0.91	0.92	0.82	0.85	0.98	0.96
Gradient Boosting	0.91	0.87	0.10	0.19	0.91	0.87	0.81	0.74	0.97	0.94
Support Vector Machine	0.93	0.87	0.11	0.14	0.93	0.87	0.86	0.74	0.97	0.93
K-Nearest Neighbors	0.88	0.88	0.08	0.18	0.88	0.89	0.77	0.77	0.95	0.95
Decision Tree	0.82	0.83	0.17	0.27	0.82	0.84	0.64	0.67	0.82	0.84
Naive Bayes	0.86	0.84	0.13	0.18	0.86	0.84	0.72	0.68	0.91	0.89
Neural Network	0.92	0.95	0.07	0.05	0.92	0.95	0.83	0.89	0.97	0.97

Note: Shaded portions indicate top two best performing models for the given metrics.

. The advanced model performance metrics indicate that for BFI_4 values, logistic regression and support vector machine models are relatively better than the other models. On the other hand, for BFI_6 values, neural network and random forest models have the better predictive power amongst the AI-ML models used for this study. However, for both BFI_4 and BFI_6 values, the neural network models stand out as relatively better performers considering the AUROC scores.

Moreover, for robustness, the performance of models is gauged using an extended set of 38 explanatory variables, i.e., the actual values and their transformations, growth rates, etc. The models are run on an extended dataset for both BFI_4 and BFI_6 values. The accuracy levels of the models ranged from 0.82 to 0.91 for BFI_4 values, and 0.86 to 0.93 for BFI_6 values. The F1-score of the models hovered between 0.86 to 0.91 for BFI_4 values and 0.86 to 0.93 for BFI_6 values. Also, the advanced performance metrics for these models correspond to the values obtained for models based only on a set of 21 explanatory variables. Even in the case of the extended dataset, it was observed that the neural network models have better predictive power amongst the AI-ML models used for this study. While gradient boosting models’ predictive power compares well with that of neural networks, they lag in terms of the likelihood ratio and Youden’s index. Therefore, the model performance and their predictive power is found to be robust to various specifications of the explanatory variables for both BFI_4 and BFI_6 values.

4.2. Information Value of the Explanatory Variables

First and foremost, a correlogram for the explanatory variables is represented in Figure 1. The pair-wise correlation values are range-bound and do not indicate strong interdependence among explanatory variables. The model performance metrics have indicated that neural networks, followed by random forest and logistics regression models are more useful in predicting a banking crisis in the Indian context. Similarly, it was explored whether some explanatory variables have higher information content in indicating or signaling a banking crisis. Using the information value (IV) or weight of evidence, the relative importance of the explanatory variables, i.e., EWIs, was assessed. It was observed that the following variables had the highest IV: credit-to-deposit ratio, yield on treasury bills, CPI inflation, credit-to-GDP ratio, and credit-to-GDP gap.

It is not surprising that these variables were found to have higher information value amongst the set of EWIs. Variables like credit-to-deposit, credit-to-GDP, and their gap capture the level of exuberance in the lending by banks, and are major early warning signals indicating an impending crisis. Similarly, the yield on treasury bills signals the underlying liquidity and market risk conditions. Likewise, CPI inflation has a direct bearing on borrower performance, with plausible spillovers on banking performance.

4.3. Policy Implications

Our empirical findings have three uses from a policy perspective. The results indicate that AI-ML models can predict banking crises with reasonable accuracy and stable performance, even when advanced performance metrics are considered. Firstly, these AI-ML models can be used alongside the statistical and econometric models by central banks and policy makers to compare crisis predictions, providing additional insights into incipient stress within the banking sector. Also, the AI-ML models can effectively handle big data, and policy makers can develop prediction models using high-frequency data coupled with standard macroeconomic and other real variables to improve the predictive power of the models. Finally, the AI-ML models can be used as testing grounds for picking up variables with the best information content to signal an impending crisis and incorporate them in the econometric modeling for achieving better results. AI-ML routines are easy to implement compared to standard econometric packages, making them the go-to solutions for developing analytical insights using public domain data. This not only enables information for policy makers, but also provides it for interested stakeholders to gauge the performance of the banking system in a cost-effective manner.

5. Conclusions

The findings of the study prima facie indicate that the AI-ML models used here can predict a banking crisis in the Indian context with a reasonable level of accuracy. This is in line with earlier studies, which have found neural network and random forest models to be performing relatively better than other models for predicting a banking crisis. Further, the models’ predictions are based on a host of EWIs that encompass macroeconomic, external, and domestic risk factors faced by banks. The results also indicate that EWIs have adequate information content to signal an impending crisis within the banking system.

Although, like most AI-ML models, the study does not indicate any causative link between EWIs and the target variable, viz., banking crisis, they can be used alongside econometrics models to improve the overall model accuracy. While econometric modeling is driven by theory, at times it becomes challenging from a modeling perspective to choose the best set of explanatory variables and arrive at the optimal model specification. However, a model developer can benefit from the use of AI-ML models used in this study to select the most important features from a larger set of explanatory variables, which can then be deployed for econometric modeling. Alongside that, it can guide policy makers in tracking EWIs with high information content, even if they are not used in the regular models. Particularly, a few EWIs like the credit-to-deposit ratio, credit-to-GDP, credit-to-GDP gap, CPI inflation, and yield on treasury bills were found to have relatively higher information content in predicting a banking crisis. Further, all AI-ML models present an inherent tradeoff between inclusion and exclusion error rates. While advanced metrics allow for the selection of a model with balanced performance, a policy maker can choose a model that best serves or balances its mandate to avoid a crisis with a low cost of disruption.

In future applications, the framework outlined in this study could be expanded to forecast crises or stress at the level of individual financial entities, and to comprehend the propagation of crises throughout different segments of the financial system. Additionally, the models employed in this study are adept at capturing crisis scenarios using both specific (BFI_4) and broad (BFI_6) risk indicators. The advantage of utilizing a broad-based target variable lies in its ability to incorporate factors that represent the business models of banks more accurately. This facilitates a refined calibration of crisis identification tailored to specific economies. Moreover, the framework can be employed to predict crises across various countries by adjusting the crisis target variables to reflect the unique features of each country’s financial system.