Risk and Risk Management in the Credit Card Industry

Using account level credit-card data from six major commercial banks from January 2009 to December 2013, we apply machine-learning techniques to combined consumer-tradeline, credit-bureau, and macroeconomic variables to predict delinquency. In addition to providing accurate measures of loss probabilities and credit risk, our models can also be used to analyze and compare risk management practices and the drivers of delinquency across the banks. We find substantial heterogeneity in risk factors, sensitivities, and predictability of delinquency across banks, implying that no single model applies to all six institutions. We measure the efficacy of a bank’s risk-management process by the percentage of delinquent accounts that a bank manages effectively, and find that efficacy also varies widely across institutions. These results suggest the need for a more customized approached to the supervision and regulation of financial institutions, in which capital ratios, loss reserves, and other parameters are specified individually for each institution according to its credit-risk model exposures and forecasts.


I. Introduction
The financial crisis of 2007-2009 highlighted the importance of risk management at financial institutions. Particular attention has been given, both in the popular press and the academic literature, to the risk management practices and policies at the mega-sized banks at the center of the crisis. Few dispute that risk management at these institutions-or the lack thereof-played a central role in shaping the subsequent economic downturn. Despite the recent focus, however, the risk management policies of individual institutions largely remain black boxes.
In this paper, we examine the practice of risk management and its implications of six major U.S. financial institutions using computationally intensive "machine-learning" techniques applied to an unprecedentedly large sample of account-level credit-card data.
The consumer-credit market is central to understanding risk management at large institutions for two reasons. First, consumer credit in the United States has grown explosively over the past three decades, totaling $3.3 trillion at the end of 2014. From the early 1980s to the Great Recession, U.S. household debt as a percentage of disposable personal income doubled, although declining interest rates have meant that the debt service ratios have grown at a lower rate. Second, algorithmic decision-making tools, including the use of scorecards based on "hard" information, have, have become increasingly common in consumer lending (Thomas, 2000). Given the larger amount of data as well as the larger number of decisions compared to commercial credit lending, this reliance on algorithmic decision-making should not be surprising. However, the implications of these tools for risk management, for individual financial institutions and their investors, and for the economy as a whole, are still unclear.
Because effective implementation of the above risk-management strategies requires banks to be able to identify accounts that are likely to default, we build predictive models to classify accounts as good or bad. The dependent variable is an indicator variable equal to 1 if an account becomes 90 days past due (delinquent) over the next two, three, or four quarters. Independent variables include individual-account characteristics such as the current balance, utilization rate, and purchase volume; individual-borrower characteristics from a large credit bureau such as the number of accounts an individual has outstanding, the number of other accounts that are delinquent, and the credit score; and macroeconomic variables including home prices, income, and unemployment statistics. In all, we construct 87 distinct variables.
We find that this ratio ranges from less than one, implying that the bank was more likely to cut the lines of good accounts than those that eventually went into default, to over 13, implying the bank was highly accurate in targeting bad accounts. While these ratios vary over time, the cross-sectional ranking of the institutions remains relatively constant, suggesting that certain firms are either better at forecasting delinquent accounts or view line cuts as a beneficial risk-management tool.
Using these variables, we compare three modeling techniques-logistic regression, decision trees using the C4.5 algorithm, and random forest. The models are all tested out of sample as if they were being implemented at that point in time, i.e., no future data were used as inputs in these tests. All models perform reasonably well, but the decision trees tend to perform the best in terms of classification rates. In particular, we compare the models based on well-known measures such as precision and recall, and statistics that combine them such as the F-Measure and kappa statistics. 2 There is, however, a great deal of cross-sectional and temporal heterogeneity. As expected, the performance of all models declines as the forecast horizon increases.
However, the performance of the models for each bank remains relatively stable over time (we test the models semi-annually starting in 2010Q4 through the end of our sample period 2013Q4). Across banks we find a great deal of heterogeneity in classification accuracy. For example, at the two-quarter forecast horizon, the mean F-Measure ranges from 63.8% at the worst performing bank to 81.6% at the best.
We find that the decision trees and random-forest models outperform logistic regression with respect to both measures. 3 We also estimate the potential dollar savings from active risk management using these machine-learning models. The basic strategy is to first classify accounts as good or bad using the above models, and then cut the credit lines of the bad accounts. The cost savings depend on 1) the model accuracy and 2) how aggressively banks cut credit lines.
Kappa statistics show similar variability.
2 Precision is defined as the proportion of positives identified by a technique that are truly positive. Recall is the proportion of positives that is correctly identified. The F-Measure is defined as the harmonic mean of precision and recall, and is meant to describe the balance between precision and recall. The kappa statistic measures performance relative to random classification. See Figure 2 for further details.
3 These F-Measures represent the mean F-Measure for a given bank over time.
14 June 2015 Risk Management for Credit Cards Page 5 of 31 The potential cost of this strategy is cutting credit lines of good accounts, thereby alienating customers and losing future revenues. We follow Khandani, et al.'s (2010) methodology to estimate the value added of our models and report the cost savings for various degrees of line cuts (ranging from doing nothing to cutting the account limit to the current balance).
To include the cost of alienating customers, we conservatively assume that customers incorrectly classified as bad will pay off their current balances and close their accounts.
Therefore, the bank will lose out on all future revenues from such customers.
With respect to this measure, we find that our models all perform well. Assuming that cutting the lines of bad accounts would save a run-up of 30% of the current balance, we find that implementing our decision tree models would save about 55% relative to taking no action for the two-quarter-horizon forecasts. When we extend the forecast horizon, the models do not perform as well and the cost savings decline to about 25% and 22% at the three-and four-quarter horizons, respectively. These figures vary considerably across banks. The bank with the greatest cost savings had a value-added of 76%, 46%, and 35% across the forecast horizons; the bank with the smallest cost savings would only stand to gain 47%, 14%, and 9% by implementing our models across the three horizons. Of course, there are many other aspects of a bank's overall risk management program, so the quality of risk management strategy of these banks cannot be ranked solely on the basis of these results, but the results do suggest that there is substantial heterogeneity in the risk management tools and effective strategies available to banks.
The remainder of the paper is organized as follows. In Section II, we describe our dataset and discuss the security issues surrounding it and the sample-selection process used. In Section III we outline the model specifications and our approach to constructing 14 June 2015 Risk Management for Credit Cards Page 6 of 31 useful variables that serve as inputs to the algorithms we employ. We also describe the machine-learning framework for creating more powerful forecast models for individual banks, and present our empirical results. We apply these results to analyze bank risk management and key risk drivers across banks in Section IV. We conclude in Section V.

II. The Data
A major U.S. financial regulator has engaged in a large-scale project to collect detailed credit-card data from several large U.S. financial institutions. As detailed below, the data contains internal account-level data from the banks merged with consumer data from a large U.S. credit bureau, comprising over 500 million records over a period of six years. It is a unique dataset that combines the detailed data available to individual banks with the benefits of cross-sectional comparisons across banks.
The underlying data contained in this dataset is confidential, and therefore has strict terms and conditions surrounding the usage and dissemination of results to ensure the privacy of the individuals and the institutions involved in the study. A third-party vendor is contracted to act as the intermediary between the reporting financial institutions, the credit bureau, and the regulatory agency and end users at the regulatory agency are not able to identify any individual consumers from the data. We are also prohibited from presenting results that would allow the identification of the banks from which the data are collected.

A. Unit of Analysis
The credit-card dataset is aggregated from two subsets we refer to as account-level and credit-bureau data. The account-level data is collected from six large U.S. financial

B. Sample Selection
The data collection by the financial regulator for supervisory purposes started in January 2008. For regulatory reasons, the banks from which the data have come have changed over time though the total number has stayed at eight or less. However, the collection has always covered the bulk of the credit-card market. Mergers and acquisitions have also altered the population over this period.
Our final sample consists of six financial institutions, chosen because they have reliable data spanning our sample period. Although data collection commenced in January 2008, our sample starts in 2009Q1 to coincide with the start of the credit-bureau data collection. Our sample period runs through the end of 2013. 5 We are forced to draw a randomized subsample from the entire population of data because of the very large size of the data. For the largest banks in our dataset, we sample 2.5% of the raw data. However, as there is substantial heterogeneity in the size of the credit-card portfolios across the institutions, we sample 10%, 20%, and 40% from the Page 9 of 31 smallest three banks in our sample. The reason is simply to render the sample sizes comparable across banks so that differences in the amount of data available for the machine-learning algorithms are not driving the results.
These subsamples are selected using a simple random sampling method. Starting with the January 2008 data, each of the credit-card accounts is given an 18-digit unique identifier based on the encrypted account number. The identifiers are simple sequences starting at some constant and increasing by one for each account. The individual accounts retain their identifiers and can therefore be tracked over time. As new accounts are added to the sample in subsequent periods, they are assigned unique identifiers that increase by one for each account. 6 Once the account-level sample is established, we merge it with the credit-bureau data. This process also requires care because the reporting frequency and historical coverage differ between the two datasets. In particular, the account-level data is reported monthly beginning in January 2008, while the credit-bureau data is reported quarterly beginning in the first quarter of 2009. We merge the data using the link file provided by the vendor at the monthly level to retain the granularity of the account-level data. Because we merge the quarterly credit-bureau data with the monthly account-level data, each credit-As accounts are charged off, sold, or closed, they simply drop out of the sample and the unique identifier is permanently retired. We therefore have a panel dataset that tracks individual accounts through time (a necessary condition for predicting delinquency) and also reflects changes in the financial institutions' portfolios over time.
14 June 2015 Risk Management for Credit Cards Page 10 of 31 bureau observation is repeated three times in the merged sample. However, we retain only the quarter-ending months for our models in this paper.

III. Empirical Design and Models
We consider three basic types of credit-card delinquency models: C4.5 decision tree models, logistic regression, and random-forest models. In addition to running a series of "horse races" between these models, we seek a better understanding of the conditions under which each type of model may be more useful. In particular, we are interested in how the models compare over different time horizons, changing economic conditions, and across banks.
We use the open-source software package Weka to run our machine-learning models. 7 Weka offers a wide collection of open-source machine-learning algorithms for data mining. We use Weka's J48 classifier, which implements the C4.5 algorithm developed by Ross Quinlan (1993) (see, Frank, Hall, and Witten (2011)), because of its combination of speed, performance, and interpretability. This algorithm is a decision tree learner. We compare the results with those obtained using logistic regression models and random forests, also available in Weka, and include the same variables as in the decision trees.
More specifically, we use a logistic regression model with a quadratic penalty function, i.e. a ridge logistic regression. This is the Weka implementation of logistic regression as per Cessie and van Houwelingen (1992). The likelihood is expressed as the following logistic function: The objective function is ( ) In all, we have 87 attributes in the models composed of account-level, credit-bureau, and macroeconomic data. 8 We acknowledge that, in practice, banks tend to segment their portfolios into distinct categories when using logistic regression and estimate different models on each segment. However, for our analysis, we do not perform any such segmentation. Our rationale is that our performance metric is solely based on classification accuracy. While it may be true that segmentation results in models that are more tailored to individual segments such as prime vs. subprime borrowers, thus potentially increasing forecast accuracy, we relegate this case to future research. For our current purposes, the number of attributes should be sufficient to approach the maximal forecast accuracy using logistic regression. We also note that decision tree models are well suited to aid in the segmentation process, and thus could be used in conjunction with logistic regression, but again leave this for future research. 9

A. Attribute Selection
Although there are few papers in the literature that have detailed account-level data to benchmark our features, we believe we have selected a set that adequately represents current industry standards, in part based on our collective experience. Glennon et al.
(2008) is one of the few papers with data similar to ours. These authors use industry experience and institutional knowledge to select and develop account-level, credit-bureau, Page 13 of 31 and macroeconomic attributes. We start by selecting all possible candidate attributes that can be replicated from Glennon et al. (2008,  We also merge macroeconomic variables to our sample using the five-digit ZIP code associated with the account. While we do not have a long time series of macro trends in our sample, there is a significant amount of cross-sectional heterogeneity that we use to pick up macro trends.

B. Dependent Variable
Our dependent variable is delinquency status. For the purposes of this study, we define delinquency as a credit-card account greater than or equal to 90 days past due. This differs from the standard accounting rule by which banks typically charge off accounts that are 180 days or more past due. However, it is rare for an account that is 90 days past due to be recovered, and is therefore common practice within the industry to use 90 days past due as a conservative definition of default. This definition is also consistent in the literature (see, e.g., Glennon et al. (2008) and Khandani et al. (2010)). We forecast all of our models over three different time horizons-two, three, and four quarters out-to classify whether or not an account becomes delinquent within those horizons.
14 June 2015 Risk Management for Credit Cards Page 14 of 31

C. Model Timing
To predict delinquency, we estimate separate machine-learning model every six months starting with the period ending 2010Q4. 10 The optimal length of the training window involves a tradeoff between increasing the amount of training data available and the stationarity of the training data (hence its relevance for predicting future performance). We use a rolling window of two years as the length of the training window to balance these two considerations. In particular, we combine the data from the most recent quarter with the data from 12 months prior to form a training sample. For example, the model trained on data ending in 2010Q4 contains the monthly credit-card accounts in 2009Q4 and 2010Q4. The average training sample thus contains about two million individual records, depending on the institution and time period. In fact, these rolling windows incorporate up to 24 months of information each because of the lag structure of some of the variables (e.g., year over year change in the HPI), and an addition 12 months over which an account could become 90 days delinquent.
We estimate these models at each point in time as if we were in that time period, i.e., no future data is ever used as inputs to a model, and require a historical training period and a future testing period. For example, a model for 2010Q4 is trained on data up to and including 2010Q4, but no further. Table 2 defines the dates for the training and test samples of each of our models.
14 June 2015 Risk Management for Credit Cards Page 15 of 31

D. Measuring Performance
The goal of our delinquency prediction models is to classify credit-card accounts into two categories: accounts that become 90 days or more past due within the next n quarters ("bad" accounts), and accounts that do not ("good" accounts). Therefore, our measure of performance should reflect the accuracy with which our model classifies the accounts into these two categories.
One common way to measure performance of such binary classification models is to calculate precision and recall. In our model, precision is defined as the number of correctly predicted delinquent accounts divided by the predicted number of delinquent accounts, while recall is defined as the number of correctly predicted delinquent accounts divided by the actual number of delinquent accounts. Precision is meant to gauge the number of false positives (accounts predicted to be delinquent that stayed current) while recall gauges the number of false negatives (accounts predicted to stay current that actually went into default).
We also consider two statistics that combine precision and recall, the F-measure and the kappa statistic. The F-Measure is defined as the harmonic mean of precision and recall, and is meant to describe the balance between precision and recall. The kappa statistic measures performance relative to random classification. According to Khandani et al. (2010) and Landis and Koch (1977), a kappa statistic above 0.6 represents substantial performance. Figure 2 summarizes the definitions of these classification performance statistics measures in a so-called "confusion matrix".
In the context of credit-card portfolio risk management, however, there are accountspecific costs and benefits associated with the classification decision that these 14 June 2015 Risk Management for Credit Cards Page 16 of 31 performance statistics fail to capture. In the management of existing lines of credit, the primary benefit of classifying bad accounts before they become delinquent is to save the lender the run-up that is likely to occur between the current time period and the time at which the borrower goes into default. On the other hand, there are costs associated with incorrectly classifying accounts as well. For example, the bank may alienate customers and lose out on potential future business and profits on future purchases.
To account for these possible gains and losses, we use a cost-sensitive measure of performance to compute the "value added" of our classifier, as in Khandani et al. (2010), by assigning different costs to false positives and false negatives, and approximating the total savings that our models would have brought if they had been implemented. Our valueadded approach is able to assign a dollar-per-account savings (or cost) of implementing any classification model. From the lender's perspective, this provides an intuitive and practical method for choosing between models. From a supervisory perspective, we can assign deadweight costs of incorrect classifications by aggregate risk levels to quantify systemic risk levels.
Following Khandani et al. (2010), our value-added function is derived from the confusion matrix. Ideally, we would like to achieve 100% true positives and true negatives, implying correct classification of all accounts, delinquent and current. However, any realistic classification will have some false positives and negatives, which will be costly.
To quantify the value-added of classification, Khandani et al. (2010) define the profit with and without a forecast as follows: where B C is the current account balance; B D is the balance at default; P M is the profitability margin; and TP, FN, FP, and TN are defined according to the confusion matrix. Note that Eq.
[3] is broken down into a savings from lowering balances (the first term) less a cost of misclassification (the second term).
To generate a value-added for each model, the authors then compare the savings from the forecast profit (  forecast ) with the benefit of perfect foresight. The savings from perfect foresight can be calculated by multiplying the total number of bad accounts (TN + FP) by the run up (B D -B C ). The ratio of the model forecast savings (Eq. [3]) to the perfect foresight case can be written as: where we substitute [1 − (1 + ) − ] for the profitability margin, r is the discount rate, and N is the discount period.

IV. Classification Results
In this section we report the results of our classification models by bank and time.
There are on average about 6.1 million accounts each month in our sample. Table 1 shows the sample sizes over time. There is a significant amount of heterogeneity in terms of delinquencies across institutions and time (see Figure 1). Delinquency rates necessarily increase with the forecast horizon, since the longer horizons include the shorter ones. Page 18 of 31 Annual delinquency rates range from 1.36% to 4.36%, indicating that the institutions we are studying have very different underwriting and/or risk-management strategies.
We run individual classification models for each bank over time; separate models are estimated for each forecast horizon for each bank. Because our data ends in 2014Q2, we can only test the three-and four-quarter-horizon models on the training periods ending in 2012Q2 and 2012Q4, respectively. 11

A. Nonstationary Environments
A fundamental concern for all prediction algorithms is generalization, i.e., whether models will continue to perform well on out-of-sample data. This is particularly important when the environment that generates the data is itself changing, and therefore the out-ofsample data is almost guaranteed to come from a different distribution than the training data. This concern is particularly relevant for financial forecasting given the nonstationarity of financial data as well as the macroeconomic and regulatory environments.
And our sample period, which starts on the heels of the 2008 financial crisis and the ensuing recession, only heightens these concerns.
We address overfitting primarily by testing out of sample. Our decision tree models also allow us to control the degree of in-sample fitting by controlling what is known as the pruning parameter, which we refer to as M. This parameter acts as the stopping criterion for the decision tree algorithm. For example, when M = 2, the algorithm will continue to attempt to add additional nodes to the leaves of the tree until there are two instances (accounts) or less on each leaf, and an additional node would be statistically significant. As M increases, the in-sample performance will degrade because the algorithm stops even though there may be potentially statistically significant splits remaining. However, our outof-sample performance may actually increase for a while because the nodes blocked by increasing M are overfitting the sample. Eventually, however, even the out-of-sample performance degrades as M becomes sufficiently high.
To find a suitable value of M for our machine-learning models, we conduct overfitting tests on data from a select bank by varying the M parameter from 2 to 5,000.

B. Model Results
In this section we show the results of the comparison of our three modeling approaches-decision trees, logistic regression, and random forests. The random-forest models are estimated with 20 random trees. 12 To preview the results and help visualize the effectiveness of our models in terms of discriminating between good and bad accounts, we plot the model-derived risk ranking Page 20 of 31 versus an account's credit score at the time of the forecast in Figure 3 for Bank 2. Accounts are rank-ordered based on a logistic regression model for a two-quarter forecast horizon.
Green points represent accounts that were current at the end of the forecast horizon; blue points represent accounts 30 days past due; yellow points represent accounts 60 days past due; and red points represent accounts 90 days or more past due. We plot each account's credit-bureau score on the horizontal axis because it is a key variable used in virtually every consumer-default prediction model, and serves as a useful comparison to the machine-learning forecast.
This plot show that while credit scores discriminate between good and bad accounts to a certain degree (the red 90+ days past due accounts do tend to cluster to the left region of the horizontal axis with lower credit scores), the C4.5 decision tree model is very effective in rank-ordering accounts in terms of riskiness. In particular, the red 90+ days past due points cluster heavily at the top of the graph, implying that the machine-learning forecasts are highly effective in identifying accounts that eventually become delinquent. 13 Table 3 shows the precision and recall for our models. We also provide the true positive and false positive rates. The results are given by bank, time, and forecast horizon for each model type. The statistics are calculated for the classification threshold that maximizes the respective model's F-Measure to provide a reasonable balance between good precision and recall.
Although selecting a modeling threshold based on the test data does introduce some look-ahead bias, we use this approach when presenting the results for two reasons. First, banks are likely to calibrate classification models using an expected delinquency rate to 13 Analogous plots for our logistic regression and random-forest models look very similar.
14 June 2015 Risk Management for Credit Cards Page 21 of 31 select the acceptance threshold. We do not separately model delinquency rates and view the primary purpose of our classifiers as the rank-ordering of accounts. To this end, we are less concerned with forecasting the realized delinquency rates than rank-ordering accounts based on risk of delinquency. Therefore, the main role of the acceptance threshold for our purposes is for exposition and to make fair comparisons across models.
Second, the performance statistics we report-the F-measure and the Kappa There are a few noteworthy points here. First, for each bank, the optimal threshold remains relatively constant over time, which means that it should be easy for a bank to select a threshold based on past results and get an adequate forecast. Second, in the cases where the selected threshold varies over time, the lines are still quite flat. For example, in our C4.5 decision tree models in Figure A1, the optimal thresholds cluster by bank and the curves are very flat between 20% to 70% threshold values for the F-measure and the kappa statistics. For the random-forest models in Figure A3, the lines are not as flat, but the optimal thresholds tend to cluster tightly for each bank. In sum, it is important to remember that the goal of a bank would not be to maximize the F-measure in any case, and as long as the selected threshold is selected using any reasonable strategy, our sensitivity Page 22 of 31 analysis demonstrates that it would, in all likelihood, only have a minimal effect on our main results.
Each of the models achieves a very high true positive rate which is not surprising given the low default rates. The false positive rates are reasonable, between 11% and 38% for the two-quarter horizon models. However, as the forecast horizon increases, the models become less accurate and the false positive rates increase for each bank. Table 4 presents the F-Measure and kappa statistics by bank and time. As mentioned above, the F-measure and kappa statistics show that the C4.5 and randomforest models outperform the logistic regression models. The performance of the models declines as the forecast horizon increases. Figures 4 and 5 present the F-measures and kappa statistic graphically for the six banks. The C4.5 and random-forest models tend to consistently out-perform the logistic regression, regardless of the forecast horizon, for each statistic. Table 5 presents the value-added for each of the models, which represents the potential gain from employing a given model versus passive risk management. Under this metric, the C4.5 and random-forest models outperform the logistic regression models. We plot the comparisons in Figure 6 for the two quarter forecast horizon models. 14 For the two quarter forecast horizon, the C4.5 models produce an average per bank cost savings of between 45.2% and 75.5%. The random forests yield similar values, The results are similar in that the C4.5 and random-forest models outperform logistic regression. All the value-added results assume a run-up of 30% and profitability margin of about 13.5%. Page 23 of 31 between 47.0% and 74.4%. The logistic regressions fare much worse based on the bank average values because Banks 1 and 2 show two periods of negative value addedmeaning that the models did such a poor job of classifying accounts that the bank would have been better off not managing accounts at all. Even omitting these negative instances, the logistic models tend to underperform the others.
There is substantial heterogeneity in value-added across banks as well. Figure 7 plots the value added for all six banks for each model type. All models are based on a twoquarter forecast horizon. Bank 3 is always at the top of the plots meaning that our models are performing the best. Bank 4 tends to be the lowest (although still has a positive valueadded) and the other four banks cluster in between.
Moving to three-and four-quarter forecast horizons, the model performance declines and as a result the value-added declines. However, the C4.5 trees and random forests remain positive and continue to outperform logistic regression. Although the relative performance degrades somewhat, our machine-learning models still provide positive value at the longest forecast horizons.

C. Risk Management Across Institutions
In this section, we examine risk management practices across institutions. First, we compare the credit-line management behavior across institutions. Second, we examine how well individual institutions target bad accounts. In credit cards, cutting lines is a very common tool used by banks to manage their risks and one we can analyze given our dataset.
As of each test date, we take the accounts which were predicted to default over a given horizon for a given bank, and analyze whether the bank cut its credit line or not. We use the predicted values from our models to simulate the banks' real problems and avoid any look-ahead bias. In Table 6 and Figure 9 we compute the mean of the ratio of the percent of lines cut for defaulted accounts to the percent of lines cut on all accounts. A ratio greater than 1 implies that the bank is effectively targeting accounts that turn out to be bad and cutting their credit lines at a disproportionately greater rate than they are cutting all accounts, a sign of effective risk management practices. Similarly, a ratio less than one implies the opposite. 15 The results show a significant amount of heterogeneity across banks. For example, We report the ratio for each quarter between the model prediction and the end of the forecast horizon because cutting lines earlier is better if indeed they turn out to become delinquent. Figure 9 shows that three banks (2, 3, and 5) are very effective at cutting lines of accounts predicted to become delinquent-they are between 4.8 and 13.2 times more likely to target accounts predicted to default than the general portfolio. In contrast, Banks 4 and 6 underperform, rarely cut lines of accounts predicted to default. Bank 1 tends to cut the same number of good and bad accounts. There is no clear pattern to banks' targeting of bad accounts across the forecast horizon.
14 June 2015 Risk Management for Credit Cards Page 25 of 31 Of course, these results are not conclusive because banks have other risk management strategies in addition to cutting lines, and our efficacy measure relies on the accuracy of our models. However, these empirical results show that, at a minimum, risk management policies differ significantly across major credit-card issuing financial institutions.

D. Attribute Analysis
A common criticism of machine-learning algorithms is that they are essentially black boxes, with results that are difficult to interpret. For example, given the chosen pruning and confidence limits of our decision tree models, the estimated decision trees tend to have about 100 leaves. The attributes selected vary across institutions and time, hence it is very difficult to compare the trees because of their complexity. Therefore, the first goal of our attribute analysis is to develop a method for interpreting the results of our machinelearning algorithms. The single decision-tree models learned using C4.5 are particularly intuitive.
We propose a relatively straightforward approach for combining the results of the decision tree output that captures the results by generating an index based on three main criteria. We start by constructing the following three metrics for each attribute in each the attribute. The logic of the C4.5 classifier is that, in general, the higher up on the tree the attribute is (i.e., the lower the leaf number), the more important is it.
Therefore, the attributes will be sorted in reverse order; that is, the variable with the lowest mean minimum leaf number would be ranked first. This statistic is computed for each tree.
3. Indicator variable equal to 1 if the attribute appears in the tree and 0 otherwise: We combine the results of multiple models over time to derive a bank-specific attribute ranking based on the number of times attributes are selected in a given model. For example, we run six separate C4.5 models for each bank using a two quarter forecast horizon. This ranking criterion is the number of times (between zero and six) that a given attribute is selected to a model. This statistic is meant to capture the stability of an attribute over time.
We combine the above statistics into a single ranking measure by standardizing each to have a mean of 0 and standard deviation of 1 and summing them by attribute.
Attributes that do not appear in a model are assigned a score equal to the minimum of the standardized distribution. We then combine the scores for all unique bank-forecast horizon combinations and rank the attributes. This leaves us with 18 individual scores for each attribute used to rank them by importance. The most important attributes should have higher scores and appear near the top of the list and be raked lower (i.e., attribute 1 is the most important).
In all, 78 of the 87 attributes are selected in at least one model. Table 7 shows the mean attribute rankings across all models, by forecast horizon, and by bank. More important attributes are ranked lower. The table is sorted based on the mean ranking for each attribute across all 18 bank-forecast horizon pairs. Columns 2-4 show the mean ranking by forecast horizon and columns 5-10 show the mean ranking by bank.
It is reassuring that the top ranking variables-days past due, behavioral score, credit score, actual payment over minimum payment, one month change in utilization, etc.-are intuitive. For example, accounts that start out delinquent (less than 90 days) are most likely to become 90 days past due, regardless of the forecast horizon or bank.
Looking across forecast horizons, we do not see much variation. In fact, the pairwise Spearman rank correlations between the attribute rankings (for all 78 attributes that appear in at least one model) are between 89.8% and 94.3%.
However, there is a substantial amount of heterogeneity across banks, as suggested by the pairwise rank correlations between banks which range from 46.5% to 80.3%. This suggests that the key risk factors affecting delinquency vary across banks. For example, the change in one-month utilization (i.e., the percentage change in the drawdown of the credit line) has an average ranking between 2.0 and 4.0 for Banks 1, 2, and 5 but ranks between 10.3 and 15.7 for Banks 3, 4, and 6. For risk managers, this is a key attribute because managing drawdown and preventing run-up prior to default is central to managing creditcard risk. Large variation across banks in other attributes such as whether an account has entered into a workout program, the total fees, and whether an account is frozen further suggest that banks have different risk management strategies.
Overall, the results in Table 7  There is also substantial heterogeneity across banks in how macroeconomic variables affect their customers. Macroeconomic variables are more predictive for Banks 2 and 6 at a two-quarter forecast horizon, while for Bank 6, macroeconomic variables are captured as important factors at the one-year forecast horizon. The macroeconomic variables are only in the most important 20 attributes for Bank 2 and 6 in a two-quarter forecast horizon and for Bank 6 at the one-year forecast horizon. Although they are not the most important attributes, their ranking score is still relatively high and shows that the macroenvironment has a significant impact on consumer credit risk.
As mentioned above, we had also drawn the data three other times before. Using the

V. Conclusion
In this study, we employ a unique, very large dataset consisting of anonymized information from six large banks collected by a financial regulator to build and test decision-tree, regularized logistic regression, and random-forest models for predicting We also analyze and compare risk-management practices across the banks and compare drivers of delinquency across institutions. We find that there is substantial heterogeneity across banks in terms of risk factors and sensitivities to those factors.
Therefore, no single model is likely to capture the delinquency tendencies across all Our study provides an in-depth illustration of the potential benefits that big data and machine-learning techniques can bring to consumers, risk managers, shareholders, and regulators, all of whom have a stake in avoiding unexpected losses and reducing the cost of consumer credit. Moreover, when aggregated across a number of financial institutions, the predictive analytics of machine-learning models provide a practical means for measuring systemic risk in one of the most important and vulnerable sectors of the economy. We plan to explore this application in ongoing and future research.         This table shows the value added results by bank, time, and forecast horizon for each model type. The statistics are based on the acceptance threshold that maximizes the respective statistic for a given bank-time-model. Value added is defined in Eq. (4). Each value assed assumes a margin of 5% (r = 5%), a run-up of 30% ((B d -B r )/B d ), and a discount horizon of three years (N = 3). The numbers represent the percentage cost savings of implementing each model versus passive risk management. The profit margin is used to estimate the opportunity cost of a false negative so that mis-classifying more profitable accounts is more costly.  This table describes how banks manage credit lines. The numbers in the table represent the ratio of the percentage of accounts predicted to default whose credit lines were cut divided by the total percentage of accounts whose credit lines were cut. A ratio greater than one means a bank is likely actively targeting credit-card accounts to manage risk. The models are as defined above.    Tables 3-6 are combined and attributes are assigned a score based on 1) the number of instances classified, 2) the minimum leaf on each tree they appear, and 3) the number of models for which they are selected. The scores are standardized and summed to generate an importance metric for each attribute for each bank-forecast horizon pair. More important attributes are ranked lower. The table is sorted based on the mean ranking for each attribute across all bank-forecast horizon pairs. Columns 2-4 show the mean ranking by forecast horizon and columns 5-10 show the mean ranking by bank. In all, 78 of the 87 attributes were selected in at least one model. This figure shows the relative delinquency rates over time. Due to data confidentiality restrictions, we do not report the actual delinquency rates over time. Each line represents an individual bank over time. The delinquency rates are all reported relative to the bank with the lowest two quarter delinquency rate in 2010Q4.

Figure 2 Performance Statistics
The figure plots the model-derived risk ranking versus an account's credit score at the time of the forecast for Bank 2. Accounts are rank-ordered based on a logistic regression model for a two quarter forecast horizon. Green points are accounts that were current at the end of the forecast horizon; blue points are 30 days past due; yellow points are 60 days past due; and red points are 90+ days past due.     These figures plot the Value Added as defined by Eq. (4) versus run-up. The statistics plotted are for the two quarter horizon forecasts. Clockwise from the top left, the figures show the value added for C4.5 decision trees, logistic regression, and random-forest models. Note the vertical axis is cut off at -100% and the logistic regression models for bank 1, bank 2, and bank 3 are negative for low values of run-up. The figures show how well banks target bad accounts and cut their credit lines relative to randomly selecting lines to cut. The targeted line ratio is defined as the percentage of accounts that our models predict to become delinquent whose lines are cut relative to the total percentage of accounts whose lines are cut. A ratio of one (log of zero) means a bank is no more active in cutting credit lines of cards classified as bad than accounts classified as good. Higher ratios signal more active risk management. The ratios for each bank are plotted on a log scale. The plots show the ratios for each quarter following our forecast through the end of the forecast horizon. Clockwise from the top left, the figures show the value added for C4.5 decision trees, logistic regression, and random-forest models.