A multicriteria model for assessing item importance and risk using operational data from military supply chains

1. Introduction

The Defense Logistics Agency (DLA) uses over 6,000 different vendors as sources of supply for a variety of commodities, ranging from fresh fruit to parts for aircraft engines. During periods of supply chain disruption (e.g. due to a natural disaster), analysts must filter the list of potentially affected vendors to identify specific risks to logistics operations and then assess those risks for military service effects. The DLA manages millions of items, each identified by its national item identification number (NIIN). While vendors may have different risk profiles, ultimately it is the items that they supply that are critical to the military services. Thus, an understanding of how risk is associated with items supplied is a key concern. This paper describes an approach to address the need for measuring the risk and importance associated with items (NIINs) stored within a military supply chain.

Using a multiobjective decision analysis (MODA) methodology, risk and importance indices were developed for the analysis of items. In the proposed approach, the modeler must identify operational elements (e.g. items) and related operational metrics. These metrics are then classified as contributing to either risk or importance within the supply chain. Then, the analyst develops value functions and weights for each attribute to translate the raw operational data into risk and importance indices that have a common interpretable scale. The proposed methodology could also be used within commercial supply chains, subject to the identification of operationally relevant data sources that are useful within those settings.

The paper is organized into sections, providing a brief literature review, an overview of the underlying methodology and risk assessment metrics and an illustrative application of the methodology to item risk. We provide some conclusions about future areas of investigation in the final section.

2. Literature review

This section presents an overview of the literature that informed methodological decisions. The purpose of the review is to identify problem elements, metrics and models that were useful in meeting the research goals and methods. A key requirement for a potential methodology is that the required data elements should be readily available without significant effort or that the required model inputs can be readily created via informed domain experts or analysts.

Christopher and Peck (2004) notes, “as supply chains become more complex as a result of global sourcing, …, supply chain risk increases”. The paper concludes that a key challenge is to “manage and mitigate that risk through creating more resilient supply chains”. Henry and Ramirez-Marquez (2012), defines resilience as a function of time using a quantitative framework based on a disruptive event, followed by system disruption, a disrupted state, system recovery and finally moving to a stable recovery state. Military acquisition planners need to account for system behavior over time under conditions of disruption when making sourcing decision (e.g. supply selection, back up suppliers, etc.) and inventory purchasing over time. What has become clear is that the idea of treating the supply chain as a complex adaptive system, see the seminal paper by Choi et al. (2001), is even more applicable today than it was over 20 years ago.

The study of supply chain risk has been a central area of research and practice for decades. A seminal paper by Sheffi (2001) lays the groundwork for identifying some of the key issues to consider when trying to mitigate risk within the operation of a supply chain. Specifically, the key idea of the paper is that while a first-order event (i.e. a terrorist attack) might have initiated the disruption, it is often the secondary response to the event that needs to be considered in the planning. For example, in both the response to the September 11, 2001 event and the response to the pandemic, the direct drivers of major disruptions to supply chains were not the event itself but rather governmental responses to the event.

Building on these concepts, Rice and Sheffi (2005) introduce the concept of resilience within supply chain management. In this view, supply chain management should consider not only reducing the vulnerability to disruptions but also strategically planning for the reduction of the time that it takes to return to normal operations. Designing for resiliency recognizes that the likelihood of a disruption cannot be totally eliminated and thus, designing the supply chain for recovery should be a key overall strategic goal. To be able to incorporate risk into the supply chain design process, we need to be able to measure risk in a quantifiable manner.

Thus, a focus within the early 2000s within the supply chain literature concerned how to measure risk, what factors to include during the risk assessment process and how to formulate mathematical models as part of the risk assessment process. Gaonkar and Viswanadham (2007) present a framework for assessing the risk within supply chains. It describes the risk management problem and the need to address the problem at the three major levels of the enterprise: strategic, operational and tactical. Gaonkar and Viswanadham (2007) identify sources, consequences, drivers and mitigation strategies as the basic constructs within supply chain risk management. It classifies supply chain risk problems into three main areas: deviation (change from the expected), disruption (radical transformation leading to nonavailability) and disaster (temporary irrecoverable shutdown of the supply chain). Finally, Gaonkar and Viswanadham (2007) propose an integer quadratic programming model for supply chain partner selection. The objective of the model was “to choose suppliers such that the expected shortfall in supply, in the face of supplier disruptions, is minimized.” Thus, it should be clear that understanding risks within a supply chain requires considering how vendors and the items they produce contribute to the risk profile. The approach proposed in this paper focuses on quantifying risk based on operationally extracted data on items and vendors.

While it is beyond the scope of this article to provide a comprehensive review of the supply chain risk management literature, it is beneficial to note that considerable effort has been taken to understand the drivers and responses to risk within the literature. For a more substantial review of the literature, the interested reader is encouraged to review the following survey papers: Ho et al. (2015) present a comprehensive literature review of research from 2003 to 2013 within the area of supply chain risk management. During this time frame, the primary areas of investigation included defining risk, classifying types of risk and identifying supply chain risk factors. Some of these risk factors are essential to include in the proposed risk model of this paper. Heckmann et al. (2015) provide a comprehensive review of the literature and subsequent quantitative methods applied within the area of supply chain risk management. It finds that supply chain risk sources (network, process, etc.), exposure and characteristics are vital perspectives when considering supply chain vulnerability. The complexity of some of these papers limits their application in practice. Fan and Stevenson (2018) provide a review of over 354 articles from the 2000–2016 time frame and find that the bulk of the research concentrates on identifying risk types and formulating mitigation strategies. It identifies ten potential research gaps within the literature, which overall suggest the need for holistic models, better monitoring, benchmarking, cost-benefit analysis and a better understanding of how position within the supply chain affects risk. One significant item to note is that the supplier’s position within the supply chain is called out as needing additional research.

In the following, we focus on papers that contributed key insights supporting our modeling approach. In particular, Loredo et al. (2015) motivated a solution approach involving the use of operational data. Dong (2006) develops the concept of “robustness” to describe the ability of a supply chain network to “carry out its functions despite some damage done to it.” Dong (2006) presents a system-wide approach to quantifying the robustness index of supply chain networks. The approach considers both network structural robustness and network functional robustness when a supply chain is faced with disruption, disaster, contingency and terrorism. Dong (2006) measures the efficiency of the network, where efficiency “attempts to measure the effectiveness of the network configuration to accomplish its function.” The model represents the network as a graph of links and nodes, where nodes represent locations in the network (e.g. distribution, production, assembly and storage) and links represent some interaction between the nodes. The approach uses the connectedness of the graph to construct various measures. In general, the methodology represents a static representation at a particular point in time. This representation permits an analysis of weak links within the network and their potential effects. At the time of the writing of the paper, the methodology had not been tested within a realistic case study. A possible shortcoming of the work is the amount of structural data that is required and how to represent the data within a computer. This shortcoming makes the approach difficult to apply to a network as complex as DLA’s supply chain, with all its connections and the varying nature of the connections over time. However, the idea of taking a system-wide approach to building a robustness index contributed to our methodology.

Kinra et al. (2020) describe a methodology to assess high-impact, low-frequency events within a supply chain. A high-impact, low frequency event is an event that is so rare that the determination of its likelihood of occurrence is effectively guesswork. Kinra et al. (2020) ignore the problem of estimating the chance of the event and instead focus on determining the vulnerability of the supply chain to unknown events. The methodology provides a heuristic for risk based on the worst-case scenario analysis, with the time of disruption as an input parameter. The methodology requires estimates of disruption time and determines the maximum loss to operations due to the disruption. The computation of loss is based on every disrupted node in the supply chain and the total loss determined across the supply chain. The heuristic shows the parts of the supply chain where it may be possible to have large losses. The goal is to identify these weak links to focus efforts to mitigate the losses. The methodology is illustrated on a small, three-level supply chain involving suppliers, a warehouse and a customer. Possible shortcomings of the work include that the disruption’s impact is measured at a particular instant and the lack of modeling for partial recoveries or temporary supplies. The methodology also suffers from similar issues as Dong (2006) in that the data to represent the network may be difficult to obtain. In addition, the methodology assumes a simplistic accounting of disruption costs; however, due to its simplicity, it may more readily lend itself to what-if analysis.

Ziegenbein and Baumgart (2006) present a quantitative approach to measuring the probability of occurrence and the financial impact of disruptions in a supply chain. The proposed methodology allows managers to gather information about the magnitude and risk of disruptions to a supply chain. The goal is to assist businesses in estimating the value/cost associated with the disruption and quantifying its impact. The methodology uses a probabilistic analysis rather than a subjective response. The main inputs to the methodology are the number of suppliers and the number of sourced items demanded per supplier. The main outputs include the business interruption value and its impact on the expected gross margin. The authors analyze a number of scenarios involving problems such as forecast error (minor), product quality (medium) and fire/flooding (severe). The shortcomings of the work include the number of assumptions that must be made to model business operations, such as ordering lot-for-lot and inventory operating policies. The focus on business interruption value makes it more difficult to translate the model to a military supply chain context.

Gaudenzi and Borghesi (2006) describe an application of the analytical hierarchical process (AHP) in identifying and managing risk within a supply chain. The process evaluates daily risks to supply chain integrity, such as on-time delivery, completed orders, correct orders, damaged products, etc. These and other metrics are used to measure and evaluate suppliers. The paper proposes a tool for managers to assess a supplier based on past performance and available data using the AHP methodology. The analysis starts by identifying specific risks (damaged products) and assigning causes and priorities to each risk. Then, the causes are examined to identify the origin of the risk. The approach produces a set of metrics and options for managing the risks. Since the approach is based on AHP, there is a significant interaction required to provide input for the model, and the modeling approach is primarily subjective in nature. Also, the analysis requires significant work if there are a large set of vendors. An advantage of the methodology is that it results in a risk (index) associated with each supplier.

Fritsche (2016) presents an Excel-based tool that facilitates the assessment of risks within a supply chain. The approach identifies a set of areas that may contribute to risk and allows managers and domain experts to subjectively grade each area based on best practices used within those areas to control, mitigate or contribute to risk. The tool includes five risk areas: managerial, sourcing, warehousing/transportation, inventory/production and information systems. Questions in each of the five areas are answered during the assessment process. Each area is weighted as to its contribution to risk. The scores for each area’s questions are tabulated and can be compared to those of other organizations. The approach allows for stakeholders to evaluate their supply chain’s risk landscape with minimal quantitative requirements in a relatively short time span. The approach facilitates self-awareness of the practices and issues relative to how operations affect supply chain risk and may help organizations identify areas for improvement. The shortcoming of the work is that it is entirely subjective; however, the approach is simple, facilitates what-if analysis and can be quickly performed because it does not require significant data collection effort.

Moore and Loredo (2013) describe the results of an analysis for the U.S. Air Force to identify supply chain risks. The main result is a comprehensive list of the risks that could affect the Air Force’s supply chain. The report does not detail a model, but instead focuses on listing risks and their supporting factors. The main risks identified are due to natural disasters, acts of war, terrorism/sabotage and accidents. In addition, some environmental operating risks are also identified, such as business environment (taxes, customs, currency devaluation, lawsuits, economic recessions, labor, strikes, etc.), market environment and technological uncertainty. One highlighted area for supply chains is the flow of raw materials, specifically the fact that China is the main producer of aluminum, which is a key raw material in aircraft production. Furthermore, the report summarizes a review of weapon systems and case studies to compare how risk was managed in the past. The findings of the report are useful for the identification of sources of risk and providing guidance on what to consider during a risk assessment.

Loredo et al. (2015) present a methodology for measuring risk within Army supply chains for items and vendors. The approach uses the existing operational data to form risk indices. It defines a risk index as “a heuristic score for (1) the likelihood that vendor could fail to supply the item and (2) a heuristic score for measuring the consequences on Army weapon systems” (Loredo et al., 2015, p. 9). The model includes various operational factors that may influence risk. The risk indices measure overall risk for a vendor by summing all risk values for items that vendors supply. The study was limited by the availability of data (NIIN and vendor characteristics) within the existing operational systems. The work led to an automated tool that can be used on a periodic basis to query the existing data, execute the risk index computations and summarize the analysis of risk based on factors of interest.

Loredo et al. (2015) develops risk measures based on factors within the supply chain. The main areas include demand fluctuation, funding uncertainty and long-lead times. It defines NIIN risk as

NIIN risk measures individual items within the overall analysis and is used to “create a weight for a NIIN on the system.” NIIN risk is a component of computing an overall system (supply chain) risk. The report summarizes the various data sources used to compute the components of NIIN risk. The risk scores are standardized between 0 and 1. Vendor failure risk does not represent a probability but rather another heuristic score that is constructed from additional factors related to vendors.

As can be noted from the VendorFailRisk equation, the conceptualization of vendor risk is functionally related to status, revenue, contract characteristics and risks associated with the location of the vendor as related to natural disasters. Notice that each of the factors has an equal weight (9 factors, divided by 9). The definition of the individual components is provided in Loredo et al. (2015).

In summary, the reviewed literature involves three major approaches: (1) quantitative, (2) qualitative and (3) blended quantitative/qualitative methods. The quantitative methods involve the application of supply chain analysis methods, sometimes utilizing mathematical programming or graph theoretic constructs. The qualitative methods focus on identifying the management process and issues that are salient in understanding supply chain risk. The blended methods tend to utilize qualitative methods for structuring the elements and quantitative methods for developing indices that attempt to measure supply chain risk. Based on the review, we concluded that an approach that blends quantitative analysis with the ability to include subjective (qualitative) factors could be a useful approach, especially if it could be driven by operational data. Thus, we focused on the approach in Loredo et al. (2015).

A compelling aspect of the approach in Loredo et al. (2015) is the ability to compute risk by item and vendor and “roll up” the analysis of risk across various factors. However, there are possibly very serious issues with the mathematical basis for the approach taken to develop the individual risk indices. The first issue is that the values of the individual components are not standardized on a common scale. Secondly, the metrics assume that the individual factors contribute equally to the overall risk index. Because of these two issues, we utilize the theory of MODA to avoid the issues of equal weighting and a lack of common domains and scales. The next section presents the application of MODA to the development of risk indices.

3. Overview of methodology and metrics

This section provides an overview of the basic components and steps within the MODA methodology applied to the risk index development process. The standard MODA methodology is laid out in Parnell et al. (2013). The key step of this process that is applicable to this research is crafting the decision objectives and value measures (or value function hierarchy). This process provides stakeholders with a common framework for identifying and quantifying the most relevant factors in the decision.

An initial step in the MODA process is to identify attributes and risk/importance measures. Within the MODA methodology, a value function hierarchy (VFH) is the major tool to structure multiple characteristics into independent and nonoverlapping groups of criteria. The VFH represents a value tree that encapsulates the attributes that are important to the decision context. Figure 1 illustrates a notional value hierarchy tree for assessing item risk. This tree has only one level, but in general there can be multiple levels, each contributing upwards to the overall measure of the decision context.

As illustrated in Figure 1, the attributes associated with item characteristics can serve as the traditional measures for the application of the MODA methodology. The values of the attributes (as observed from operational data) serve as the raw scores that will be normalized to a common measurement scale by the specification value functions. The value functions represent a mapping from raw scores to a scale (0–100) that represents the attribute’s contribution to the value represented by the tree. In this research, we develop two value function hierarchies: one for item risk and another for item importance. Each hierarchy will result in an overall index, one for item risk and one for item importance. Once the value function hierarchies for item risk and item importance are developed, the value functions for the attributes of each hierarchy are specified.

Each value function has an x-axis and a y-axis, where the x-axis is the scale of the raw score (e.g. production lead time) and the y-axis is a unit-less value measure on the scale from 0 to 100. Continuous value functions typically follow four basic shapes: linear, concave, convex and S-curve, as illustrated in Figure 2. Depending on the impact of each value measure, value functions could be either monotonically increasing or decreasing, as indicated in Figure 2. As suggested in Kirkwood (1997), the shape of value functions is determined by consulting with subject matter experts. Once the general shape is determined, the experts identify the increase/decrease in value from a specific incremental increase in the measure scale. Repeating this multiple times up to the maximum on the measure scale produces a piecewise linear function. The functions illustrated in Figure 2 were produced in a linear piecewise fashion.

Each attribute has raw score, xi based on its natural scale, which is standardized to a common scale by applying a value function, v(xi,), where

• xi is the raw score of measure i on the x-axis of the value function and

• v(xi) is the value of measure i on the y-axis of the value function.

Attribute values are combined into an overall value using weights and an additive model. In this research, the combined overall value is called an index. A higher overall value indicates more risk (or importance) for the item for the given attributes, weights, value functions and raw scores.

The weights depend on both the criticality of the attribute and the impact of varying the score of attributes. A swing weight matrix is one of the well-known methods to determine the weights. This method assesses measure weights by “swinging” the attribute score from its worst to its best. Parnell and Trainor (2009) discuss this method in detail with examples. There are various ways, besides the swing weight matrix, to elicit weights from stakeholders (Clemen and Reilly, 2014; Kirkwood, 1997). After the value functions have been specified, the attribute criteria can be ranked and the results recorded within a swing weight matrix so that an overall index can be formed.

To summarize, using the MODA methodology as outlined here enables the evaluation of the risk based on multiple characteristics extracted from operational data sources. The process starts with specifying a value function hierarchy (i.e. selecting the component (item) measures). After the VFH has been specified, data are collected and normalized, and then mapped to indices (for risk and importance). Finally, an analysis can be completed based on the results. In this step, the scores for the supply chain component (item) can be analyzed. The purpose of this step is to guide further work by identifying assumptions, criteria, parameters and factors that are significant to the overall analysis of risk. The risk or importance index value is a single numerical score, and the output of an additive value function model is based on the value functions and the swing weights applied to each supply chain component (item). After performing an analysis of the risk and importance indices for various supply chain grouping criteria, a sensitivity analysis of the weights and scores can be performed.

We can gain valuable insights by performing a sensitivity analysis on the elements of the MODA model and observing how the results change. Weights and value functions are the elements that can be subjected to sensitivity analysis to gain further insights on the riskiest entities and their influence on overall supply chain risk. The application of the MODA methodology to the building of risk indices mitigates one of the problems associated with the approach suggested within Loredo et al. (2015) by providing a rational and well-established basis for combining measures that have different scales and values within the context of measuring supply chain risk. In the following section, we provide guidance for analysts to select and interpret value functions.

4. Illustrative application

In this section, we apply the MODA model to example item data for the DLA supply chain. The purpose of this section is illustrative in nature that is, the results are based on an assumed item risk model for the purpose of illustrating the construction of the model and its internal computations.

First, we provide some illustrative implementation details related to data integrity. This illustration should inform potential users of the methodology on the significance of effective data capture in the model. In addition, these details illustrate some of the potential effects of inaccurate and incomplete data on the interpretation of the results. For fields used in the prototype tool, we filtered the raw numeric data for missing, negative, zero, positive or other values. This allows the user to assess the quality of the raw score data supporting the item indices before transformation by the value functions. A straightforward statistical summary of the raw numeric fields is useful. This information supports a better understanding of the distribution, mean and variance of the raw scores. For categorical data a frequency analysis is useful. This analysis can highlight invalid or missing data values. Based on this input analysis, the analyst can decide to exclude or subset the data to separately analyze the results based on the factors identified during the data quality analysis.

Next, after assessing the quality of the raw data, computing the risk and importance indices can proceed.

Finally, we complete a MODA post-processing analysis to understand the risk and importance indices within the context of their item populations. We examine the distribution of the risk and importance indices across the sample DLA item population. For vendors, we would examine the risk and importance indices across the DLA vendor population.

5. Summary and future work

Using the MODA methodology, a supply chain analyst can identify operational elements (e.g. items, vendors, etc.) and their associated operational metrics and classify the metrics as attributes contributing to either risk or importance within the supply chain. Then, the analyst can supply inputs such as value functions and categorical rankings of attributes to translate the raw operational data into risk and importance indices that have a common interpretable scale (i.e. between 0 and 100). From the computed MODA indices, the analyst can identify those elements that contribute the most and the least to operational risk. The MODA index approach generalizes readily to other supply chain elements, such as vendors. The element indices can be made available to decision-makers using operational dashboards. In addition, while the model was illustrated for DLA data, the application of the methodology is general and could be readily applied to other military services’ supply chains.

Because the indices are computed from operational data, a future extension of this research should investigate how often the data used within the analysis should be refreshed. Some of the data elements used to define risk and importance indices may not vary significantly with respect to time, while other data elements may change on a regular basis. In addition, the population of potential elements may change over time. For example, some items are phased out. From a methodological standpoint, there is no distinction made between elements that change regularly based on operational conditions and those that do not. This does not limit the methodology, but it does indicate that the effectiveness of the results should be monitored over time. The tracking of a history of indices over time may be of potential benefit. This would be a natural extension to the dashboard concept.

One additional possibility for future extension of the proposed methodology is examining how to combine the indices into an overall index. First, the methodology does address this issue in a limited way by suggesting a combined risk and importance index; however, the methodology suggested here could be improved in a number of ways. For example, a weighted overall index could be constructed. In addition to combining risk and importance, since the methodology can provide an index for any supply chain component, a natural issue would be how to combine those indices into an overall supply chain index. The MODA approach to developing multilevel value hierarchies could be useful in exploring this extension. This would be consistent with the overall approach and the MODA theory proposed in this paper.