Predicting success in United States Air Force pilot training using machine learning techniques

Highlights

• This research examines specialized undergraduate pilot training candidate data.

• Machine learning techniques are utilized to obtain insights on candidate success.

• Results show tree-based machine learning techniques outperform neural networks.

• This work informs future selection criteria for pilot training candidates.

Abstract

The chronic pilot shortage that has plagued the United States Air Force over the past three years poses a national-level problem that senior military members are working to overcome. Unfortunately, not all pilot candidates successfully complete the necessary training requirements to become fully qualified Air Force pilots, which wastes critical time and resources and only further exacerbates the pilot shortage problem. Therefore, it is important for the Air Force to carefully consider whom they select to attend pilot training. This research examines historical specialized undergraduate pilot training (SUPT) candidate data leveraging a variety of machine learning techniques to obtain insights on candidate success. Computational experimentation is performed to determine how selected machine learning techniques and their respective hyperparameters affect solution quality. Results reveal that the extremely randomized tree machine learning technique can achieve nearly 94% accuracy in predicting candidate success. Additional analysis indicates degree type and commissioning source are the most important features in determining candidate success. Ultimately, this research can inform the modification of future SUPT candidate selection criteria and other related Air Force personnel policies.

Introduction

United States Air Force (USAF) senior leaders have repeatedly expressed concerns about the service's pilot shortage. In 2017, then Secretary of the Air Force (SECAF), Heather Wilson, disclosed that the military branch was 2,000 pilots below sustainable manning levels. She further stated that this shortfall was poised to “break the force” [1]. The situation has not improved in the intervening years; if anything, it has grown more dire. As of early 2020, USAF senior leaders reported the shortfall had increased to 2,100 pilots [2]. This research seeks to mitigate this shortfall by reducing pilot candidate attrition during training. The goal is to determine what, if any, factors make a pilot candidate more likely to successfully complete their training, and ultimately to use this knowledge to influence future pilot selection processes.

The genesis of the current pilot shortfall can, in part, be traced back to the sequestration-driven federal budgets of 2013–2015. Unpredictable funding levels prohibited implementation of a forward-looking personnel policy and led to the limited contemporary population of junior pilots. The USAF depends on younger pilots to replace senior pilots. This allows senior pilots to serve in administrative staff positions wherein they gain headquarters (i.e., corporate-level) experience and are generally provided a reprieve from frequent deployments. The contemporary pilot shortfall has instead forced senior pilots to remain in the cockpits, reduced the pilot presence on headquarters' staffs, and increased fears that large numbers of senior pilots will leave the service due to the fatigue caused by their operational commitments. This has caused a ripple effect throughout the USAF, disrupting the operations of other career fields as they surge to fill the manning void left on headquarters’ staffs.

Correcting the pilot shortage requires a multi-faceted approach. In 2020, then United States Air Force Chief of Staff (CSAF) General David Goldfein acknowledged that there is no “silver bullet” that will solve the problem [2]. A concentrated effort on several fronts to increase recruiting, reduce training attrition, improve training timeline efficiency, and enhance retention are all necessary efforts to ameliorate the USAF pilot shortage. The USAF has already instigated efforts on many of these fronts. Air Combat Command is revamping fighter pilot training attempting to streamline the current 40 month pipeline [3]. The Air Force Personnel Center has offered retention bonuses to senior pilots in exchange for a multi-year service commitment [4]. The United States Air Force Academy has sent an increasing number of its graduates into the pilot training pipeline [5]. Notably, Air Education and Training Command has made a concentrated effort to improve Specialized Undergraduate Pilot Training (SUPT), the initial skills training for pilots in the USAF [6]. However, efforts to reduce training attrition without sacrificing quality standards are less well-developed. Such is the focus of this study.

Given the extensive time and monetary commitments required to train a pilot [7], a candidate failure during training represents a significant loss to the USAF and is counterproductive in the attempt to reduce the current pilot shortfall. Therefore, in this study we examine how modern machine learning techniques may be used to build a classifier that helps to predict candidate success in SUPT. By doing so, we present a method to reduce SUPT attrition as a component of the larger endeavor to reduce the USAF pilot shortfall.

In adopting this perspective, our research extends the application of machine learning in human resource and educational tasks to the military environment. In the civilian sector, machine learning has been applied to hiring practices [8], job performance prediction [9], employee retention [10], and recruitment [11]. [12,13] provide thorough literature reviews in this regard. As is the case in numerous machine learning tasks, neural networks take a prominent role in human resource applications (e.g., Refs. [14,15]. Other modern techniques such as deep learning [16] and generative adversarial networks [17] are also beginning to take hold.

Educational data mining (EDM) is another closely related field that leverages machine learning on educational data to improve pedagogical practice [18,19]. It is broadly defined and encompasses analysis that empowers the decision making of myriad actors (e.g., teachers, course developers, administrators, etc.). Relevant applications include the design of e-learning education systems [20,21], the analysis of pedagogical strategies [22], student performance prediction [23,24], and personalized feedback dissemination [25]. [26] reviewed contemporary EDM applications finding that most machine learning algorithms can be leveraged; additionally, the authors provide a summary of how each algorithm is specifically employed. For further information on this field of study, we refer an interested reader to the comprehensive surveys by Refs. [27,28].

Despite these advancements, machine learning for human resource management or pedagogical applications in the military is much less developed; a limited number of examples exist in these areas leveraging operations research techniques of any variety (e.g., Refs. [[29], [30], [31], [32]]. The relative dearth of research is noteworthy. Although related, military personnel and training systems are significantly different from their civilian counterparts; they are insular systems wherein all training and promotion occur endogenously.

Nevertheless, the DoD is increasingly looking to artificial intelligence and machine learning to increase the efficacy and efficiency of its processes [33]. With specific regard to recruiting, practitioners have recently applied machine learning to identify promising candidates for special operations forces [34]. Our research illustrates how such an approach can be extended to USAF pilot selection, another resource and time expensive training pipeline. The methods utilized herein are not necessarily meant to supplant human expertise, but rather to supplement it. That is, the machine learning algorithms examined provide decision support to humans tasked with selecting USAF pilot candidates that have the highest probability of successfully graduating. This research also illustrates how insights from the models developed herein can be used for alternative military personnel tasks (e.g., training pipeline monitoring and human resource allocation). In so doing, our research takes initial steps towards adapting the varied insights drawn from machine learning in civilian human resource management and education to the military setting.

The remaining sections of this manuscript are structured as follows. In Section 2, we provide an overview of the USAF pilot training pipeline, the empirical basis for current application testing, and an overview of recent SUPT attrition statistics. In Section 3, we provide an empirical exploration of numerous machine learning techniques to identify which subset are most promising for the development of an effective classifier. In Section 4, we illustrate the efficacy of each technique, provide experimental results relating to hyperparameter selection, and determine the relative importance of the data features. Notably, we demonstrate counter-intuitive behavior regarding the superior performance of relatively simpler techniques over others generally considered more capable. In Section 5, we discuss the relevance of our quantitative findings to USAF senior policymakers, propose a composite pilot selection index, and enumerate potential policy recommendations. Finally, in Section 6, we provide closing remarks and directions of future inquiry.