Pilot performance and workload whilst using an angle of attack system

.


Introduction
Loss of control in flight (LOC-I) is the main category of fatal accidents for all sectors of aviation and all aircraft types (IATA, 2019) (GAJSC and Loss of Control, 2012).A failure to maintain adequate airspeed leading to a stall, resulting in a loss of control in flight (LOC-I) event is frequently cited as the primary causal factor in National Transportation Safety Board (NTSB) accident reports involving general aviation (GA) aircraft (NTSB, 2022) (Smith and Bromfield, 2022).An aircraft may stall at any airspeed but only when the critical angle of attack (AoA) is exceeded.The stall speed (the airspeed at which a stall may occur) varies with bank angle, normal acceleration, aircraft configuration and centre of gravity.Therefore, it is not a reliable indicator of the proximity to stall or the point at which critical AoA is exceeded (Federal Aviation Administration, 2016a).In the United States, the highest proportion of GA LOC-I events occur in the circuit (or pattern) in close proximity to an airport (Hart, 2015).Between 2008 and 2014, in the United States, 35% of all LOC-I events occurred in the take-off and climb phases of flight whilst 43% occurred in the approach (straight in or circuit) and landing phases of flight.
The NTSB studies for the period 2008 to 2014 inclusive, show that the majority of LOC-I events take place within close proximity of an airport (Hart, 2015).The report identified 996 LOC-I events of which 352 were fatal, with events occurring in different flight phases of the pattern/circuit in Visual Meteorological (VMC) and Instrument Meteorological (IMC) conditions as well as straight-in approaches.(Table 1).A review of LoC-I events indicated that weather was as a contributory factor, with 25% of all LOC-I events occurred during IMC conditions (Hart, 2015).
LOC-I events, their causal and contributory factors were analysed for each segment of the circuit in VMC/IMC conditions, for events including stalls.The analysis showed that the top six most common scenarios in which a stall occurred were Engine Failure after take-off (EFATO), steep turn manoeuvre, base to finals turn, glide approach or forced landing, sideslip manoeuvre or the missed approach/go-around.
In an attempt to mitigate LOC-I events, the Federal Aviation Administration (FAA) recently relaxed the certification overhead for design and installation of AoA systems (FAA, 2014) although there is a lack of empirical and rigorous evidence to substantiate the benefits.AoA systems have been in use since the inception of powered flight.

A review of AoA literature 1.History
Angle attack systems have been used as an aid to flying since the first powered flight by the Wright brothers in 1903.Their 'angle of incidence' indicator consisted of a small vane and angle pointer attached to the outboard strut connecting the upper and lower wings (Tuomela, 1965).With the advent of high-speed heavy piston engine aircraft and jets in World War II and shortly afterwards the U.S. Navy was an early adopter of angle of attack technologies.The key driver for this was the need for controlled carrier deck landings from 1957 onwards with the majority of U.S. Navy aircraft being fitted with a variety of angle of attack measuring instruments (Gracey, 1958).Use of angle of attack and glide path indication became standard procedure for the U.S. Navy (Hurt, 1960) since AoA is not affected by gross weight, bank angle, load factor, velocity, density altitude or position errors of the airspeed indicator and high angles of attack.To complement this, the U.S. Navy adopted the attitude flying technique (Hurt, 1960), 'attitude plus power equals performance'.They noted that flying the correct AoA meant flying the correct airspeed is an essential requirement for a stable carrier approach and therefore used elevator as the primary means to control for airspeed and power setting as the primary means to control rate of descent.With regards to presentation of angle of attack, studies were conducted to assess whether or not the presentation of angle of attack information in pilot training would enhance pilot learning (Forest, 1969).The report concluded that there was no significant difference between pilot groups trained with or without angle of attack instrumentation fitted in combination with airspeed instrumentation.This indifference was attributed to the benefits of using AoA being negated by the additional learning required to use AoA and airspeed in combination.In addition, it was felt that contact (flying training) time was of more importance than ground school education time in the use of AoA.Notwithstanding these points, the study recommended further research should be conducted into the use of AoA in lieu of airspeed and for the use of AoA in instrument flight.

Previous research
In 1971 Gee et al. of NASA, conducted an in-flight evaluation of the use of an angle of attack system fitted to a modified Piper PA-30 Twin Comanche (Gee et al., 1971).The assessment was conducted to investigate safety improvements in performance and flight safety with the use of angle of attack systems using a wing-mounted vane and electronic computer unit.The first test programme focused on flying in the low speed region of flight where accident rates are notably their highest (NTSB, 1972) at that time.Limited benefits were realized and the report concluded that the use of angle of attack did not represent significant improvement in pilot performance or safety.It should be noted that the test aircraft used was a relatively low powered twin-engine aircraft, atypical of the GA fleet.It is likely that the undesirable longitudinal periodic pitching motion (phugoid), lateral/directional stability (Fink and Freeman, 1969) combined with and a substantial time lag in the display of AoA (up to 2 s), attributed to this overall assessment.The phugoid motion is a lightly damped low-frequency oscillation in speed, which couples with changes in pitch attitude and height (Cook, 2013).Hefley's study in 1983, considered pilot workload during the carrier landing whilst using AoA systems (Heffley, 1983).This this study used subjective pilot evaluation only and this challenging flying environment demanded compensatory, pursuit and pre-cognitive pilot behavior.The study suggested that pilot workload increased during the approach as a result of the phugoid mode being triggered during the turn to finals.
Experimental evaluations of the effectiveness of AoAs were conducted by Dillman et al., in 2017 as part of the FAA PEGASAS Programme initiated to enhance GA safety, accessibility, and sustainability by partnering the FAA with a national network of world-class researchers, educators and industry leaders (PEGASAS, 2022).This study within the overall research programme, attempted to evaluate the effectiveness of AoA displays at three separate, independent flight academies.The study used three different aircraft makes/models, different pilot cohorts, different airports/runways with different landing aids in diverse weather conditions (Dillman et al., 2017).Results were inconclusive, however qualitative feedback suggested that pilots felt that these systems were beneficial.The reader is directed to Bromfield & Dillman (Bromfield and Dillman, 2015), Le Vie (Le Vie, 2014) and Campbell (Campbell, 2019) for a more detailed review of AoA research and AoA effectiveness.

Current trends
Despite these apparent limitations, AoA systems have grown in popularity in the GA sector since the 1980's.With the FAA's relaxation of certification requirements in 2014 (FAA, 2014) there has been considerable growth in the number and type of AoA systems available in the GA.Each system employs different methods of sensing and presentation, highlighting a lack of standardization in the market.Education and training material in relation to the use of AoA from regulators and manufacturers is also limited.There are several different categories of design (See Table 2) (e.g.Index, Quantitative or Pictorial (Everett et al., 2018)).One of the most common formats of AoA display on the market is the 'traffic light' (Index) design.
This display uses a series of illuminated LED segment to indicate 'optimal' AoA for given flight conditions.The display uses both colour and LED segment position/layout to provide redundancy gain (similar to traffic lights) following an in flight calibration procedure.The device has limited pictorial representation and limited use of knowledge (shape/ colour).

Human factors considerations
The management of airspeed or AoA is a compensatory tracking task (McRuer and Krendel, 1965) where the pilot monitors the difference between desired and actual airspeed or angle of attack and adjusts accordingly.Flying too slow or too high an AoA requires the pilot to pitch the aircraft nose down by pushing the pitch control stick/yoke forward whereas flying too fast/too low an AoA requires nose up by pulling the control backwards (Federal Aviation Administration, 2016b).
The aircraft uses pressure probe sensor to determine airspeed and either a vane or pressure probe to determine angle of attack.Both sensors require calibration to correct for the position errors due to local airflow upwash/downwash and each sensor also has an inherent time delay due to build-up of air pressure or change in direction of vane opposing friction in the mechanism.
The pilot in the loop compensatory tracking task can be broken down into the classical information processing model (Fig. 1) (Bromfield et al., 2015) and the pilot uses the visual modality/sensory channel to scan the instrument panel and monitor airspeed/angle of attack.During a final approach for example, the pilot flies the aircraft 'heads up' and 'eyes out' to maintain an aiming point in the out of the window view of the runway, with occasional reference to the airspeed indicator or angle of attack display on the cockpit instrument panel.In most GA aircraft, the airspeed indicator uses a fixed circular scale with moving pointer to present quantitative indicated airspeed (corrected for sensor position errors).Pilot perception of the aircraft state is determined by the observed differences between desired and actual airspeed or angle of attack.Pilot decisions are determined by the observed differences which drive the required actions (primary control inputs) and the control order and system dynamics of the pilot in the loop system differs for airspeed and angle of attack.Angle of attack can be considered as zero order/position control corresponding to rotational movement of the aircraft in flight by use of the pitch control, however airspeed can be considered as first order/rate (or speed) control for linear movement of the aircraft also using pitch control.In general, the higher the control order, the more demanding the task (Bridger, 2018).This difference manifests itself as a variation of the effective time delay with control order, the higher the control order the greater the effective time delay (McRuer and Jex, 1967).
AoA displays present angle of attack directly to the pilot rather than using airspeed as a surrogate which tends to be unreliable.The direct presentation of AoA reduces time delays, enhances situation awareness providing more precise information regarding proximity to stall and achievement of optimum flight performance/flightpath by maintaining optimum angle of attack if a given flight condition.Presentation of AoA in the cockpit may require the pilot to modify their aircraft instrument scan pattern, possibly lead to an increase in workload.Anecdotal evidence, suggests that military pilots -already trained in the use of AoAconsider AoA essential to safe and efficient flying.The accuracy of an AoA system and display is dependent upon the quality of the calibration process for given aircraft configurations.An incorrectly calibrated AoA can give pilots the wrong information, degrading their situation awareness.AoA displays provide direct display of the angle of attack rather than using airspeed as a surrogate which is unreliable.

ICON A5
The aerofoil shaped needle sweeps with respect to AoA. High above; approach horizontal.

Redundancy Gain
The display utilises both colour and position for redundancy gain (similar to traffic lights).
Colour is used as a redundancy check for the needle position and numerical values.Less salient than an index display.
The display utilises colour to enhance, or check, understanding of the needle position.Less salient than an index display.

Pictorial Realism
Limited pictorial realism.The shape (and therefore direction) of the chevrons help to identify appropriate action.
No pictorial realism.The pilot has to decode the numerical and positional information to generate a mental picture of the situation.
Good pictorial realism achieved by shaping the needle.However, approach position is where one would expect cruise (wings level) flight to be.

Knowledge in the World
Limited use of knowledge in the world (shape and colour).Some conflicting use of colour consistency with respect to other displays.
Good use of knowledge in the world features such as bugs on the scale to remind the pilot of important regions such as approach.
Use of additional presentation features to denote approach region.Display is still free from clutter.

Rationale
Previous AoA studies have lacked GA context, rigorous quantitative and qualitative assessment of pilot workload, pilot performance and human factors considerations.A further study was proposed utilising one type of AoA display (Index) commonly used in GA sector to gather additional research data with respect to the effects of the simulated AoA system on pilot performance and workload using flight scenarios with simulated weather conditions to engage the pilot.The experimental hypotheses were: Hypothesis Ho 1 : • There is no change to the level of pilot performance when AoA is displayed in the cockpit for nominal and off-nominal flight scenarios in the pattern/circuit.

Hypothesis Ho 2 :
• There is no change to the level of pilot workload when AoA displayed in the cockpit for nominal and off-nominal flight scenarios in the pattern/circuit.

Methods
A series of simulated flight scenarios was developed to evaluate pilot performance and workload using a fixed-based flight simulator and simulation model of the Cirrus SR20 airplane.

Participants
Twenty pilot volunteers (Mean Age: 20.75 years) were recruited with licence types including student PPL (60%), Full PPL (25%), Selflaunched Motor Glider (5%), and Unclassified or Expired (10%).Total hours of experience ranged from 5 h to 160 h (Median = 32.5 h, Appendix A, Table A1).All pilots held a current medical certificate with normal or corrected normal vision and most frequently flew single engine piston aeroplanes (80%) or Self-launch Motor Gliders (SLMG) (20%).All data was gathered in accordance with university ethics policies and procedures (Application 121115).Data was anonymised and aggregated and securely stored using the Onlinesurveys.ac.uk (formerly BoS) system.

Equipment
A fixed-base engineering flight simulator using commercially available desktop flight simulation software (X-Plane, 2022) was used to conduct all tests (Fig. 2).The external visual environment consisted of 150 • horizontal field of view by 25 • vertical field of view using three 60" High Definition LCD/TV screens, suitably positioned for selected flight scenarios based on the circuit/traffic pattern.The simulator used an approximate replica of a Cirrus SR20 cockpit with basic instrument panel, floor mounted stick, pedals, toe brakes, and pedestal mounted throttle, brakes, flaps and elevator trim.For this study, the floor mounted stick was fitted with mechanical springs.Simulator data output comprising airspeed, angle of attack, heading, latitude/longitude and height was recorded at a frequency of 4 Hz.The low-wing, low-tail aeroplane flight dynamics model based on the Cirrus SR20 was independently validated for flight performance using actual flight test data.Two interchangeable, basic instrument panels were available: one standard and one fitted with supplementary AoA display positioned on top of the panel within the pilots forward line of sight (Fig. 3).

Experimental design
The experimental design comprised a 2 × 4 within-subjects repeated measures design.
Independent variables were AoA presentation ('on' or 'off') and flight scenario ('normal take-off', 'normal approach', 'engine failure after take-off or EFATO' and 'glide approach').The dependent variables were pilot workload using NASA-TLX and pilot performance as measured by deviation from mean AoA and airspeed and Root Mean Square Error (RMSE) AoA and airspeed during all flight scenarios.The repeated measures factor (with/without AoA display) was pseudorandomised and presented in a counterbalanced order.The flight scenarios were sequenced so that they were completed in a pseudorandomised/counterbalanced order with AoA/without AoA and by flight scenario e.g.Participant 1: Scenario 1,2,3,4 with No AoA followed by 4,2,3,1 with AoA, Participant 2: Scenario 1,2,4,3 with No AoA followed by 3,1,4,2 with AoA etc.
Pilots' flight performance was calculated from two different performance measures during flight, namely: airspeed and angle of attack during all scenarios.Variation in flight performance between the target and the actual (baseline and test flight) was elected as opposed to a zscore since z-score provides a numerical value in which a score deviates from the mean, whereas with the present data the variation is from the target flight performance measure (Harris, 1999).This is important, as the mean performance of the pilot was not under examination, however, the deviation from the target performance measure was evaluated.While a pilot can elect, within reason (e.g.aircraft limitations), to fly at any given altitude during the downwind leg of a circuit, it is not feasible Fig. 2. Experimental apparatus.
M.A. Bromfield et al. as there are strict altitude requirements for completing circuits at aerodromes which take into consideration rate of descent, in addition to the aircraft glide capabilities.Hence, for light aircraft circuit height is generally 1000 ft above ground level (AGL).Similarly, on approach or during climb, there are also limits in terms of airspeed, some of which result from the engine capacity of the aircraft, and others from the airframe design.Therefore, variations from these prescribed targets provide a more accurate assessment of pilots' abilities, based on the feedback provided.

Procedure
Simulated weather was incorporated into scenario design to provide increased realism, challenge the pilots and potentially trigger LoC-I events.Simulated crosswind for the approaches was set slightly higher than the maximum demonstrated crosswind limit for the Cirrus SR20 (21 kts) (Aircraft, 2011) (Design, 2010).Tuning of the scenarios to increase the likelihood of LOC-I accidents was achieved by iteration to ensure that limits were close to limits of aircraft and (student) pilot capabilities in the simulator environment (Appendix A, Table A2).A 25 kt crosswind for the approaches was found to be appropriate, lower limits of 20 kts were not sufficiently challenging resulting in a lower probability of LOC-I.In simulator experiments, tests can be designed to challenge pilots in a safe environment using scenarios that may result in LOC-I.
Each participant received a 10-min verbal pre-flight briefing including the use of AoA and a 10 min video briefing with respect to aircraft type (Cirrus SR20), normal and emergency procedures, cockpit controls, instrumentation, AoA display, radio-telephony communication, airfield location and weather environment.All participants were also briefed in the use of the basic, un-weighted NASA-TLX method (NASA, 2003) for the assessment of workload with each participant performing an individual flight scenario and workload assessment before progressing to the next scenario (Table 3).

Data collection
In order to assess pilot performance and workload during the proposed flight scenarios, objective measurements of performance were made using flight data output from the engineering flight simulator and    subjective measurements of pilot workload using NASA-TLX and selfassessment after each scenario.

Pilot performance measures
For each flight scenario, 51 flight variables were recorded at a sampling frequency of 4 Hz.The principal flight variables of indicated airspeed (kts), angle of attack (degrees), height above ground level (ft) and distance to touchdown (ft) were used to characterise performance in all scenarios.Data was exported in text file format, reduced, analysed and plotted using commonly used tools for flight test data analysis Excel, Matlab and/or Datplot (Bromfield and Belberov, 2016).

Workload measures
The NASA-TLX method was implemented using an GDPR and ISO 27001 compliant online survey tool for convenience and to accelerate data reduction and analysis (JISC, 2022).The subjective ratings assessment enables total workload to be derived from the mean scores of the sub-scales: mental demand, physical demand, temporal demand, own performance, effort and frustration.Weighted NASA-TLX requires more time to complete and was therefore not utilised for this series of experiments for expediency.All subjects performed all tasks with and without AoA displayed within the cockpit environment in a pseudo-randomised sequence.

Statistical analysis
Statistical analysis of the results were conducted using the statistical processing tool SPSS with a repeated-measures ANOVA (Field, 2005) and independent variables of AoA Display/No AoA Display and flight scenarios.To avoid Type I error (rejecting the null hypothesis H 0 , when it is true) significance testing was performed at p < 0.05 level using a Bonferroni correction tests were conducted with all 20 participants.The two-tailed tests were used to determine if display of angle of attack had a direct and/or statistically significant effect on pilot performance (airspeed and AoA) and workload (NASA-TLX).Objective pilot performance for all subjects (n=20) was assessed by using mean, standard deviation and RMSE for airspeed and angle of attack for each scenario.Subjective workload ratings were assessed using un-weighted NASA-TLX.

Results
Experimental results for pilot performance using simulator data output and workload using NASA-TLX for all 20 participants was reduced and analysed using statistical processes as described in the method section.

Pilot performance
Each pilot's performance was plotted for each flight scenario with and without AoA display using principal flight variables of indicated airspeed (kts), angle of attack (degrees) and height above ground level (ft).One typical example, with no AoA display executing a normal approach (Scenario 2), the angle of attack is seen to vary during the approach, reaching a maximum at the point of touchdown (Fig. 4).Indicated airspeed also varies considerably and the descent profile characterised by the variation of height with time is also seen to vary.When AoA was displayed for the same approach scenario (Fig. 5), angle of attack is seen to be steady with little variation up to the point of touchdown.Airspeed is also less variable and the descent more stable.Analysis of all pilots' performance as measured by airspeed an angle of attack management with/without AoA display is presented in the following sections.Tests for statistical significance using repeated measures ANOVA [39], sample multi-variate, within-subject tests using conservative Greenhouse-Geisser assumptions of sphericity with corrected values (p < 0.05).A summary of the statistical analysis of pilot performance is given in Appendix A, Table A3.

Indicated airspeed
During a normal take-off (Scenario 1) differences in mean airspeed were negligible with the AoA display (79.9 kts) or without the AoA display (79.2 kts) (Fig. 6, Scenario 1).Error bars plotted on all graphs represent calculated range of errors as determined by statistical analysis using SPSS.Airspeeds were slightly lower than the recommended target airspeed for normal take-off to 300' AGL of 85 kts in the Cirrus SR 20 Pilot Operating Handbook (Aviation Administration, 2011).Statistical analysis confirmed that differences in mean airspeed with/without AoA display were nonsignificant [F(1, 19) = 0.114, p < 0.05] with a result of η 2 = 0.006 indicating a small effect size.With respect to RMSE airspeed when executing a normal take-off (Fig. 7, Scenario 1) with and without AoA displayed in the cockpit the RMSE for airspeed was 4.4 kts and 6 kts respectively, however tests for statistical significance at the level of p < 0.05 showed this to be nonsignificant [F(1,19) = 2.654, p < 0.05] with a result of η^2 = 0.123 indicating a small effect size.
For the normal approach (Fig. 6, Scenario 2) with and without AoA displayed the mean airspeed was approximately 87 kts and 95 kts respectively.Airspeed when AoA was not displayed was 8 kts (9.2%) higher than with AoA displayed and statistically significant [F(1, 19) = 7.236, p < 0.05] with a result of η^2 = 0.276 indicating a small to medium effect size. .The mean airspeed was 15 kts higher than that recommended in the Pilot Operating Handbook (80 kts) when AoA was not displayed.During a normal approach (Fig. 7,Scenario 2) with and without AoA displayed the RMSE airspeed was approximately 7 kts and Fig. 4. Pilot performance: Scenario 2 -normal approach No AoA.M.A. Bromfield et al. 11 kts respectively.RMSE Airspeed when AoA was not displayed was 4 kts (57%) greater than when AoA displayed and highly significant [F(1, 19) = 9.488, p < 0.01] with a result of η^2 = 0.333 indicating a small to medium effect size.
When considering emergency procedures, the glide approach (Fig. 6, Scenario 3) with and without AoA displayed the mean airspeed was approximately 90 kts and 85 kts respectively.Airspeed when AoA was not displayed was 5 kts (5.6%) lower than with AoA displayed and statistically significant [F(1, 19) = 7.223, p < 0.05] with a result of η^2 = 0.275 indicating a small to medium effect size. .Mean airspeed was approximately 10 kts higher than that recommended in the Pilot Operating Handbook when AoA was not displayed.For emergency procedures, during the glide approach (Fig. 7, Scenario 3) with and without AoA displayed the RMSE airspeed was approximately 8 kts and 9 kts respectively and nonsignificant [F(1, 19) = 2.329, p < 0.05] with a result of η^2 = 0. 109 indicating a small effect size.

Angle of attack
For a normal take-off with and without AoA displayed in the cockpit (Fig. 8, Scenario 1) the mean angle of attack was approximately 3.8 • and differences were nonsignificant [F(1, 19) = 0.001, p < 0.05] with a result of η^2 = 0.000 indicating a small effect size..This angle of attack is within the optimum (target) range of AoA of 3.5-4.5 • as indicated by    illumination of two green 'doughnut segments of the simulated AoA display.With respect to RMSE AoA when executing a normal take-off (Fig. 9, Scenario 1) with and without AoA displayed in the cockpit the RMSE for AoA was 0.8 and 1.0 • respectively, tests for statistical significance at the level of p < 0.05 () showed this to be nonsignificant [F(1, 19) = 1.876, p < 0.05] with a result of η^2 = 0.090 indicating a small effect size.
During a normal approach with and without AoA displayed in the cockpit (Fig. 8, Scenario 2), the mean angle of attack was 3.3 and 2 • respectively.Mean AoA was 1.2 • higher when AoA was displayed, closer to the optimum AoA differences were highly significant [F(1, 19) = 13.317,p < 0.01] with a result of η^2 = 0.412 indicating a small to medium effect size. .During a normal approach (Fig. 9, Scenario 2) with and without AoA displayed the RMSE AoA was approximately 1.0 and 1.3 • respectively but differences were and nonsignificant [F(1, 19) = 3.039, p < 0.05] with a result of η^2 = 0.138 indicating a small effect size. .
In the emergency procedure glide approach, the mean AoA was 4 • when AoA was displayed and within the optimum AoA range (Fig. 8, Scenario 3).When AoA was not displayed, the mean AoA was 5 • and outside of the upper boundary of the optimum AoA range.Differences between mean AoAs with and without AoA displayed were highly significant [F(1, 19) = 12.824, p < 0.01] with a result of η^2 = 0.403 indicating a small to medium effect size..In glide approach (Fig. 9, Scenario 3) with and without AoA displayed the RMSE AoA was approximately 1.5 and 1.6 • respectively and nonsignificant [F(1, 19) = 0.003, p < 0.05] with a result of η^2 = 0.000 indicating a small effect size.

Workload (NASA-TLX)
Workload was self-assessed by each pilot and recorded following completion of each task using unweighted NASA-TLX.The sub-measures of mental demand, physical demand, temporal demand, own performance, effort, and frustration were used to determine total workload.A summary of the statistical analysis of pilot workload is given in Appendix A, Table A4.

Normal take-off (scenario 1)
During a normal take-off and initial climb (Fig. 10), moderately higher levels of mental demand (+0.3), physical demand (+0.4), temporal demand (+0.6), own performance (+0.1) were recorded with greater changes seen in effort (+1.2) and frustration remained the same when AoA was displayed.Total workload was moderately higher (+0.5) when using the AoA.However, tests for statistical significance using repeated measures ANOVA, sample multi-variate, within-subject tests using conservative Greenhouse-Geisser assumptions of sphericity with corrected values showed that differences with/without AoA display for total workload were nonsignificant [F(1,19) = 0.713, p < 0.05] with a result of η^2 = 0.036 indicating a small effect size. .All sub-measures were also nonsignificant.

Performance interactions
Since the desired flightpath requires appropriate airspeed and AoA angle of attack of an aircraft, these vary with the given phase of flight associated with the given scenario.Therefore, comparisons of interactions were considered for either the take-off or approach scenarios, hence the main interactions of interest were Normal Take-off (Scenario 1) vs Engine Failure after Take-off (Scenario 4) with/without AoA and Normal Approach (Scenario 2) vs Glide Approach (Scenario 3), with/ without AoA.For mean airspeed, highly significant interaction effects were present for Normal Approach (Scenario 2) vs Glide Approach (Scenario 3), with/without AoA display [F(1,19) = 21.435,p < 0.001] with a result of η^2 = 0.530 indicating a medium to large effect size..No other interaction effects were present.For RMSE airspeed, no interaction effects were present.For mean AoA, highly significant interaction effects were present for Normal Approach (Scenario 2) vs Glide Approach (Scenario 3), with/without AoA display [F(1,19) = 25.671,p < 0.001] with a result of η^2 = 0.575 indicating a medium to large effect size..No other interaction effects were present.For RMSE AoA no interaction effects were present.

Workload interactions
Similar to performance, only comparisons of interactions were considered for either the take-off or approach scenario being Normal Take-off (Scenario 1) vs Engine Failure after Take-off (Scenario 4) with/ without AoA and Normal Approach (Scenario 2) vs Glide Approach (Scenario 3), with/without AoA.No interaction effects were present for Total Workload or sub-measures.

Discrete events (stalls)
A number of stall events were observed whilst conducting emergency procedures during execution of the normal and emergency procedures (Table 4).In total, three stall events were recorded involving three different participants, one when the AoA was displayed and two when not displayed.During the 1st occurrence of a stall event during simulated EFATO when AoA was displayed, the pilot attempted to return to the runway in use at less than 500' AGL.The pilot was seen to enter a stall/spin from which recovery was not successful.

Discussion
Although angle of attack systems have been in use since the beginning of powered flight, there has been a lack of rigorous studies to evaluate their effectiveness using quantitative and qualitative data analysis.This study has used quantitative measurements of pilot performance and workload, supported by statistical analysis to evaluate the effectiveness of a representative, commercially available AoA display in a simulation environment for a range of scenarios with twenty pilots.Median pilot experience was 32.5 h and all student pilots were unfamiliar with AoA displays as these are uncommon in training aircraft.Therefore, they were unlikely to bring pre-conceptions as to the effectiveness or not of AoA displays.Use of pilots with significantly more flying hours (or military experience) may have skewed the results through exposure to AoA displays in non-training aircraft.
During the normal take-off and climb to 300′ AGL the presence of an AoA display had no significant effect (p < 0.05) on the management of airspeed or angle of attack and there was no significant effect on workload.Each AoA display is required to be calibrated for a given aircraft make/model, MTOW, flight phase and configuration (e.g.flaps up/down, power on/off).Manufacturers usually recommend that the AoA is calibrated for the 'optimum' (worst case) scenario and therefore the activation of the 'green circle' represents a single 'optimum' condition, usually the approach with flaps down, idle power and slow, decelerated stall (1 kt/s).The AoA in the climb for a typical light aircraft is slightly lower than the approach, therefore likely to provide less than optimal performance.
For the normal approach, significant differences (p < 0.05) were observed when the AoA display was not present.Airspeeds were significantly higher (+8 kts) and subject to more variation (+4 kts).The  mean airspeed was 15 kts higher than the recommended final approach speed of 80 kts but only 5 kts higher when AoA was displayed.This is consistent with U.S. Navy findings and adoption of the attitude flying technique for stable carrier landings (Hurt, 1960).At lower airspeeds in order to maintain constant lift, higher AoA is required, consistent with known theory.Large variations in airspeed and rate of descent are normally associated with unstable approaches and these were observed for a number of pilots when the AoA display was not present.With respect to workload, mental demand (+0.8) and temporal demand (+0.9) were significantly higher when AoA was displayed.It is recognised that the approach and landing phase of flight is the most task intensive phase of flight and the lack of familiarity with AoA concepts and display systems may have contributed to this perceived increase in mental/temporal demand, that said, differences in total workload were nonsignificant.
During the glide approach when AoA was displayed, the mean airspeed was significantly higher and the mean AoA was significantly lower (p < 0.05), as observed with the normal approach scenario.When AoA was displayed, the angle of attack was within the optimum range (3.5-4.5 • ) and this was not the case when AoA was not displayed.Mental demand (+1.1), temporal demand (+1.3) and total workload (+0.8) were significantly higher (p < 0.05) when AoA was displayed.This is probably due to the combination of lack of familiarity with the AoA display and tasks demands of a time critical emergency landing/ glide approach.
The engine failure after take-off scenario showed no significant difference (p < 0.05) in pilot performance when AoA was displayed in the cockpit, however, mental demand was significantly higher (+1.2) and total workload also higher (+0.5) when AoA was displayed.
Analysis of previous fatal accidents showed that glide approaches/ forced landings and EFATO were two of the six most common scenarios in which stall events occurred leading to loss of control in flight.Experimental results in this study indicate that an appropriately calibrated AoA display could provide benefit not only in the case of glide approaches/forced landings but also for normal approaches resulting in perhaps, more stable approaches reducing the likelihood of loss of control in flight.Contrary to earlier studies by Forrest (Forest, 1969), this study has shown that the limited education and training in the use of AoA can yield benefits.
During the experiments, three stall events occurred, one during a glide approach when AoA was not displayed.Two more occurred during EFATO one with AoA displayed and one without.During the former, the pilot entered a stall/spin whilst attempting a turnback to the airfield following engine failure at low altitude above ground level.This highlights the need for appropriate pilot decision making during time critical emergency situationssafety technologies are of little use if inappropriate decisions are made.
With respect to the AoA display position, this was placed within line of sight/forward view out of the cockpit window of the pilot.This is likely to have encouraged 'heads up' and 'eyes out' flying in the normal and glide approaches and may have further contributed to performance improvements with minor increases in workload.

Conclusions
This study has used quantitative measurements of pilot performance and workload (n=20), supported by statistical analysis to evaluate the effectiveness of a representative, commercially available AoA display in a simulation environment for a range of scenarios with twenty pilots.
Experimental results have shown that pilot performance is improved when AoA is displayed in the cockpit for nominal and off-nominal flight scenarios during the approach phase of flight in the pattern/circuit.The null hypothesis, Ho 1 is therefore rejected for the given AoA display during this phase of flight only.
In relation to pilot workload, results have shown that during the approach phase of flight, there is a moderate but tolerable increase in pilot workload and the null hypothesis, Ho 2 is rejected for this phase of flight in nominal and off-nominal flight scenarios in the pattern/circuit.
The use of such an AoA display may assist pilots to maintain angle of attack within the optimum range and hence reduce occurrences of unstable approaches.The use of an AoA display during emergency scenarios may have significant effects on total workload, mental and temporal demand if the pilot has not been adequately educated and trained to use it.Differences training should be considered to ensure safe use of AoA displays during normal and emergency procedures.This study was limited to the evaluation of a 'traffic light' AoA design/layout, calibrated for the 'worst case' scenario of stall warning.Different calibrations are required for different phases of flight when stalls might occur, where the 'optimum' angle of attack may be different (e.g.climb out).

Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Fig. 3 .
Fig. 3. Flight simulator cockpit panel with simulated AoA display (top left of panel).
Notes: *Normal, acceptable flying tolerance for the pilot of a Single Engine Piston (SEP) light aircraft in the traffic pattern (Aviation Administration, 2011).
M.A.Bromfield et al.

Fig. 6 .
Fig. 6.Effect of AoA display on mean airspeed by scenario.
M.A.Bromfield et al.

Fig. 9 .
Fig. 9. Effect of AoA Display on RMSE Angle of Attack by Scenario.
ExampleAlpha Systems Eagle The display segments illuminate with respect to AoA. High above; approach centre.Teledyne Avionics Dial format, needle sweeps counter clockwise with respect to percentage total AoA.

Table 3
simulated flight scenarios.

Table 4
Number of stall events detected.

Table A3
Summary of Statistical Analysis of Pilot Performance Results (p < 0.05)

Table A4
Summary of Statistical Analysis of Pilot Workload Results (p<0.05)