Skip to main content
SpringerLink
Log in

Controversy in human factors constructs and the explosive use of the NASA-TLX: a measurement perspective

  • Original Article
  • Published:
Cognition, Technology & Work Aims and scope Submit manuscript

A Commentary to this article was published on 20 May 2014

A Commentary to this article was published on 20 May 2014

A Commentary to this article was published on 20 May 2014

Abstract

Situation awareness and workload are popular constructs in human factors science. It has been hotly debated whether these constructs are scientifically credible, or whether they should merely be seen as folk models. Reflecting on the works of psychophysicist Stanley Smith Stevens and of measurement theorist David Hand, we suggest a resolution to this debate, namely that human factors constructs are situated towards the operational end of a representational–operational continuum. From an operational perspective, human factors constructs do not reflect an empirical reality, but they aim to predict. For operationalism to be successful, however, it is important to have suitable measurement procedures available. To explore how human factors constructs are measured, we focused on (mental) workload and its measurement by questionnaires and applied a culturomic analysis to investigate secular trends in word use. The results reveal an explosive use of the NASA Task Load Index (TLX). Other questionnaires, such as the Cooper Harper rating scale and the Subjective Workload Assessment Technique, show a modest increase, whereas many others appear short lived. We found no indication that the TLX is improved by iterative self-correction towards optimal validity, and we argue that usage of the NASA-TLX has become dominant through a Matthew effect. Recommendations for improving the quality of human factors research are provided.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Adams S (1998) Practical considerations for measuring Situational Awareness. In: Proceedings for the third annual symposium and exhibition on situational awareness in the tactical air environment, Piney Point, MD, pp 157–164

  • Boring EG, Stevens SS (1936) The nature of tonal brightness. Proc Natl Acad Sci 22:514–521

    Article  Google Scholar 

  • Bridgman PW (1927) The logic of modern physics. MacMillan, New York

    Google Scholar 

  • Byers JC, Bittner AC, Hill SG (1989) Traditional and raw task load index (TLX) correlations: are paired comparisons necessary? In: Mital A (ed) Advances in industrial ergonomics and safety 1. Taylor and Francis, London, pp 481–485

    Google Scholar 

  • Campbell NR (1920) Physics: the elements. Cambridge University Press, Cambridge

    Google Scholar 

  • Campbell NR, Jeffreys H (1938) Symposium: measurement and its importance for philosophy. Proc Aristot Soc Suppl Vol 17:121–151

    Google Scholar 

  • Dawes RM, Smith TL (1985) Attitude and opinion measurement. In: Lindzey G, Aronson E (eds) The handbook of social psychology. Random House, New York, pp 509–566

    Google Scholar 

  • Dekker SWA (2013) On the epistemology and ethics of communicating a Cartesian consciousness. Saf Sci 56:96–99

    Article  Google Scholar 

  • Dekker S, Hollnagel E (2004) Human factors and folk models. Cogn Technol Work 6:79–86

    Article  Google Scholar 

  • Dekker SW, Woods DD (2002) Maba-maba or abracadabra? progress on human-automation co-ordination. Cogn Technol Work 4:240–244

    Article  Google Scholar 

  • Dekker S, Nyce JM, Van Winsen R, Henriqson E (2010) Epistemological self-confidence in human factors research. J Cogn Eng Decis Mak 4:27–38

    Article  Google Scholar 

  • Dickinson J, Byblow WD, Ryan LA (1993) Order effects and the weighting process in workload assessment. Appl Ergon 24:357–361

    Article  Google Scholar 

  • Ferguson A, Myers CS, Bartlett RJ, Banister H, Bartlett FC, Brown W et al (1938) Quantitative estimates of sensory events: interim report. Br Assoc Adv Sci 108:277–334

    Google Scholar 

  • Ferguson A, Myers CS, Bartlett RJ, Banister H, Bartlett FC, Brown W et al (1940) Quantitative estimates of sensory events. Adv Sci 1:331–349

    Google Scholar 

  • Flach JM (1995) Situation awareness: proceed with caution. Hum Factors 37:149–157

    Article  Google Scholar 

  • Hancock PA (1996) Effects of control order, augmented feedback, input device and practice on tracking performance and perceived workload. Ergonomics 39:1146–1162

    Article  Google Scholar 

  • Hancock PA, Szalma JL (2004) On the relevance of qualitative methods for ergonomics. Theor Issues Ergon Sci 5:499–506

    Article  Google Scholar 

  • Hand DJ (1996) Statistics and the theory of measurement. J R Stat Soc A 159:445–492

    Article  Google Scholar 

  • Hand DJ (2004) Measurement: theory and practice. Arnold, London

    Google Scholar 

  • Hand DJ (2005) Size matters—how measurement defines our world. Significance 2:81–83

    Article  MathSciNet  Google Scholar 

  • Hand DJ, Keynes M (1993) Letter to the Editor: Velleman PF, Wilkinson L (1993) Nominal, ordinal, interval, and ratio typologies are misleading. Am Stat 47, 65–72: Comments by Huberty and Hand; rejoinder by Velleman and Wilkinson. Am Stat 47:314–315

  • Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload, North Holland Press, Amsterdam, pp 139–183. http://humanfactors.arc.nasa.gov/groups/TLX/downloads/NASA-TLXChapter.pdf

  • Hendy KC, Hamilton KM, Landry LM (1993) Measuring subjective workload: when is one scale better than many? Hum Factors 35:579–601

    Google Scholar 

  • Hill SG, Iavecchia HP, Byers JC, Bittner AC Jr, Zaklade AL, Christ RE (1992) Comparison of four subjective workload rating scales. Hum Factors 34:429–439

    Google Scholar 

  • Ioannidis JP (2012) Why science is not necessarily self-correcting. Perspect Psychol Sci 7:645–654

    Article  Google Scholar 

  • Kantowitz BH (1992) Selecting measures for human factors research. Hum Factors 34:387–398

    Google Scholar 

  • Khurshid A, Sahai H (1993) Scales of measurements: an introduction and a selected bibliography. Qual Quant 27:303–324

    Article  Google Scholar 

  • Liu YL, Wickens CD (1994) Mental workload and cognitive task automaticity: an evaluation of subjective and time-estimation metrics. Ergonomics 37:1843–1854

    Article  Google Scholar 

  • Lord FM (1953) On the statistical treatment of football numbers. Am Psychol 8:750–751

    Article  Google Scholar 

  • Luce RD, Suppes P (2002) Representational measurement theory. In: Pashler H, Wixted J (eds) Stevens’ handbook of experimental psychology, vol 4. Methodology in experimental psychology, 3rd edn. Wiley, New York, pp 1–41

  • Marcus-Roberts HM, Roberts FS (1987) Meaningless statistics. J Educ Behav Stat 12:383–394

    Article  Google Scholar 

  • Marks LE (2006) S.S. Stevens: a brief scientific biography. In: Fechner Day 2006: Proceedings of the twenty-second annual meeting of the international society for psychophysics, St. Albans, England

  • Mather M, Cacioppo JT, Kanwisher N (2013) Introduction to the Special Section: 20 Years of fMRI—what has it done for understanding cognition? Persp Psychol Sci 8:41–43

    Article  Google Scholar 

  • Merton RK (1968) The Matthew effect in science. Science 159:56–63

    Article  Google Scholar 

  • Michel J-B, Shen YK, Aiden AP et al (2010) Quantitative analysis of culture using millions of digitized books. Science 331:176–182

    Article  Google Scholar 

  • Miller GA (1974) Stanley Smith Stevens: 1906–1973. Am J Psychol 87:279–288

    Google Scholar 

  • Moroney WF, Biers DW, Eggemeier FT, Mitchell JA (1992) A comparison of two scoring procedures with the NASA Task Load Index in a simulated flight task. In: Proceedings of the IEEE national aerospace and electronics conference 2, Dayton, OH, pp 734–740

  • NASA (1986) Task Load Index (NASA-TLX). v. 1.0. Paper and pencil package (instruction manual). NASA Ames Research Center, Moffett Field, CA. http://humanfactors.arc.nasa.gov/groups/TLX/downloads/TLX.pdf

  • NASA (2014) Task Load Index paper and pencil version. NASA Ames Research Center, Moffett Field, CA. http://humansystems.arc.nasa.gov/groups/TLX/downloads/TLXScale.pdf

  • Nygren TE (1991) Psychometric properties of subjective workload measurement techniques: implications for their use in the assessment of perceived mental workload. Hum Factors 33:17–33

    Google Scholar 

  • Parasuraman R, Sheridan TB, Wickens CD (2008) Situation awareness, mental workload, and trust in automation: viable, empirically supported cognitive engineering constructs. J Cogn Eng Decis Mak 2:140–160

    Article  Google Scholar 

  • Reid GB, Shingledecker CA, Eggemeier FT (1981) Application of conjoint measurement to workload scale development. Proc Hum Factors Ergon Soc Annu Meet 25:522–526

    Google Scholar 

  • Rubio S, Díaz E, Martín J, Puente JM (2004) Evaluation of subjective mental workload: a comparison of SWAT, NASA-TLX, and Workload Profile methods. Appl Psychol 53:61–86

    Article  Google Scholar 

  • Siegel S (1956) Nonparametric statistics for the behavioral sciences. McGraw-Hill, New York

    MATH  Google Scholar 

  • Stevens SS (1946) On the theory of scales of measurement. Science 103:677–688

    Article  MATH  Google Scholar 

  • Stevens SS (1959) Measurement, psychophysics and utility. In: Churchman CW, Ratoosh P (eds) Measurement: definitions and theories. Wiley, New York, pp 18–63

    Google Scholar 

  • Stevens SS (1961) To honor Fechner and repeal his law. Science 133:80–86

    Article  Google Scholar 

  • Teghtsoonian R (2001) S. S. Stevens. In: Smelser NJ, Baltes PB (eds) International encyclopedia of the social and behavioral sciences 22:15104–15108. Pergamon, Oxford

  • Townsend JT, Ashby FG (1984) Measurement scales and statistics: the misconception misconceived. Psychol Bull 96:394–401

    Article  Google Scholar 

  • Underwood BJ (1957) Psychological research. Prentice-Hall, Englewood Cliffs, NJ

    Google Scholar 

  • Uttal WR (2008) Time, space, and number in physics and psychology. Sloan Publishing. http://j.b5z.net/i/u/2084689/f/Online_Time_Space_Number.pdf

  • Velleman PF, Wilkinson L (1993) Nominal, ordinal, interval, and ratio typologies are misleading. Am Stat 47:65–72

    Google Scholar 

Download references

Acknowledgments

I thank Dr. Dimitra Dodou for useful comments and for helping to create Table 3.

Author information

Authors and Affiliations

  1. Department BioMechanical Engineering, Faculty of Mechanical, Maritime and Materials Engineering, Delft University of Technology, Mekelweg 2, 2628 CD, Delft, The Netherlands

    J. C. F. de Winter

Authors
  1. J. C. F. de Winter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to J. C. F. de Winter.

Additional information

This original paper is discussed in the commentaries available at: doi:10.1007/s10111-014-0276-0; doi:10.1007/s10111-014-0277-z; doi:10.1007/s10111-014-0278-y.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

de Winter, J.C.F. Controversy in human factors constructs and the explosive use of the NASA-TLX: a measurement perspective. Cogn Tech Work 16, 289–297 (2014). https://doi.org/10.1007/s10111-014-0275-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10111-014-0275-1

Keywords

Search

Navigation