Skip to main content
SpringerLink
Log in

Selection or Training? A Two-Fold Test of the Validity of Job-Analytic Ratings of Trainability

  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

Decisions to use selection devices or training are typically based on subject matter expert (SME) judgments concerning the trainability of job components, including tasks and knowledge, skills, abilities, and other characteristics (KSAOs). The studies reported here examined the validity of SME trainability judgments from two perspectives. We found that (1) different SME groups' judgments related differently to actual learning changes in KSAOs covered in a training program and (2) sources of variability thought to be unrelated to the job influenced trainability ratings. Implications for the selection of SMEs in job analysis applications intended to determine the trainability of various job components are outlined.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

REFERENCES

  • Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Monterey, CA: Brooks-Cole.

    Google Scholar 

  • Arvey, R. D., Davis, G. A., McGowen, S. L., & Dipboye, R. L. (1982). Potential sources of bias in job analytic processes. Academy of Management Journal, 25, 618–629.

    Google Scholar 

  • Arvey, R. D., Passino, E. M., & Lounsbury, J. W. (1977). Job analysis results as influenced by sex of incumbents and sex of analyst. Journal of Applied Psychology, 62, 411–416.

    Google Scholar 

  • Brief, A. P., Burke, M. J., George, J. M., Robinson, B. S., & Webster, J. (1988). Should negative affectivity remain an unmeasured variable in the study of job stress. Journal of Applied Psychology, 73, 193–198.

    Google Scholar 

  • Brief, A. P., & Motowidlo, S. J. (1986). Prosocial organizational behaviors. Academy of Management Review, 10, 710–725.

    Google Scholar 

  • Burke, M. J., Brief, A. P., & George, J. M. (1993). The role of negative affectivity in understanding relations between self-reports of stressors and strains: A comment on the applied psychology literature. Journal of Applied Psychology, 78, 402–412.

    Google Scholar 

  • Cain, P. S., & Green, B. F. (1983). Reliability of selected ratings available from the Dictionary of Occupational Titles. Journal of Applied Psychology, 68, 155–165.

    Google Scholar 

  • Chen, P. Y., & Spector, P. E. (1991). Negative affectivity as the underlying cause of correlations between stressors and strains. Journal of Applied Psychology, 76, 398–407.

    Google Scholar 

  • Cornelius, E. T., Denisi, A. S., & Blencoe, A. G. (1984). Expert and naive raters using the PAQ: Does it matter? Personnel Psychology, 37, 453–464.

    Google Scholar 

  • Cornelius, E. T., & Lyness, K. S. (1980). A comparison of holistic and decomposed judgment strategies in job analysis by job incumbents. Journal of Applied Psychology, 65, 155–163.

    Google Scholar 

  • Danserau, F., Jr., Graen, G., & Haga, W. J. (1975). A vertical dyad linkage approach to leadership within formal organizations: A longitudinal investigation of the role making process. Organizational Behavior and Human Performance, 13, 46–78.

    Google Scholar 

  • Ford, J. K., & Noe, R. (1987). Self-assessed training needs: The effects of attitudes towards training, managerial level, and function. Personnel Psychology, 45, 511–527.

    Google Scholar 

  • Ford, J. K., Smith, E. M., Sego, D. J., & Quiñones, M. A. (1991). Impact of task experience and individual factors on training-emphasis ratings. Journal of Applied Psychology, 78, 583–590.

    Google Scholar 

  • Friedman, L., & Harvey, R. J. (1986). Can raters with reduced job descriptive information provide accurate position analysis questionnaire (PAQ) ratings? Personnel Psychology, 39, 779–789.

    Google Scholar 

  • Ganzach, Y. (1994). Theory and configurality in expert and layperson judgment. Journal of Applied Psychology, 79, 439–448.

    Google Scholar 

  • Gerhart, B. (1988). Sources of variance in incumbent perceptions of job complexity. Journal of Applied Psychology, 73, 154–162.

    Google Scholar 

  • Goldstein, I. (1986). Training in Organizations. Pacific Grove, CA: Brooks/Cole.

    Google Scholar 

  • Green, S. B., & Stutzman, T. (1986). An evaluation of methods to select respondents to structured job analysis questionnaires. Personnel Psychology, 39, 543–564.

    Google Scholar 

  • Hackman, J. R., & Oldham, G. R. (1975). Development of the Job Diagnostic Survey. Journal of Applied Psychology, 60, 159–170.

    Google Scholar 

  • Harvey, R. J. (1991). Job analysis. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of Industrial and Organizational Psychology, v. 2, chpt. 7. Palo Alto: CPP.

    Google Scholar 

  • Harvey, R. J., & Wilson, M. A. (1994). Assessing the redundancy between dichotomous versus multipoint job analysis scales: The critical role of the does-not-apply rating. Paper presented at the convention of the Society for Industrial and Organizational Psychology, Nashville.

  • Hattie, J., & Cooksey, R. W. (1984). Procedures for assessing the validities of tests using the “known-groups” method. Applied Psychological Measurement, 8, 295–305.

    Google Scholar 

  • Hughes, G. L., & Prien, E. P. (1989). Evaluation of task and job skill linkage judgments used to develop test specifications. Personnel Psychology, 42, 283–292.

    Google Scholar 

  • James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98.

    Google Scholar 

  • James, L. R., & Jones, A. P. (1980). Perceived job characteristics and job satisfaction: An examination of reciprocal causation. Personnel Psychology, 33, 97–135.

    Google Scholar 

  • James, L. R., & Tetrick, L. E. (1986). Confirmatory analytic tests of three causal models relating job perceptions to job satisfaction. Journal of Applied Psychology, 71, 77–82.

    Google Scholar 

  • Kozlowski, S. W. J., Kirsch, M. P., & Chao, G. T. (1986). Job knowledge, ratee familiarity, conceptual similarity, and halo error: An exploration. Journal of Applied Psychology, 71, 45–49.

    Google Scholar 

  • Kruglanski, A. W. (1989). The psychology of being “right”: The problem of accuracy in social perception and cognition. Psychological Bulletin, 106(3), 395–409.

    Google Scholar 

  • Kunzo, J. P., & York, C. M. (1994). Field data on the assessment of incumbent expertise in job analysis. Paper presented at Society for Industrial and Organizational Psychology, Nashville.

  • Landy, F. (1993). Job analysis and job evaluation: The respondent's perspective. In H. Schuler, J. L. Farr, & M. Smith (Eds.), Personnel selection and assessment. Individual and organizational perspectives (pp. 75–90). Hillsdale, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87, 72–107.

    Google Scholar 

  • Landy, F. J., & Vasey, J. (1991). Job analysis: The composition of SME samples. Personnel Psychology, 44, 27–50.

    Google Scholar 

  • Levine, E. L. (1983). Everything you always wanted to know about job analysis. Tampa: Mariner.

    Google Scholar 

  • Mathieu, J. E., Farr, J. L., Shankster, L. J., Landy, F. J., & Kohler, S. S. (1994). An examination of the convergence between incumbents' and managers' job analysis ratings. Paper presented at Society for Industrial and Organizational Psychology, Nashville.

  • Morgeson, F. P., & Campion, M. A. (1997). Social and cognitive sources of potential inaccuracy in job analysis. Journal of Applied Psychology, 82, 627–655.

    Google Scholar 

  • Mumford, M. D., Weeks, J. L., Harding, F. D., & Fleishman, E. A. (1987). Measuring occupational difficulty: A construct validation against training criteria. Journal of Applied Psychology, 72, 578–587.

    Google Scholar 

  • Murphy, K. R. (1991). Criterion issues in performance appraisal research. Organizational Behavior and Human Decision Processes, 50, 45–50.

    Google Scholar 

  • Murphy, K. R., & Cleveland, J. N. (1995). Understanding Performance Appraisal. Thousand Oaks, CA: Sage.

    Google Scholar 

  • O'Connor, E. J., & Barret, G. V. (1980). Informational cues and individual differences as determinants of subjective perceptions of task enrichment. Academy of Management Journal, 23, 697–716.

    Google Scholar 

  • Riggs, M. L. (1989). The development of self-efficacy and outcome scales for general applica tions. Paper presented at the Annual Conference of The Society for Industrial and Organizational Psychology, Boston, MA.

  • Sanchez, J. I., & Fraser, S. L. (1992). On the choice of scales for task analysis. Journal of Applied Psychology, 77(4), 545–553.

    Google Scholar 

  • Schmitt, N., & Cohen, S. A. (1989). Internal analyses of task ratings by job incumbents. Journal of Applied Psychology, 74, 96–104.

    Google Scholar 

  • Schneider, B. (1987). The people make the place. Personnel Psychology, 40, 437–453.

    Google Scholar 

  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.

    Google Scholar 

  • Smith, J. E., & Hakel, M. D. (1979). Convergence among data sources, response bias, and reliability and validity of a structured job analysis questionnaire. Personnel Psychology, 32, 677–693.

    Google Scholar 

  • Society for Industrial and Organizational Psychology (1987). Principles for the Validation and Use of Personnel Selection Procedures. Author.

  • Spector, P. E., & Jex, S. M. (1991). Relations of job characteristics from multiple data sources withe employee affect, absence, turnover intentions, and health. Journal of Applied Psychology, 76, 46–53.

    Google Scholar 

  • Spector, P. E., Jex, S. M., & Chen, P. Y. (1994). Relations of incumbent affect-related personality traits with incumbent and objective measures of characteristics of jobs. Journal of Organizational Behavior, 15, 1–7.

    Google Scholar 

  • Speilberger, C. D., Gorsuch, R. L., & Lushene, R. E. (1970). Manual for the State-Trait Anxiety Inventory. Palo Alto, CA: Consulting Psychologists Press, Inc.

    Google Scholar 

  • Staw, B. M., Bell, N. E., & Clausen, J. A. (1986). The dispositional approach to job attitudes: A lifetime longitudinal test. Administrative Science Quarterly, 31, 56–77.

    Google Scholar 

  • Staw, B. M., & Ross, J. (1985). Stability in the midst of change: A dispositional approach to job attitudes. Journal of Applied Psychology, 70, 469–480.

    Google Scholar 

  • Stone, E. F. (1979). Field independence and perceptions of job characteristics: A laboratory investigation. Journal of Applied Psychology, 64, 305–310.

    Google Scholar 

  • Sulsky, L. M., & Balzer, W. K. (1988). Meaning and measurement of rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology, 73, 497–506.

    Google Scholar 

  • Taber, T. D., & Taylor, E. (1990). A review and evaluation of the psychometric properties of the Job Diagnostic Survey. Personnel Psychology, 43, 467–500.

    Google Scholar 

  • Viteles, M. J. (1932). Industrial psychology. New York: Norton.

    Google Scholar 

  • U.S. Department of Labor, Manpower Administration (1972). Handbook for analyzing jobs. Washington, D.C.: U.S. Government Printing Office.

    Google Scholar 

  • U.S. Department of Labor (1991). Dictionary of occupational titles (4th ed., revised). Washington, D.C.: U.S. Government Printing Office.

    Google Scholar 

  • Williams, L. J., Gavin, M. B., & Williams, M. L. (1996). Measurement and nonmeasurement processes with negative affectivity and employee attitudes. Journal of Applied Psychology, 81, 88–101.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Southwest Missouri State University, Springfield, MO, 65804

    Robert G. Jones

  2. Florida International University, USA

    Juan I. Sanchez, Marie Williams & Steve White

  3. Southwest Missouri State University, USA

    Gowri Parameswaran, Jeanne Phelps & Carol Shoptaugh

Authors
  1. Robert G. Jones
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juan I. Sanchez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gowri Parameswaran
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jeanne Phelps
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Carol Shoptaugh
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marie Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Steve White
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jones, R.G., Sanchez, J.I., Parameswaran, G. et al. Selection or Training? A Two-Fold Test of the Validity of Job-Analytic Ratings of Trainability. Journal of Business and Psychology 15, 363–389 (2001). https://doi.org/10.1023/A:1007804815480

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1007804815480

Search

Navigation