For decades, there has been a quiet murmur of existential discontent within industrial-organizational (I-O) psychology. This has taken many forms, such as calls to mind the science-practice gap (Briner & Rousseau, Reference Briner and Rousseau2011), expressions of concern over the usefulness of I-O psychology’s general approach to science (Highhouse & Zickar, Reference Highhouse and Zickar1997), and calls to increase our influence on and efforts to improve the world at large (Maynard & Ferdman, Reference Maynard and Ferdman2009). Despite decades of commentary encouraging actions to address these concerns, little has changed, and this murmur has in recent years become a bit louder and more insistent, in part because the increasingly rapid pace of technological change, the changing nature of work itself, has made these weaknesses more problematic, more destructive, and more obvious. In short, we are poised to plunge headfirst into our own obsolescence.
In this chapter, my first goal is to explain how we reached this point by describing five key threats to I-O psychology that set us up for this dive. My second goal is to describe some troubling outcomes of these threats so far, to more clearly illustrate why these threats must be addressed. To summarize these outcomes, I-O practice has pulled far ahead of academia in terms of technological expertise, yet in an absolute sense, neither practice nor academia are particularly current or competitive in terms of their understanding of or approach to technology. Third, I provide a list of four recommendations that I believe will turn us toward a better path, one which fully embraces an interdisciplinary future for our field.
1.1 A Perfect Storm for Irrelevance
Some of the threats to I-O psychology I will next describe were created by I-O itself, or more specifically, its culture and common practices, whereas other threats reflect market conditions or the realities of the technological world we now find ourselves in. I will describe these threats in an order of increasing compounding; in other words, each reason is made worse by the reasons that came before it, and in combination, they may be lethal.
1.1.1 Threat #1: Developing Theory for Its Own Sake Is Popular but Not Typically Useful
Numerous I-O researchers over the past decade have noted that I-O psychology literature is becoming more oriented toward an unusual and harmful type of theory development (e.g., Campbell & Wilmot, Reference Campbell, Wilmot, Ones, Anderson, Viswesvaran and Sinangil2018). To illustrate, consider Table 1.1, which contains a list of titles of articles published in the Journal of Applied Psychology from 2018 Issue 1 alongside those published in 1988 Issue 1, thirty years earlier. Even a brief study of this table reveals a noticeable priority shift. Whereas 1988 articles develop measures, investigate effects, and compare methods, 2018 articles are more likely to present theories, test models, and propose mediators. Importantly, my listing of these titles is not to somehow shame or minimize the contributions of either set of researchers or their findings; instead, I use this to illustrate just how abstract and theory-oriented much published I-O psychology research has now become in relation to the I-O psychology of yesteryear. If you have been staying current on the I-O literature, this also should not be at all surprising.
Table 1.1 Seven most recent studies across three journals
Note. JAP = Journal of Applied Psychology; IJHCS = International Journal of Human-Computer Studies
So what might be less obvious to I-Os is that this idea, that the purpose of research is to propose theory, puts our field not only in contrast to the historical roots of I-O psychology but also to virtually all research literatures on I-O–related technologies created outside our field. In contrast to I-O theory-building research, technology and the way it is typically researched is highly concrete. In the third column of Table 1.1, I have added a list of recent articles from a respected outlet in the field of human-computer interaction (HCI), an interdisciplinary field that falls at the intersection point between psychology and computer science. In that column, you will find much of the same language of 1988 JAP, with lots of measuring, evaluating, and exploring, yet relatively few papers concerning theory as an overarching goal. A cynical traditionalist might interpret this to mean that HCI is 30 years behind I-O, whereas a futurist might interpret it to mean that HCI’s increasing popularity must be driven by this applied focus. The truth, as usual, is likely somewhere in the middle. At the very least, this difference reflects a real mismatch between the typical goals of technologists and the typical goals of (publishing) I-O psychologists.
1.1.2 Threat #2: Research on Technology as Yet-More-Stimuli is Artificially Limiting
In the classic language of psychology, technologies are stimuli. They are designed by humans to realize an intended purpose, but once they exist and are in use, they are inherently part of the situations in which people find themselves. People make decisions regarding how to interact with those technologies, or they react as those technologies are forced upon them. Unfortunately, psychology has historically considered and defined its stimuli quite poorly (Gibson, Reference Gibson1960). This is most obvious in social psychology, where even today, stimuli are often developed for use in a single study without extensive pilot testing to ensure that those stimuli are in fact valid representations of whatever they are intended to represent. This might be attributed to the focus of the field; psychology is, as evidenced by its own name, primarily the study of people’s mental states and not the things happening to those people. But such a simple treatment belies the complexity of the world in which people exist. Lewin (Reference Lewin1936) already knew this when he stated, “Every psychological event depends upon the state of the person and at the same time on the environment, although their relative importance is different in different cases” (p. 12). Despite many calls since that time to better integrate both the person and the situation (Ekehammar, Reference Ekehammar1974), it remains a challenge even today.
When researchers adopt this classic stance, consciously or not, they limit the types of questions that they ask of technology and the approaches they take to studying it. In psychology, such researchers typically default to a stance in which technology takes the form of a well-defined and specific cause, something to either be manipulated by an experimenter or passively recorded in a correlational study, evidenced by research questions like, “Do mobile devices harm measurement?” The reality of technology’s relationship with people is more complex, which is recognized explicitly in other fields. For example, in a highly influential article in the field of Management Information Systems, Orlikowski (Reference Orlikowski1992) presented a non-recursive model of workplace technology in which people create and change technology, technology in turn influences organizational policies and norms, and those policies and norms in turn influence how people treat technology; additionally, the technology itself changes how people work, as shown in Figure 1.1. This is a much more flexible and useful approach to studying technology than the simple and uninformative meta-research question “what does technology do to people?” pervasive in psychology and management, the existence of which is in part caused by Threat #1.
Figure 1.1 Orlikowski (1992) model of workplace technology
Additionally, due to this limited view of technology, specific technologies are often ill defined and misapplied. Grawitch, Winton, Mudigonda, and Buerck (Reference Grawitch, Winton, Mudigonda and Buerck2017) made this argument convincingly and phrased in a way relatable to psychologists: “technology is more than just error” (Grawitch et al., Reference Grawitch, Winton, Mudigonda and Buerck2017). Importantly, this operationalization of misapplication is not unique to I-O psychology; for example, in media psychology, which is a field that studies the effects of various technologies on human psychology as its primary purpose, researchers still appear to have a significant bias toward investigating psychological concerns instead of technological ones (Reeves, Yeykelis, & Cummings, Reference Reeves, Yeyekelis and Cummings2016). In short, because we are trained as psychologists, it is seductive to focus on psychology alone in our research. In the modern world, this approach is often not particularly useful.
To remain relevant, we need to be active, integrative, and increasingly interdisciplinary. In contrast to this charge, psychology’s mind-set about technology is generally passive, reactive, and siloed. It encourages researchers to sit back and wait until technologies are implemented, often wreaking some degree of havoc upon the world; only when the dust has settled does it become appropriate to begin sifting through what has happened and try to make sense of it. This is, furthermore, reinforced by Threat #1, because one needs to be a passive observer to develop a theory that is only to be tested with confirmatory hypothesis testing, an approach in stark contrast to the natural sciences, where pushing the boundaries of knowledge through invention and discovery are the raison d’être. When is the last time you recall an academic I-O psychologist inventing something new, trustworthy, and immediately useful to practitioners? Although there are a few examples (e.g., De Corte, Sackett, & Lievens, Reference De Corte, Sackett and Lievens2011), they are rare, scattered, and tend to fall on the “industrial” side of I-O. It does not need to be this way.
1.1.3 Threat #3: Both Psychology and Technology Are Moving Targets, but Technology Is Worse
The most common epistemology among modern social scientists is likely post-positivism. Many I-O psychologists are not aware of this philosophy of science underlying their research, so I shall take a moment to explore it. Post-positivism, in brief, asserts that there is some “true” state of the world. In statistical terms, these are populations, and within those populations, various relationships, both causal and correlational, are true. So for example, perhaps in the true world, conscientiousness is indeed an emergent state of a person’s brain that affects how they behave. We can never know this “true” world; instead, we must make inferences about it via observation, data collection, and statistical tests. Given certain assumptions, we can state with some degree of confidence that our observations in our own world reflect this true world. If I were to stop there, I would be describing the most common philosophical framework behind most modern natural sciences, logical positivism. This approach works quite well when measuring the behavior of atoms, or planets, or biological systems, because these relationships are quite stable. The fundamental forces of the universe (i.e., think E = MC2) will not change over time or because we observe them. In psychology as currently studied, this is not a safe assumption. When I conduct a research study to observe the usefulness of Facebook metadata in predicting human behavior, I have no reason to assume between this study and the next that (1) Facebook will be the same, (2) the population using Facebook will be the same, (3) the capabilities of Facebook will be the same, (4) the data being produced by Facebook will be the same, (5) people will behave the same way on Facebook, and so on. Facebook is a living, reactive system, just as the people who use it are themselves complex biological systems. Thus, logistical positivism is not enough for psychology, because (1) researchers need to interpret what they find through these various lenses to make sense of what they find and (2) even if true scores exist, these scores may change over time between one study and the next. Post-positivism is thus a common refinement of logical positivism that adds these caveats: that we must always reflect upon our own influence, as researchers, on the systems we are researching and also recognize that causal forces from outside the scope of our studies might change the nature of our observations even as we make those observations.
To make this a bit more relatable, realize that post-positivism is the philosophical framework that enables us to conduct meta-analyses of psychological constructs that we explicitly expect to change over time; if we did not believe true scores could move around depending upon when the study was conducted and the assumptions surrounding it at the time, we would expect later meta-analytic estimates to only become more precise, not to fundamentally change. If the true-score relationship between conscientiousness and job performance in 1991 was ρ=.22 (Barrick & Mount, Reference Barrick and Mount1991), in a logical positivist framework, we would also expect ρ=.22 in 2091, although measured more precisely. But I suspect most I-O psychologists do not have such an expectation. Jobs will change, people will change, and that number is going to change with them; it is only a matter of how quickly. Thus, even if you have never articulated what post-positivism involves, you probably have an intuitive understanding of it; it is hard-baked into the very foundations of our field.
Why this is critical is that the study of technology on human behavior relies on post-positivism too, although it takes a somewhat different shape. You, as a researcher, do not have the power to personally change the ρ=.22 mentioned above. If the true score is .22 in an organization today, it is very likely to be close to .22 a year or two from now. It may drift over the long term, if the job itself changes, or society changes, or some other “big” thing changes. But it is not something that a researcher, as an individual, can influence. In contrast, modern technologies are constantly being developed, designed, and redesigned by humans according to human needs. Modern technologies are updated continuously with the intent of continuous improvement. Thus, human decisions and behaviors actively change true scores between technologies and other variables in ways that are unlikely when examining relationships between psychological constructs alone. If we believe a technology is ineffective in its purpose (i.e., some desirable effect caused by the technology is too weak), we may redesign the technology to increase its effectiveness (i.e., to increase its true score effect). There may be a ceiling to this true effect, given particular design considerations within a particular technology, but there is no clear way to know where either our observed or true scores are in relation to that ceiling.
We have seen the negative effects of assuming technology to be much more stable than it actually is in all areas of I-O psychology where technologies are studied. It is particularly strongly evidenced by the decades-long arguments in our literature regarding assessment center validity (cf., Klimoski & Brickner, Reference Klimoski and Brickner1987; Jackson, Michaelides, Dewberry, & Kim, Reference Jackson, Michaelides, Dewberry and Kim2016). The assessment center method, like all selection methods, is a technology, designed by humans to assess other humans’ KSAOs. Assessment centers are typically defined by certain common design characteristics, such as the use of multiple raters and exercises (International Taskforce on Assessment Center Guidelines, 2015), but the details vary dramatically – by purpose, by constructs assessed, by methods employed, by exercises selected, by rater populations sampled, and so on. Thus, as a technology, assessment centers are multidimensional. They incorporate and combine multiple distinct technologies, each with their own quirks, effects, and design considerations. For example, leaderless group discussion is an assessment exercise, and therefore a selection method, and therefore a technology. It can be designed well or designed poorly, and these design considerations are also multidimensional. This logic can similarly be applied to every technology contained within any assessment center, keeping in mind that some assessment centers may not even overlap with others in terms of the specific technologies employed. This is a startling level of interactive complexity, once the true number of dimensions involved are considered accurately. Furthermore, as the assessment center method has developed, the specific design considerations related to each of these issues have changed; an assessment center designed to the guidelines of 2015 might not have even been referred to as an “assessment center” twenty years earlier. To even investigate the “validity of assessment centers” as such in this context is an absolute waste of researcher time and effort. Although the futility of this approach has been recognized to an increasing degree in the last few years (e.g., Kuncel & Sackett, Reference Kuncel and Sackett2014), it took decades to get here. In other technology-oriented literatures within I-O psychology, we face this same road ahead again and again.
As we dig deeper into any technology, whether speaking of the technologies that enable co-located work or the technologies that enable online assessment or the technologies that enable chatbots to teach people leadership skills, the effects of human-contributed variance on true scores will only become more complex. The value of evaluating technologies as if they behave similarly to psychological constructs will remain similarly fruitless. For our field to remain relevant in this new technology-driven landscape, we cannot afford to repeat this same path across every technology-focused research stream within I-O psychology (Landers & Behrend, Reference Landers and Behrend2017). This also builds on Threat #2 in that we should not react continuously for decades to every innovative technology as it becomes popular, a new stimulus that has appeared suitable for study, pretending that each incarnation of it in our research literature is a random sample from some grand population of technologies. This is unreasonable. And building on Threat #1, neither should we pretend that new technological advancements are simply new versions of technologies we have already studied; our default position should not be to scramble for existing theory as a comfortable and familiar crutch (e.g., Chamorro-Premuzic, Winsborough, Sherman, & Hogan, Reference Chamorro-Premuzic, Winsborough, Sherman and Hogan2016).
1.1.4 Threat #4: I-O Psychologists Are Not Adequately Trained in Technology
Until recently, it appeared that I-O psychologists, especially those in academia, did not consider technology, as distinct concept needing focused training, to be integral to the field. This is evidenced by Tett, Walser, Brown, Simonet, and Tonidandel’s (Reference Tett, Walser, Brown, Simonet and Tonidandel2013) report on the 2011 SIOP Graduate Program survey, which in part assessed the degree to which both “substantive” and “methods” topics were covered in I-O psychology programs. Technology did not even make the list of questions, and among what was asked, the most technology-oriented competency area was “human factors.” Perhaps unsurprisingly, zero doctoral programs surveyed included this in their curriculum. The next year, Byrne et al. (Reference Byrne, Hayes, McPhail, Hakel, Cortina and McHenry2014), writing an article inspired by a Society for Industrial and Organizational Psychology (SIOP) panel discussion centered on Tett et al.’s work, described new competency priorities for graduate training in I-O; the word “technology” does not even appear in their work. It is understandable not to focus on technology in an I-O psychology graduate program, but this suggests that even just a few years ago, in terms of training new I-Os, technology was not even on the proverbial radar, despite better understanding of technology appearing among the concerns of both I-O students (Harris & Hollman, Reference Harris and Hollman2013) and I-O practitioners (Church, Reference Church1998; Silzer & Cober, Reference Silzer and Cober2010).
Things have certainly changed in the last five years. In 2015, Guzzo, Fink, King, Tonidandel, and Landis (Reference Guzzo, Fink, King, Tonidandel and Landis2015) called for I-O psychology to formally respond to the sudden popularity of big data. To inspire I-Os, they provided several examples of I-O work in the big data space already. Yet all their citations to I-O’s work in this area appeared in working papers, unpublished manuscripts, and a single published book, all of which were written or published that same year. Importantly, the term “big data” in its current usage has been around since at least 2008, but the concept of analytics at scale had existed for decades before that (Boyd & Crawford, Reference Boyd and Crawford2012). From this timeline, it is straightforward to conclude that I-O fell a bit behind modern analytics. In response to Guzzo et al.’s article, Aiken and Hanges (Reference Aiken and Hanges2015) called to integrate some degree of modern data science into the core I-O graduate curriculum, including programming skills and modern predictive modeling, primarily suggesting that I-O students should read more books and consider supplementing their own educations by participating in massive online courses on data science until I-O faculty teach themselves enough to in turn teach seminars on the topic. As they noted, “This is not just something that would be nice to see; this is an imperative, and our graduate training needs to reflect this imperative immediately” (p. 544). The threat of technology to I-O became so plain to SIOP that in 2016, the Executive Board established the Future Scanning Task Force to assess threats to the future existence of both SIOP and I-O psychology in general brought by the changing world of work, and to provide recommendations regarding these threats. Understanding technology emerged as a major theme. In 2018, the Executive Board promoted this Task Force to become an Ad Hoc Committee, meaning it will be likely to continue advising the Executive Board for some time. Additionally, two technology-oriented columns intended to teach I-Os about technology now appear in the Industrial-Organizational Psychologist: Poeppelman and Sinar’s (Reference Poeppelman and Sinar2016) “The Modern App” and Landers’ (Reference Landers and Behrend2017) “Crash Course in I-O Technology.” The push from within for I-O psychologists to understand technology, regardless of application domain, has never been higher.
Despite this increasing pressure, in terms of both initial and continuing education, I-O psychology is struggling to respond. The sudden demand for a new skillset that most academic I-O psychologists do not have means that there are relatively few people capable of teaching this skillset currently employed to teach graduate students or lead SIOP workshops. This too is changing, although slowly, and Aiken and Hanges’ (Reference Aiken and Hanges2015) recommendation to outsource these needs to computer science departments in the interim is unlikely to be successful. Computer scientists have quite dissimilar needs from psychologists in terms of programming expertise, and I-O psychologists are different still. I have chatted with students in I-O graduate programs where this is currently recommended, and, universally, I have heard complaints of perceived relevance and value. I-O psychologists completing programming courses in computer science departments creates the same problem as I-O psychologists completing statistics courses in mathematics departments; it is difficult to understand why what you are learning is useful, and it this kind of contextualization that is presently most critical.
1.1.5 Threat #5: It Is Easier to Bury Our Heads in the Sand
Although this may seem a minor point, it is still worth noting that field momentum is a difficult force to counter. In other words, I-O psychology is a difficult and unwieldy ship to steer. As a field, we are generally decentralized, and SIOP, the European Association of Work and Organizational Psychology (EAWOP), and other national I-O organizations can only do so much. In the case of SIOP, it is a volunteer-run organization, which means that it is in the interests of its leadership to avoid courting controversy. There are no licensure programs or graduate program certification programs to leverage a field-wide shift. Thus, the organization cannot simply tell graduate programs to run themselves differently for the good of the field; instead, committees must be formed, debate the issues, and make recommendations, which the programs can then choose to heed or ignore. This adds significant complexity to decision-making and, more critically, adds a lot of time. I-O psychology, as a field, is about as far from “agile” as is possible, and it is hurting us.
Additionally, finger-pointing is already common. I have heard from numerous I-O academic researchers that this is ultimately the problem of practitioners; academia, after all, can only move so fast. I have also heard from numerous I-O practitioners that the problem is ultimately one of academics; after all, the field has changed, so the training must adapt too. Frankly, neither of these perspectives is productive, as both simply encourage their respective constituencies to “stay the course” on a course that is already off-track. The truth is that I-O psychology, as a field, will live or die together, because these problems are all interconnected (Aguinis, Bradley, & Brodersen, Reference Aguinis, Bradley and Brodersen2014), a so-called “wicked problem” (Behrend & Landers, Reference Behrend and Landers2017). The problem with our field’s bifurcation is particularly salient in light of Threats #1 – #4. Although practitioners are at the very forefront of exploratory applied research, following and learning about new technologies literally as they change in front of them, it is extremely difficult for any of them to publish in I-O journals given the apparent need to propose novel theory in a confirmatory framework with well-established parameters in every paper.
1.2 Storm Damage So Far
Together, these five threats are interactive; they cause more damage in combination than their individual effects would suggest. This interaction has already manifested itself in at least three ways that promise to become worse if not mitigated soon.
1.2.1 Practitioners Lead the Way in Technology Because Academia Forces Them To
What brought the limitations of academia’s approach into greatest relief for me, and really the inspiration for this chapter, come from the results of the first ever SIOP Machine Learning Competition at the SIOP 2018 conference (Putka et al., Reference Putka, Schwall, Taylor, Bateman, Beatty, Jin and Walmsley2018). In this competition, 17 teams of either academics or practitioners attempted a prediction problem using an authentic turnover dataset provided by a volunteer organization. The dataset was quite large (for I-O research) and complex, with hundreds of variables, systematic missingness, and longitudinal characteristics, among numerous other features. Each team was tasked with creating the predictive model that would hold up the best in a hold-out sample using whatever techniques they had at their disposal. Additionally, teams received feedback on the quality of their models each week for about a month in the form of a leaderboard. Importantly, although academic-practitioner teams were permitted, none formed. At the end of the competition, the top four scoring teams were asked to present on their methods at SIOP. It was revealed that the four winning teams consisted entirely of practitioners.
What is striking about that story, to me, is that academic researchers in both the natural and other social sciences, including the rest of applied psychology, lead the way. This is where academics in universities are intended to bring the greatest value, by standing at the forefront of knowledge, unconstrained by organizational politics and the bottom line. Yet, in this competition, the very best minds in machine learning and predictive modeling in I-O psychology were all among practitioners. And perhaps more importantly, very few of the skills used by any of those teams are traditionally taught in I-O psychology programs. Instead, these were all skills picked up in personal professional development, by both the academics and the practitioners, and the practitioners were, as a group, more successful. This suggests that practitioners, or at least academic-practitioner teams, should be leading the charge in our research literature to define best practices and explore the value of all this technology appearing in the employee selection and retention space. So why are there so few such articles? Why are most of the articles we see still building theory of limited practical use?
A troubling truth is that I-O practice, as it is exists right now, is not particularly evidence-based (Briner & Rousseau, Reference Briner and Rousseau2011). Although this statement prima facie may suggest that practitioners are the problem, the reality is that academia is equally, if not more, to blame. I-O practice does not generally benefit from I-O academia in its current state, because academia is no longer supplying much practical theory. Practitioners instead must create, interpret, and market their own brand of evidence. There is little motivation for practitioners to adopt and employ academic research for which they see little value. There are only two ways I see for academia to compete for attention in this situation. The first is for I-O psychology academic researchers to transition to the role that academic research has traditionally filled in the natural sciences: inventors and testers of new, trusted technologies. The second is to encourage academic-practitioner partnerships in which academics learn from practitioners, translate for a broader audience, test the ideas, and publish their findings collaboratively. We have seen calls for the second approach for a long while; perhaps it is time to try the first approach, as well. For example, outside psychology in academic engineering fields, new inventions routinely appear, and patents are a major source of revenue for such programs. But unlike the creations of industry, inventions created by these academic departments tend to address more fundamental challenges that industry is unlikely to spend its time and resources investigating, given a higher risk of failure. This is because academia typically serves a social good; it creates fundamental advances in our scientific understanding of phenomena that might not be cost effective for a single organization to pioneer yet benefit all (Behrend & Landers, Reference Behrend and Landers2017). I-O psychology historically did the same; it is time to return to our roots.
1.2.2 Existing Discussions of I-O Technology Reveal Significant Knowledge Gaps
Arthur and Villado (Reference Arthur and Villado2008) reminded researchers that characteristics of people (i.e., constructs) and the technologies being used to assess them (i.e., methods) are in fact different things. The existence of such an article, or more specifically, the legitimate need for it then and now, suggests the sort of thinking that might have necessitated it: “because psychological constructs are familiar, anything worth studying is probably a construct.” This might be called the psychology scholarship heuristic: because most concepts of interest in psychology have traditionally been constructs, constructs are therefore the most important subject of research. It is a default philosophical orientation. But such an orientation is limiting and harmful for I-O psychology when exploring technology, because it places artificial limits on both research and practice. Two examples from the I-O literature will illuminate the issue.
First, Adler, and Boyce (Reference Adler and Boyce2016) made a rather forceful statement regarding the data science brand of predictive modeling in which the specific causes of a model’s predictive ability are not explainable by humans:
In our view, surrender to using “black box” solutions – when we don’t understand why those solutions work – may in isolated cases be expedient but is simply not a long-term option for building our science … What distinguishes us as advisors to organizations around talent issues is in part our grasp of the conceptual frameworks we can apply to develop those insights and produce those hypotheses a priori in addition, of course, to the discipline and techniques for empirically testing those frameworks and hypotheses.
On its face, this comment seems like a reasonable stance and sound advice. It echoes the old criticisms of “dustbowl empiricism” in the earlier days of I-O psychology, a time that many current I-Os are glad is dead and buried. It suggests, quite reasonably, that prediction without understanding the constructs involved is not worth the effort. However, it also closes us off to possibilities. What it reveals to me, as someone who follows recent developments in computer science research, is a disparity between what Adler and Boyce believe the interpretability of black box solutions are and what computer science researchers believe their interpretability could become. In the computer science research area of neural network modeling, commonly called “deep learning” and what Adler and Boyce are most likely referring to as “black box solutions,” there is currently a substantial effort to create approaches and visualizations that will help explain precisely why these models predict outcomes so well, and why they do so better, in general, than any predictive modeling approach we currently commonly employ in I-O psychology. Although these approaches are in their infancy, they are certainly in development.
Additionally, by wholesale discounting “black box solutions,” I-O psychology closes itself to the possibility that there may be specific situations or contexts in which understanding why a model predicts well is legitimately a secondary goal. For example, if you could employ a model predicting turnover with an R2 of .45 using traditional regression-based modeling or an R2 of .55 using convolutional neural networks, and those predicted scores themselves were correlated .8, would you automatically turn to the .45, simply because it is more explainable? I suggest you probably would not. That inter-algorithmic reliability of .8 is evidence that the sorts of variables being weighted more heavily in the regression are likely the same ones being picked up in the neural network approach, and if the ΔR2 of .10 is generalizable out-of-sample, the neural network model emerges as a clearly superior choice for practical decision-making. Thus, automatically discounting black box solutions both (1) reveals an ignorance of the research in computer science currently underway to improve interpretation of such solutions, which is likely to become mainstream within the next five years anyway, and (2) forces I-O psychologists to wait until those approaches already exist, rather than working collaboratively with computer scientists or data scientists to ensure they meet the needs of I-O psychology. Once again, this sort of stance puts us as passive consumers of technology rather than active builders of the technologies that would most benefit our field.
A second example comes from Chamorro-Premuzic, Winsborough, Sherman, and Hogan (Reference Chamorro-Premuzic, Winsborough, Sherman and Hogan2016), who attempted to re-brand various technology trends as reincarnations of existing I-O practices: “gamified assessments are the digital equivalent of situational judgment tests, digital interviews represent computerized versions of traditional selection interviews, and professional social networks, such as LinkedIn, are the modern equivalent of a resumé and recommendation letters” (p. 622). Much like Adler and Boyce’s (Reference Adler and Boyce2016) recommendations, there is an intuitive appeal to this approach. They make technologies that seem alien and foreign relatable within the comfortable, warm embrace of existing theory and practice in I-O psychology. However, this has the same effect as in Adler and Boyce’s treatment; it limits our possibilities and betrays a lack of expertise in these specific technologies. To illustrate, consider their treatment of gamification, which is never defined except to list “SJT and self-report” as the non-digital I-O equivalent, along with descriptions of the products of three companies they label as gamification: Knack, Pymetrics, and Tinder. As Armstrong, Ferrell, Collmus, and Landers (Reference Landers2016) explain in a response article, the area of game-thinking in assessment is quite broad and dissimilar to traditional I-O assessment methods, encompassing both game-based assessment, in which a full assessee experience is designed, and gamification, in which existing assessments are modified using lessons from the game-design literature. For example, they describe personality surveys in which narrative elements have been added and simulations in which animation and sound effects have been added. None of this is to say that a situational judgment test is not one example of gamification, but rather that defining gamification as digital situational judgment tests closes off numerous possibilities for I-O psychology to grow along with modern technology. And beyond that, there is no reason to assume that our current theoretical understanding of situational judgment tests is adequate to understand the full spectrum of game-related changes that could be made to situational judgment tests administered via the internet. Situational judgment tests are not constructs that instantiate themselves as different technologies over time, and they vary widely even within the label of “situational judgment test.” Once again, this implicit view creates an artificial limit and belies an ignorance of what is possible with the technology, not only as it exists today, but as it will exist in the future.
1.2.3 Published I-O Psychology Is Becoming (Even) Less Useful
One of the core challenges to I-O psychology in recent years has been the migration of I-O psychologists to business schools (Aguinis, Bradley, & Brodersen, Reference Aguinis, Bradley and Brodersen2014). This has happened for several reasons, but most cynically, a primary draw is because business schools can pay much better salaries and may even provide cash bonuses for publication (Luthans, 2017). Realistically, this is not something with which I-O psychology will ever be able to compete. Psychology departments, historically, are situated in either Colleges of Liberal Arts or Colleges of Science. In most universities, faculty are expected to bring funding to their institution by seeking external funding, through grants and contracts, and sharing indirect costs. In short, a funding model has developed for colleges and universities in which college expenses are covered by faculty research (Zusman, Reference Zusman, Altbach, Berdahl and Gumport2005), a situation that is increasingly common worldwide (Polster, Reference Polster2007). Because of the significant tuition currently paid by students seeking Master’s degrees in Business Administration (MBA), faculty in business schools typically do not face the same expectation, and there is also a sizable pot of money from which to pay lucrative salaries. Thus, whereas faculty in the rest of the university are expected to supplement their own salaries with external funding that they must themselves apply for, business schools leverage their MBA-driven funding model to lure faculty that are perceived as “the best of the best” from disciplines relevant to business. Increasingly, this includes psychologists, typically I-O and social psychologists.
This pattern is not by itself a problem for I-O psychology. If I-Os in business schools continued to publish I-O psychology research, it would not particularly matter where they were employed at the time. But what has happened instead is that the norms and values of business schools, and particularly of organizational behavior (OB), have changed the type of research that business school I-O psychologists deem valuable and important (Lefkowitz, Reference Lefkowitz2008). Because business schools were historically seen as less “serious” than traditional academic disciplines, their faculties needed to fight for their relevance to universities, and a major outcome of that struggle was the development of the theory fetish described earlier as Threat #1. Over the last two decades, as I-O psychologists have left for business schools but continued to publish in I-O psychology, they have increasingly brought these business schools’ values with them into I-O journals. Now, such thinking seems to have infected mainstream academic I-O psychology, field-wide.
Considering that the business school community has known these values to have created an existential problem that they have been grappling with for decades (Pfeffer & Fong, Reference Pfeffer and Fong2002), it is unfortunate that I-O psychology continues to import them freely. The negative effects are significant, yet these values have spread like a cancer, gradually nudging I-O psychology journals to publish a different sort of work than they did before. As more I-O psychology journals fall to this influence, we create a research literature that is “overly abstract, pedantic, and somewhat pretentious” (Campbell & Wilmot, Reference Campbell, Wilmot, Ones, Anderson, Viswesvaran and Sinangil2018). With such values and such a literature, academics become increasingly siloed within not just I-O psychology but within their own narrowly defined research areas, and practitioners see decreasing value in I-O research to further their own organizational and job-related goals. Academics write papers for an ever-shrinking number of other academics within a tightly defined research area, while practitioners keep their own research proprietary, behind the organizational curtain. At the end of this road, no one learns anything new or useful from anyone else within the I-O community. Realistically, it would probably never get quite so apocalyptic as that, but the field is already too far down this path. We need to stop now and reverse course.
1.3 Recommendations for a Brighter Future
With so much gloom in these pages, it might sound like I am saying that I-O psychology is doomed to failure. To be clear, I do not think that it is. I-O psychology can and does bring substantial value to people in organizations, but we are currently straying far from the path that would most directly bring about a positive vision in the future; our value to people in organizations is high but currently diminishing, and I would like to stop this trend before the situation deteriorates further. If we wish to become undisputed experts in the domain of understanding, predicting, and changing human behavior in organizations – a righteous and appropriate goal, I would argue – there are many threats ahead. More tangibly, if we do not want Silicon Valley to “disrupt” I-O psychology and render us voiceless, we need to fix it now. To that end, I have developed four key recommendations for the field, two focusing upon academia and two bridging academia and practice.
First, it does I-O psychology no good to be “OB-lite,” pursing esoteric business-school-values-inspired theory-building as the primary goal of published academic work, but at half the salaries. I-O psychology will never win that fight, but more importantly, we should not want to win that fight. The value of I-O psychology has traditionally been its ability to walk the tightrope between science and practice, integrating them both into a cohesive whole for the betterment of organizations. Technology has become central to practice, yet our science is not only woefully behind but actively trying to diminish that importance in the name of theory building, and in the name of staying nestled where we are comfortable. To reintegrate, academic I-O psychology must abandon business school values. The origins of our field are as an interdisciplinary applied psychology (Zickar & Highhouse, Reference Zickar and Highhouse2017), and we need to return to this view. It is this approach that made I-O psychology useful in the first place, an attractive recruiting pool for the very business schools that now threaten us. We must not lose that aspect of our identity; we are psychologists first (Adler & Boyce, Reference Adler and Boyce2016). Yet we should not be psychologists only. We must recognize that our field is already interdisciplinary in nature; integrating and studying technology, incorporating existing technology research into our own expertise, is merely another extension of this interdisciplinarity. Even business school research has a role to play in an interdisciplinary I-O psychology; it simply should not define I-O psychology. For those I-O psychologists already in business schools, I urge you to heed Zickar and Highhouse’s recommendations: seek joint appointments in psychology and forge explicit, documented ties with your institution’s psychology department. If you want your PhD students to be able to call themselves I-O psychologists, ensure they complete coursework in I-O psychology and interact with psychologists; do not put the burden solely on yourself to teach them to be an “I-O in a business school environment.” It is not the same, and it never will be.
Second, we must repair the problems already emergent within academic I-O psychology. Most critically, journal editors must be more open to non-theoretical contributions. Importantly, this does not imply “atheoretical,” nor does it imply that the theory we incorporate must be psychological in nature. For example, simply presenting data, a null hypothesis significance test, and an effect size without any context is not particularly useful in scientific research literature. This is atheoretical; it does not involve theory. However, if a researcher can make a compelling case that the test improves our understanding of some existing theory or leads to an interesting theoretical question that has not yet been answered, each of these are valid science. High-quality non-theoretical research contributions will call upon theory but do not seek to build new theory until sufficient information is known to be confident in that label. Individual studies do not need to “create theory” to be useful; instead, they can develop, test, comment, create new questions, or provide context for theory, among other purposes. Journal editors should not discard research simply because it does not present boxes and arrows linking concepts together alongside what two or three reviewers consider to be convincing narration. In the technology space, this is particularly relevant regarding exploratory research, the first scientific poke at what could become new research domains for I-O psychology. In the current publishing environment of our “top tier” journals, exploratory research related to a novel, untested technology is essentially unpublishable. This must change.
Third, we must improve training in technology for both academics and practitioners. For my part, I have tried to contribute one partial solution to this problem by releasing free, open-access course materials in data science intended for social scientists (http://datascience.tntlab.org). This course can be used to teach a one-semester graduate-level course in the statistical programming language R (Culpepper & Aguinis, Reference Culpepper and Aguinis2011), starting from zero prior exposure and ending with web apps, natural language processing, and machine learning. Alternatively, it can be used to self-teach, using both the resources I provided and those found in websites providing interactive coding instruction and practice. From the feedback I have received on it, I know that I-O academics, I-O graduate students, and I-O practitioners have all been completing it; it does meet a need. But this is alone is insufficient. New graduate courses, retraining efforts among I-O psychologists working in organizations, data science groups in large consultancies, and other such formal efforts are needed. We cannot rely on grassroots technology evangelism alone; there are simply too many people currently undertrained in technology and lacking the skills they need to compete in the modern I-O environment. Sheets et al. (Chapter 2, this volume) provide specific, concrete steps for program chairs and I-O faculty, graduate students, and professionals to take to help narrow this gap. If all their suggestions were implemented field-wide, by both institutions and individuals, we would be in a much better position than we are now.
Fourth, I-O psychology must allow itself to become truly interdisciplinary. As mentioned before, I-O psychology has always had an interdisciplinary flavor to it. Historically, we have borrowed concepts and ideas from other areas of psychology, such as personality and social psychology, applied them to the context of employee management, and used that to develop advice for practitioners, whether in the form of practical theories, guidelines, or simple recommendations. We even integrated the field of statistics and helped realize its human psychology applications as the field of psychometrics. It is time to expand this effort to explicitly include technology, to formally blend I-O psychology research and practice with fields like computer science, data science, and human-computer interaction. Adler and Boyce (Reference Adler and Boyce2016) stated, “We are I-O psychologists, not human resource technologists or data scientists” (p. 642). Although a true statement, this does not imply that human resource technology and data science should not be a part of modern I-O psychology. They absolutely must. As Ducey et al. (Reference Ducey, Guenole, Weiner, Herleman, Gibby and Delany2015) argued, I-O psychologists should “join business analysts, data scientists, statisticians, mathematicians, and economists in creating the vanguard of expertise as we acclimate to the reality of analytics in the world of big data” (pp. 555–556). Their statement is specific to big data, but the view it implicitly endorses is broader than that. It suggests integrating these other fields into I-O psychology while integrating I-O psychology back into these other fields. This is what we must work toward. We cannot retreat into our siloes if we wish to have any impact on the world of work as it continues to change.
1.4 Conclusion
I-O psychology is at a crossroads. Down one path, we turn toward business school values, building ever-more-complex theories to better understand and explore every minute detail of organizational functioning, a rigorous but not particularly useful science to people trying to enact change within those organizations, staking a claim to exhaustive understanding of psychological constructs as what defines us. In the other direction, we embrace our own foundations as an interdisciplinary and applied psychology, integrating our field with the disciplines surrounding it, contributing to those fields while being augmented by them, forging our own unique identity, building a practical science, working shoulder to shoulder with all those working to understand employee behavior in the modern workplace, regardless of their discipline of origin. I do not know which way I-O will turn, but I hope it is toward this latter, brighter future.
