Important considerations in using statistical procedures to control for nuisance variables in non-experimental studies
Section snippets
What does statistical control involve?
In attempting to control for the effects of confounding variables, multiple regression analysis is commonly used (Austin et al., 2002, Stone-Romero, 2007). Frequently, researchers have used a hierarchical strategy in which control variables are entered first and then a hypothesized predictor variable is entered. The increment to R2 (i.e., the squared semi-partial correlation) added by the predictor provides an estimate of the unconfounded predictor effect (i.e., the unique predictor variance).
Theory as the determinant of whether a statistical control strategy is appropriate
Although much of the research conducted on human resource management topics is intended to advance our understanding of causal relationships, as is noted above, researchers frequently have used a non-experimental research design. With such a design, “variables that are assumed causes are measured, as opposed to being manipulated” (Stone-Romero, 2007, p. 520). An obvious weakness of a non-experimental design is that, if a relationship is found between an assumed causal variable and an assumed
Judge and Cable (2004): an example of statistical control in practice
Judge and Cable (2004) were interested in the effect of a person's height on subsequent earnings. Table 1 summarizes the major findings of their Studies 2–4 (Study 1 is not discussed because, unlike the other studies, in Study 1 weight reflected a judgment rather than actual pounds). Having controlled for nuisance variables (i.e., gender, age, and weight in Studies 2 and 3; gender and weight in Study 4), Judge and Cable found the three regression coefficients for height were significant at a p <
An analysis of recent research that used a statistical control strategy
In beginning this project, my perception was that the degree of shared variance between an original predictor variable and the corresponding residual predictor was not reported in most studies (Becker, 2005; did not gather data on the reporting of shared variance). In order to assess the accuracy of this impression, 59 studies were reviewed that used multiple regression analysis to control for confounding variables that were published in 2005 or 2006 in the Journal of Applied Psychology or
Much ado about nothing?
In this paper, the importance of reporting the shared variance between the original predictor variable and the residual predictor variable has been stressed. Reporting this information is important for two reasons. First, it focuses attention on whether the same construct is likely to be reflected by the two predictors. Second, information on shared predictor variance can offer insight into unexpected research findings. The first issue has already been addressed in some detail. The second issue
More extreme examples of the confusion statistical control can cause
A recent study by Schreurs, Derous, De Witte, Proost, Andriessen and Glabeke (2005) provides a clear example of the confusion that the use of statistical control procedures can cause. Utilizing a non-experimental study, these researchers examined the assumed effects of three recruiter characteristics (i.e., warmth, informativeness, and competence) on job applicant attraction to an organization, intention to apply for a position, and whether an application was submitted. Based upon their review
The generalizability of findings resulting from the use of a statistical control strategy
If the results of research on a given topic do not generalize, scientific advancement is difficult (Hunter & Schmidt, 2004). Given this fact, researchers typically are interested in the external validity of their results (Stone-Romero, 2002). For researchers using a statistical control strategy, the issue of external validity merits careful consideration. Meehl (1970) succinctly addressed the problematic nature of statistical control with regard to the generalizability of research findings —
Additional issues for consideration in using a statistical control strategy
For years, statistical experts (e.g. Meehl, 1970, Schwab, 2005) have stressed that theory should determine whether it is appropriate to control for confounding variables. Despite this fact, Becker (2005) found that in over 50% of the articles he reviewed no theoretical rationale was provided for at least one of the nuisance variables. Assuming a researcher has made a coherent theoretical case for the control of nuisance variables, such variables should be controlled. Yet, as is apparent from
Concluding remarks
This article considered the results of several recent studies (e.g., Ambrose and Cropanzano, 2003, Schreurs et al., 2005) and focused heavily on an article by Judge and Cable (2004). In selecting these studies, my intent was not to question the contributions these studies made (each has several impressive attributes), nor was it to embarrass the authors. Rather, my intention was to demonstrate the confusing findings that can result from the use of a statistical control strategy. The paper by
References (33)
- et al.
A longitudinal analysis of organizational fairness: An examination of reactions to tenure and promotion decisions
Journal of Applied Psychology
(2003) - et al.
Rhythms of life: Antecedents and outcomes of work-family balance in employed parents
Academy of Management Journal
(2005) - et al.
History of research methods in industrial and organizational psychology
Self-Efficacy: The exercise of control
(1997)Potential problems in the statistical control of variables in organizational research: A qualitative analysis with recommendations
Organizational Research Methods
(2005)- et al.
Applicant attraction to organizations and job choice: A meta-analytic review of the correlates of recruiting outcomes
Journal of Applied Psychology
(2005) - et al.
- et al.
Applied multiple regression/correlation analysis for the behavioral sciences
(2003) - et al.
Exploring applicant pool quantity and quality: The effects of early recruitment practice strategies, corporate advertising, and firm reputation
Personnel Psychology
(2004) - et al.
Work value congruence and intrinsic career success: The compensatory roles of leader-member exchange and perceived organizational support
Personnel Psychology
(2004)
Issues in multiple regression
American Journal of Sociology
Employer treatment of employees during a community crisis: The role of procedural and distributive justice
Journal of Business and Psychology
Information processing, knowledge development, and strategic supply chain performance
Academy of Management Journal
Methods of meta-analysis
Employees' goal orientations, the quality of leader–member exchange, and the outcomes of job performance and job satisfaction
Academy of Management Journal
The effect of physical height on workplace success and income: Preliminary test of a theoretical model
Journal of Applied Psychology
Cited by (65)
Have we controlled properly? Problems with and recommendations for the use of control variables in information systems research
2024, International Journal of Information ManagementEconometrics and archival data: Reflections for purchasing and supply management (PSM) research
2022, Journal of Purchasing and Supply ManagementCitation Excerpt :This brings us to a second issue with devising fitting GLMs to archival data: the misuse, and especially the overuse, of control variables (Breaugh, 2006). Mathematical presentations of the GLM (Mauro, 1990; Neuhaus and Jewell, 1993; Yatchew and Griliches, 1985) and methodological discussions of control variables (Breaugh, 2006, 2008; Carlson and Wu, 2012; Spector and Brannick, 2011) indicate control variables should only be included if they satisfy three conditions: (i) they are significantly related to the dependent variable conditional on the focal predictor and control variables, (ii) they do not reside theoretically downstream from the focal predictor, and (iii) their inclusion does not unduly affect the interpretation of the focal predictor. Regarding the mathematics, Mauro (1990) demonstrates that in models with an identity link function (e.g., OLS), for an omitted variable (Z) to impact the estimate of a focal predictor (X) in the presence of k control variables (C1, C2, …Ck), Z must be both partially correlated with X conditional on C1, C2, …Ck and partially correlated with the dependent variable conditional on X and C1, C2, …Ck.
Analyzing the women's empowerment and food security nexus in rural areas of Azad Jammu & Kashmir, Pakistan: By giving consideration to sense of land entitlement and infrastructural facilities
2020, Land Use PolicyCitation Excerpt :These variables were not included as control variables in the model. According to Breaugh(2008), control variables would make the parameter estimates, such as the path coefficients and regressions, not reflect the original independent variables. Rather, they would reflect the portions of the original independent variables that are not related to the control variables.
Does proactivity matter? the importance of initiative selling tactics for headquarters acceptance of subsidiary initiatives
2019, Journal of International ManagementSanction severity and employees’ information security policy compliance: Investigating mediating, moderating, and control variables
2018, Information and ManagementCitation Excerpt :Although the impact of PSS on CI is significant at the p = .05 level, the variance of CI that is explained by PSS is small (R2 = .015). This indicates that the variance of CI is influenced by other factors and sets the stage for investigating other meaningful explanatory variables [43,53–55]. Next, we investigated whether the positive relationship between the PSS and CI is moderated by other variables by adding the other variables to the base model.