Are proxy interviews associated with biased earnings reports? Marital status and gender effects of proxy

Abstract

Social science findings routinely rely on proxy-reported economic data in household surveys. A typical assumption is that this information is not biased compared to self-reports, but empirical findings on the issue are mixed. Using a dataset that links workers in the 2004 Survey of Income and Program Participation to their W-2 tax records, we estimate the effects of reporting status (proxy vs. self) on the magnitude and direction of measurement bias in earnings data and explore whether these effects are heterogeneous across gender and marital status. A slight downward bias in proxy-reported earnings is observed; however, these effects are associated with demographic variables. For married workers, proxies do not contribute substantial bias in earnings measurement regardless of the target respondent’s gender. However, for single female workers, proxy interviews are a significant source of downward bias in earnings estimates. The implications of these findings are discussed.

Introduction

Social science findings routinely rely on economic data in household surveys. Although not often recognized, the quality of these data hinge on proxy responders accurately reporting someone else’s economic situation, such as their earnings. Among major household surveys in the US, proxy responses can constitute nearly one-half (e.g., Current Population Survey, CPS) to one-third (e.g., Survey of Income and Program Participation, SIPP) of adult interviews. Proxy interviews reduce survey costs and nonresponse (Todorov and Kirchner, 2000). A possible drawback of longstanding concern, which we consider herein, is more pervasive response bias.

As noted by Alwin (2007:53–54), response bias consists of two components, – systematic error and random error. Systematic error, which is often of concern to researchers due to its greater potential to distort measurement, reflects consistent deviations (e.g., over- or under-reporting) from the true value. Random errors, while not necessarily leading to significant bias, are still undesirable because they reduce the efficiency of statistical estimates and can cause attenuation bias when they are present in independent variables of a regression model.

In this study, we examine the degree to which reporting status (proxy vs. self) contributes to bias in earnings data and whether the effects are heterogeneous across gender and marital status. This issue deserves more attention for several reasons. Earnings constitute a vital measure of a person’s financial and labor market circumstance, and systematic differences between self- and proxy-reported earnings and the actual value could lead to biases in statistical inferences based on such data. Although researchers usually assume that proxy responses have no systematic errors, the issue remains unclear. While some studies have found little proxy bias in earnings (e.g., Bound and Krueger, 1991, Mellow and Sider, 1983), others suggest a downward bias (e.g., Hill, 1987, Reynolds and Wenger, 2012). Moreover, whether proxy response bias varies across subpopulations has not been fully considered in the literature, despite its potential implications for cross-sectional estimates and long-term trends, such as the gender wage gap (Lee and Lee, 2012, Reynolds and Wenger, 2012).

Our analysis uses a rich dataset that links workers in the Survey of Income and Program Participation (SIPP) to their own Internal Revenue Service (IRS) tax records. These linked data provide “gold standard” earnings data (Bollinger, 1998, Kim and Tamborini, 2012b) that can be utilized to assess the accuracy of survey earnings. This method allows us to provide baseline national estimates of the magnitude and direction of proxy response bias in SIPP earnings. A primary analytic interest is to assess whether proxy bias varies across key demographic variables, focusing on the target respondent’s gender and marital status. Although researchers often implicitly assume that proxy bias is constant across the population, our results reveal significant differences by these two attributes. Demographic characteristics of persons using proxy respondents, thus, may affect measurement error in survey data.

Researchers do not often consider proxy respondents to be a key element of survey methodology. Yet, proxy interviews are very common in surveys. In the US, household surveys such as the Current Population Survey, Consumer Expenditure Survey, and the Survey of Consumer Finances use a designated household member to respond for all household members. The implication is that nearly half of survey responses are based on proxy interviews. In survey designs aiming to maximize self-response, such as the SIPP, self-interviews are attempted with all adult individuals in the household before a proxy interview is attempted (U.S. Census, 2001). This design feature lowers proxy responses but, nonetheless, proxies still constitute roughly one-third of total SIPP responses.

Proxy interviews are beneficial because they reduce survey costs and the nonresponse rate, which can make the sample more representative. A concern is potentially greater response bias (Alwin, 2007:152). Also salient, as recent papers using the CPS (Reynolds and Wenger, 2012, Fig. 1) and the Panel Study of Income Dynamics (Lee and Lee, 2012) point out, the demographic (e.g., gender) composition of respondents who use proxy interviews in the same survey can change over time, which can influence measurement of long-term trends, such as the evolution of the gender wage gap, if proxy bias is present.

The bulk of empirical research about proxy reporting and response bias has been in the areas of public health and epidemiology rather than the labor market. Topics include the adequacy of proxy reports for measures of activity during post-acute care among stroke patients (Jette et al., 2012), alcohol consumption (Graham and Jackson, 1993), prior pesticide use (Johnson et al., 1993), smoking (Navarro, 1999, Soulakova et al., 2009), health care experiences (Elliott et al., 2008), and disability (Todorov and Kirchner, 2000). Empirical attention has also focused on proxy reports of family status variables, such as fertility intention (Williams and Thomson, 1985) and parental characteristics (Hauser and Wong, 1989, Looker, 1989, Wagmiller, 2009).

In terms of earnings, there are plausible reasons for both more and less measurement error due to proxies (Mathiowetz and Lair, 1994, Moore, 1988). An intuitively appealing postulation is that proxies increase bias because they have more recall error, comprehend interview questions differently (Tourangeau et al., 2000), or overlook comprehensive wage information such as deferred wages or second jobs (Brown et al., 2001). Furthermore, proxy responders, who often have their own interview, have a higher reporting load, which may increase interview fatigue (Moore, 1990).

Alternatively, proxy interviews may enhance data quality. Proxies may know more about the subject matter than the target individual, or they may be subject to less social desirability pressure (Brown et al., 2001, Tourangeau et al., 2000). In explaining lower voter turnout estimates among proxy responses in the CPS, Highton (2005) argues that proxy responders were less influenced by social desirability bias to inflate voter participation. In terms of earnings, proxy respondents may feel less pressure to inflate low earnings or, conversely, to understate high earnings. Proxies also may be more willing to disclose sensitive information.

In a review of the literature, Moore (1988) concludes that there is not substantial evidence that data quality is altered by reporting status, but the issue is unclear given the dearth of research on the topic. Since Moore’s review, a number of studies have appeared, but the evidence remains inconclusive. Bound and Krueger (1991) use a longitudinal 1977 and 1978 CPS file matched to Social Security records (capped earnings) and find little bias in proxy-reported earnings (see also Mellow and Sider, 1983). More recent studies, such as Reynolds and Wenger (2012), use CPS data alone and find that proxy interviews bias earnings downward. Cristia and Schwabish (2009), while not focusing on proxy effects, document downward bias in proxy-reported earnings using the 1996 SIPP matched to tax records (see also, Hill, 1987).

Few empirical studies assess whether proxy-reporting bias varies across different study populations. This is surprising as demographic variables can be seen, theoretically, as potentially influencing reporting bias. Of special interest in this study is the target respondent’s marital status. Although proxy interviews are widely used by married persons, they also are utilized by a non-trivial number of unmarried workers (see Table 1). Because married individuals often manage their household finances jointly (Kenney, 2006, Knoll et al., 2012), spouse proxies may be less prone to response bias than non-spouse proxies. Put differently, proxies who answer questions for single individuals are probably less familiar with the target respondent’s personal finances. Indeed, studies in other contexts have demonstrated that the degree of closeness between the proxy and the target respondent matters (Wagmiller, 2009). This point is raised by Elliott et al. (2008), which find more accurate estimates of elderly health care experiences among spouse proxies than other proxies (see also Goldscheider and Kaufman, 1996). The same pattern could be evident for earnings.

Gender is another potentially salient variable. As Reynolds and Wenger (2012:14) assert, gendered norms and beliefs can influence how men and women report their own or other’s earnings. Accordingly, when the target respondent is male, proxy respondents may tend to overstate earnings, partly to conform to cultural expectations related to the male ‘breadwinner.’ Alternatively, when the target respondent is female, proxy respondents may tend to understate earnings; for example, if women’s market work is undervalued. Although this issue has not been examined directly in the literature, recent studies by Lee and Lee (2012) and Reynolds and Wenger (2012) identify changes in the gender distribution of respondents using proxy interviews over time as influencing estimates of the evolution of the gender wage gap using the Panel Study of Income Dynamics (PSID) and the CPS respectively.¹

What would be a good methodological approach to address the concerns addressed above? Often the most ideal approach matches survey data to an external source of similar kind (Calderwood and Lessof, 2009). However, because validation data, such as in this study, are often unavailable or difficult to access, many studies use only survey data. Although valuable, such studies lack a benchmark and, therefore, cannot discern the direction of the bias (e.g., whether self-reported earnings were over-reported or proxy-reported earnings were under-reported). To address this concern, an increasing number of scholars call for the utilization of third-party validation data (e.g., Lee and Lee, 2012, Reynolds and Wenger, 2012).

An endemic concern for all studies, particularly those that use only survey data, is selection bias. For example, married individuals use proxy interviews to a greater degree than single respondents. Marital status, in turn, is a correlate of earnings. Or, suppose that both low and high earners disproportionately use proxy interviews because they are unavailable due to working long hours. These subpopulations associate with distinct directions of response bias in earnings (Kim and Tamborini, 2012a). To address these concerns, researchers include socioeconomic and demographic control variables in their regression models.

Additionally, a recent literature has made use of longitudinal survey data to exploit individual changes in interview status (Bollinger and Hirsch, 2012, Lee and Lee, 2012, Reynolds and Wenger, 2012). By comparing the change in earnings for those who change from self to proxy status from time 0 to time 1 against those who do not using the CPS Outgoing Rotation Group, researchers can control for time-invariant unobserved heterogeneity and estimate proxy error as the difference in the change in earnings between the two groups. Although valuable, this approach also faces limitations. For example, we cannot rule out reverse causality; a change in interview status may be a function of a change in earnings, rather than vice versa. Also, if low earners are more likely to rely on proxy interviews even after controlling for usual covariates, researchers will find negative proxy effects even if proxy and self-reports are similar. Alternatively, if high earners differentially rely on proxies, then the effects may attenuate toward zero given the prevalence of under-reporting among higher earners. Survey data alone cannot solve this problem.

Against this backdrop, the current study brings national survey data linked to Federal tax records to bear on an investigation of the effects of reporting status on measurement error in survey earnings data. We investigate the following hypotheses.

Hypothesis 1

Proxy reports of earnings relative to self-reports are downward biased.

As noted, even though proxy-reports can be more accurate than self-reports, studies generally show a small downward bias in proxy-reported earnings. Moreover, a substantial body of evidence shows that survey earnings are generally slightly downward biased compared to administrative earnings (Akee, 2011, Bollinger, 1998, Bound and Krueger, 1991, Kim and Tamborini, 2012a, Kim and Tamborini, 2012b, Gottschalk and Huynh, 2010).

Hypothesis 2

Proxy response bias in earnings is heterogeneous across demographic groups.

As discussed above, proxy response bias may vary by the gender and marital status of the target respondent. In this context, we test two hypotheses.

Hypothesis 2a

The proxy bias in earnings is larger for unmarried workers than for married workers.

Hypothesis 2b

The proxy bias in earnings is more negative for female workers than for male workers.