Correcting Measures of Relationship between Aggregate-Level Variables for Both Unreliability and Correlated Errors: An Empirical Example

doi:10.1006/ssre.1998.0619

Social Science Research

Volume 27, Issue 2, June 1998, Pages 218-234

https://doi.org/10.1006/ssre.1998.0619 Get rights and content

Abstract

Many social scientists conducting macro-level analyses use aggregate-level variables (weighted sums of the characteristics of the individuals within the macro-level units: e.g., means, percents, or rates) to describe macro-level units (e.g., schools, cities, or states). Researchers have long known that aggregate-level variables are likely to be more reliable than individual-level variables. More recently, they have recognized the possibility of estimating the reliability of aggregate-level variables using information on the variability of these variables and variability of the individual-level characteristics (on which they are based). These reliability estimates can be used to correct measures of association at the aggregate-level for unreliability. When correcting measures of relationship, however, researchers will often need to consider the correlated errors generated when the same set of individuals responds to the same questions used to create the aggregate-level variables (the problem of joint sampling). This paper shows how to estimate these correlated measurement errors and use them, and aggregate-level reliability coefficients, to correct measures of relationship at the aggregate-level. These techniques are applied to data from the Senate National Election Studies to illustrate the impact of these corrections.

References (24)

L.L. Hargens et al.
A new approach to referees' assessments of manuscripts
Social Science Research
(1990)
R.M. O'Brien
The reliability of composites of referee assessments in manuscripts
Social Science Research
(1991)
C.J. Coluton et al.
Measuring neighborhood context for young children in an urban area
American Journal of Community Psychology
(1996)
L.J. Cronbach et al.
The Dependability of Behavioral Measurement: Theory of Generalizability for Score Profiles
(1972)
R.S. Erikson et al.
Statehouse Democracy
(1993)
K.F. Ferraro
Validity, reliability, and hostility: A comment on Longshore
Social Forces
(1984)
R.V. Hogg et al.
Introduction to Mathematical Statistics
(1978)
B.S. Jones et al.
The reliability of aggregated public opinion measures
American Journal of Political Science
(1996)
P.F. Lazarsfeld et al.
On the relation between individual and collective properties
A Sociological Reader on Complex Organizations
(1969)
D.W. Levine
True scores, error, reliability, and unit of analysis in environment and behavior research
Environment and Behavior
(1994)

D.W. Levine

Why choose one level of analysis? And other issues in multilevel research

Environment and Behavior

(1996)

Cited by (6)

Multiculturalism in the classroom: Ethnic attitudes and classmates' beliefs
2013, International Journal of Intercultural Relations
Citation Excerpt :
First, we aggregated the three original items at the classroom level, and computed α for the 38 classrooms, which was 0.91. Second, we calculated the reliability coefficient suggested by O’Brien (1998) which indicates the extent to which the mean scores for individual students can be regarded as parallel items of the aggregated measure. This reliability coefficient was r = 0.73.
As classmates form an important social reference group for preadolescents, this study examined the relation between classmates’ multicultural beliefs and the ethnic attitudes of fifth- and sixth-grade ethnic majority students (ages 9–13). Thirty-eight school classes in the Netherlands participated. We focused on the native Dutch (n = 548) but information from all students (N = 838) was used to assess classmates’ beliefs and other classroom variables. Multilevel analyses showed that classmates’ multicultural beliefs were positively related to students’ ethnic out-group attitudes, especially for children who felt more accepted by their peers, and negatively to their attitude toward the Dutch in-group. These links were mediated by students’ personal multicultural beliefs, and they were independent of the proportion of co-ethnics in the classroom. Results indicate that it is important to look at specific peer groups (i.e. classmates) to understand children's interethnic attitudes.
Ethnic attitudes of minority students and their contact with majority group teachers
2012, Journal of Applied Developmental Psychology
Citation Excerpt :
Cronbach's alpha was .81. We also calculated the reliability coefficient suggested by O'Brien (1998), which indicates the extent to which the mean scores for individual students can be regarded as parallel items of the aggregated measure. This reliability coefficient was satisfactory, r = .77.
Research on intergroup attitudes in children has focused on contact with out-group peers but neglected the role of adults. This cross-sectional self-report study examined the association between the ethnic attitudes of 174 minority (Turkish- and Moroccan–Dutch) preadolescents (ages 9–13) and the perceived interpersonal relationships with their (native Dutch) majority group teachers. Majority classmates (97 native Dutch children) were included as a control group. Analyses showed that Turkish and Moroccan students who had better relationships with their Dutch teacher had more positive attitudes toward the Dutch out-group, especially in relatively segregated classrooms. The positive aspect of the relationship (‘closeness’) was more important than the negative aspect (‘conflict’). The link between out-group attitudes and the relationship with majority teachers was not significant for the majority students. Findings show that contact theory can be applied to interactions between children and important adults, and that student-teacher relationships can contribute to more harmonious ethnic relations.
Ethnic incongruence and the student-teacher relationship: The perspective of ethnic majority teachers
2012, Journal of School Psychology
Citation Excerpt :
First, we aggregated the three original items at the classroom level, and computed Cronbach's α for the 36 classrooms, which was .82. Second, we calculated the reliability coefficient suggested by O'Brien (1998), which indicates the extent to which the mean scores for individual students can be regarded as parallel items of the aggregated measure. This reliability coefficient was sufficient, r = .76.
Among 36 ethnic-Dutch school teachers in the Netherlands, the present study examined the role of ethnic incongruence in perceived student–teacher relationship quality. Teachers rated their relationships with 59 Turkish-Dutch, 62 Moroccan-Dutch, and 109 ethnic-Dutch students attending grades 4 through 6 (M_age = 10.81 years, SD = 1.05). On average, relationships were less favorable for the Moroccan-Dutch students only. However, the effects of ethnic incongruence were most pronounced among students with strong perceived inattention/hyperactivity and among teachers endorsing lower levels of multiculturalism (the view that different cultures deserve equal treatment). Results support the notion that ethnically incongruent relationships may be perceived as less favorable than ethnically congruent ones due to cultural misunderstandings and intergroup bias. Practical implications are discussed.
Affluence and influence: Economic inequality and political power in America
2012, Affluence and Influence: Economic Inequality and Political Power in America
Affluence and influence: Economic inequality and political power in America
2012, Affluence and Influence: Economic Inequality and Political Power in America
In-group bias in the classroom: The role of co-ethnic and other-ethnic peers and multiculturalism
2011, Anales de Psicologia

^☆: Address reprint requests to Robert M. O'Brien, Department of Sociology, University of Oregon, Eugene, OR 97403.

^☆☆: A. Etzioni

View full text

Regular ArticleCorrecting Measures of Relationship between Aggregate-Level Variables for Both Unreliability and Correlated Errors: An Empirical Example☆,☆☆

Abstract

Social Science Research

Social Science Research

Measuring neighborhood context for young children in an urban area

American Journal of Community Psychology

The Dependability of Behavioral Measurement: Theory of Generalizability for Score Profiles

Statehouse Democracy

Validity, reliability, and hostility: A comment on Longshore

Social Forces

Introduction to Mathematical Statistics

The reliability of aggregated public opinion measures

American Journal of Political Science

On the relation between individual and collective properties

A Sociological Reader on Complex Organizations

True scores, error, reliability, and unit of analysis in environment and behavior research

Environment and Behavior

Why choose one level of analysis? And other issues in multilevel research

Environment and Behavior

Regular Article
Correcting Measures of Relationship between Aggregate-Level Variables for Both Unreliability and Correlated Errors: An Empirical Example☆,☆☆