Evaluation of social network measurement instruments

doi:10.1016/S0378-8733(99)00007-6

Social Networks

Volume 21, Issue 2, April 1999, Pages 111-130

https://doi.org/10.1016/S0378-8733(99)00007-6 Get rights and content

Abstract

This paper evaluates the reliability and validity of network measurement instruments for measuring social support. The authors present and discuss the results from eight experiments which were designed to analyze the quality of four measurement scales: (1) binary, (2) categorical, (3) categorical with labels, and (4) line production, as well as two measurement techniques for listing alters (free recall and recognition). Reliability and validity were estimated by the true score multitrait–multimethod (MTMM) approach. Meta-analysis of factors affecting the reliability and the validity of network measurement was done by multiple classification analysis (MCA). The results show that the binary scale and the first presentation of measurement instruments are the least reliable. Surprisingly, the two data collection techniques (free recall and recognition) yield equally reliable data.

Introduction

The quality of survey data from complete social networks can be affected by many characteristics of the measurement instrument. Some of the factors that can affect response behavior, e.g., are (a) the wording of questions, (b) the response scale, (c) the data collection method, and (d) the context of the questions. The overall quality of measurement in a given network thus depends upon several factors whose impact on data quality has seldom been thoroughly analyzed and controlled. This state of affairs results from a lack of systematic research into the issues of data quality in the field of social network analysis. Research work on measurement issues mainly focuses on the questions of measurement validity, reliability, accuracy and measurement error (Wasserman and Faust, 1994, p. 56).

There are, however some studies which have focused mainly on the question of network data quality (Hammer, 1984; Sudman, 1985, Sudman, 1988; Hlebec, 1993; Brewer and Webster, 1997). These papers are especially important for the experimental design used in this study, which focuses on the test–retest stability of complete networks. Each of these studies has compared two basic data collection techniques frequently used in the survey collection of network data: free recall and recognition. In all above mentioned studies, free recall and recognition yielded different measured egocentric networks. Previous research indicated little difference (when the networks were small) between the free recall and recognition techniques in the assessment of (a) important ties, (b) most recent contacts, and (c) most frequent contacts. The recognition technique gave much better results when networks were larger and ties were weaker. One aim of this study is to test whether the recognition data collection technique is more stable than that of free recall, since differences in reliability could significantly alter the interpretation of results.

Much work has been done on topics such as respondent accuracy (e.g., Killworth and Bernard, 1976; Bondonio, 1998; Casciaro, 1998), characteristics of the measured networks (e.g., Burt, 1984; Marsden, 1987; Wellman and Wortley, 1990), comparison of the measured networks using different network generators (e.g., Bernard et al., 1987, Bernard et al., 1990; Campbell and Lee, 1991), and characteristics of the measured ties (e.g., Marsden and Campbell, 1984; Burt, 1986). Owing to the complexity of the data structure, there is still need for an extensive and systematic evaluation of survey measurement instruments in terms of test–retest reliability of measurement in the field of social network analysis. Nevertheless, in evaluating survey measurement instruments when measuring variables, there are some approaches which are applicable to social network analysis. The first such evaluation of survey measurement instruments, by two stage meta-analysis, was by Andrews (1984), who analyzed the quality of American and Canadian surveys. Together with Willem Saris and several other European social scientists, they established an international group on methodology and comparative survey research (IRMCS). Their extensive and fruitful work (Saris and van Meurs, 1990, Ferligoj and Hlebec, 1998; Ferligoj et al., 1995; Saris and Münnich, 1995; Scherpenzeel, 1995) contributed substantially to knowledge about the quality of survey measurement instruments. Their results inspired the work of Ferligoj and Hlebec (1995), Ferligoj and Hlebec (1998), who first used the multitrait–multimethod (MTMM) approach to estimate the reliability of complete network measurements. They reported that the binary scale is the least reliable scale, at least when compared with an 11-point ordinal scale or a line drawing scale, regardless of the order of presentation. In the present study, two five-point ordinal scales have also been included.

Saris and Münnich (1995) and Ferligoj and Hlebec (1998) have also reported that factors, such as the order of repetition and the time between two successive presentations of the measurement instrument, can also have substantial impact on the quality estimates. Therefore, the order of presentation and the time between presentations have been included in the experimental design of this study. Finally, the content of the network name generators — social support — was selected on the basis of the characteristics of the experimental groups: eight classes of third year high school students.

This paper presents and discusses the results from eight experiments that were designed to analyze systematically the impact of different measurement characteristics on the reliability and validity of complete network data. In the first phase of this study, estimates of test–retest reliability, validity and method effects are obtained for each set of relationships in each of eight classes, using the MTMM approach. In the second phase, the effects of the characteristics of the measurement instruments used in different classes are analyzed to explain the variability of the estimates for the reliability, validity and method effects. A secondary analysis of MTMM results is done by multiple classification analysis (MCA). The two-stage procedure described is similar to that used by Saris and Münnich (1995).

Section snippets

Experimental design

In this study, data were collected regarding social support relationships among third year students in a high school in Gimnazija Bežigrad in Ljubljana, the capital city of Slovenia. On average, there were 31 students, aged 17, in each of eight classes. Four name generators² (traits) were used — exchange of study materials, exchange of information in the case of long-term illness, invitation to a birthday party, and discussion of important personal matters.

Mean levels of data quality

In Table 3, summary statistics for the validity and the reliability coefficients over eight classes are presented. Within each class, 12 reliability and 12 validity estimates were obtained for both the original and the reversed measures. The overall mean reliability of 12 reliability coefficients for the original questions is 0.881, and 0.882 for the reversed. The mean overall validity coefficient is 0.981 for original questions, and 0.974 for the reversed.

As is evident in Table 3, the validity

Conclusions

The results from four meta-analyses show that the domain of social support, as measured by the binary scale, is the least reliable when the recognition data collection technique is used. In contrast, when free recall is used, a smaller number of dyads is reported and all scales are equally reliable. It seems that when a full list of membership is available, it should be used in any measurement procedure to simplify the reporting task for respondents, and to increase the number of reported ties.

Acknowledgements

The authors are grateful to the members of IRMCS for their critical comments and especially to Willem Saris and Brendan Bunting for their useful suggestions on the text.

References (28)

H.R. Bernard et al.
Comparing four different methods for measuring personal social networks
Social Networks
(1990)
D. Bondonio
Predictors of accuracy in perceiving informal social network
Social Networks
(1998)
K.E. Campbell et al.
Name generators in surveys of personal networks
Social Networks
(1991)
T. Casciaro
Seeing things clearly: social structure, personality, and accuracy in social network perception
Social Networks
(1998)
M. Hammer
Explorations into the meaning of social network interview data
Social Networks
(1984)
S. Sudman
Experiments in the measurement of the size of social networks
Social Networks
(1985)
S. Sudman
Experiments in measuring neighbour and relative social networks
Social Networks
(1988)
F.M. Andrews
Construct validity and error components of survey measures: a structural modeling approach
Public Opinion Quarterly
(1984)
H.R. Bernard et al.
How much of a network does the GSS and RSW dredge up?
Social Networks
(1987)
Brewer, D.D., Webster, C.M., 1997. Patterns in the recall and recognition of friends by residents of a university...

R.S. Burt

Network items and the general social survey

Social Networks

(1984)

R.S. Burt

A note on sociometric order in the general social survey network data

Social Networks

(1986)

De Wit, H., Billiet, J., 1995. The MTMM design: back to founding fathers. In: Saris, W.E., Münnich (Eds.), The...

Ferligoj, A., Hlebec, V., 1995. Reliability of network measurements. In: Ferligoj, A., Kramberger, T. (Eds.),...

Cited by (67)

Social networks, secondary syringe exchange, and opioid agonist therapy retention among people who inject drugs in Hartford, CT
2024, International Journal of Drug Policy
Opioid agonist therapies (OAT) and harm reduction such as syringe service programs (SSP) have been shown to be effective in preventing adverse outcomes such as overdose deaths, HIV and Hepatitis C infections among people who inject drugs (PWID). The importance of social network influence on disease transmission is well established, yet the interplay between harm reduction and network structures is, generally, not well understood. This study aims to analyze how social networks can mediate the harm reduction effects associated with secondary exchange through syringe service programs (SSP) and opioid agonist therapies (OAT) among injection network members.
Sociometric data on networks on people who inject drugs from Hartford, CT, which were collected in 2012–2013, provided assessment of risk behaviors among 1574 injection network members, including participation in OAT and SSP. Subject's network characteristics were examined in relation to retention in OAT, as well as secondary syringe exchange using exponential random graph model (ERGM) and regression.
Based on the analysis, we found that probability of individuals being retained in OAT was positively associated with the OAT retention status of their peers within the network. Using simulations, we found that higher levels of positive correlation of OAT retention among network members can result in reduced risk of transmission of HIV to network partners on OAT. In addition, we found that secondary syringe exchange engagement was associated with higher probability of sharing of paraphernalia and unsterile needles at the network level.
Understanding how networks mediate risk behaviors is crucial for making progress toward ending the HIV epidemic.
Friends, family, and family friends: Predicting friendships of Dutch women
2022, Social Networks
Friends are an important source of well-being but people differ in who they consider to be friends. With a unique quantitative test of such differences based on 17,650 social relations of 706 Dutch women (aged 18–41), of whom 40% were considered friends, we examined (a) which kind of personal relations were typically identified as friends (e.g., family, colleague), (b) how this linked to relationship closeness, face-to-face and non-face-to-face contact, and (c) whether these relationship characteristics of friendships differed with age. Most friends were met at school (>70%) and 20% of family were considered friends. Friendships were often close relationships with more non-face-to-face contact, while meeting in person was less predictive. Relatively older women reported fewer friends. Even in this homogenous sample with multiple measures of tie strength, friendships were difficult to predict and often overlapped with other social roles, meaning that researchers should be careful in using friendship as distinct category.
Collecting survey-based social network information in work organizations
2022, Social Networks
Collecting social network data among organization members using surveys is challenging and requires a well-considered strategy. Based on extensive past experience with collecting social network information in work organizations with surveys, we identify and discuss four major elements of the data collection process, all linked with and dependent on the specific research question and objective: 1) negotiating access to the organization; 2) identifying the network’s boundary, the relevant formal organizational structures that affect social networks, and the sampling approach; 3) deciding how to approach research subjects and collect network data; and, 4) providing useful and ethically-sensitive feedback to the organization and its members. Decisions on each of these elements and their co-alignment, particularly with respect to the chosen research question, is crucial to a successful study. We offer guidelines and provide examples for each of these elements.
Collecting large personal networks in a representative sample of Dutch women
2021, Social Networks
In this study we report on our experiences with collecting large personal network data (25 alters) from a representative sample of Dutch women. We made use of GENSI, a recently developed tool for network data collection using interactive visual elements that has been shown to reduce respondent burden. A sample of 758 women between the ages of 18 and 40 were recruited through the LISS-panel; a longitudinal online survey of Dutch people. Respondents were asked to name exactly 25 alters, answer sixteen questions about these alters (name interpreter questions), and assess all 300 alter-alter relations. Nearly all (97%) respondents reported on 25 alters. Non-response was minimal: 92% of respondents had no missing values, and an additional 5% had fewer than 10% missing values. Listing 25 alters took 3.5  $\pm$  2.2 (mean  $\pm$  SD) minutes, and reporting on the ties between these alters took 3.6  $\pm$  1.3 min. Answering all alter questions took longest with a time of 15.2  $\pm$  5.3 min. The majority of respondents thought the questions were clear and easy to answer, and most enjoyed filling in the survey. Collecting large personal networks can mean a significant burden to respondents, but through the use of visual elements in the survey, it is clear that it can be done within reasonable time, with enjoyment and without much non-response.
Sociocultural mechanisms of conflict: Combining topic and stochastic actor-oriented models in an analysis of Afghanistan, 1979–2001
2020, Poetics
How do sociocultural dynamics shape conflict? We develop a relational understanding of how social relations, culture, and conflict are interwoven. Using this framework, we examine how combatants’ associations with cultural elements affect the interpersonal relationships underlying conflict dynamics, as well as how these relationships engender associations to cultural elements. To do so, we first introduce a novel analytical approach that synthesizes computational textual analysis and stochastic actor-oriented models of longitudinal networks. We then use our approach to analyze a two-level socio-semantic graph representing both the cultural domain and social relationships of prominent militants operating in one Afghan province, Balkh, between 1979 and 2001. Our results indicate that militants’ interpersonal comradeships rely, in part, on their connections to cultural elements and relative power. Comradeship, in turn, fosters militants’ connections to cultural elements. We conclude by discussing how conflict studies can continue to build on insights from cultural sociology, as well as how cultural sociology and socio-semantic network research can benefit from further engaging conflict studies and developing our analytical approach. We also highlight provisional insights into endogenous mechanisms of conflict resolution and cultural change.
Modeling policy networks: The case of Natura 2000 in Croatian forestry
2019, Forest Policy and Economics
Natura 2000, the largest network of protected areas in the world, is the EU's approach to the protection of its habitats and species. National implementations of the directives that define it should be based on scientific argumentation but practice has showed that they are affected by stakeholders, whose power relations and different normative standpoints have led to very diverse formal and practical implementations of the Network. Croatia as the 28th EU member country has faced the same process, where a working group has been established in order to define which forest areas should be protected under Natura 2000 and which management measures should be implemented to protect forest-dependent species.
The study follows the activities of the working group, with a goal to assess how the interpersonal influence and inter-organizational power relations have affected its decisions. Analysis is focused on the usage of social influence network theory and models of network dynamics. Primary data is drawn from 4 questionnaires distributed to 43 respondents.
Although the decisions of the working groups should have been based on scientific argumentation, 71% of variance in final policy preferences can be explained through social influence model. Results also show polarization of actors on the forestry and nature protection sector, both within the working group and between the organizations represented in it. Forestry sector was dominant on both levels, and most of the decisions are closer to its position. Modeled outcome decisions both from the interpersonal and inter-organizational level are better predictors of actual decisions than the initial opinions. This points to applicability of the presented models as decision support tools in different stages of the policy cycle; for example, in the evaluation stage they can be used to assess the empowerment of marginalized groups and to assess conditions for more consensual decision making. Results also show strong increase frequency and strength of cross-sectoral ties, which may be beneficial for the subsequent steps of the implementation of the network in Croatia.

View all citing articles on Scopus

¹: E-mail: [email protected].

View full text

Evaluation of social network measurement instruments

Abstract

Introduction

Section snippets

Experimental design

Mean levels of data quality

Conclusions

Acknowledgements

Social Networks

Social Networks

Social Networks

Social Networks

Social Networks

Social Networks

Social Networks

Construct validity and error components of survey measures: a structural modeling approach

Public Opinion Quarterly

How much of a network does the GSS and RSW dredge up?

Social Networks

Network items and the general social survey

Social Networks

A note on sociometric order in the general social survey network data

Social Networks