Improving Traditional Nonresponse Bias Adjustments: Combining Statistical Properties with Social Theory

Improving Traditional Nonresponse Bias Adjustments: Combining Statistical Properties with Social... Abstract Declining response rates have led to increasing reliance on nonresponse adjustment as a way to reduce the risk of nonresponse bias. Unfortunately, the auxiliary variables used in most surveys frequently do not satisfy the sine qua non for effective adjustment: significant associations with both nonresponse and the survey variables. We describe an approach to selecting weight variables that identifies candidates, such as voting and volunteering, not usually considered. In an analysis of the 2012 General Social Survey (GSS), we show that voting eligibility, voter turnout, and candidate choice meet the statistical conditions for nonresponse bias reduction, as does volunteering. Voting and volunteering are strongly associated with participation in the GSS and also associated with a wide array of variables measured in the GSS. Adjustments using either voting benchmarks or volunteering benchmarks result in significant changes to GSS estimates compared to traditional adjustments. As this approach shows promise, we identify several lines of research needed to inform its implementation. 1. INTRODUCTION Response rates in US surveys of the general public have been declining for more than a half century, a trend that shows no sign of reversing. Although lower response rates do not necessarily increase nonresponse bias (Groves and Peytcheva 2008), they raise the risk of such bias. As a result, surveys increasingly rely on nonresponse adjustments to protect against this risk. Weighting can reduce the risk of nonresponse bias by bringing the distribution(s) of respondent characteristics into line with the distribution(s) either for the full sample or for the population. But the effectiveness of this procedure depends on the properties of the variables (known as “auxiliary” variables) used to construct the weights. The ideal auxiliary variables for nonresponse adjustment are strongly associated with both nonresponse and the survey variables of interest (e.g., Little 1986; Little and Vartivarian 2005; Groves 2006,). Yet in many surveys, the auxiliary variables are associated with only one or the other (e.g., Kreuter, Olson, Wagner, Yan, Ezzati-Rice et al. 2010), if either. In practice, adjustments for nonresponse usually model the likelihood of an interview, resulting in the selection of variables that are associated with nonresponse, but not the survey variables of interest. In addition to typically leading to little bias reduction, this has the unfortunate effect of reducing the precision of the estimates due to the variability of the weights.1 The main sampling frames for the US general population—lists of addresses and telephone numbers—are variable-poor, and the adjustment variables typically chosen from data bases to which the frames can be linked or from observations made during data collection usually do not satisfy the twin conditions necessary for reducing nonresponse bias. In particular, The aggregate nature of census estimates, to which addresses and phone numbers are most often merged, combined with their limitation to mainly demographic topics generally lead to ineffective nonresponse bias adjustments (Biemer and Peytchev 2013). For example, the average age in a county, ZIP code, or even Census Block Group, is not highly predictive of individual-level nonresponse and even less predictive of individual responses to most survey questions. Because census aggregate data lack both conditions and the resulting weight adjustments add very little variability, use of these data has been found to neither help (by reducing bias) nor hurt (by increasing variance) the accuracy of survey estimates (Biemer and Peytchev 2013). Individual-level commercial data has problems of matching, missing data, and data quality and, probably as a result, the data has been found to be of little value in adjusting for nonresponse (Peytchev and Raghunathan 2013; Pasek, Jang, Cobb, Dennis, and Disogra 2014; West, Wagner, Hubbard, and Gu 2015). Paradata, such as number of call attempts and whether anyone ever refused, can dramatically improve model fit in nonresponse adjustments, because they reflect actual nonresponse; that is, they predict final nonresponse on the basis of prior refusals and noncontacts. But although the number of call attempts, for example, can be a near perfect predictor of nonresponse, it has been found to lack association with survey variables (Curtin, Presser, and Singer 2000). Thus, weighting using this approach can lead to loss of precision in survey estimates (due to variation in the weights) without commensurate reduction in nonresponse bias (Wagner, Valliant, Hubbard, and Jiang 2014). The other main source of paradata—interviewer observations—might, in principle, be more informative for nonresponse adjustment, but its utility is diminished by substantial measurement error (West 2013). Summary demographic estimates for the population from the Census Bureau are probably the most frequently used auxiliary information, but a meta-analysis of nonresponse bias studies found that nonresponse bias in demographic characteristics and nonresponse bias in substantive survey estimates were generally unrelated (Peytcheva and Groves 2009). In sum, there is increased reliance on nonresponse adjustments, yet the auxiliary variables used to make adjustments often lack at least one of the two essential statistical properties for such variables: significant associations with survey nonresponse and with a wide array of substantive survey variables. In the next sections, we identify voting and volunteering as two candidates for nonresponse adjustments that are theoretically and empirically motivated from the vantage point of a social science perspective; demonstrate the results of using voting and volunteering to adjust estimates in a major national survey; and discuss future research that is needed to further explore this approach. 2. CIVIC DUTY Civic duty—or social integration—figures prominently in theoretical treatments of both survey participation (Dillman 1978; Goyder 1987; Groves and Couper 1998; Groves, Singer, and Corning 2000) and a diverse array of substantive matters ranging from political activity to subjective well-being (Amaya and Presser 2017). Moreover, two key civic behaviors—voting and volunteering—have been shown to be associated with survey participation. Thus, these indicators seem particularly promising for use as auxiliary variables in nonresponse adjustment. 2.1 Voting One of the greatest challenges to political surveys is that political participation is positively associated with survey participation (e.g., Knack 1992; Voogt 2004; Keeter, Kennedy, Dimock, Best, and Craighill 2006). Paradoxically, this relationship presents an opportunity to correct for survey nonresponse. This is because voting is also associated with many seemingly unrelated survey variables, such as health outcomes (Subramanian, Huijts, and Perkins 2009; Shin and McCarthy 2013). Thus, indicators of political participation are prime candidates for variables associated with both nonresponse and a range of survey variables of interest. For a somewhat different reason, candidate choice may also be affected by nonresponse. Support for some candidates could be seen as socially undesirable, potentially suppressing survey participation among these voters. 2.1.1 Strengths of using political participation and orientation One of the advantages of using voting and support for a particular candidate is that very high-quality benchmark estimates are available for both. The Federal Election Commission (FEC) releases counts and rates for voting among the eligible adult population, along with the allocation of support for each candidate. 2.1.2 Limitations of using political participation and orientation The main limitation of using voting as an adjustment variable is that record-check studies have consistently found it to be overreported in surveys, much like other socially desirable behaviors (for an overview, see Tourangeau, Rips, and Rasinski 2000). Recent research, however, suggests that much of the observed error may be due to errors in the records or matching procedures (Berent, Krosnick, and Lupia 2016), and several studies have concluded that nonvoters are disproportionately likely to be survey nonrespondents (Tourangeau, Groves, and Redline 2010; Jackman and Spahn 2014). Nonetheless, misreporting could affect our findings. With respect to political orientation, we found no studies showing that Republicans versus Democrats and conservatives versus liberals vary in the propensity to overreport voting. There is some evidence, however, that respondents overreport having voted for the winning candidate (Wright 1993). We return to this issue below. 2.2 Volunteering Research has shown a strong link between volunteering and survey participation. For example, Couper, Singer, and Kulka (1998) found that those who reported being involved in community groups and organizations were more likely to have returned their decennial census form. Likewise, Abraham, Helms, and Presser (2009) found that the response rate to the American Time Use Survey (ATUS) was 35% higher among those who reported having done volunteer work in the preceding year than among those who had reported not volunteering (the volunteering questions were asked in the Current Population Survey from which the ATUS draws its sample). Indeed, this evidence led Abraham, Helms, and Presser (2009: 1162–1163) to suggest that the volunteering estimate from the annual Current Population Survey (CPS) volunteering supplement could be a good nonresponse adjustment factor in general population surveys. 2.2.1 Strengths of using volunteering There are at least three major strengths of using volunteering for nonresponse adjustments. First, there is an up-to-date benchmark, as annual population estimates are available from the CPS September Supplement. Second, because the CPS is also an interviewer-administered survey, the population estimate used in adjustments is subject to overreporting due to social desirability as is the survey being adjusted. Third, since the CPS is a survey rather than a variable-poor administrative data source, volunteering estimates can be obtained for subgroups to further improve the adjustments. For example, adjustment cells could be formed by volunteering by sex by age and by education, rather than ignoring the interactions and adjusting to the marginal distributions of volunteering, sex, age, and education. 2.2.2 Limitations of using volunteering Although the CPS is a large national survey with a very high response rate, the CPS estimates are themselves subject to nonresponse bias. In addition, if one uses the CPS volunteering estimates for adjustments in a self-administered survey, mode differences may lead to bias. Of course, these same limitations apply to conventional CPS adjustments using demographic characteristics like age, race, and sex. 3. CASE STUDIES In this section, we present two empirical evaluations of theoretically motivated adjustments for potential nonresponse bias, one using voting and the other using volunteering. We analyze the 2012 General Social Survey (GSS), a nationally representative probability-based survey, with 1,974 completed interviews. The 2012 GSS subsampled nonrespondents to reduce cost and increase the response rate and achieved an AAPOR Response Rate 5 (AAPOR 2015) of 71.4%. In Study 1, we use all 1,974 interviews. In Study 2, we use the random subsample of 1,299 interviews asked the volunteering questions. The GSS does not implement elaborate nonresponse or any poststratification adjustments. Instead, it relies on a sample-based geographic nonresponse adjustment that is equal to the inverse of the response rate in the primary sampling units (PSUs, or “National Frame Areas” in the GSS documentation). Thus, the GSS nonresponse adjustments differ from those of many other surveys, which include poststratification to key demographic characteristics of the population. As a result, we augmented the GSS weights, in order to broaden our conclusions to the wider array of surveys that apply demographic poststratification. The properties of the auxiliary data determine whether sample-based or population-based weighting adjustments are applied. Sample-based adjustments require data on both respondents and nonrespondents. Commonly, weighting class adjustments, response propensity models, or weighting class adjustments based on propensity strata are then used to adjust the weights back to the full sample. Population-based adjustments, by contrast, do not require data on nonrespondents and rely on population counts or distributions for the auxiliary variables. Weights are calibrated to those known population benchmarks. This calibration can be done using different statistical approaches, typically through poststratification when the full cross-classification for the population is known and used, and raking (through iterative methods) to satisfy only marginal distributions. Because weights are calibrated to the population rather than the sample, these adjustments correct for both coverage and nonresponse bias. (For a good overview of nonresponse adjustments, see Brick and Kalton 1996.) We use calibration, since our main interest is to explore the use of auxiliary data for which only population totals are known. For each study, we computed two sets of calibration weights. The first set applies demographic population controls2 to the GSS weights that account for selection probability and the geographic nonresponse adjustment, using iterative proportionate fitting (often referred to as “raking”). In the second set, we add controls for voting (Study 1) and volunteering (Study 2). This led to an expected increase in the variability of the weights. One plus the square of the coefficient of variation of the weights (1 + CV(w)2, or 1 + L) is a useful approximation of the expected loss in precision due to weighting (Kish 1965). For Study 1, this measure increased from 1.73 to 2.13, and for Study 2, it increased from 1.80 to 2.15. The differences in weighted estimates with each set of weights yield the nonresponse bias that is identified by adding the substantive variables in the adjustment, conditional on the adjustment model that includes the GSS geographic nonresponse adjustments and the additional calibration to demographic controls. There are three types of evaluations that are of interest, which we present for each study: Is the auxiliary information associated with nonresponse? We show the magnitude of the bias in the weighted voting estimates (voting eligibility, voting among eligible voters, and candidate choice among voters) and in the weighted volunteering estimates. Is the auxiliary information associated with the survey variables? We present the association between the voting-related or volunteering-related adjustment variables and the survey variables among the respondents. Does the use of the auxiliary information lead to changes in the weighted survey estimates?We evaluate the extent to which adding the constructed voting or volunteering variable to the weighting adjustments leads to changes in the survey estimates. In addition, we also present the impact on total error, using the mean square error (MSE) to combine the bias and variance tradeoff into a single measure. Section 4.1 presents Study 1, section 4.2 presents Study 2, and section 4.3 shows the impact on MSE in both studies. 3.1 Study 1: Voting The survey included the following two voting questions, for those eligible to vote: “In 2008, you remember that Obama ran for President on the Democratic ticket against McCain for the Republicans. Do you remember for sure whether or not you voted in that election?” And if answered affirmatively, it was immediately followed by, “Did you vote for Obama or McCain?” Based on the responses to these questions, the respondents were classified into one of five mutually exclusive categories (unweighted counts in parentheses): Not eligible to vote (159)—in additional to citizenship, this group also includes age-ineligibility due to the time lag between the election in 2008 and the survey in 2012. Did not vote, do not know, or refused (521)—this includes 22 who said that they did not know whether they voted and four who refused to answer. Voted, for Obama (795). Voted, for McCain (472). Voted, for another candidate (27). There were forty-five respondents who reported having voted, but did not name a candidate. Those who self-identified as strong Democrat to Independent were imputed as Obama voters, those who self-identified as Independent, leaning towards Republican to strong Republican were imputed as McCain voters, and the remaining three respondents were randomly assigned to a candidate. The estimates are virtually identical if these respondents are excluded from the analysis or if the imputation is done based on political ideology (liberal to conservative). Control totals were then calculated using a combination of population estimates for noncitizens, age-ineligibility, and non-voting from the American Community Survey (ACS) and the CPS (US Census Bureau 2012), and number of voters and candidate choice from the FEC (Federal Election Commission 2009). The CPS and FEC estimates of number of voters differ by only 0.1% (131,144,000 versus 131,313,820). The ACS and CPS estimates have variances themselves that are ignored in practice due to their relatively small magnitudes (due to the large sample sizes of the ACS and CPS) and complexity in reflecting them in estimation (Dever and Valliant 2010; Dever and Valliant 2016). They are not the focus in this study, but we encourage further investigations on this topic, especially in the case of surveys that implement control totals with larger variances. 3.1.1 Conditional relationship between the voting adjustment variables and nonresponse Figure 1 shows that the GSS estimated percentage of the population eligible to vote in 2008 based on citizenship and age in 2008 was 90%, while the CPS-derived population estimate was 84%. This difference is not unique to the GSS—surveys in other countries have shown that immigrants (many, if not most, of whom would not be eligible to vote) participate in surveys at lower rates (Rendall, Tomassini, and Elliot 2003). Those eighteen to twenty-one years old at the time of the 2012 survey, who were ineligible to vote in 2008, are also likely to have responded at lower rates (Mulry 2014), and the regular age-based weighting adjustments do not take individual years of birth into account, but instead use broader age categories. Figure 1. View largeDownload slide GSS Weighted Estimates (Using GSS Weights Also Adjusted to Population Demographic Characteristics) and Population Benchmark Estimates for Percent of the Adult Population in 2012 Who Were Eligible to Vote in 2008, Percent of the Voting-Eligible Population Who Voted in 2008, and Percent of Voters in 2008 By Candidate Choice. Bars represent standard errors. Figure 1. View largeDownload slide GSS Weighted Estimates (Using GSS Weights Also Adjusted to Population Demographic Characteristics) and Population Benchmark Estimates for Percent of the Adult Population in 2012 Who Were Eligible to Vote in 2008, Percent of the Voting-Eligible Population Who Voted in 2008, and Percent of Voters in 2008 By Candidate Choice. Bars represent standard errors. Of possibly even greater importance is the effect of the adjustment on the estimated percent who voted in 2008, among those eligible to vote. Figure 1 shows that the weighted estimate (72%) is substantially higher than the actual voter turnout in 2008 of 64% (US Census Bureau 2012). The bias is also in the direction that would be expected from social desirability—the overreporting of voting in surveys. The third component of the additional adjustment, candidate choice, also shows substantial imbalance in the respondent pool (figure 1). The GSS nonresponse adjustment with demographic controls estimates 57% to have voted for Barack Obama and 41% for John McCain (a sixteen-percentage point difference)—far from the actual election outcome of 53% for Obama and 46% for McCain (a 7-percentage point difference), which is reflected in the estimates using candidate choice in the poststratification. This overrepresentation of voters supporting Obama may seem surprising since most pre-election polls underestimated support for Obama (Cohn 2014), but as Merkle and Edelman (2009) demonstrated, even the direction of the bias in political support can be a function of the survey protocol. We also note that, just as for voter turnout, part of this bias may be due to social desirability—the tendency to overreport support for the winning candidate. Nonetheless, the observed difference in the GSS underscores the potential importance of this variable in postsurvey adjustments, if Obama and McCain supporters also differ on key survey variables. This is examined next. 3.1.2 Relationship between the voting adjustment variables and survey variables Table 1 shows the correlations of the self-reported eligibility to vote, voting in 2008 (among eligible voters), and voting for Obama (among voters) with responses to fifteen survey questions selected prior to the analysis to represent a diverse array of attitudes and behaviors, and asked of all respondents. All three voting-related questions show significant associations with the survey variables, with voting (conditional on eligibility) and candidate choice (among self-reported voters) yielding some moderate in size. Interestingly, the significant correlations are somewhat different across the three components (voting eligibility, voting, and candidate choice). All of the selected survey variables were correlated with at least one of the three voting variables, and the magnitudes of the statistically significant correlations for the other variables varied widely, from 0.03 to 0.31. Table 1. Selection-Weighted Correlations of Voting Eligibility, Having Voted in 2008 (Among Eligible to Vote), and Voting for Obama (Among Self-Reported Voters) with Fifteen GSS Survey Variables Eligible to vote n = 1,974 Voted in 2008 n = 1,815 Voted for Obama n = 1,294 Fair or poor health 0.01 −0.13a 0.13a Very or pretty happy −0.01 0.09a −0.08a Life exciting −0.04 0.05 −0.07a Most people try to be helpful 0.07a 0.19a −0.01 Most try to take advantage −0.10a −0.13a 0.03a Most people can be trusted 0.04 0.18a −0.04a Family income below average −0.01 −0.14a 0.09a Read newspaper every day 0.10a 0.16a 0.02 No religion −0.05a 0.00 0.19a Donated blood in past year 0.00 0.08a 0.00a Donated to charity in past year 0.14a 0.25a −0.09a Support birth control to teens 14 to 16 years old −0.06a −0.06a 0.21 A same-sex female couple can bring up a child just as well −0.08a 0.04 0.31 Oppose capital punishment −0.08a 0.02 0.31 Courts are too harsh with criminals −0.11a −0.09a 0.19 Eligible to vote n = 1,974 Voted in 2008 n = 1,815 Voted for Obama n = 1,294 Fair or poor health 0.01 −0.13a 0.13a Very or pretty happy −0.01 0.09a −0.08a Life exciting −0.04 0.05 −0.07a Most people try to be helpful 0.07a 0.19a −0.01 Most try to take advantage −0.10a −0.13a 0.03a Most people can be trusted 0.04 0.18a −0.04a Family income below average −0.01 −0.14a 0.09a Read newspaper every day 0.10a 0.16a 0.02 No religion −0.05a 0.00 0.19a Donated blood in past year 0.00 0.08a 0.00a Donated to charity in past year 0.14a 0.25a −0.09a Support birth control to teens 14 to 16 years old −0.06a −0.06a 0.21 A same-sex female couple can bring up a child just as well −0.08a 0.04 0.31 Oppose capital punishment −0.08a 0.02 0.31 Courts are too harsh with criminals −0.11a −0.09a 0.19 NOTE.—Using the nonresponse-adjusted poststratified weights yields similar results. a Significant at the 0.05 level, based on the Wald F-statistic, accounting for the complex survey design. Table 1. Selection-Weighted Correlations of Voting Eligibility, Having Voted in 2008 (Among Eligible to Vote), and Voting for Obama (Among Self-Reported Voters) with Fifteen GSS Survey Variables Eligible to vote n = 1,974 Voted in 2008 n = 1,815 Voted for Obama n = 1,294 Fair or poor health 0.01 −0.13a 0.13a Very or pretty happy −0.01 0.09a −0.08a Life exciting −0.04 0.05 −0.07a Most people try to be helpful 0.07a 0.19a −0.01 Most try to take advantage −0.10a −0.13a 0.03a Most people can be trusted 0.04 0.18a −0.04a Family income below average −0.01 −0.14a 0.09a Read newspaper every day 0.10a 0.16a 0.02 No religion −0.05a 0.00 0.19a Donated blood in past year 0.00 0.08a 0.00a Donated to charity in past year 0.14a 0.25a −0.09a Support birth control to teens 14 to 16 years old −0.06a −0.06a 0.21 A same-sex female couple can bring up a child just as well −0.08a 0.04 0.31 Oppose capital punishment −0.08a 0.02 0.31 Courts are too harsh with criminals −0.11a −0.09a 0.19 Eligible to vote n = 1,974 Voted in 2008 n = 1,815 Voted for Obama n = 1,294 Fair or poor health 0.01 −0.13a 0.13a Very or pretty happy −0.01 0.09a −0.08a Life exciting −0.04 0.05 −0.07a Most people try to be helpful 0.07a 0.19a −0.01 Most try to take advantage −0.10a −0.13a 0.03a Most people can be trusted 0.04 0.18a −0.04a Family income below average −0.01 −0.14a 0.09a Read newspaper every day 0.10a 0.16a 0.02 No religion −0.05a 0.00 0.19a Donated blood in past year 0.00 0.08a 0.00a Donated to charity in past year 0.14a 0.25a −0.09a Support birth control to teens 14 to 16 years old −0.06a −0.06a 0.21 A same-sex female couple can bring up a child just as well −0.08a 0.04 0.31 Oppose capital punishment −0.08a 0.02 0.31 Courts are too harsh with criminals −0.11a −0.09a 0.19 NOTE.—Using the nonresponse-adjusted poststratified weights yields similar results. a Significant at the 0.05 level, based on the Wald F-statistic, accounting for the complex survey design. 3.1.3 Change in survey estimates with the addition of the voting-related variables Does the addition of the voting variable in the weighting adjustments alter the survey estimates? Figure 2 shows that the changes are generally small, but three of the fifteen estimates are significantly different.3 The largest change is 4%. The addition of voting contributed to most of the shifts, though voting eligibility seems to be the driving factor in two cases. The full estimates under each set of weights, differences, and significance tests are included in Appendix A. The changes in the estimates were substantially larger when less elaborate population demographic controls were used in the weighting (results not presented). Figure 2. View largeDownload slide Differences in Weighted Estimates for Fifteen GSS Variables, Relative to the Nonresponse and Poststratified Estimates Without Voting Eligibility, Voting, and Candidate Choice. * Significant difference between weighted estimates using GSS nonresponse adjustment with raking to age, sex, education, and race, while adding voting eligibility, voting, and candidate choice to the weighting adjustment, accounting for the complex survey design. Figure 2. View largeDownload slide Differences in Weighted Estimates for Fifteen GSS Variables, Relative to the Nonresponse and Poststratified Estimates Without Voting Eligibility, Voting, and Candidate Choice. * Significant difference between weighted estimates using GSS nonresponse adjustment with raking to age, sex, education, and race, while adding voting eligibility, voting, and candidate choice to the weighting adjustment, accounting for the complex survey design. 3.2 Study 2: Volunteering Approximately two-thirds (1,299) of the 2012 GSS respondents were randomly assigned to be asked the two questions from the 2012 CPS September Supplement on volunteering that CPS uses to construct its volunteering estimate: “Since [CURRENT MONTH] 1st of last year, have you done any volunteer activities through or for an organization?” And if the respondent did not respond “yes,” it was followed by: “Sometimes people don’t think of activities they do infrequently or activities they do for children’s schools or youth organizations as volunteer activities. Since [CURRENT MONTH] 1st of last year, have you done any of these types of volunteer activities?” Respondents who answered “yes” to either question were classified as having volunteered in the past year (613), and those who said “no” to both questions were classified as not having volunteered (681). Five respondents who refused the two questions were randomly imputed into the two categories, four into the first group and one into the second group. The population proportion for volunteering used in weighting was derived from the 2012 CPS data. 3.2.1 Conditional relationship between the volunteering adjustment variable and nonresponse We first compare the GSS weighted estimate for volunteering, including demographic adjustments, with the CPS estimate. Figure 3 shows an exceptionally large difference, with the GSS estimate about twice as large (48%) as the CPS-based adjusted estimate (26%), much larger than the difference for voting in Study 1. As both the GSS and the CPS are interviewer-administered surveys using the same two volunteering questions, we attribute the difference to nonresponse. Although one might argue that the context of the surveys, one on social topics and the other on economic topics, could induce different levels of social desirability reporting bias. The same is true for nonresponse—those agreeing to participate in a social survey may be more socially involved. Figure 3. View largeDownload slide GSS Weighted Estimates (Using GSS Weights Also Adjusted to Population Demographic Characteristics) and CPS Population Benchmark Estimates for Percent of the Adult Population in 2012 Who Volunteered. Bars represent standard errors. Figure 3. View largeDownload slide GSS Weighted Estimates (Using GSS Weights Also Adjusted to Population Demographic Characteristics) and CPS Population Benchmark Estimates for Percent of the Adult Population in 2012 Who Volunteered. Bars represent standard errors. 3.2.2 Relationship between the volunteering adjustment variable and survey variables The volunteering variable was significantly correlated with more than half the selected GSS variables, shown in table 2. The pattern is very similar to the correlations with voting, presented in the middle column in table 2, which is consistent with expectations under the shared mechanism of civic duty. As in Study 1 with voting, not all variables were correlated with volunteering, and among the statistically significant correlations, they ranged from 0.07 to as high as 0.32. Table 2. Selection-Weighted Correlations of Volunteering with 15 GSS Survey Variables Volunteered (n = 1,299) Fair or poor health −0.12a Very or pretty happy 0.07a Life exciting 0.11a Most people try to be helpful 0.05 Most try to take advantage −0.14a Most people can be trusted 0.14a Family income below average −0.08a Read newspaper every day 0.05 No religion −0.07a Donated blood in past year 0.15a Donated to charity in past year 0.32a Support birth control to teens 14 to 16 years old −0.04 A same-sex female couple can bring up a child just as well −0.01 Oppose capital punishment −0.04 Courts are too harsh with criminals 0.01 Volunteered (n = 1,299) Fair or poor health −0.12a Very or pretty happy 0.07a Life exciting 0.11a Most people try to be helpful 0.05 Most try to take advantage −0.14a Most people can be trusted 0.14a Family income below average −0.08a Read newspaper every day 0.05 No religion −0.07a Donated blood in past year 0.15a Donated to charity in past year 0.32a Support birth control to teens 14 to 16 years old −0.04 A same-sex female couple can bring up a child just as well −0.01 Oppose capital punishment −0.04 Courts are too harsh with criminals 0.01 NOTE.—Using the nonresponse-adjusted poststratified weights yields similar results. a Significant at the 0.05 level, based on the Wald F-statistic, accounting for the complex survey design. Table 2. Selection-Weighted Correlations of Volunteering with 15 GSS Survey Variables Volunteered (n = 1,299) Fair or poor health −0.12a Very or pretty happy 0.07a Life exciting 0.11a Most people try to be helpful 0.05 Most try to take advantage −0.14a Most people can be trusted 0.14a Family income below average −0.08a Read newspaper every day 0.05 No religion −0.07a Donated blood in past year 0.15a Donated to charity in past year 0.32a Support birth control to teens 14 to 16 years old −0.04 A same-sex female couple can bring up a child just as well −0.01 Oppose capital punishment −0.04 Courts are too harsh with criminals 0.01 Volunteered (n = 1,299) Fair or poor health −0.12a Very or pretty happy 0.07a Life exciting 0.11a Most people try to be helpful 0.05 Most try to take advantage −0.14a Most people can be trusted 0.14a Family income below average −0.08a Read newspaper every day 0.05 No religion −0.07a Donated blood in past year 0.15a Donated to charity in past year 0.32a Support birth control to teens 14 to 16 years old −0.04 A same-sex female couple can bring up a child just as well −0.01 Oppose capital punishment −0.04 Courts are too harsh with criminals 0.01 NOTE.—Using the nonresponse-adjusted poststratified weights yields similar results. a Significant at the 0.05 level, based on the Wald F-statistic, accounting for the complex survey design. 3.2.3 Change in survey estimates with the addition of the volunteering variable Figure 4 shows that the addition of volunteering to the weighting adjustment led to significantly different estimates for five of the fifteen variables, just as in the voting adjustment, but only two of the variables were shared. The magnitudes of the shifts in the estimates are larger than those in figure 2—the largest change is 13%—but so are the standard errors since the volunteering variables were administered to only two-thirds of the respondents.4 The estimates under each set of weights, differences, and significance tests are included in Appendix B. Figure 4. View largeDownload slide Relative Differences in Weighted Estimates for Fifteen GSS Variables after Including Volunteering in the Adjustments. * Significant difference between weighted estimates using GSS nonresponse adjustment with raking to age, sex, education, and race, while adding volunteering to the weighting adjustment, accounting for the complex survey design. Figure 4. View largeDownload slide Relative Differences in Weighted Estimates for Fifteen GSS Variables after Including Volunteering in the Adjustments. * Significant difference between weighted estimates using GSS nonresponse adjustment with raking to age, sex, education, and race, while adding volunteering to the weighting adjustment, accounting for the complex survey design. 3.3 Impact on Total Error Our focus in Study 1 and Study 2 was the relationship between the auxiliary information with nonresponse and the survey variables, and the resultant ability to adjust the survey estimates. In our introduction, we noted that a poor adjustment variable would disproportionately increase the variance of the estimate through increased weight variation, relative to the change in the estimate. A measure of total error to evaluate the variance and bias tradeoff is the mean square error (MSE), which is equal to the sum of the bias squared (⁠ By2 ⁠) and the variance (⁠ Vy ⁠) of the estimate from variable (⁠ y ⁠): MSE=By2+Vy ⁠. For the estimate of the bias, we treat the estimate using the augmented weights that include the voting variables or the volunteering variable as the unbiased estimate. Table 3 shows the MSEs for the fifteen GSS variables, and the relative change in MSEs from adding the voting (Study 1) and volunteering (Study 2) variables. Despite the modest sample sizes that increase the relative importance of the variance estimates, the bias reduction was sufficient to reduce the overall MSEs for almost all the estimates. On average, the MSEs were reduced by 4% from adding voting and by 27% from adding volunteering. The larger benefit from adding volunteering is consistent with the large respondent deviation from the benchmark for volunteering (figure 3), which allowed for larger shifts in the weighted estimates in Study 2. The reduction in MSE is also highly dependent on the other variables in the adjustment. The reduction in MSE when adding voting was almost four times larger (change in MSE of -14.4% instead of -3.9%) when simpler, dichotomized versions of the demographic variables were used in the weight calibration. Our estimate of the reduction in MSE is also likely an underestimate, since the usual linearization approach to variance estimation, used on the GSS, does not capitalize on the stabilizing effect of calibration to variables that are highly correlated with the survey variable of interest. Future studies should examine the impact of the variance estimation approach. Table 3. Estimated MSEs and Relative Change in MSEs for the Fifteen GSS Survey Variables, from Adding Voting (Study 1) and Volunteering (Study 2) Voting (Study 1) Volunteering (Study 2) MSE prior to voting adjustment (A) MSE post voting adjustment (B) Relative change (B-A)/A MSE prior to volunteering adjustment (C) MSE post volunteering adjustment (D) Relative change (D-C)/C Fair or poor health 0.02% 0.02% −2.4% 0.06% 0.04% −31.1% Very or pretty happy 0.01% 0.01% 5.3% 0.02% 0.02% −4.6% Life exciting 0.04% 0.04% 4.4% 0.12% 0.07% −36.3% Most people try to be helpful 0.04% 0.03% −17.5% 0.05% 0.05% 11.0% Most try to take advantage 0.04% 0.04% −1.0% 0.12% 0.09% −24.0% Most people can be trusted 0.04% 0.04% −4.9% 0.11% 0.08% −25.2% Family income below average 0.02% 0.02% 5.0% 0.03% 0.03% 10.4% Read newspaper every day 0.04% 0.04% −11.9% 0.04% 0.04% −11.6% No religion 0.01% 0.01% −17.3% 0.06% 0.02% −57.3% Donated blood in past year 0.02% 0.02% −5.2% 0.08% 0.04% −55.8% Donated to charity in past year 0.04% 0.03% −27.9% 0.36% 0.06% −82.2% Support birth control to teens 14 to 16 years old 0.03% 0.03% 4.8% 0.05% 0.03% −32.5% A same-sex female couple can bring up a child just as well 0.06% 0.05% −16.8% 0.07% 0.05% −27.0% Oppose capital punishment 0.02% 0.02% −5.2% 0.05% 0.04% −22.5% Courts are too harsh with criminals 0.01% 0.01% 32.4% 0.01% 0.01% −12.8% Average relative change in MSE −3.9% −26.8% Voting (Study 1) Volunteering (Study 2) MSE prior to voting adjustment (A) MSE post voting adjustment (B) Relative change (B-A)/A MSE prior to volunteering adjustment (C) MSE post volunteering adjustment (D) Relative change (D-C)/C Fair or poor health 0.02% 0.02% −2.4% 0.06% 0.04% −31.1% Very or pretty happy 0.01% 0.01% 5.3% 0.02% 0.02% −4.6% Life exciting 0.04% 0.04% 4.4% 0.12% 0.07% −36.3% Most people try to be helpful 0.04% 0.03% −17.5% 0.05% 0.05% 11.0% Most try to take advantage 0.04% 0.04% −1.0% 0.12% 0.09% −24.0% Most people can be trusted 0.04% 0.04% −4.9% 0.11% 0.08% −25.2% Family income below average 0.02% 0.02% 5.0% 0.03% 0.03% 10.4% Read newspaper every day 0.04% 0.04% −11.9% 0.04% 0.04% −11.6% No religion 0.01% 0.01% −17.3% 0.06% 0.02% −57.3% Donated blood in past year 0.02% 0.02% −5.2% 0.08% 0.04% −55.8% Donated to charity in past year 0.04% 0.03% −27.9% 0.36% 0.06% −82.2% Support birth control to teens 14 to 16 years old 0.03% 0.03% 4.8% 0.05% 0.03% −32.5% A same-sex female couple can bring up a child just as well 0.06% 0.05% −16.8% 0.07% 0.05% −27.0% Oppose capital punishment 0.02% 0.02% −5.2% 0.05% 0.04% −22.5% Courts are too harsh with criminals 0.01% 0.01% 32.4% 0.01% 0.01% −12.8% Average relative change in MSE −3.9% −26.8% Table 3. Estimated MSEs and Relative Change in MSEs for the Fifteen GSS Survey Variables, from Adding Voting (Study 1) and Volunteering (Study 2) Voting (Study 1) Volunteering (Study 2) MSE prior to voting adjustment (A) MSE post voting adjustment (B) Relative change (B-A)/A MSE prior to volunteering adjustment (C) MSE post volunteering adjustment (D) Relative change (D-C)/C Fair or poor health 0.02% 0.02% −2.4% 0.06% 0.04% −31.1% Very or pretty happy 0.01% 0.01% 5.3% 0.02% 0.02% −4.6% Life exciting 0.04% 0.04% 4.4% 0.12% 0.07% −36.3% Most people try to be helpful 0.04% 0.03% −17.5% 0.05% 0.05% 11.0% Most try to take advantage 0.04% 0.04% −1.0% 0.12% 0.09% −24.0% Most people can be trusted 0.04% 0.04% −4.9% 0.11% 0.08% −25.2% Family income below average 0.02% 0.02% 5.0% 0.03% 0.03% 10.4% Read newspaper every day 0.04% 0.04% −11.9% 0.04% 0.04% −11.6% No religion 0.01% 0.01% −17.3% 0.06% 0.02% −57.3% Donated blood in past year 0.02% 0.02% −5.2% 0.08% 0.04% −55.8% Donated to charity in past year 0.04% 0.03% −27.9% 0.36% 0.06% −82.2% Support birth control to teens 14 to 16 years old 0.03% 0.03% 4.8% 0.05% 0.03% −32.5% A same-sex female couple can bring up a child just as well 0.06% 0.05% −16.8% 0.07% 0.05% −27.0% Oppose capital punishment 0.02% 0.02% −5.2% 0.05% 0.04% −22.5% Courts are too harsh with criminals 0.01% 0.01% 32.4% 0.01% 0.01% −12.8% Average relative change in MSE −3.9% −26.8% Voting (Study 1) Volunteering (Study 2) MSE prior to voting adjustment (A) MSE post voting adjustment (B) Relative change (B-A)/A MSE prior to volunteering adjustment (C) MSE post volunteering adjustment (D) Relative change (D-C)/C Fair or poor health 0.02% 0.02% −2.4% 0.06% 0.04% −31.1% Very or pretty happy 0.01% 0.01% 5.3% 0.02% 0.02% −4.6% Life exciting 0.04% 0.04% 4.4% 0.12% 0.07% −36.3% Most people try to be helpful 0.04% 0.03% −17.5% 0.05% 0.05% 11.0% Most try to take advantage 0.04% 0.04% −1.0% 0.12% 0.09% −24.0% Most people can be trusted 0.04% 0.04% −4.9% 0.11% 0.08% −25.2% Family income below average 0.02% 0.02% 5.0% 0.03% 0.03% 10.4% Read newspaper every day 0.04% 0.04% −11.9% 0.04% 0.04% −11.6% No religion 0.01% 0.01% −17.3% 0.06% 0.02% −57.3% Donated blood in past year 0.02% 0.02% −5.2% 0.08% 0.04% −55.8% Donated to charity in past year 0.04% 0.03% −27.9% 0.36% 0.06% −82.2% Support birth control to teens 14 to 16 years old 0.03% 0.03% 4.8% 0.05% 0.03% −32.5% A same-sex female couple can bring up a child just as well 0.06% 0.05% −16.8% 0.07% 0.05% −27.0% Oppose capital punishment 0.02% 0.02% −5.2% 0.05% 0.04% −22.5% Courts are too harsh with criminals 0.01% 0.01% 32.4% 0.01% 0.01% −12.8% Average relative change in MSE −3.9% −26.8% 3.4 Evaluation of Reduction of Bias Against Population Benchmarks The analysis in Sections 3.1 and 3.2 focused on the relative change in estimates when voting or volunteering are included in the weight calibration adjustment. Section 3.3 presented estimates of MSE, assuming that the augmented weights produce less bias in the survey estimates. In this instance, we have the ability to evaluate this fundamental assumption, using our population benchmarks. To do this, we used the voting adjusted weights to estimate volunteering and the volunteering adjusted weights to estimate voting—the two estimates for which we have benchmark values (from the CPS and the FEC, respectively). Table 4 shows that by adding voting and volunteering to the adjustments, the estimates for volunteering and voting moved by 1.3 and by 2.5 percentage points toward the benchmark, respectively (from 47.9% to 46.6% and from 72.2% to 69.7%). Most of the bias remained, but the volunteering adjustment removed a greater proportion of the bias—over a quarter of the estimated bias ([(72.2–69.7)/(72.2–63.7)]*100 = 29.4%). Table 4. Effect on the Estimate for Volunteering after Weighting for Voting and Effect on the Estimate for Voter Turnout in 2008 from Weighting for Volunteering and Benchmark Estimates from the Current Population Survey and the Federal Election Commission 1. GSS nonresponse adjustment and calibration to demographic characteristics 2. Adding voting to (1) 3. Adding volunteering to (1) 4. Benchmark values (from the CPS and the FEC) n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Volunteering 1,299 47.9% (1.73) 46.6% (1.77) 26.4% (0.2) Voting 1,188 72.2% (1.57) 69.7% (1.83) 63.7% (–) 1. GSS nonresponse adjustment and calibration to demographic characteristics 2. Adding voting to (1) 3. Adding volunteering to (1) 4. Benchmark values (from the CPS and the FEC) n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Volunteering 1,299 47.9% (1.73) 46.6% (1.77) 26.4% (0.2) Voting 1,188 72.2% (1.57) 69.7% (1.83) 63.7% (–) Table 4. Effect on the Estimate for Volunteering after Weighting for Voting and Effect on the Estimate for Voter Turnout in 2008 from Weighting for Volunteering and Benchmark Estimates from the Current Population Survey and the Federal Election Commission 1. GSS nonresponse adjustment and calibration to demographic characteristics 2. Adding voting to (1) 3. Adding volunteering to (1) 4. Benchmark values (from the CPS and the FEC) n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Volunteering 1,299 47.9% (1.73) 46.6% (1.77) 26.4% (0.2) Voting 1,188 72.2% (1.57) 69.7% (1.83) 63.7% (–) 1. GSS nonresponse adjustment and calibration to demographic characteristics 2. Adding voting to (1) 3. Adding volunteering to (1) 4. Benchmark values (from the CPS and the FEC) n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Volunteering 1,299 47.9% (1.73) 46.6% (1.77) 26.4% (0.2) Voting 1,188 72.2% (1.57) 69.7% (1.83) 63.7% (–) Unlike all the weights presented in this study, the GSS public use file weights are not calibrated to population distributions on demographic characteristics. Compared to the GSS public use file weighted estimates, our weights that are calibrated to age, sex, education, and race reduce bias, prior to the addition of voting or volunteering (results not presented). This was not the focus of our investigation—we show that despite demographic adjustments, use of voting and volunteering can further reduce bias in weighted estimates. However, this also suggests that the GSS should consider calibration to population demographic distributions, regardless of whether adjustment to voting or volunteering variables is adopted. 4. SUMMARY AND DISCUSSION Our results show promise in the use of voting and volunteering to inform postsurvey adjustments. Both voting and volunteering variables were substantially skewed in the pool of GSS respondents. Both were also modestly correlated with a variety of other GSS variables. The voting variable was comprised of three nested variables (voting eligibility, voting, and candidate choice). Consistent with the notion of civic duty, the correlations with survey variables were most similar between voting (middle column in figure 2) and volunteering (figure 4). When the voting variables were included, more than a third of the resulting weighted estimates changed. Most of the changes in the estimates were attributable to the addition of voting as opposed to voting eligibility or candidate choice. It is possible that candidate choice would play a larger factor depending on the survey design features—such as the topic of the survey (e.g., religion, education) and the sponsor (e.g., values associated with the sponsoring organization). The inclusion of volunteering also significantly altered the weighted estimates, but the magnitude of the effect was larger and the impact was mostly on different variables. By comparison, the changes in the estimates attributable to the two conventional adjustments for nonresponse (sample-based geographic nonresponse adjustment and population-based demographic calibration) were fewer and smaller in magnitude (results not shown). For the survey practitioner, these finding suggest the deliberate inclusion of substantive survey questions that have dependable external benchmark estimates. Our examples used the ACS, CPS, and the election results from the NEC. There are numerous such benchmarks, and the choice should be informed by the topic of the survey—for example, a health survey might use enrollment in federal and state health insurance plans. Moreover, the payoff to this approach may be larger than our results suggest in the many surveys with response rates substantially lower than the GSS’s relatively high 71%. For the survey statistician, this approach means that it is not sufficient to request the inclusion of demographic questions in order to create weighting adjustments. The statistician responsible for weighting should be involved during the survey design and could investigate different sources of benchmark estimates. For the student of survey error, there are many challenging questions to be addressed. How do measurement differences in the survey and benchmark data affect estimates? How could such effects be minimized? How should systematic and variable errors in these benchmarks be best incorporated? This approach to constructing weighting adjustments may be novel for probability-based surveys, but nondemographic variables have been used in weighting of data from nonprobability-based surveys where inference depends entirely or almost entirely on the selection of statistical models (Schonlau, van Soest, and Kapteyn 2007; DiSogra, Cobb, Chan, and Dennis 2011; Fahimi, Barlas, Thomas, and Buttermore 2015). For example, researchers at the Pew Research Center compared estimates from nine samples coming from eight organizations using a single instrument (Pew Research Center 2016). Based on benchmark estimates from large-scale national probability-based surveys, one sample showed the least bias in its survey estimates. The key differentiating factor for that sample was the vendor’s use of substantive survey measures in the weighting adjustments, including political ideology and party membership. We note this parallel application, but acknowledge that the tolerance levels for error in postsurvey adjustments are markedly different for surveys that rely on probability-based inference, calling for a much higher level of scrutiny. Like most other postsurvey adjustments, the proposed approach can do more harm than good, if implemented blindly. Bias can be induced, rather than reduced, by using population totals or estimates without examining how their properties differ from the properties of the corresponding survey measures. Part of our motivation for presenting two studies was that the adjustments in the two were subject to different error sources: population totals derived from administrative records (Study 1) versus from another survey (Study 2). Future research could focus on whether there are conditions in which one is more desirable than the other. For the present, we are encouraged that the findings were consistent across the two studies suggesting robustness to the conclusions. One potentially important limitation in Study 1 is the inability to correct for measurement error in reported voting. This may affect the magnitudes of some of the estimates, but seems unlikely to change the general conclusions.5 Nonetheless, future work could address this potential problem, as well as explore the use of other types of substantive variables for adjustment purposes. The following list includes key questions: How can measurement error in responses to voting-related questions be reduced? Some have proposed different ways of asking the questions (Belli, Traugott, Young, and McGonagle 1999; Zeglovits and Kritzinger 2014), while others have offered statistical adjustments (Voogt 2005; Katz and Katz 2010). What other substantive variables could be considered for adjustment? For example, the Health Information and National Trends Survey (HINTS) employs estimates from the National Health Interview Survey (NHIS) for having been diagnosed with cancer and health insurance status (Cantor, Coa, Crystal-Mansour, Davis, and Dipko et al. 2009; Peytchev, Ridenhour, and Krotki 2010; West et al. 2015). What other administrative records contain potentially useful adjustment variables? For example, the Centers for Medicare & Medicaid Services’ (CMS) Medicare Coverage Database contains information that may be relevant for certain kinds of surveys. How should methods reflect the uncertainty in population control totals, especially when they are estimates themselves? While presidential candidate choice is a population count after the election, estimates from surveys (such as CPS or NHIS) have sampling variances, which can negatively bias the variance estimates. The CPS volunteering estimate in Study 2 is subject to sampling error, but it is very small relative to the magnitude of the GSS variance estimates. This would not be the case when the survey being conducted is larger or when the control totals are derived from a smaller survey. When estimates from other surveys are used, how important is the assumption that the measurement procedures be the same across the surveys? In a survey with a low response rate, departures from the assumption might not be critical, as the error source being corrected may be the dominant one. Nonetheless, the measurement properties of the external data and the survey data ought to be as similar as possible and, when they are not, the question is whether the benefits, from a total survey error perspective, outweigh the costs. Finally, it is worth noting that our methodological inquiry has produced important substantive findings. We have demonstrated that the GSS, probably the most rigorously done public opinion survey in the United States, substantially underrepresents nonvoters, those who do no volunteer work, and noncitizens. Future work could usefully explore the consequences of this for the many different kinds of analyses based on the GSS (and possibly on other surveys, as well). This article was written while the first author was a Research Assistant Professor at the Institute for Social Research at the University of Michigan, Ann Arbor, MI, USA. The authors are grateful for the constructive comments from the anonymous reviewers and the editor. Footnotes 1 Multiple imputation models the survey variables instead of the likelihood of an interview to reduce the adverse impact on variance estimates (Peytchev 2012; Rässler and Schnell 2004), but its ability to reduce bias still depends on auxiliary information that is associated with both nonresponse and the survey variables (Little and Vartivarian 2005). In addition, this approach is limited to cases where information is available for the full sample and, thus, does not permit the use of population benchmarks. 2 Age (18–29, 30–44, 45–59, 60, and older), sex, education (less than high school diploma; high school diploma, GED, or some college but no four-year degree; and bachelor's or higher degree), and race (white versus other). 3 Statistical testing used the stacking approach in which the data under the different weighting scenarios are stacked (e.g., Aldworth, Barnett, Cribb, Davis, Foster et al. 2013), and tests conducted between scenarios. The Rao-Scott Chi-Square was used, which also applies a correction for the complex survey design. 4 Differences between the voting and volunteering results are not due to the fact that the voting questions were asked of the total sample and the volunteering questions were asked of only a random subsample. All the voting results are essentially the same if based on only the cases asked the volunteering questions. 5 If measurement error alone accounted for the differences in eligibility, for example, more than half of the noncitizens would have had to report being citizens or citizens 14-17 years of age in 2008 would have had to have reported they voted in 2008. Similarly, if misreporting were solely responsible for the results, 17% of those who voted for McCain would have had to misreport voting for Obama. Appendix A Weighted Estimates with GSS Nonresponse Adjustments and Poststratification to Demographic Characteristics, and with the Addition of Poststratification to Voting Eligibility, Voter Turnout, and Candidate Choice 1. GSS nonresponse adjustment and poststratification to demographic characteristics 2. Adding voting eligibility 3. Adding voting eligibility and voting 4. Adding voting eligibility, voting, and candidate choice Difference (4–1) p-value n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 1,306 24.3% (1.36) 24.2% (1.42) 25.2% (1.51) 25.0% (1.50) 0.7% 0.14 Very or pretty happy 1,964 87.6% (0.98) 87.8% (0.96) 87.4% (1.02) 87.5% (1.02) −0.2% 0.52 Life exciting 1,296 52.9% (2.01) 53.5% (2.00) 52.9% (2.03) 53.0% (2.05) 0.1% 0.86 Most people try to be helpful 1,328 47.6% (1.70) 47.8% (1.76) 46.9% (1.75) 46.7% (1.76) −0.9% 0.08 Most try to take advantage 1,326 42.2% (1.99) 42.3% (2.02) 42.6% (2.03) 42.7% (2.04) 0.5% 0.35 Most people can be trusted 1,331 34.2% (1.92) 34.6% (1.93) 33.9% (1.90) 33.8% (1.92) −0.4% 0.41 Family income below average 1,952 31.3% (1.37) 31.1% (1.37) 31.8% (1.44) 31.7% (1.44) 0.3% 0.38 Read newspaper every day 1,301 28.1% (1.91) 28.0% (1.94) 27.6% (1.96) 27.4% (1.94) −0.8% <0.05 No religion 1,967 20.1% (1.02) 20.2% (1.07) 20.0% (1.03) 19.6% (1.03) −0.5% 0.13 Donated blood in past year 1,301 13.5% (1.26) 13.5% (1.28) 13.1% (1.28) 13.0% (1.33) −0.5% 0.13 Donated to charity in past year 1,298 73.6% (1.55) 73.4% (1.62) 72.3% (1.69) 72.4% (1.68) −1.2% <0.05 Support birth control to teens 14 to 16 years old 1,269 57.6% (1.74) 57.9% (1.79) 58.3% (1.82) 57.9% (1.81) 0.3% 0.56 A same-sex female couple can bring up a child just as well 1,230 47.3% (2.26) 47.7% (2.24) 47.1% (2.21) 46.4% (2.22) −0.9% <0.05 Oppose capital punishment 1,824 33.6% (1.33) 34.2% (1.31) 34.0% (1.30) 33.4% (1.30) −0.1% 0.76 Courts are too harsh with criminals 1,777 14.5% (0.79) 14.8% (0.88) 15.1% (0.98) 14.8% (0.98) 0.3% 0.40 1. GSS nonresponse adjustment and poststratification to demographic characteristics 2. Adding voting eligibility 3. Adding voting eligibility and voting 4. Adding voting eligibility, voting, and candidate choice Difference (4–1) p-value n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 1,306 24.3% (1.36) 24.2% (1.42) 25.2% (1.51) 25.0% (1.50) 0.7% 0.14 Very or pretty happy 1,964 87.6% (0.98) 87.8% (0.96) 87.4% (1.02) 87.5% (1.02) −0.2% 0.52 Life exciting 1,296 52.9% (2.01) 53.5% (2.00) 52.9% (2.03) 53.0% (2.05) 0.1% 0.86 Most people try to be helpful 1,328 47.6% (1.70) 47.8% (1.76) 46.9% (1.75) 46.7% (1.76) −0.9% 0.08 Most try to take advantage 1,326 42.2% (1.99) 42.3% (2.02) 42.6% (2.03) 42.7% (2.04) 0.5% 0.35 Most people can be trusted 1,331 34.2% (1.92) 34.6% (1.93) 33.9% (1.90) 33.8% (1.92) −0.4% 0.41 Family income below average 1,952 31.3% (1.37) 31.1% (1.37) 31.8% (1.44) 31.7% (1.44) 0.3% 0.38 Read newspaper every day 1,301 28.1% (1.91) 28.0% (1.94) 27.6% (1.96) 27.4% (1.94) −0.8% <0.05 No religion 1,967 20.1% (1.02) 20.2% (1.07) 20.0% (1.03) 19.6% (1.03) −0.5% 0.13 Donated blood in past year 1,301 13.5% (1.26) 13.5% (1.28) 13.1% (1.28) 13.0% (1.33) −0.5% 0.13 Donated to charity in past year 1,298 73.6% (1.55) 73.4% (1.62) 72.3% (1.69) 72.4% (1.68) −1.2% <0.05 Support birth control to teens 14 to 16 years old 1,269 57.6% (1.74) 57.9% (1.79) 58.3% (1.82) 57.9% (1.81) 0.3% 0.56 A same-sex female couple can bring up a child just as well 1,230 47.3% (2.26) 47.7% (2.24) 47.1% (2.21) 46.4% (2.22) −0.9% <0.05 Oppose capital punishment 1,824 33.6% (1.33) 34.2% (1.31) 34.0% (1.30) 33.4% (1.30) −0.1% 0.76 Courts are too harsh with criminals 1,777 14.5% (0.79) 14.8% (0.88) 15.1% (0.98) 14.8% (0.98) 0.3% 0.40 1. GSS nonresponse adjustment and poststratification to demographic characteristics 2. Adding voting eligibility 3. Adding voting eligibility and voting 4. Adding voting eligibility, voting, and candidate choice Difference (4–1) p-value n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 1,306 24.3% (1.36) 24.2% (1.42) 25.2% (1.51) 25.0% (1.50) 0.7% 0.14 Very or pretty happy 1,964 87.6% (0.98) 87.8% (0.96) 87.4% (1.02) 87.5% (1.02) −0.2% 0.52 Life exciting 1,296 52.9% (2.01) 53.5% (2.00) 52.9% (2.03) 53.0% (2.05) 0.1% 0.86 Most people try to be helpful 1,328 47.6% (1.70) 47.8% (1.76) 46.9% (1.75) 46.7% (1.76) −0.9% 0.08 Most try to take advantage 1,326 42.2% (1.99) 42.3% (2.02) 42.6% (2.03) 42.7% (2.04) 0.5% 0.35 Most people can be trusted 1,331 34.2% (1.92) 34.6% (1.93) 33.9% (1.90) 33.8% (1.92) −0.4% 0.41 Family income below average 1,952 31.3% (1.37) 31.1% (1.37) 31.8% (1.44) 31.7% (1.44) 0.3% 0.38 Read newspaper every day 1,301 28.1% (1.91) 28.0% (1.94) 27.6% (1.96) 27.4% (1.94) −0.8% <0.05 No religion 1,967 20.1% (1.02) 20.2% (1.07) 20.0% (1.03) 19.6% (1.03) −0.5% 0.13 Donated blood in past year 1,301 13.5% (1.26) 13.5% (1.28) 13.1% (1.28) 13.0% (1.33) −0.5% 0.13 Donated to charity in past year 1,298 73.6% (1.55) 73.4% (1.62) 72.3% (1.69) 72.4% (1.68) −1.2% <0.05 Support birth control to teens 14 to 16 years old 1,269 57.6% (1.74) 57.9% (1.79) 58.3% (1.82) 57.9% (1.81) 0.3% 0.56 A same-sex female couple can bring up a child just as well 1,230 47.3% (2.26) 47.7% (2.24) 47.1% (2.21) 46.4% (2.22) −0.9% <0.05 Oppose capital punishment 1,824 33.6% (1.33) 34.2% (1.31) 34.0% (1.30) 33.4% (1.30) −0.1% 0.76 Courts are too harsh with criminals 1,777 14.5% (0.79) 14.8% (0.88) 15.1% (0.98) 14.8% (0.98) 0.3% 0.40 1. GSS nonresponse adjustment and poststratification to demographic characteristics 2. Adding voting eligibility 3. Adding voting eligibility and voting 4. Adding voting eligibility, voting, and candidate choice Difference (4–1) p-value n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 1,306 24.3% (1.36) 24.2% (1.42) 25.2% (1.51) 25.0% (1.50) 0.7% 0.14 Very or pretty happy 1,964 87.6% (0.98) 87.8% (0.96) 87.4% (1.02) 87.5% (1.02) −0.2% 0.52 Life exciting 1,296 52.9% (2.01) 53.5% (2.00) 52.9% (2.03) 53.0% (2.05) 0.1% 0.86 Most people try to be helpful 1,328 47.6% (1.70) 47.8% (1.76) 46.9% (1.75) 46.7% (1.76) −0.9% 0.08 Most try to take advantage 1,326 42.2% (1.99) 42.3% (2.02) 42.6% (2.03) 42.7% (2.04) 0.5% 0.35 Most people can be trusted 1,331 34.2% (1.92) 34.6% (1.93) 33.9% (1.90) 33.8% (1.92) −0.4% 0.41 Family income below average 1,952 31.3% (1.37) 31.1% (1.37) 31.8% (1.44) 31.7% (1.44) 0.3% 0.38 Read newspaper every day 1,301 28.1% (1.91) 28.0% (1.94) 27.6% (1.96) 27.4% (1.94) −0.8% <0.05 No religion 1,967 20.1% (1.02) 20.2% (1.07) 20.0% (1.03) 19.6% (1.03) −0.5% 0.13 Donated blood in past year 1,301 13.5% (1.26) 13.5% (1.28) 13.1% (1.28) 13.0% (1.33) −0.5% 0.13 Donated to charity in past year 1,298 73.6% (1.55) 73.4% (1.62) 72.3% (1.69) 72.4% (1.68) −1.2% <0.05 Support birth control to teens 14 to 16 years old 1,269 57.6% (1.74) 57.9% (1.79) 58.3% (1.82) 57.9% (1.81) 0.3% 0.56 A same-sex female couple can bring up a child just as well 1,230 47.3% (2.26) 47.7% (2.24) 47.1% (2.21) 46.4% (2.22) −0.9% <0.05 Oppose capital punishment 1,824 33.6% (1.33) 34.2% (1.31) 34.0% (1.30) 33.4% (1.30) −0.1% 0.76 Courts are too harsh with criminals 1,777 14.5% (0.79) 14.8% (0.88) 15.1% (0.98) 14.8% (0.98) 0.3% 0.40 Appendix B Weighted Estimates with GSS Nonresponse Adjustments and Poststratification to Demographic Characteristics, and with the Addition of Poststratification to Volunteering GSS nonresponse adjustment and poststratification to demographic characteristics Adding volunteering Difference p-value n Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 633 22.1% (1.85) 23.7% (2.03) 1.6% <0.05 Very or pretty happy 1,295 89.0% (1.15) 88.4% (1.28) −0.6% 0.17 Life exciting 630 54.6% (2.62) 52.3% (2.74) −2.2% <0.05 Most people try to be helpful 658 49.4% (2.21) 49.4% (2.33) −0.1% 0.95 Most try to take advantage 656 40.9% (2.75) 43.0% (3.02) 2.1% <0.05 Most people can be trusted 660 31.5% (2.69) 29.5% (2.90) −2.0% 0.10 Family income below average 1,284 31.2% (1.67) 31.8% (1.86) 0.6% 0.34 Read newspaper every day 1,298 28.9% (1.92) 28.1% (1.92) −0.7% 0.33 No religion 1,294 21.4% (1.31) 23.3% (1.53) 1.9% <0.05 Donated blood in past year 634 14.3% (2.17) 12.4% (1.90) −1.9% 0.05 Donated to charity in past year 632 73.1% (2.13) 67.6% (2.52) −5.6% <0.05 Support birth control to teens 14 to 16 years old 1,268 57.3% (1.73) 58.7% (1.83) 1.4% 0.06 A same-sex female couple can bring up a child just as well 1,230 46.8% (2.25) 48.2% (2.27) 1.4% 0.07 Oppose capital punishment 1,204 33.9% (1.74) 35.2% (1.92) 1.3% 0.08 Courts are too harsh with criminals 1,182 14.0% (1.13) 13.6% (1.12) −0.4% 0.40 GSS nonresponse adjustment and poststratification to demographic characteristics Adding volunteering Difference p-value n Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 633 22.1% (1.85) 23.7% (2.03) 1.6% <0.05 Very or pretty happy 1,295 89.0% (1.15) 88.4% (1.28) −0.6% 0.17 Life exciting 630 54.6% (2.62) 52.3% (2.74) −2.2% <0.05 Most people try to be helpful 658 49.4% (2.21) 49.4% (2.33) −0.1% 0.95 Most try to take advantage 656 40.9% (2.75) 43.0% (3.02) 2.1% <0.05 Most people can be trusted 660 31.5% (2.69) 29.5% (2.90) −2.0% 0.10 Family income below average 1,284 31.2% (1.67) 31.8% (1.86) 0.6% 0.34 Read newspaper every day 1,298 28.9% (1.92) 28.1% (1.92) −0.7% 0.33 No religion 1,294 21.4% (1.31) 23.3% (1.53) 1.9% <0.05 Donated blood in past year 634 14.3% (2.17) 12.4% (1.90) −1.9% 0.05 Donated to charity in past year 632 73.1% (2.13) 67.6% (2.52) −5.6% <0.05 Support birth control to teens 14 to 16 years old 1,268 57.3% (1.73) 58.7% (1.83) 1.4% 0.06 A same-sex female couple can bring up a child just as well 1,230 46.8% (2.25) 48.2% (2.27) 1.4% 0.07 Oppose capital punishment 1,204 33.9% (1.74) 35.2% (1.92) 1.3% 0.08 Courts are too harsh with criminals 1,182 14.0% (1.13) 13.6% (1.12) −0.4% 0.40 GSS nonresponse adjustment and poststratification to demographic characteristics Adding volunteering Difference p-value n Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 633 22.1% (1.85) 23.7% (2.03) 1.6% <0.05 Very or pretty happy 1,295 89.0% (1.15) 88.4% (1.28) −0.6% 0.17 Life exciting 630 54.6% (2.62) 52.3% (2.74) −2.2% <0.05 Most people try to be helpful 658 49.4% (2.21) 49.4% (2.33) −0.1% 0.95 Most try to take advantage 656 40.9% (2.75) 43.0% (3.02) 2.1% <0.05 Most people can be trusted 660 31.5% (2.69) 29.5% (2.90) −2.0% 0.10 Family income below average 1,284 31.2% (1.67) 31.8% (1.86) 0.6% 0.34 Read newspaper every day 1,298 28.9% (1.92) 28.1% (1.92) −0.7% 0.33 No religion 1,294 21.4% (1.31) 23.3% (1.53) 1.9% <0.05 Donated blood in past year 634 14.3% (2.17) 12.4% (1.90) −1.9% 0.05 Donated to charity in past year 632 73.1% (2.13) 67.6% (2.52) −5.6% <0.05 Support birth control to teens 14 to 16 years old 1,268 57.3% (1.73) 58.7% (1.83) 1.4% 0.06 A same-sex female couple can bring up a child just as well 1,230 46.8% (2.25) 48.2% (2.27) 1.4% 0.07 Oppose capital punishment 1,204 33.9% (1.74) 35.2% (1.92) 1.3% 0.08 Courts are too harsh with criminals 1,182 14.0% (1.13) 13.6% (1.12) −0.4% 0.40 GSS nonresponse adjustment and poststratification to demographic characteristics Adding volunteering Difference p-value n Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 633 22.1% (1.85) 23.7% (2.03) 1.6% <0.05 Very or pretty happy 1,295 89.0% (1.15) 88.4% (1.28) −0.6% 0.17 Life exciting 630 54.6% (2.62) 52.3% (2.74) −2.2% <0.05 Most people try to be helpful 658 49.4% (2.21) 49.4% (2.33) −0.1% 0.95 Most try to take advantage 656 40.9% (2.75) 43.0% (3.02) 2.1% <0.05 Most people can be trusted 660 31.5% (2.69) 29.5% (2.90) −2.0% 0.10 Family income below average 1,284 31.2% (1.67) 31.8% (1.86) 0.6% 0.34 Read newspaper every day 1,298 28.9% (1.92) 28.1% (1.92) −0.7% 0.33 No religion 1,294 21.4% (1.31) 23.3% (1.53) 1.9% <0.05 Donated blood in past year 634 14.3% (2.17) 12.4% (1.90) −1.9% 0.05 Donated to charity in past year 632 73.1% (2.13) 67.6% (2.52) −5.6% <0.05 Support birth control to teens 14 to 16 years old 1,268 57.3% (1.73) 58.7% (1.83) 1.4% 0.06 A same-sex female couple can bring up a child just as well 1,230 46.8% (2.25) 48.2% (2.27) 1.4% 0.07 Oppose capital punishment 1,204 33.9% (1.74) 35.2% (1.92) 1.3% 0.08 Courts are too harsh with criminals 1,182 14.0% (1.13) 13.6% (1.12) −0.4% 0.40 REFERENCES AAPOR ( 2015 ). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys ( 8th ed .), Oakbrook Terrace, IL: American Association for Public Opinion Research . Abraham K. , Helms S. , Presser S. ( 2009 ). “How Social Processes Distort Measurement: The Impact of Survey Nonresponse on Estimates of Volunteer Work in the United States,” American Journal of Sociology , 114 , 1129 – 1165 . Google Scholar Crossref Search ADS Aldworth J. , Barnett S. , Cribb D. , Davis T. R. , Foster M. S. , Harter R. , Packer L. E. , Spagnola K. , Bose J. , Hughes A. ( 2013 ). 2013 National Survey on Drug Use and Health: Statistical Inference Report. Rockville, Maryland: RTI International. Amaya A. , Presser S. ( 2017 ). “ Nonresponse Bias for Univariate and Multivariate Estimates of Social Activities and Roles ,” Public Opinion Quarterly , 81 , 1 – 36 . Belli R. F. , Traugott M. W. , Young M. , McGonagle K. A. ( 1999 ). “ Reducing Vote Overreporting in Surveys: Social Desirability, Memory Failure, and Source Monitoring ,” Public Opinion Quarterly , 63 , 90 – 108 . Google Scholar Crossref Search ADS Berent M. K. , Krosnick J. A. , Lupia A. ( 2016 ). “ Measuring Voter Registration and Turnout in Surveys Do Official Government Records Yield More Accurate Assessments ?” Public Opinion Quarterly , 80 , 597 – 621 . Google Scholar Crossref Search ADS Biemer P. P. , Peytchev A. ( 2013 ). “ Using Geocoded Census Data for Nonresponse Bias Correction: An Assessment ,” Journal of Survey Statistics and Methodology , 1 , 24 – 44 . Google Scholar Crossref Search ADS Brick J. M. , Kalton G. ( 1996 ). “ Handling Missing Data in Survey Research ,” Statistical Methods in Medical Research , 5 , 215 – 238 . Google Scholar Crossref Search ADS PubMed Cantor D. , Coa K. , Crystal-Mansour S. , Davis T. , Dipko S. , Sigman R. ( 2009 ). Health Information National Trends Survey 2007 (HINTS 2007) Final Report . Bethesda, MD : National Cancer Institute . Cohn N. ( 2014 ). “The Upshot: Why Polls Tend to Undercount Democrats,” The New York Times, http://www.nytimes.com/2014/10/30/upshot/why-polls-tend-to-undercount-democrats.html? abt=0002&abg=1. Couper M. P. , Singer E. , Kulka R. A. ( 1998 ). “ Participation in the 1990 Decennial Census: Politics, Privacy, Pressures ,” American Politics Quarterly , 26 , 59 – 80 . Google Scholar Crossref Search ADS Curtin R. , Presser S. , Singer E. ( 2000 ). “ The Effects of Response Rate Changes on the Index of Consumer Sentiment ,” Public Opinion Quarterly , 64 , 413 – 428 . Google Scholar Crossref Search ADS PubMed Dever J. A. , Valliant R. ( 2016 ). “ General Regression Estimation Adjusted for Undercoverage and Estimated Control Totals ,” Journal of Survey Statistics and Methodology , 4, 289 – 318 . Google Scholar Crossref Search ADS Dever J. A. , Valliant R. L. ( 2010 ). “ A Comparison of Variance Estimators for Poststratification to Estimated Control Totals ,” Survey Methodology , 36 , 45 – 56 . Dillman D. A. ( 1978 ). Mail and Telephone Surveys . New York : Wiley . DiSogra C. , Cobb C. , Chan E. , Dennis J. M. ( 2011 ). “Calibrating Non-Probability Internet Samples with Probability Samples Using Early Adopter Characteristics.” Joint Statistical Meetings of the American Statistical Association. Fahimi M. , Barlas F. M. , Thomas R. K. , Buttermore N. ( 2015 ). “ Scientific Surveys Based on Incomplete Sampling Frames and High Rates of Nonresponse ,” Survey Practice , 8 . Federal Election Commission ( 2009 ). Federal Elections 2008: Election Results for the US President, the US Senate and the US House of Representatives. Federal Election Commission. Goyder J. ( 1987 ). The Silent Minority: Nonrespondents on Sample Surveys . Cambridge : Polity Press . Groves R. M. ( 2006 ). “ Nonresponse Rates and Nonresponse Bias in Household Surveys ,” Public Opinion Quarterly , 70 , 646 – 675 . Google Scholar Crossref Search ADS Groves R. M. , Couper M. P. ( 1998 ). Nonresponse in Household Interview Surveys . New York : Wiley . Groves R. M. , Peytcheva E. ( 2008 ). “ The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis ,” Public Opinion Quarterly , 72 , 167 – 189 . Google Scholar Crossref Search ADS Groves R. M. , Singer E. , Corning A. ( 2000 ). “ Leverage-Saliency Theory of Survey Participation—Description and an Illustration ,” Public Opinion Quarterly , 64 , 299 – 308 . Google Scholar Crossref Search ADS PubMed Jackman S. , Spahn B. T. ( 2014 ). “Why Does the American National Election Study Overestimate Voter Turnout?” Available at http://polmeth.wustl.edu/media/Paper/polmeth2014.pdf. Katz J. N. , Katz G. ( 2010 ). “ Correcting for Survey Misreports Using Auxiliary Information with an Application to Estimating Turnout ,” American Journal of Political Science , 54 , 815 – 835 . Google Scholar Crossref Search ADS Keeter S. , Kennedy C. , Dimock M. , Best J. , Craighill P. ( 2006 ). “ Gauging the impact of growing nonresponse on estimates from a national RDD telephone survey ,” Public Opinion Quarterly , 70 , 759 – 779 . Google Scholar Crossref Search ADS Kish L. ( 1965 ). Survey Sampling . New York : John Wiley . Knack S. ( 1992 ). “ Civic Norms, Social Sanctions, and Voter Turnout ,” Rationality and Society , 4 , 133 – 156 . Google Scholar Crossref Search ADS Kreuter F. , Olson K. , Wagner J. , Yan T. , Ezzati-Rice T. M. , Casas-Cordero C. , Lemay M. , Peytchev A. , Groves R. M. , Raghunathan T. E. ( 2010 ). “ Using Proxy Measures and Other Correlates of Survey Outcomes to Adjust for Nonresponse: Examples from Multiple Surveys ,” Journal of the Royal Statistical Society, Series A (Statistics in Society) , 173 , 389 . Google Scholar Crossref Search ADS Little R. J. A. ( 1986 ). “ Survey Nonresponse Adjustments for Estimates of Means ,” International Statistical Review , 54 , 139 – 157 . Google Scholar Crossref Search ADS Little R. J. , Vartivarian S. ( 2005 ). “ Does Weighting for Nonresponse Increase the Variance of Survey Means ?” Survey Methodology , 31 , 161 – 168 . Merkle D. M. , Edelman M. ( 2009 ). “ An Experiment on Improving Response Rates and Its Unintended Impact on Survey Error ,” Survey Practice (March) , 2 . Mulry M. H. ( 2014 ). “Measuring Undercounts for Hard-to-Survey Groups.” in Hard-to-Survey Populations , eds. T. Roger , E. Brad , P. J. Timothy , B. Nancy , M. W Kirk . Cambridge : Cambridge University Press . Pasek J. , Jang S. M. , Cobb C. L. , Dennis J. M. , Disogra C. ( 2014 ). “ Can Marketing Data Aid Survey Research? Examining Accuracy and Completeness in Consumer-File Data ,” Public Opinion Quarterly , 78 , 889 – 916 . Google Scholar Crossref Search ADS Pew Research Center ( 2016 ). “Evaluating Online Nonprobability Surveys.” Available at http://www.pewresearch.org/files/2016/04/Nonprobability-report-May-2016-FINAL.pdf. Peytchev A. ( 2012 ). “ Multiple Imputation for Unit Nonresponse and Measurement Error ,” Public Opinion Quarterly , 76 , 214 – 237 . Google Scholar Crossref Search ADS Peytchev A. , Raghunathan T. ( 2013 ). “Evaluation and Use of Commercial Data for Nonresponse Bias Adjustment.” American Association for Public Opinion Research annual conference, Boston, MA, May 16. Peytchev A. , Ridenhour J. , Krotki K. ( 2010 ). “ Differences Between RDD Telephone and ABS Mail Survey Design: Coverage, Unit Nonresponse, and Measurement Error ,” Journal of Health Communication: International Perspectives , 15 , 117 – 134 . Google Scholar Crossref Search ADS Peytcheva E. , Groves R. M. ( 2009 ). “ Using Variation in Response Rates of Demographic Subgroups as Evidence of Nonresponse Bias in Survey Estimates ,” Journal of Official Statistics , 25 , 193 – 201 . Rässler S. , Schnell R. ( 2004 ). Multiple imputation for unit-nonresponse versus weighting including a comparison with a nonresponse follow-up study. Diskussionspapiere//Friedrich-Alexander-Universität Erlangen-Nürnberg, Lehrstuhl für Statistik und Ökonometrie, Nurenberg, Germany. Rendall M. , Tomassini C. , Elliot D. J. ( 2003 ). “ Estimation of Annual International Migration from the Labor Force Surveys of the United Kingdom and the Continental European Union ,” Statistical Journal of the United Nations ECE , 20 , 219 – 234 . Schonlau M. , van Soest A. , Kapteyn A. ( 2007 ). “ Are ‘Webographic’ or attitudinal questions useful for adjusting estimates from Web surveys using propensity scoring ?” Survey Research Methods , 1 , 9 . Shin M. E. , McCarthy W. J. ( 2013 ). “ The Association between County Political Inclination and Obesity: Results from the 2012 Presidential Election in the United States ,” Preventive Medicine , 57 , 721 – 724 . Google Scholar Crossref Search ADS PubMed Subramanian S. V. , Huijts T. , Perkins J. M. ( 2009 ). “ Association between Political Ideology and Health in Europe ,” European Journal of Public Health , 19 , 455 – 457 . Google Scholar Crossref Search ADS PubMed Tourangeau R. , Groves R. M. , Redline C. D. ( 2010 ). “ Sensitive Topics and Reluctant Respondents: Demonstrating a Link between Nonresponse Bias and Measurement Error ,” Public Opinion Quarterly , 74 , 413 – 432 . Google Scholar Crossref Search ADS Tourangeau R. , Rips L. J. , Rasinski K. A. ( 2000 ). The Psychology of Survey Response . Cambridge, U.K.; New York : Cambridge University Press . U.S. Census Bureau ( 2012 ). Voting and Registration in the Election of November 2008. US Census Bureau. Voogt R. J. J. ( 2005 ). “ An Alternative Approach to Correcting Response and Nonresponse Bias in Election Research ,” Acta Politica , 40 , 94 – 116 . Google Scholar Crossref Search ADS Voogt R. J. J. ( 2004 ). “Nonresponse Bias, Response Bias and Stimulus Effects in Election Research.” PhD Faculty of Social Sciences, University of Amsterdam. Wagner J. , Valliant R. , Hubbard F. , Jiang L. ( 2014 ). “ Level-of-Effort Paradata and Nonresponse Adjustment Models for a National Face-to-Face Survey ,” Journal of Survey Statistics and Methodology , 2 , 410 – 432 . Google Scholar Crossref Search ADS West B. T. ( 2013 ). “ An examination of the quality and utility of interviewer observations in the National Survey of Family Growth ,” Journal of the Royal Statistical Society: Series A (Statistics in Society) , 176 , 211 – 225 . Google Scholar Crossref Search ADS West B. T. , Wagner J. , Hubbard F. , Gu H. ( 2015 ). “ The Utility of Alternative Commercial Data Sources for Survey Operations and Estimation: Evidence from the National Survey of Family Growth ,” Journal of Survey Statistics and Methodology , 3 , 240 – 264 . Google Scholar Crossref Search ADS Westat ( 2015 ). Health Information National Trends Survey 4 (HINTS 4) Cycle 4: Methodology Report . Bethesda, MD : National Cancer Institute . Wright G. C. ( 1993 ). “ Errors in Measuring Vote Choice in the National Election Studies, 1952-88 ,” American Journal of Political Science , 37 , 291 – 316 . Google Scholar Crossref Search ADS Zeglovits E. , Kritzinger S. ( 2014 ). “ New Attempts to Reduce Overreporting of Voter Turnout and Their Effects ,” International Journal of Public Opinion Research , 26 , 224 – 234 . Google Scholar Crossref Search ADS © The Author 2018. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Survey Statistics and Methodology Oxford University Press

Improving Traditional Nonresponse Bias Adjustments: Combining Statistical Properties with Social Theory

Loading next page...
 
/lp/ou_press/improving-traditional-nonresponse-bias-adjustments-combining-4gFiyrJYCk
Publisher
Oxford University Press
Copyright
© The Author 2018. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please email: journals.permissions@oup.com
ISSN
2325-0984
eISSN
2325-0992
D.O.I.
10.1093/jssam/smx035
Publisher site
See Article on Publisher Site

Abstract

Abstract Declining response rates have led to increasing reliance on nonresponse adjustment as a way to reduce the risk of nonresponse bias. Unfortunately, the auxiliary variables used in most surveys frequently do not satisfy the sine qua non for effective adjustment: significant associations with both nonresponse and the survey variables. We describe an approach to selecting weight variables that identifies candidates, such as voting and volunteering, not usually considered. In an analysis of the 2012 General Social Survey (GSS), we show that voting eligibility, voter turnout, and candidate choice meet the statistical conditions for nonresponse bias reduction, as does volunteering. Voting and volunteering are strongly associated with participation in the GSS and also associated with a wide array of variables measured in the GSS. Adjustments using either voting benchmarks or volunteering benchmarks result in significant changes to GSS estimates compared to traditional adjustments. As this approach shows promise, we identify several lines of research needed to inform its implementation. 1. INTRODUCTION Response rates in US surveys of the general public have been declining for more than a half century, a trend that shows no sign of reversing. Although lower response rates do not necessarily increase nonresponse bias (Groves and Peytcheva 2008), they raise the risk of such bias. As a result, surveys increasingly rely on nonresponse adjustments to protect against this risk. Weighting can reduce the risk of nonresponse bias by bringing the distribution(s) of respondent characteristics into line with the distribution(s) either for the full sample or for the population. But the effectiveness of this procedure depends on the properties of the variables (known as “auxiliary” variables) used to construct the weights. The ideal auxiliary variables for nonresponse adjustment are strongly associated with both nonresponse and the survey variables of interest (e.g., Little 1986; Little and Vartivarian 2005; Groves 2006,). Yet in many surveys, the auxiliary variables are associated with only one or the other (e.g., Kreuter, Olson, Wagner, Yan, Ezzati-Rice et al. 2010), if either. In practice, adjustments for nonresponse usually model the likelihood of an interview, resulting in the selection of variables that are associated with nonresponse, but not the survey variables of interest. In addition to typically leading to little bias reduction, this has the unfortunate effect of reducing the precision of the estimates due to the variability of the weights.1 The main sampling frames for the US general population—lists of addresses and telephone numbers—are variable-poor, and the adjustment variables typically chosen from data bases to which the frames can be linked or from observations made during data collection usually do not satisfy the twin conditions necessary for reducing nonresponse bias. In particular, The aggregate nature of census estimates, to which addresses and phone numbers are most often merged, combined with their limitation to mainly demographic topics generally lead to ineffective nonresponse bias adjustments (Biemer and Peytchev 2013). For example, the average age in a county, ZIP code, or even Census Block Group, is not highly predictive of individual-level nonresponse and even less predictive of individual responses to most survey questions. Because census aggregate data lack both conditions and the resulting weight adjustments add very little variability, use of these data has been found to neither help (by reducing bias) nor hurt (by increasing variance) the accuracy of survey estimates (Biemer and Peytchev 2013). Individual-level commercial data has problems of matching, missing data, and data quality and, probably as a result, the data has been found to be of little value in adjusting for nonresponse (Peytchev and Raghunathan 2013; Pasek, Jang, Cobb, Dennis, and Disogra 2014; West, Wagner, Hubbard, and Gu 2015). Paradata, such as number of call attempts and whether anyone ever refused, can dramatically improve model fit in nonresponse adjustments, because they reflect actual nonresponse; that is, they predict final nonresponse on the basis of prior refusals and noncontacts. But although the number of call attempts, for example, can be a near perfect predictor of nonresponse, it has been found to lack association with survey variables (Curtin, Presser, and Singer 2000). Thus, weighting using this approach can lead to loss of precision in survey estimates (due to variation in the weights) without commensurate reduction in nonresponse bias (Wagner, Valliant, Hubbard, and Jiang 2014). The other main source of paradata—interviewer observations—might, in principle, be more informative for nonresponse adjustment, but its utility is diminished by substantial measurement error (West 2013). Summary demographic estimates for the population from the Census Bureau are probably the most frequently used auxiliary information, but a meta-analysis of nonresponse bias studies found that nonresponse bias in demographic characteristics and nonresponse bias in substantive survey estimates were generally unrelated (Peytcheva and Groves 2009). In sum, there is increased reliance on nonresponse adjustments, yet the auxiliary variables used to make adjustments often lack at least one of the two essential statistical properties for such variables: significant associations with survey nonresponse and with a wide array of substantive survey variables. In the next sections, we identify voting and volunteering as two candidates for nonresponse adjustments that are theoretically and empirically motivated from the vantage point of a social science perspective; demonstrate the results of using voting and volunteering to adjust estimates in a major national survey; and discuss future research that is needed to further explore this approach. 2. CIVIC DUTY Civic duty—or social integration—figures prominently in theoretical treatments of both survey participation (Dillman 1978; Goyder 1987; Groves and Couper 1998; Groves, Singer, and Corning 2000) and a diverse array of substantive matters ranging from political activity to subjective well-being (Amaya and Presser 2017). Moreover, two key civic behaviors—voting and volunteering—have been shown to be associated with survey participation. Thus, these indicators seem particularly promising for use as auxiliary variables in nonresponse adjustment. 2.1 Voting One of the greatest challenges to political surveys is that political participation is positively associated with survey participation (e.g., Knack 1992; Voogt 2004; Keeter, Kennedy, Dimock, Best, and Craighill 2006). Paradoxically, this relationship presents an opportunity to correct for survey nonresponse. This is because voting is also associated with many seemingly unrelated survey variables, such as health outcomes (Subramanian, Huijts, and Perkins 2009; Shin and McCarthy 2013). Thus, indicators of political participation are prime candidates for variables associated with both nonresponse and a range of survey variables of interest. For a somewhat different reason, candidate choice may also be affected by nonresponse. Support for some candidates could be seen as socially undesirable, potentially suppressing survey participation among these voters. 2.1.1 Strengths of using political participation and orientation One of the advantages of using voting and support for a particular candidate is that very high-quality benchmark estimates are available for both. The Federal Election Commission (FEC) releases counts and rates for voting among the eligible adult population, along with the allocation of support for each candidate. 2.1.2 Limitations of using political participation and orientation The main limitation of using voting as an adjustment variable is that record-check studies have consistently found it to be overreported in surveys, much like other socially desirable behaviors (for an overview, see Tourangeau, Rips, and Rasinski 2000). Recent research, however, suggests that much of the observed error may be due to errors in the records or matching procedures (Berent, Krosnick, and Lupia 2016), and several studies have concluded that nonvoters are disproportionately likely to be survey nonrespondents (Tourangeau, Groves, and Redline 2010; Jackman and Spahn 2014). Nonetheless, misreporting could affect our findings. With respect to political orientation, we found no studies showing that Republicans versus Democrats and conservatives versus liberals vary in the propensity to overreport voting. There is some evidence, however, that respondents overreport having voted for the winning candidate (Wright 1993). We return to this issue below. 2.2 Volunteering Research has shown a strong link between volunteering and survey participation. For example, Couper, Singer, and Kulka (1998) found that those who reported being involved in community groups and organizations were more likely to have returned their decennial census form. Likewise, Abraham, Helms, and Presser (2009) found that the response rate to the American Time Use Survey (ATUS) was 35% higher among those who reported having done volunteer work in the preceding year than among those who had reported not volunteering (the volunteering questions were asked in the Current Population Survey from which the ATUS draws its sample). Indeed, this evidence led Abraham, Helms, and Presser (2009: 1162–1163) to suggest that the volunteering estimate from the annual Current Population Survey (CPS) volunteering supplement could be a good nonresponse adjustment factor in general population surveys. 2.2.1 Strengths of using volunteering There are at least three major strengths of using volunteering for nonresponse adjustments. First, there is an up-to-date benchmark, as annual population estimates are available from the CPS September Supplement. Second, because the CPS is also an interviewer-administered survey, the population estimate used in adjustments is subject to overreporting due to social desirability as is the survey being adjusted. Third, since the CPS is a survey rather than a variable-poor administrative data source, volunteering estimates can be obtained for subgroups to further improve the adjustments. For example, adjustment cells could be formed by volunteering by sex by age and by education, rather than ignoring the interactions and adjusting to the marginal distributions of volunteering, sex, age, and education. 2.2.2 Limitations of using volunteering Although the CPS is a large national survey with a very high response rate, the CPS estimates are themselves subject to nonresponse bias. In addition, if one uses the CPS volunteering estimates for adjustments in a self-administered survey, mode differences may lead to bias. Of course, these same limitations apply to conventional CPS adjustments using demographic characteristics like age, race, and sex. 3. CASE STUDIES In this section, we present two empirical evaluations of theoretically motivated adjustments for potential nonresponse bias, one using voting and the other using volunteering. We analyze the 2012 General Social Survey (GSS), a nationally representative probability-based survey, with 1,974 completed interviews. The 2012 GSS subsampled nonrespondents to reduce cost and increase the response rate and achieved an AAPOR Response Rate 5 (AAPOR 2015) of 71.4%. In Study 1, we use all 1,974 interviews. In Study 2, we use the random subsample of 1,299 interviews asked the volunteering questions. The GSS does not implement elaborate nonresponse or any poststratification adjustments. Instead, it relies on a sample-based geographic nonresponse adjustment that is equal to the inverse of the response rate in the primary sampling units (PSUs, or “National Frame Areas” in the GSS documentation). Thus, the GSS nonresponse adjustments differ from those of many other surveys, which include poststratification to key demographic characteristics of the population. As a result, we augmented the GSS weights, in order to broaden our conclusions to the wider array of surveys that apply demographic poststratification. The properties of the auxiliary data determine whether sample-based or population-based weighting adjustments are applied. Sample-based adjustments require data on both respondents and nonrespondents. Commonly, weighting class adjustments, response propensity models, or weighting class adjustments based on propensity strata are then used to adjust the weights back to the full sample. Population-based adjustments, by contrast, do not require data on nonrespondents and rely on population counts or distributions for the auxiliary variables. Weights are calibrated to those known population benchmarks. This calibration can be done using different statistical approaches, typically through poststratification when the full cross-classification for the population is known and used, and raking (through iterative methods) to satisfy only marginal distributions. Because weights are calibrated to the population rather than the sample, these adjustments correct for both coverage and nonresponse bias. (For a good overview of nonresponse adjustments, see Brick and Kalton 1996.) We use calibration, since our main interest is to explore the use of auxiliary data for which only population totals are known. For each study, we computed two sets of calibration weights. The first set applies demographic population controls2 to the GSS weights that account for selection probability and the geographic nonresponse adjustment, using iterative proportionate fitting (often referred to as “raking”). In the second set, we add controls for voting (Study 1) and volunteering (Study 2). This led to an expected increase in the variability of the weights. One plus the square of the coefficient of variation of the weights (1 + CV(w)2, or 1 + L) is a useful approximation of the expected loss in precision due to weighting (Kish 1965). For Study 1, this measure increased from 1.73 to 2.13, and for Study 2, it increased from 1.80 to 2.15. The differences in weighted estimates with each set of weights yield the nonresponse bias that is identified by adding the substantive variables in the adjustment, conditional on the adjustment model that includes the GSS geographic nonresponse adjustments and the additional calibration to demographic controls. There are three types of evaluations that are of interest, which we present for each study: Is the auxiliary information associated with nonresponse? We show the magnitude of the bias in the weighted voting estimates (voting eligibility, voting among eligible voters, and candidate choice among voters) and in the weighted volunteering estimates. Is the auxiliary information associated with the survey variables? We present the association between the voting-related or volunteering-related adjustment variables and the survey variables among the respondents. Does the use of the auxiliary information lead to changes in the weighted survey estimates?We evaluate the extent to which adding the constructed voting or volunteering variable to the weighting adjustments leads to changes in the survey estimates. In addition, we also present the impact on total error, using the mean square error (MSE) to combine the bias and variance tradeoff into a single measure. Section 4.1 presents Study 1, section 4.2 presents Study 2, and section 4.3 shows the impact on MSE in both studies. 3.1 Study 1: Voting The survey included the following two voting questions, for those eligible to vote: “In 2008, you remember that Obama ran for President on the Democratic ticket against McCain for the Republicans. Do you remember for sure whether or not you voted in that election?” And if answered affirmatively, it was immediately followed by, “Did you vote for Obama or McCain?” Based on the responses to these questions, the respondents were classified into one of five mutually exclusive categories (unweighted counts in parentheses): Not eligible to vote (159)—in additional to citizenship, this group also includes age-ineligibility due to the time lag between the election in 2008 and the survey in 2012. Did not vote, do not know, or refused (521)—this includes 22 who said that they did not know whether they voted and four who refused to answer. Voted, for Obama (795). Voted, for McCain (472). Voted, for another candidate (27). There were forty-five respondents who reported having voted, but did not name a candidate. Those who self-identified as strong Democrat to Independent were imputed as Obama voters, those who self-identified as Independent, leaning towards Republican to strong Republican were imputed as McCain voters, and the remaining three respondents were randomly assigned to a candidate. The estimates are virtually identical if these respondents are excluded from the analysis or if the imputation is done based on political ideology (liberal to conservative). Control totals were then calculated using a combination of population estimates for noncitizens, age-ineligibility, and non-voting from the American Community Survey (ACS) and the CPS (US Census Bureau 2012), and number of voters and candidate choice from the FEC (Federal Election Commission 2009). The CPS and FEC estimates of number of voters differ by only 0.1% (131,144,000 versus 131,313,820). The ACS and CPS estimates have variances themselves that are ignored in practice due to their relatively small magnitudes (due to the large sample sizes of the ACS and CPS) and complexity in reflecting them in estimation (Dever and Valliant 2010; Dever and Valliant 2016). They are not the focus in this study, but we encourage further investigations on this topic, especially in the case of surveys that implement control totals with larger variances. 3.1.1 Conditional relationship between the voting adjustment variables and nonresponse Figure 1 shows that the GSS estimated percentage of the population eligible to vote in 2008 based on citizenship and age in 2008 was 90%, while the CPS-derived population estimate was 84%. This difference is not unique to the GSS—surveys in other countries have shown that immigrants (many, if not most, of whom would not be eligible to vote) participate in surveys at lower rates (Rendall, Tomassini, and Elliot 2003). Those eighteen to twenty-one years old at the time of the 2012 survey, who were ineligible to vote in 2008, are also likely to have responded at lower rates (Mulry 2014), and the regular age-based weighting adjustments do not take individual years of birth into account, but instead use broader age categories. Figure 1. View largeDownload slide GSS Weighted Estimates (Using GSS Weights Also Adjusted to Population Demographic Characteristics) and Population Benchmark Estimates for Percent of the Adult Population in 2012 Who Were Eligible to Vote in 2008, Percent of the Voting-Eligible Population Who Voted in 2008, and Percent of Voters in 2008 By Candidate Choice. Bars represent standard errors. Figure 1. View largeDownload slide GSS Weighted Estimates (Using GSS Weights Also Adjusted to Population Demographic Characteristics) and Population Benchmark Estimates for Percent of the Adult Population in 2012 Who Were Eligible to Vote in 2008, Percent of the Voting-Eligible Population Who Voted in 2008, and Percent of Voters in 2008 By Candidate Choice. Bars represent standard errors. Of possibly even greater importance is the effect of the adjustment on the estimated percent who voted in 2008, among those eligible to vote. Figure 1 shows that the weighted estimate (72%) is substantially higher than the actual voter turnout in 2008 of 64% (US Census Bureau 2012). The bias is also in the direction that would be expected from social desirability—the overreporting of voting in surveys. The third component of the additional adjustment, candidate choice, also shows substantial imbalance in the respondent pool (figure 1). The GSS nonresponse adjustment with demographic controls estimates 57% to have voted for Barack Obama and 41% for John McCain (a sixteen-percentage point difference)—far from the actual election outcome of 53% for Obama and 46% for McCain (a 7-percentage point difference), which is reflected in the estimates using candidate choice in the poststratification. This overrepresentation of voters supporting Obama may seem surprising since most pre-election polls underestimated support for Obama (Cohn 2014), but as Merkle and Edelman (2009) demonstrated, even the direction of the bias in political support can be a function of the survey protocol. We also note that, just as for voter turnout, part of this bias may be due to social desirability—the tendency to overreport support for the winning candidate. Nonetheless, the observed difference in the GSS underscores the potential importance of this variable in postsurvey adjustments, if Obama and McCain supporters also differ on key survey variables. This is examined next. 3.1.2 Relationship between the voting adjustment variables and survey variables Table 1 shows the correlations of the self-reported eligibility to vote, voting in 2008 (among eligible voters), and voting for Obama (among voters) with responses to fifteen survey questions selected prior to the analysis to represent a diverse array of attitudes and behaviors, and asked of all respondents. All three voting-related questions show significant associations with the survey variables, with voting (conditional on eligibility) and candidate choice (among self-reported voters) yielding some moderate in size. Interestingly, the significant correlations are somewhat different across the three components (voting eligibility, voting, and candidate choice). All of the selected survey variables were correlated with at least one of the three voting variables, and the magnitudes of the statistically significant correlations for the other variables varied widely, from 0.03 to 0.31. Table 1. Selection-Weighted Correlations of Voting Eligibility, Having Voted in 2008 (Among Eligible to Vote), and Voting for Obama (Among Self-Reported Voters) with Fifteen GSS Survey Variables Eligible to vote n = 1,974 Voted in 2008 n = 1,815 Voted for Obama n = 1,294 Fair or poor health 0.01 −0.13a 0.13a Very or pretty happy −0.01 0.09a −0.08a Life exciting −0.04 0.05 −0.07a Most people try to be helpful 0.07a 0.19a −0.01 Most try to take advantage −0.10a −0.13a 0.03a Most people can be trusted 0.04 0.18a −0.04a Family income below average −0.01 −0.14a 0.09a Read newspaper every day 0.10a 0.16a 0.02 No religion −0.05a 0.00 0.19a Donated blood in past year 0.00 0.08a 0.00a Donated to charity in past year 0.14a 0.25a −0.09a Support birth control to teens 14 to 16 years old −0.06a −0.06a 0.21 A same-sex female couple can bring up a child just as well −0.08a 0.04 0.31 Oppose capital punishment −0.08a 0.02 0.31 Courts are too harsh with criminals −0.11a −0.09a 0.19 Eligible to vote n = 1,974 Voted in 2008 n = 1,815 Voted for Obama n = 1,294 Fair or poor health 0.01 −0.13a 0.13a Very or pretty happy −0.01 0.09a −0.08a Life exciting −0.04 0.05 −0.07a Most people try to be helpful 0.07a 0.19a −0.01 Most try to take advantage −0.10a −0.13a 0.03a Most people can be trusted 0.04 0.18a −0.04a Family income below average −0.01 −0.14a 0.09a Read newspaper every day 0.10a 0.16a 0.02 No religion −0.05a 0.00 0.19a Donated blood in past year 0.00 0.08a 0.00a Donated to charity in past year 0.14a 0.25a −0.09a Support birth control to teens 14 to 16 years old −0.06a −0.06a 0.21 A same-sex female couple can bring up a child just as well −0.08a 0.04 0.31 Oppose capital punishment −0.08a 0.02 0.31 Courts are too harsh with criminals −0.11a −0.09a 0.19 NOTE.—Using the nonresponse-adjusted poststratified weights yields similar results. a Significant at the 0.05 level, based on the Wald F-statistic, accounting for the complex survey design. Table 1. Selection-Weighted Correlations of Voting Eligibility, Having Voted in 2008 (Among Eligible to Vote), and Voting for Obama (Among Self-Reported Voters) with Fifteen GSS Survey Variables Eligible to vote n = 1,974 Voted in 2008 n = 1,815 Voted for Obama n = 1,294 Fair or poor health 0.01 −0.13a 0.13a Very or pretty happy −0.01 0.09a −0.08a Life exciting −0.04 0.05 −0.07a Most people try to be helpful 0.07a 0.19a −0.01 Most try to take advantage −0.10a −0.13a 0.03a Most people can be trusted 0.04 0.18a −0.04a Family income below average −0.01 −0.14a 0.09a Read newspaper every day 0.10a 0.16a 0.02 No religion −0.05a 0.00 0.19a Donated blood in past year 0.00 0.08a 0.00a Donated to charity in past year 0.14a 0.25a −0.09a Support birth control to teens 14 to 16 years old −0.06a −0.06a 0.21 A same-sex female couple can bring up a child just as well −0.08a 0.04 0.31 Oppose capital punishment −0.08a 0.02 0.31 Courts are too harsh with criminals −0.11a −0.09a 0.19 Eligible to vote n = 1,974 Voted in 2008 n = 1,815 Voted for Obama n = 1,294 Fair or poor health 0.01 −0.13a 0.13a Very or pretty happy −0.01 0.09a −0.08a Life exciting −0.04 0.05 −0.07a Most people try to be helpful 0.07a 0.19a −0.01 Most try to take advantage −0.10a −0.13a 0.03a Most people can be trusted 0.04 0.18a −0.04a Family income below average −0.01 −0.14a 0.09a Read newspaper every day 0.10a 0.16a 0.02 No religion −0.05a 0.00 0.19a Donated blood in past year 0.00 0.08a 0.00a Donated to charity in past year 0.14a 0.25a −0.09a Support birth control to teens 14 to 16 years old −0.06a −0.06a 0.21 A same-sex female couple can bring up a child just as well −0.08a 0.04 0.31 Oppose capital punishment −0.08a 0.02 0.31 Courts are too harsh with criminals −0.11a −0.09a 0.19 NOTE.—Using the nonresponse-adjusted poststratified weights yields similar results. a Significant at the 0.05 level, based on the Wald F-statistic, accounting for the complex survey design. 3.1.3 Change in survey estimates with the addition of the voting-related variables Does the addition of the voting variable in the weighting adjustments alter the survey estimates? Figure 2 shows that the changes are generally small, but three of the fifteen estimates are significantly different.3 The largest change is 4%. The addition of voting contributed to most of the shifts, though voting eligibility seems to be the driving factor in two cases. The full estimates under each set of weights, differences, and significance tests are included in Appendix A. The changes in the estimates were substantially larger when less elaborate population demographic controls were used in the weighting (results not presented). Figure 2. View largeDownload slide Differences in Weighted Estimates for Fifteen GSS Variables, Relative to the Nonresponse and Poststratified Estimates Without Voting Eligibility, Voting, and Candidate Choice. * Significant difference between weighted estimates using GSS nonresponse adjustment with raking to age, sex, education, and race, while adding voting eligibility, voting, and candidate choice to the weighting adjustment, accounting for the complex survey design. Figure 2. View largeDownload slide Differences in Weighted Estimates for Fifteen GSS Variables, Relative to the Nonresponse and Poststratified Estimates Without Voting Eligibility, Voting, and Candidate Choice. * Significant difference between weighted estimates using GSS nonresponse adjustment with raking to age, sex, education, and race, while adding voting eligibility, voting, and candidate choice to the weighting adjustment, accounting for the complex survey design. 3.2 Study 2: Volunteering Approximately two-thirds (1,299) of the 2012 GSS respondents were randomly assigned to be asked the two questions from the 2012 CPS September Supplement on volunteering that CPS uses to construct its volunteering estimate: “Since [CURRENT MONTH] 1st of last year, have you done any volunteer activities through or for an organization?” And if the respondent did not respond “yes,” it was followed by: “Sometimes people don’t think of activities they do infrequently or activities they do for children’s schools or youth organizations as volunteer activities. Since [CURRENT MONTH] 1st of last year, have you done any of these types of volunteer activities?” Respondents who answered “yes” to either question were classified as having volunteered in the past year (613), and those who said “no” to both questions were classified as not having volunteered (681). Five respondents who refused the two questions were randomly imputed into the two categories, four into the first group and one into the second group. The population proportion for volunteering used in weighting was derived from the 2012 CPS data. 3.2.1 Conditional relationship between the volunteering adjustment variable and nonresponse We first compare the GSS weighted estimate for volunteering, including demographic adjustments, with the CPS estimate. Figure 3 shows an exceptionally large difference, with the GSS estimate about twice as large (48%) as the CPS-based adjusted estimate (26%), much larger than the difference for voting in Study 1. As both the GSS and the CPS are interviewer-administered surveys using the same two volunteering questions, we attribute the difference to nonresponse. Although one might argue that the context of the surveys, one on social topics and the other on economic topics, could induce different levels of social desirability reporting bias. The same is true for nonresponse—those agreeing to participate in a social survey may be more socially involved. Figure 3. View largeDownload slide GSS Weighted Estimates (Using GSS Weights Also Adjusted to Population Demographic Characteristics) and CPS Population Benchmark Estimates for Percent of the Adult Population in 2012 Who Volunteered. Bars represent standard errors. Figure 3. View largeDownload slide GSS Weighted Estimates (Using GSS Weights Also Adjusted to Population Demographic Characteristics) and CPS Population Benchmark Estimates for Percent of the Adult Population in 2012 Who Volunteered. Bars represent standard errors. 3.2.2 Relationship between the volunteering adjustment variable and survey variables The volunteering variable was significantly correlated with more than half the selected GSS variables, shown in table 2. The pattern is very similar to the correlations with voting, presented in the middle column in table 2, which is consistent with expectations under the shared mechanism of civic duty. As in Study 1 with voting, not all variables were correlated with volunteering, and among the statistically significant correlations, they ranged from 0.07 to as high as 0.32. Table 2. Selection-Weighted Correlations of Volunteering with 15 GSS Survey Variables Volunteered (n = 1,299) Fair or poor health −0.12a Very or pretty happy 0.07a Life exciting 0.11a Most people try to be helpful 0.05 Most try to take advantage −0.14a Most people can be trusted 0.14a Family income below average −0.08a Read newspaper every day 0.05 No religion −0.07a Donated blood in past year 0.15a Donated to charity in past year 0.32a Support birth control to teens 14 to 16 years old −0.04 A same-sex female couple can bring up a child just as well −0.01 Oppose capital punishment −0.04 Courts are too harsh with criminals 0.01 Volunteered (n = 1,299) Fair or poor health −0.12a Very or pretty happy 0.07a Life exciting 0.11a Most people try to be helpful 0.05 Most try to take advantage −0.14a Most people can be trusted 0.14a Family income below average −0.08a Read newspaper every day 0.05 No religion −0.07a Donated blood in past year 0.15a Donated to charity in past year 0.32a Support birth control to teens 14 to 16 years old −0.04 A same-sex female couple can bring up a child just as well −0.01 Oppose capital punishment −0.04 Courts are too harsh with criminals 0.01 NOTE.—Using the nonresponse-adjusted poststratified weights yields similar results. a Significant at the 0.05 level, based on the Wald F-statistic, accounting for the complex survey design. Table 2. Selection-Weighted Correlations of Volunteering with 15 GSS Survey Variables Volunteered (n = 1,299) Fair or poor health −0.12a Very or pretty happy 0.07a Life exciting 0.11a Most people try to be helpful 0.05 Most try to take advantage −0.14a Most people can be trusted 0.14a Family income below average −0.08a Read newspaper every day 0.05 No religion −0.07a Donated blood in past year 0.15a Donated to charity in past year 0.32a Support birth control to teens 14 to 16 years old −0.04 A same-sex female couple can bring up a child just as well −0.01 Oppose capital punishment −0.04 Courts are too harsh with criminals 0.01 Volunteered (n = 1,299) Fair or poor health −0.12a Very or pretty happy 0.07a Life exciting 0.11a Most people try to be helpful 0.05 Most try to take advantage −0.14a Most people can be trusted 0.14a Family income below average −0.08a Read newspaper every day 0.05 No religion −0.07a Donated blood in past year 0.15a Donated to charity in past year 0.32a Support birth control to teens 14 to 16 years old −0.04 A same-sex female couple can bring up a child just as well −0.01 Oppose capital punishment −0.04 Courts are too harsh with criminals 0.01 NOTE.—Using the nonresponse-adjusted poststratified weights yields similar results. a Significant at the 0.05 level, based on the Wald F-statistic, accounting for the complex survey design. 3.2.3 Change in survey estimates with the addition of the volunteering variable Figure 4 shows that the addition of volunteering to the weighting adjustment led to significantly different estimates for five of the fifteen variables, just as in the voting adjustment, but only two of the variables were shared. The magnitudes of the shifts in the estimates are larger than those in figure 2—the largest change is 13%—but so are the standard errors since the volunteering variables were administered to only two-thirds of the respondents.4 The estimates under each set of weights, differences, and significance tests are included in Appendix B. Figure 4. View largeDownload slide Relative Differences in Weighted Estimates for Fifteen GSS Variables after Including Volunteering in the Adjustments. * Significant difference between weighted estimates using GSS nonresponse adjustment with raking to age, sex, education, and race, while adding volunteering to the weighting adjustment, accounting for the complex survey design. Figure 4. View largeDownload slide Relative Differences in Weighted Estimates for Fifteen GSS Variables after Including Volunteering in the Adjustments. * Significant difference between weighted estimates using GSS nonresponse adjustment with raking to age, sex, education, and race, while adding volunteering to the weighting adjustment, accounting for the complex survey design. 3.3 Impact on Total Error Our focus in Study 1 and Study 2 was the relationship between the auxiliary information with nonresponse and the survey variables, and the resultant ability to adjust the survey estimates. In our introduction, we noted that a poor adjustment variable would disproportionately increase the variance of the estimate through increased weight variation, relative to the change in the estimate. A measure of total error to evaluate the variance and bias tradeoff is the mean square error (MSE), which is equal to the sum of the bias squared (⁠ By2 ⁠) and the variance (⁠ Vy ⁠) of the estimate from variable (⁠ y ⁠): MSE=By2+Vy ⁠. For the estimate of the bias, we treat the estimate using the augmented weights that include the voting variables or the volunteering variable as the unbiased estimate. Table 3 shows the MSEs for the fifteen GSS variables, and the relative change in MSEs from adding the voting (Study 1) and volunteering (Study 2) variables. Despite the modest sample sizes that increase the relative importance of the variance estimates, the bias reduction was sufficient to reduce the overall MSEs for almost all the estimates. On average, the MSEs were reduced by 4% from adding voting and by 27% from adding volunteering. The larger benefit from adding volunteering is consistent with the large respondent deviation from the benchmark for volunteering (figure 3), which allowed for larger shifts in the weighted estimates in Study 2. The reduction in MSE is also highly dependent on the other variables in the adjustment. The reduction in MSE when adding voting was almost four times larger (change in MSE of -14.4% instead of -3.9%) when simpler, dichotomized versions of the demographic variables were used in the weight calibration. Our estimate of the reduction in MSE is also likely an underestimate, since the usual linearization approach to variance estimation, used on the GSS, does not capitalize on the stabilizing effect of calibration to variables that are highly correlated with the survey variable of interest. Future studies should examine the impact of the variance estimation approach. Table 3. Estimated MSEs and Relative Change in MSEs for the Fifteen GSS Survey Variables, from Adding Voting (Study 1) and Volunteering (Study 2) Voting (Study 1) Volunteering (Study 2) MSE prior to voting adjustment (A) MSE post voting adjustment (B) Relative change (B-A)/A MSE prior to volunteering adjustment (C) MSE post volunteering adjustment (D) Relative change (D-C)/C Fair or poor health 0.02% 0.02% −2.4% 0.06% 0.04% −31.1% Very or pretty happy 0.01% 0.01% 5.3% 0.02% 0.02% −4.6% Life exciting 0.04% 0.04% 4.4% 0.12% 0.07% −36.3% Most people try to be helpful 0.04% 0.03% −17.5% 0.05% 0.05% 11.0% Most try to take advantage 0.04% 0.04% −1.0% 0.12% 0.09% −24.0% Most people can be trusted 0.04% 0.04% −4.9% 0.11% 0.08% −25.2% Family income below average 0.02% 0.02% 5.0% 0.03% 0.03% 10.4% Read newspaper every day 0.04% 0.04% −11.9% 0.04% 0.04% −11.6% No religion 0.01% 0.01% −17.3% 0.06% 0.02% −57.3% Donated blood in past year 0.02% 0.02% −5.2% 0.08% 0.04% −55.8% Donated to charity in past year 0.04% 0.03% −27.9% 0.36% 0.06% −82.2% Support birth control to teens 14 to 16 years old 0.03% 0.03% 4.8% 0.05% 0.03% −32.5% A same-sex female couple can bring up a child just as well 0.06% 0.05% −16.8% 0.07% 0.05% −27.0% Oppose capital punishment 0.02% 0.02% −5.2% 0.05% 0.04% −22.5% Courts are too harsh with criminals 0.01% 0.01% 32.4% 0.01% 0.01% −12.8% Average relative change in MSE −3.9% −26.8% Voting (Study 1) Volunteering (Study 2) MSE prior to voting adjustment (A) MSE post voting adjustment (B) Relative change (B-A)/A MSE prior to volunteering adjustment (C) MSE post volunteering adjustment (D) Relative change (D-C)/C Fair or poor health 0.02% 0.02% −2.4% 0.06% 0.04% −31.1% Very or pretty happy 0.01% 0.01% 5.3% 0.02% 0.02% −4.6% Life exciting 0.04% 0.04% 4.4% 0.12% 0.07% −36.3% Most people try to be helpful 0.04% 0.03% −17.5% 0.05% 0.05% 11.0% Most try to take advantage 0.04% 0.04% −1.0% 0.12% 0.09% −24.0% Most people can be trusted 0.04% 0.04% −4.9% 0.11% 0.08% −25.2% Family income below average 0.02% 0.02% 5.0% 0.03% 0.03% 10.4% Read newspaper every day 0.04% 0.04% −11.9% 0.04% 0.04% −11.6% No religion 0.01% 0.01% −17.3% 0.06% 0.02% −57.3% Donated blood in past year 0.02% 0.02% −5.2% 0.08% 0.04% −55.8% Donated to charity in past year 0.04% 0.03% −27.9% 0.36% 0.06% −82.2% Support birth control to teens 14 to 16 years old 0.03% 0.03% 4.8% 0.05% 0.03% −32.5% A same-sex female couple can bring up a child just as well 0.06% 0.05% −16.8% 0.07% 0.05% −27.0% Oppose capital punishment 0.02% 0.02% −5.2% 0.05% 0.04% −22.5% Courts are too harsh with criminals 0.01% 0.01% 32.4% 0.01% 0.01% −12.8% Average relative change in MSE −3.9% −26.8% Table 3. Estimated MSEs and Relative Change in MSEs for the Fifteen GSS Survey Variables, from Adding Voting (Study 1) and Volunteering (Study 2) Voting (Study 1) Volunteering (Study 2) MSE prior to voting adjustment (A) MSE post voting adjustment (B) Relative change (B-A)/A MSE prior to volunteering adjustment (C) MSE post volunteering adjustment (D) Relative change (D-C)/C Fair or poor health 0.02% 0.02% −2.4% 0.06% 0.04% −31.1% Very or pretty happy 0.01% 0.01% 5.3% 0.02% 0.02% −4.6% Life exciting 0.04% 0.04% 4.4% 0.12% 0.07% −36.3% Most people try to be helpful 0.04% 0.03% −17.5% 0.05% 0.05% 11.0% Most try to take advantage 0.04% 0.04% −1.0% 0.12% 0.09% −24.0% Most people can be trusted 0.04% 0.04% −4.9% 0.11% 0.08% −25.2% Family income below average 0.02% 0.02% 5.0% 0.03% 0.03% 10.4% Read newspaper every day 0.04% 0.04% −11.9% 0.04% 0.04% −11.6% No religion 0.01% 0.01% −17.3% 0.06% 0.02% −57.3% Donated blood in past year 0.02% 0.02% −5.2% 0.08% 0.04% −55.8% Donated to charity in past year 0.04% 0.03% −27.9% 0.36% 0.06% −82.2% Support birth control to teens 14 to 16 years old 0.03% 0.03% 4.8% 0.05% 0.03% −32.5% A same-sex female couple can bring up a child just as well 0.06% 0.05% −16.8% 0.07% 0.05% −27.0% Oppose capital punishment 0.02% 0.02% −5.2% 0.05% 0.04% −22.5% Courts are too harsh with criminals 0.01% 0.01% 32.4% 0.01% 0.01% −12.8% Average relative change in MSE −3.9% −26.8% Voting (Study 1) Volunteering (Study 2) MSE prior to voting adjustment (A) MSE post voting adjustment (B) Relative change (B-A)/A MSE prior to volunteering adjustment (C) MSE post volunteering adjustment (D) Relative change (D-C)/C Fair or poor health 0.02% 0.02% −2.4% 0.06% 0.04% −31.1% Very or pretty happy 0.01% 0.01% 5.3% 0.02% 0.02% −4.6% Life exciting 0.04% 0.04% 4.4% 0.12% 0.07% −36.3% Most people try to be helpful 0.04% 0.03% −17.5% 0.05% 0.05% 11.0% Most try to take advantage 0.04% 0.04% −1.0% 0.12% 0.09% −24.0% Most people can be trusted 0.04% 0.04% −4.9% 0.11% 0.08% −25.2% Family income below average 0.02% 0.02% 5.0% 0.03% 0.03% 10.4% Read newspaper every day 0.04% 0.04% −11.9% 0.04% 0.04% −11.6% No religion 0.01% 0.01% −17.3% 0.06% 0.02% −57.3% Donated blood in past year 0.02% 0.02% −5.2% 0.08% 0.04% −55.8% Donated to charity in past year 0.04% 0.03% −27.9% 0.36% 0.06% −82.2% Support birth control to teens 14 to 16 years old 0.03% 0.03% 4.8% 0.05% 0.03% −32.5% A same-sex female couple can bring up a child just as well 0.06% 0.05% −16.8% 0.07% 0.05% −27.0% Oppose capital punishment 0.02% 0.02% −5.2% 0.05% 0.04% −22.5% Courts are too harsh with criminals 0.01% 0.01% 32.4% 0.01% 0.01% −12.8% Average relative change in MSE −3.9% −26.8% 3.4 Evaluation of Reduction of Bias Against Population Benchmarks The analysis in Sections 3.1 and 3.2 focused on the relative change in estimates when voting or volunteering are included in the weight calibration adjustment. Section 3.3 presented estimates of MSE, assuming that the augmented weights produce less bias in the survey estimates. In this instance, we have the ability to evaluate this fundamental assumption, using our population benchmarks. To do this, we used the voting adjusted weights to estimate volunteering and the volunteering adjusted weights to estimate voting—the two estimates for which we have benchmark values (from the CPS and the FEC, respectively). Table 4 shows that by adding voting and volunteering to the adjustments, the estimates for volunteering and voting moved by 1.3 and by 2.5 percentage points toward the benchmark, respectively (from 47.9% to 46.6% and from 72.2% to 69.7%). Most of the bias remained, but the volunteering adjustment removed a greater proportion of the bias—over a quarter of the estimated bias ([(72.2–69.7)/(72.2–63.7)]*100 = 29.4%). Table 4. Effect on the Estimate for Volunteering after Weighting for Voting and Effect on the Estimate for Voter Turnout in 2008 from Weighting for Volunteering and Benchmark Estimates from the Current Population Survey and the Federal Election Commission 1. GSS nonresponse adjustment and calibration to demographic characteristics 2. Adding voting to (1) 3. Adding volunteering to (1) 4. Benchmark values (from the CPS and the FEC) n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Volunteering 1,299 47.9% (1.73) 46.6% (1.77) 26.4% (0.2) Voting 1,188 72.2% (1.57) 69.7% (1.83) 63.7% (–) 1. GSS nonresponse adjustment and calibration to demographic characteristics 2. Adding voting to (1) 3. Adding volunteering to (1) 4. Benchmark values (from the CPS and the FEC) n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Volunteering 1,299 47.9% (1.73) 46.6% (1.77) 26.4% (0.2) Voting 1,188 72.2% (1.57) 69.7% (1.83) 63.7% (–) Table 4. Effect on the Estimate for Volunteering after Weighting for Voting and Effect on the Estimate for Voter Turnout in 2008 from Weighting for Volunteering and Benchmark Estimates from the Current Population Survey and the Federal Election Commission 1. GSS nonresponse adjustment and calibration to demographic characteristics 2. Adding voting to (1) 3. Adding volunteering to (1) 4. Benchmark values (from the CPS and the FEC) n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Volunteering 1,299 47.9% (1.73) 46.6% (1.77) 26.4% (0.2) Voting 1,188 72.2% (1.57) 69.7% (1.83) 63.7% (–) 1. GSS nonresponse adjustment and calibration to demographic characteristics 2. Adding voting to (1) 3. Adding volunteering to (1) 4. Benchmark values (from the CPS and the FEC) n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Volunteering 1,299 47.9% (1.73) 46.6% (1.77) 26.4% (0.2) Voting 1,188 72.2% (1.57) 69.7% (1.83) 63.7% (–) Unlike all the weights presented in this study, the GSS public use file weights are not calibrated to population distributions on demographic characteristics. Compared to the GSS public use file weighted estimates, our weights that are calibrated to age, sex, education, and race reduce bias, prior to the addition of voting or volunteering (results not presented). This was not the focus of our investigation—we show that despite demographic adjustments, use of voting and volunteering can further reduce bias in weighted estimates. However, this also suggests that the GSS should consider calibration to population demographic distributions, regardless of whether adjustment to voting or volunteering variables is adopted. 4. SUMMARY AND DISCUSSION Our results show promise in the use of voting and volunteering to inform postsurvey adjustments. Both voting and volunteering variables were substantially skewed in the pool of GSS respondents. Both were also modestly correlated with a variety of other GSS variables. The voting variable was comprised of three nested variables (voting eligibility, voting, and candidate choice). Consistent with the notion of civic duty, the correlations with survey variables were most similar between voting (middle column in figure 2) and volunteering (figure 4). When the voting variables were included, more than a third of the resulting weighted estimates changed. Most of the changes in the estimates were attributable to the addition of voting as opposed to voting eligibility or candidate choice. It is possible that candidate choice would play a larger factor depending on the survey design features—such as the topic of the survey (e.g., religion, education) and the sponsor (e.g., values associated with the sponsoring organization). The inclusion of volunteering also significantly altered the weighted estimates, but the magnitude of the effect was larger and the impact was mostly on different variables. By comparison, the changes in the estimates attributable to the two conventional adjustments for nonresponse (sample-based geographic nonresponse adjustment and population-based demographic calibration) were fewer and smaller in magnitude (results not shown). For the survey practitioner, these finding suggest the deliberate inclusion of substantive survey questions that have dependable external benchmark estimates. Our examples used the ACS, CPS, and the election results from the NEC. There are numerous such benchmarks, and the choice should be informed by the topic of the survey—for example, a health survey might use enrollment in federal and state health insurance plans. Moreover, the payoff to this approach may be larger than our results suggest in the many surveys with response rates substantially lower than the GSS’s relatively high 71%. For the survey statistician, this approach means that it is not sufficient to request the inclusion of demographic questions in order to create weighting adjustments. The statistician responsible for weighting should be involved during the survey design and could investigate different sources of benchmark estimates. For the student of survey error, there are many challenging questions to be addressed. How do measurement differences in the survey and benchmark data affect estimates? How could such effects be minimized? How should systematic and variable errors in these benchmarks be best incorporated? This approach to constructing weighting adjustments may be novel for probability-based surveys, but nondemographic variables have been used in weighting of data from nonprobability-based surveys where inference depends entirely or almost entirely on the selection of statistical models (Schonlau, van Soest, and Kapteyn 2007; DiSogra, Cobb, Chan, and Dennis 2011; Fahimi, Barlas, Thomas, and Buttermore 2015). For example, researchers at the Pew Research Center compared estimates from nine samples coming from eight organizations using a single instrument (Pew Research Center 2016). Based on benchmark estimates from large-scale national probability-based surveys, one sample showed the least bias in its survey estimates. The key differentiating factor for that sample was the vendor’s use of substantive survey measures in the weighting adjustments, including political ideology and party membership. We note this parallel application, but acknowledge that the tolerance levels for error in postsurvey adjustments are markedly different for surveys that rely on probability-based inference, calling for a much higher level of scrutiny. Like most other postsurvey adjustments, the proposed approach can do more harm than good, if implemented blindly. Bias can be induced, rather than reduced, by using population totals or estimates without examining how their properties differ from the properties of the corresponding survey measures. Part of our motivation for presenting two studies was that the adjustments in the two were subject to different error sources: population totals derived from administrative records (Study 1) versus from another survey (Study 2). Future research could focus on whether there are conditions in which one is more desirable than the other. For the present, we are encouraged that the findings were consistent across the two studies suggesting robustness to the conclusions. One potentially important limitation in Study 1 is the inability to correct for measurement error in reported voting. This may affect the magnitudes of some of the estimates, but seems unlikely to change the general conclusions.5 Nonetheless, future work could address this potential problem, as well as explore the use of other types of substantive variables for adjustment purposes. The following list includes key questions: How can measurement error in responses to voting-related questions be reduced? Some have proposed different ways of asking the questions (Belli, Traugott, Young, and McGonagle 1999; Zeglovits and Kritzinger 2014), while others have offered statistical adjustments (Voogt 2005; Katz and Katz 2010). What other substantive variables could be considered for adjustment? For example, the Health Information and National Trends Survey (HINTS) employs estimates from the National Health Interview Survey (NHIS) for having been diagnosed with cancer and health insurance status (Cantor, Coa, Crystal-Mansour, Davis, and Dipko et al. 2009; Peytchev, Ridenhour, and Krotki 2010; West et al. 2015). What other administrative records contain potentially useful adjustment variables? For example, the Centers for Medicare & Medicaid Services’ (CMS) Medicare Coverage Database contains information that may be relevant for certain kinds of surveys. How should methods reflect the uncertainty in population control totals, especially when they are estimates themselves? While presidential candidate choice is a population count after the election, estimates from surveys (such as CPS or NHIS) have sampling variances, which can negatively bias the variance estimates. The CPS volunteering estimate in Study 2 is subject to sampling error, but it is very small relative to the magnitude of the GSS variance estimates. This would not be the case when the survey being conducted is larger or when the control totals are derived from a smaller survey. When estimates from other surveys are used, how important is the assumption that the measurement procedures be the same across the surveys? In a survey with a low response rate, departures from the assumption might not be critical, as the error source being corrected may be the dominant one. Nonetheless, the measurement properties of the external data and the survey data ought to be as similar as possible and, when they are not, the question is whether the benefits, from a total survey error perspective, outweigh the costs. Finally, it is worth noting that our methodological inquiry has produced important substantive findings. We have demonstrated that the GSS, probably the most rigorously done public opinion survey in the United States, substantially underrepresents nonvoters, those who do no volunteer work, and noncitizens. Future work could usefully explore the consequences of this for the many different kinds of analyses based on the GSS (and possibly on other surveys, as well). This article was written while the first author was a Research Assistant Professor at the Institute for Social Research at the University of Michigan, Ann Arbor, MI, USA. The authors are grateful for the constructive comments from the anonymous reviewers and the editor. Footnotes 1 Multiple imputation models the survey variables instead of the likelihood of an interview to reduce the adverse impact on variance estimates (Peytchev 2012; Rässler and Schnell 2004), but its ability to reduce bias still depends on auxiliary information that is associated with both nonresponse and the survey variables (Little and Vartivarian 2005). In addition, this approach is limited to cases where information is available for the full sample and, thus, does not permit the use of population benchmarks. 2 Age (18–29, 30–44, 45–59, 60, and older), sex, education (less than high school diploma; high school diploma, GED, or some college but no four-year degree; and bachelor's or higher degree), and race (white versus other). 3 Statistical testing used the stacking approach in which the data under the different weighting scenarios are stacked (e.g., Aldworth, Barnett, Cribb, Davis, Foster et al. 2013), and tests conducted between scenarios. The Rao-Scott Chi-Square was used, which also applies a correction for the complex survey design. 4 Differences between the voting and volunteering results are not due to the fact that the voting questions were asked of the total sample and the volunteering questions were asked of only a random subsample. All the voting results are essentially the same if based on only the cases asked the volunteering questions. 5 If measurement error alone accounted for the differences in eligibility, for example, more than half of the noncitizens would have had to report being citizens or citizens 14-17 years of age in 2008 would have had to have reported they voted in 2008. Similarly, if misreporting were solely responsible for the results, 17% of those who voted for McCain would have had to misreport voting for Obama. Appendix A Weighted Estimates with GSS Nonresponse Adjustments and Poststratification to Demographic Characteristics, and with the Addition of Poststratification to Voting Eligibility, Voter Turnout, and Candidate Choice 1. GSS nonresponse adjustment and poststratification to demographic characteristics 2. Adding voting eligibility 3. Adding voting eligibility and voting 4. Adding voting eligibility, voting, and candidate choice Difference (4–1) p-value n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 1,306 24.3% (1.36) 24.2% (1.42) 25.2% (1.51) 25.0% (1.50) 0.7% 0.14 Very or pretty happy 1,964 87.6% (0.98) 87.8% (0.96) 87.4% (1.02) 87.5% (1.02) −0.2% 0.52 Life exciting 1,296 52.9% (2.01) 53.5% (2.00) 52.9% (2.03) 53.0% (2.05) 0.1% 0.86 Most people try to be helpful 1,328 47.6% (1.70) 47.8% (1.76) 46.9% (1.75) 46.7% (1.76) −0.9% 0.08 Most try to take advantage 1,326 42.2% (1.99) 42.3% (2.02) 42.6% (2.03) 42.7% (2.04) 0.5% 0.35 Most people can be trusted 1,331 34.2% (1.92) 34.6% (1.93) 33.9% (1.90) 33.8% (1.92) −0.4% 0.41 Family income below average 1,952 31.3% (1.37) 31.1% (1.37) 31.8% (1.44) 31.7% (1.44) 0.3% 0.38 Read newspaper every day 1,301 28.1% (1.91) 28.0% (1.94) 27.6% (1.96) 27.4% (1.94) −0.8% <0.05 No religion 1,967 20.1% (1.02) 20.2% (1.07) 20.0% (1.03) 19.6% (1.03) −0.5% 0.13 Donated blood in past year 1,301 13.5% (1.26) 13.5% (1.28) 13.1% (1.28) 13.0% (1.33) −0.5% 0.13 Donated to charity in past year 1,298 73.6% (1.55) 73.4% (1.62) 72.3% (1.69) 72.4% (1.68) −1.2% <0.05 Support birth control to teens 14 to 16 years old 1,269 57.6% (1.74) 57.9% (1.79) 58.3% (1.82) 57.9% (1.81) 0.3% 0.56 A same-sex female couple can bring up a child just as well 1,230 47.3% (2.26) 47.7% (2.24) 47.1% (2.21) 46.4% (2.22) −0.9% <0.05 Oppose capital punishment 1,824 33.6% (1.33) 34.2% (1.31) 34.0% (1.30) 33.4% (1.30) −0.1% 0.76 Courts are too harsh with criminals 1,777 14.5% (0.79) 14.8% (0.88) 15.1% (0.98) 14.8% (0.98) 0.3% 0.40 1. GSS nonresponse adjustment and poststratification to demographic characteristics 2. Adding voting eligibility 3. Adding voting eligibility and voting 4. Adding voting eligibility, voting, and candidate choice Difference (4–1) p-value n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 1,306 24.3% (1.36) 24.2% (1.42) 25.2% (1.51) 25.0% (1.50) 0.7% 0.14 Very or pretty happy 1,964 87.6% (0.98) 87.8% (0.96) 87.4% (1.02) 87.5% (1.02) −0.2% 0.52 Life exciting 1,296 52.9% (2.01) 53.5% (2.00) 52.9% (2.03) 53.0% (2.05) 0.1% 0.86 Most people try to be helpful 1,328 47.6% (1.70) 47.8% (1.76) 46.9% (1.75) 46.7% (1.76) −0.9% 0.08 Most try to take advantage 1,326 42.2% (1.99) 42.3% (2.02) 42.6% (2.03) 42.7% (2.04) 0.5% 0.35 Most people can be trusted 1,331 34.2% (1.92) 34.6% (1.93) 33.9% (1.90) 33.8% (1.92) −0.4% 0.41 Family income below average 1,952 31.3% (1.37) 31.1% (1.37) 31.8% (1.44) 31.7% (1.44) 0.3% 0.38 Read newspaper every day 1,301 28.1% (1.91) 28.0% (1.94) 27.6% (1.96) 27.4% (1.94) −0.8% <0.05 No religion 1,967 20.1% (1.02) 20.2% (1.07) 20.0% (1.03) 19.6% (1.03) −0.5% 0.13 Donated blood in past year 1,301 13.5% (1.26) 13.5% (1.28) 13.1% (1.28) 13.0% (1.33) −0.5% 0.13 Donated to charity in past year 1,298 73.6% (1.55) 73.4% (1.62) 72.3% (1.69) 72.4% (1.68) −1.2% <0.05 Support birth control to teens 14 to 16 years old 1,269 57.6% (1.74) 57.9% (1.79) 58.3% (1.82) 57.9% (1.81) 0.3% 0.56 A same-sex female couple can bring up a child just as well 1,230 47.3% (2.26) 47.7% (2.24) 47.1% (2.21) 46.4% (2.22) −0.9% <0.05 Oppose capital punishment 1,824 33.6% (1.33) 34.2% (1.31) 34.0% (1.30) 33.4% (1.30) −0.1% 0.76 Courts are too harsh with criminals 1,777 14.5% (0.79) 14.8% (0.88) 15.1% (0.98) 14.8% (0.98) 0.3% 0.40 1. GSS nonresponse adjustment and poststratification to demographic characteristics 2. Adding voting eligibility 3. Adding voting eligibility and voting 4. Adding voting eligibility, voting, and candidate choice Difference (4–1) p-value n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 1,306 24.3% (1.36) 24.2% (1.42) 25.2% (1.51) 25.0% (1.50) 0.7% 0.14 Very or pretty happy 1,964 87.6% (0.98) 87.8% (0.96) 87.4% (1.02) 87.5% (1.02) −0.2% 0.52 Life exciting 1,296 52.9% (2.01) 53.5% (2.00) 52.9% (2.03) 53.0% (2.05) 0.1% 0.86 Most people try to be helpful 1,328 47.6% (1.70) 47.8% (1.76) 46.9% (1.75) 46.7% (1.76) −0.9% 0.08 Most try to take advantage 1,326 42.2% (1.99) 42.3% (2.02) 42.6% (2.03) 42.7% (2.04) 0.5% 0.35 Most people can be trusted 1,331 34.2% (1.92) 34.6% (1.93) 33.9% (1.90) 33.8% (1.92) −0.4% 0.41 Family income below average 1,952 31.3% (1.37) 31.1% (1.37) 31.8% (1.44) 31.7% (1.44) 0.3% 0.38 Read newspaper every day 1,301 28.1% (1.91) 28.0% (1.94) 27.6% (1.96) 27.4% (1.94) −0.8% <0.05 No religion 1,967 20.1% (1.02) 20.2% (1.07) 20.0% (1.03) 19.6% (1.03) −0.5% 0.13 Donated blood in past year 1,301 13.5% (1.26) 13.5% (1.28) 13.1% (1.28) 13.0% (1.33) −0.5% 0.13 Donated to charity in past year 1,298 73.6% (1.55) 73.4% (1.62) 72.3% (1.69) 72.4% (1.68) −1.2% <0.05 Support birth control to teens 14 to 16 years old 1,269 57.6% (1.74) 57.9% (1.79) 58.3% (1.82) 57.9% (1.81) 0.3% 0.56 A same-sex female couple can bring up a child just as well 1,230 47.3% (2.26) 47.7% (2.24) 47.1% (2.21) 46.4% (2.22) −0.9% <0.05 Oppose capital punishment 1,824 33.6% (1.33) 34.2% (1.31) 34.0% (1.30) 33.4% (1.30) −0.1% 0.76 Courts are too harsh with criminals 1,777 14.5% (0.79) 14.8% (0.88) 15.1% (0.98) 14.8% (0.98) 0.3% 0.40 1. GSS nonresponse adjustment and poststratification to demographic characteristics 2. Adding voting eligibility 3. Adding voting eligibility and voting 4. Adding voting eligibility, voting, and candidate choice Difference (4–1) p-value n Percent Standard error Percent Standard error Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 1,306 24.3% (1.36) 24.2% (1.42) 25.2% (1.51) 25.0% (1.50) 0.7% 0.14 Very or pretty happy 1,964 87.6% (0.98) 87.8% (0.96) 87.4% (1.02) 87.5% (1.02) −0.2% 0.52 Life exciting 1,296 52.9% (2.01) 53.5% (2.00) 52.9% (2.03) 53.0% (2.05) 0.1% 0.86 Most people try to be helpful 1,328 47.6% (1.70) 47.8% (1.76) 46.9% (1.75) 46.7% (1.76) −0.9% 0.08 Most try to take advantage 1,326 42.2% (1.99) 42.3% (2.02) 42.6% (2.03) 42.7% (2.04) 0.5% 0.35 Most people can be trusted 1,331 34.2% (1.92) 34.6% (1.93) 33.9% (1.90) 33.8% (1.92) −0.4% 0.41 Family income below average 1,952 31.3% (1.37) 31.1% (1.37) 31.8% (1.44) 31.7% (1.44) 0.3% 0.38 Read newspaper every day 1,301 28.1% (1.91) 28.0% (1.94) 27.6% (1.96) 27.4% (1.94) −0.8% <0.05 No religion 1,967 20.1% (1.02) 20.2% (1.07) 20.0% (1.03) 19.6% (1.03) −0.5% 0.13 Donated blood in past year 1,301 13.5% (1.26) 13.5% (1.28) 13.1% (1.28) 13.0% (1.33) −0.5% 0.13 Donated to charity in past year 1,298 73.6% (1.55) 73.4% (1.62) 72.3% (1.69) 72.4% (1.68) −1.2% <0.05 Support birth control to teens 14 to 16 years old 1,269 57.6% (1.74) 57.9% (1.79) 58.3% (1.82) 57.9% (1.81) 0.3% 0.56 A same-sex female couple can bring up a child just as well 1,230 47.3% (2.26) 47.7% (2.24) 47.1% (2.21) 46.4% (2.22) −0.9% <0.05 Oppose capital punishment 1,824 33.6% (1.33) 34.2% (1.31) 34.0% (1.30) 33.4% (1.30) −0.1% 0.76 Courts are too harsh with criminals 1,777 14.5% (0.79) 14.8% (0.88) 15.1% (0.98) 14.8% (0.98) 0.3% 0.40 Appendix B Weighted Estimates with GSS Nonresponse Adjustments and Poststratification to Demographic Characteristics, and with the Addition of Poststratification to Volunteering GSS nonresponse adjustment and poststratification to demographic characteristics Adding volunteering Difference p-value n Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 633 22.1% (1.85) 23.7% (2.03) 1.6% <0.05 Very or pretty happy 1,295 89.0% (1.15) 88.4% (1.28) −0.6% 0.17 Life exciting 630 54.6% (2.62) 52.3% (2.74) −2.2% <0.05 Most people try to be helpful 658 49.4% (2.21) 49.4% (2.33) −0.1% 0.95 Most try to take advantage 656 40.9% (2.75) 43.0% (3.02) 2.1% <0.05 Most people can be trusted 660 31.5% (2.69) 29.5% (2.90) −2.0% 0.10 Family income below average 1,284 31.2% (1.67) 31.8% (1.86) 0.6% 0.34 Read newspaper every day 1,298 28.9% (1.92) 28.1% (1.92) −0.7% 0.33 No religion 1,294 21.4% (1.31) 23.3% (1.53) 1.9% <0.05 Donated blood in past year 634 14.3% (2.17) 12.4% (1.90) −1.9% 0.05 Donated to charity in past year 632 73.1% (2.13) 67.6% (2.52) −5.6% <0.05 Support birth control to teens 14 to 16 years old 1,268 57.3% (1.73) 58.7% (1.83) 1.4% 0.06 A same-sex female couple can bring up a child just as well 1,230 46.8% (2.25) 48.2% (2.27) 1.4% 0.07 Oppose capital punishment 1,204 33.9% (1.74) 35.2% (1.92) 1.3% 0.08 Courts are too harsh with criminals 1,182 14.0% (1.13) 13.6% (1.12) −0.4% 0.40 GSS nonresponse adjustment and poststratification to demographic characteristics Adding volunteering Difference p-value n Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 633 22.1% (1.85) 23.7% (2.03) 1.6% <0.05 Very or pretty happy 1,295 89.0% (1.15) 88.4% (1.28) −0.6% 0.17 Life exciting 630 54.6% (2.62) 52.3% (2.74) −2.2% <0.05 Most people try to be helpful 658 49.4% (2.21) 49.4% (2.33) −0.1% 0.95 Most try to take advantage 656 40.9% (2.75) 43.0% (3.02) 2.1% <0.05 Most people can be trusted 660 31.5% (2.69) 29.5% (2.90) −2.0% 0.10 Family income below average 1,284 31.2% (1.67) 31.8% (1.86) 0.6% 0.34 Read newspaper every day 1,298 28.9% (1.92) 28.1% (1.92) −0.7% 0.33 No religion 1,294 21.4% (1.31) 23.3% (1.53) 1.9% <0.05 Donated blood in past year 634 14.3% (2.17) 12.4% (1.90) −1.9% 0.05 Donated to charity in past year 632 73.1% (2.13) 67.6% (2.52) −5.6% <0.05 Support birth control to teens 14 to 16 years old 1,268 57.3% (1.73) 58.7% (1.83) 1.4% 0.06 A same-sex female couple can bring up a child just as well 1,230 46.8% (2.25) 48.2% (2.27) 1.4% 0.07 Oppose capital punishment 1,204 33.9% (1.74) 35.2% (1.92) 1.3% 0.08 Courts are too harsh with criminals 1,182 14.0% (1.13) 13.6% (1.12) −0.4% 0.40 GSS nonresponse adjustment and poststratification to demographic characteristics Adding volunteering Difference p-value n Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 633 22.1% (1.85) 23.7% (2.03) 1.6% <0.05 Very or pretty happy 1,295 89.0% (1.15) 88.4% (1.28) −0.6% 0.17 Life exciting 630 54.6% (2.62) 52.3% (2.74) −2.2% <0.05 Most people try to be helpful 658 49.4% (2.21) 49.4% (2.33) −0.1% 0.95 Most try to take advantage 656 40.9% (2.75) 43.0% (3.02) 2.1% <0.05 Most people can be trusted 660 31.5% (2.69) 29.5% (2.90) −2.0% 0.10 Family income below average 1,284 31.2% (1.67) 31.8% (1.86) 0.6% 0.34 Read newspaper every day 1,298 28.9% (1.92) 28.1% (1.92) −0.7% 0.33 No religion 1,294 21.4% (1.31) 23.3% (1.53) 1.9% <0.05 Donated blood in past year 634 14.3% (2.17) 12.4% (1.90) −1.9% 0.05 Donated to charity in past year 632 73.1% (2.13) 67.6% (2.52) −5.6% <0.05 Support birth control to teens 14 to 16 years old 1,268 57.3% (1.73) 58.7% (1.83) 1.4% 0.06 A same-sex female couple can bring up a child just as well 1,230 46.8% (2.25) 48.2% (2.27) 1.4% 0.07 Oppose capital punishment 1,204 33.9% (1.74) 35.2% (1.92) 1.3% 0.08 Courts are too harsh with criminals 1,182 14.0% (1.13) 13.6% (1.12) −0.4% 0.40 GSS nonresponse adjustment and poststratification to demographic characteristics Adding volunteering Difference p-value n Percent Standard error Percent Standard error Percentage points Rao-Scott chi-square Fair or poor health 633 22.1% (1.85) 23.7% (2.03) 1.6% <0.05 Very or pretty happy 1,295 89.0% (1.15) 88.4% (1.28) −0.6% 0.17 Life exciting 630 54.6% (2.62) 52.3% (2.74) −2.2% <0.05 Most people try to be helpful 658 49.4% (2.21) 49.4% (2.33) −0.1% 0.95 Most try to take advantage 656 40.9% (2.75) 43.0% (3.02) 2.1% <0.05 Most people can be trusted 660 31.5% (2.69) 29.5% (2.90) −2.0% 0.10 Family income below average 1,284 31.2% (1.67) 31.8% (1.86) 0.6% 0.34 Read newspaper every day 1,298 28.9% (1.92) 28.1% (1.92) −0.7% 0.33 No religion 1,294 21.4% (1.31) 23.3% (1.53) 1.9% <0.05 Donated blood in past year 634 14.3% (2.17) 12.4% (1.90) −1.9% 0.05 Donated to charity in past year 632 73.1% (2.13) 67.6% (2.52) −5.6% <0.05 Support birth control to teens 14 to 16 years old 1,268 57.3% (1.73) 58.7% (1.83) 1.4% 0.06 A same-sex female couple can bring up a child just as well 1,230 46.8% (2.25) 48.2% (2.27) 1.4% 0.07 Oppose capital punishment 1,204 33.9% (1.74) 35.2% (1.92) 1.3% 0.08 Courts are too harsh with criminals 1,182 14.0% (1.13) 13.6% (1.12) −0.4% 0.40 REFERENCES AAPOR ( 2015 ). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys ( 8th ed .), Oakbrook Terrace, IL: American Association for Public Opinion Research . Abraham K. , Helms S. , Presser S. ( 2009 ). “How Social Processes Distort Measurement: The Impact of Survey Nonresponse on Estimates of Volunteer Work in the United States,” American Journal of Sociology , 114 , 1129 – 1165 . Google Scholar Crossref Search ADS Aldworth J. , Barnett S. , Cribb D. , Davis T. R. , Foster M. S. , Harter R. , Packer L. E. , Spagnola K. , Bose J. , Hughes A. ( 2013 ). 2013 National Survey on Drug Use and Health: Statistical Inference Report. Rockville, Maryland: RTI International. Amaya A. , Presser S. ( 2017 ). “ Nonresponse Bias for Univariate and Multivariate Estimates of Social Activities and Roles ,” Public Opinion Quarterly , 81 , 1 – 36 . Belli R. F. , Traugott M. W. , Young M. , McGonagle K. A. ( 1999 ). “ Reducing Vote Overreporting in Surveys: Social Desirability, Memory Failure, and Source Monitoring ,” Public Opinion Quarterly , 63 , 90 – 108 . Google Scholar Crossref Search ADS Berent M. K. , Krosnick J. A. , Lupia A. ( 2016 ). “ Measuring Voter Registration and Turnout in Surveys Do Official Government Records Yield More Accurate Assessments ?” Public Opinion Quarterly , 80 , 597 – 621 . Google Scholar Crossref Search ADS Biemer P. P. , Peytchev A. ( 2013 ). “ Using Geocoded Census Data for Nonresponse Bias Correction: An Assessment ,” Journal of Survey Statistics and Methodology , 1 , 24 – 44 . Google Scholar Crossref Search ADS Brick J. M. , Kalton G. ( 1996 ). “ Handling Missing Data in Survey Research ,” Statistical Methods in Medical Research , 5 , 215 – 238 . Google Scholar Crossref Search ADS PubMed Cantor D. , Coa K. , Crystal-Mansour S. , Davis T. , Dipko S. , Sigman R. ( 2009 ). Health Information National Trends Survey 2007 (HINTS 2007) Final Report . Bethesda, MD : National Cancer Institute . Cohn N. ( 2014 ). “The Upshot: Why Polls Tend to Undercount Democrats,” The New York Times, http://www.nytimes.com/2014/10/30/upshot/why-polls-tend-to-undercount-democrats.html? abt=0002&abg=1. Couper M. P. , Singer E. , Kulka R. A. ( 1998 ). “ Participation in the 1990 Decennial Census: Politics, Privacy, Pressures ,” American Politics Quarterly , 26 , 59 – 80 . Google Scholar Crossref Search ADS Curtin R. , Presser S. , Singer E. ( 2000 ). “ The Effects of Response Rate Changes on the Index of Consumer Sentiment ,” Public Opinion Quarterly , 64 , 413 – 428 . Google Scholar Crossref Search ADS PubMed Dever J. A. , Valliant R. ( 2016 ). “ General Regression Estimation Adjusted for Undercoverage and Estimated Control Totals ,” Journal of Survey Statistics and Methodology , 4, 289 – 318 . Google Scholar Crossref Search ADS Dever J. A. , Valliant R. L. ( 2010 ). “ A Comparison of Variance Estimators for Poststratification to Estimated Control Totals ,” Survey Methodology , 36 , 45 – 56 . Dillman D. A. ( 1978 ). Mail and Telephone Surveys . New York : Wiley . DiSogra C. , Cobb C. , Chan E. , Dennis J. M. ( 2011 ). “Calibrating Non-Probability Internet Samples with Probability Samples Using Early Adopter Characteristics.” Joint Statistical Meetings of the American Statistical Association. Fahimi M. , Barlas F. M. , Thomas R. K. , Buttermore N. ( 2015 ). “ Scientific Surveys Based on Incomplete Sampling Frames and High Rates of Nonresponse ,” Survey Practice , 8 . Federal Election Commission ( 2009 ). Federal Elections 2008: Election Results for the US President, the US Senate and the US House of Representatives. Federal Election Commission. Goyder J. ( 1987 ). The Silent Minority: Nonrespondents on Sample Surveys . Cambridge : Polity Press . Groves R. M. ( 2006 ). “ Nonresponse Rates and Nonresponse Bias in Household Surveys ,” Public Opinion Quarterly , 70 , 646 – 675 . Google Scholar Crossref Search ADS Groves R. M. , Couper M. P. ( 1998 ). Nonresponse in Household Interview Surveys . New York : Wiley . Groves R. M. , Peytcheva E. ( 2008 ). “ The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis ,” Public Opinion Quarterly , 72 , 167 – 189 . Google Scholar Crossref Search ADS Groves R. M. , Singer E. , Corning A. ( 2000 ). “ Leverage-Saliency Theory of Survey Participation—Description and an Illustration ,” Public Opinion Quarterly , 64 , 299 – 308 . Google Scholar Crossref Search ADS PubMed Jackman S. , Spahn B. T. ( 2014 ). “Why Does the American National Election Study Overestimate Voter Turnout?” Available at http://polmeth.wustl.edu/media/Paper/polmeth2014.pdf. Katz J. N. , Katz G. ( 2010 ). “ Correcting for Survey Misreports Using Auxiliary Information with an Application to Estimating Turnout ,” American Journal of Political Science , 54 , 815 – 835 . Google Scholar Crossref Search ADS Keeter S. , Kennedy C. , Dimock M. , Best J. , Craighill P. ( 2006 ). “ Gauging the impact of growing nonresponse on estimates from a national RDD telephone survey ,” Public Opinion Quarterly , 70 , 759 – 779 . Google Scholar Crossref Search ADS Kish L. ( 1965 ). Survey Sampling . New York : John Wiley . Knack S. ( 1992 ). “ Civic Norms, Social Sanctions, and Voter Turnout ,” Rationality and Society , 4 , 133 – 156 . Google Scholar Crossref Search ADS Kreuter F. , Olson K. , Wagner J. , Yan T. , Ezzati-Rice T. M. , Casas-Cordero C. , Lemay M. , Peytchev A. , Groves R. M. , Raghunathan T. E. ( 2010 ). “ Using Proxy Measures and Other Correlates of Survey Outcomes to Adjust for Nonresponse: Examples from Multiple Surveys ,” Journal of the Royal Statistical Society, Series A (Statistics in Society) , 173 , 389 . Google Scholar Crossref Search ADS Little R. J. A. ( 1986 ). “ Survey Nonresponse Adjustments for Estimates of Means ,” International Statistical Review , 54 , 139 – 157 . Google Scholar Crossref Search ADS Little R. J. , Vartivarian S. ( 2005 ). “ Does Weighting for Nonresponse Increase the Variance of Survey Means ?” Survey Methodology , 31 , 161 – 168 . Merkle D. M. , Edelman M. ( 2009 ). “ An Experiment on Improving Response Rates and Its Unintended Impact on Survey Error ,” Survey Practice (March) , 2 . Mulry M. H. ( 2014 ). “Measuring Undercounts for Hard-to-Survey Groups.” in Hard-to-Survey Populations , eds. T. Roger , E. Brad , P. J. Timothy , B. Nancy , M. W Kirk . Cambridge : Cambridge University Press . Pasek J. , Jang S. M. , Cobb C. L. , Dennis J. M. , Disogra C. ( 2014 ). “ Can Marketing Data Aid Survey Research? Examining Accuracy and Completeness in Consumer-File Data ,” Public Opinion Quarterly , 78 , 889 – 916 . Google Scholar Crossref Search ADS Pew Research Center ( 2016 ). “Evaluating Online Nonprobability Surveys.” Available at http://www.pewresearch.org/files/2016/04/Nonprobability-report-May-2016-FINAL.pdf. Peytchev A. ( 2012 ). “ Multiple Imputation for Unit Nonresponse and Measurement Error ,” Public Opinion Quarterly , 76 , 214 – 237 . Google Scholar Crossref Search ADS Peytchev A. , Raghunathan T. ( 2013 ). “Evaluation and Use of Commercial Data for Nonresponse Bias Adjustment.” American Association for Public Opinion Research annual conference, Boston, MA, May 16. Peytchev A. , Ridenhour J. , Krotki K. ( 2010 ). “ Differences Between RDD Telephone and ABS Mail Survey Design: Coverage, Unit Nonresponse, and Measurement Error ,” Journal of Health Communication: International Perspectives , 15 , 117 – 134 . Google Scholar Crossref Search ADS Peytcheva E. , Groves R. M. ( 2009 ). “ Using Variation in Response Rates of Demographic Subgroups as Evidence of Nonresponse Bias in Survey Estimates ,” Journal of Official Statistics , 25 , 193 – 201 . Rässler S. , Schnell R. ( 2004 ). Multiple imputation for unit-nonresponse versus weighting including a comparison with a nonresponse follow-up study. Diskussionspapiere//Friedrich-Alexander-Universität Erlangen-Nürnberg, Lehrstuhl für Statistik und Ökonometrie, Nurenberg, Germany. Rendall M. , Tomassini C. , Elliot D. J. ( 2003 ). “ Estimation of Annual International Migration from the Labor Force Surveys of the United Kingdom and the Continental European Union ,” Statistical Journal of the United Nations ECE , 20 , 219 – 234 . Schonlau M. , van Soest A. , Kapteyn A. ( 2007 ). “ Are ‘Webographic’ or attitudinal questions useful for adjusting estimates from Web surveys using propensity scoring ?” Survey Research Methods , 1 , 9 . Shin M. E. , McCarthy W. J. ( 2013 ). “ The Association between County Political Inclination and Obesity: Results from the 2012 Presidential Election in the United States ,” Preventive Medicine , 57 , 721 – 724 . Google Scholar Crossref Search ADS PubMed Subramanian S. V. , Huijts T. , Perkins J. M. ( 2009 ). “ Association between Political Ideology and Health in Europe ,” European Journal of Public Health , 19 , 455 – 457 . Google Scholar Crossref Search ADS PubMed Tourangeau R. , Groves R. M. , Redline C. D. ( 2010 ). “ Sensitive Topics and Reluctant Respondents: Demonstrating a Link between Nonresponse Bias and Measurement Error ,” Public Opinion Quarterly , 74 , 413 – 432 . Google Scholar Crossref Search ADS Tourangeau R. , Rips L. J. , Rasinski K. A. ( 2000 ). The Psychology of Survey Response . Cambridge, U.K.; New York : Cambridge University Press . U.S. Census Bureau ( 2012 ). Voting and Registration in the Election of November 2008. US Census Bureau. Voogt R. J. J. ( 2005 ). “ An Alternative Approach to Correcting Response and Nonresponse Bias in Election Research ,” Acta Politica , 40 , 94 – 116 . Google Scholar Crossref Search ADS Voogt R. J. J. ( 2004 ). “Nonresponse Bias, Response Bias and Stimulus Effects in Election Research.” PhD Faculty of Social Sciences, University of Amsterdam. Wagner J. , Valliant R. , Hubbard F. , Jiang L. ( 2014 ). “ Level-of-Effort Paradata and Nonresponse Adjustment Models for a National Face-to-Face Survey ,” Journal of Survey Statistics and Methodology , 2 , 410 – 432 . Google Scholar Crossref Search ADS West B. T. ( 2013 ). “ An examination of the quality and utility of interviewer observations in the National Survey of Family Growth ,” Journal of the Royal Statistical Society: Series A (Statistics in Society) , 176 , 211 – 225 . Google Scholar Crossref Search ADS West B. T. , Wagner J. , Hubbard F. , Gu H. ( 2015 ). “ The Utility of Alternative Commercial Data Sources for Survey Operations and Estimation: Evidence from the National Survey of Family Growth ,” Journal of Survey Statistics and Methodology , 3 , 240 – 264 . Google Scholar Crossref Search ADS Westat ( 2015 ). Health Information National Trends Survey 4 (HINTS 4) Cycle 4: Methodology Report . Bethesda, MD : National Cancer Institute . Wright G. C. ( 1993 ). “ Errors in Measuring Vote Choice in the National Election Studies, 1952-88 ,” American Journal of Political Science , 37 , 291 – 316 . Google Scholar Crossref Search ADS Zeglovits E. , Kritzinger S. ( 2014 ). “ New Attempts to Reduce Overreporting of Voter Turnout and Their Effects ,” International Journal of Public Opinion Research , 26 , 224 – 234 . Google Scholar Crossref Search ADS © The Author 2018. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)

Journal

Journal of Survey Statistics and MethodologyOxford University Press

Published: Dec 1, 2018

References

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off