Using Bonus Monetary Incentives to Encourage Web Response in Mixed-Mode Household Surveys

Using Bonus Monetary Incentives to Encourage Web Response in Mixed-Mode Household Surveys Abstract This paper builds on prior research on alternative protocols for conducting mixed-mode surveys by mail where respondents are asked to complete either a web questionnaire or a paper questionnaire. We present the results from a 4 × 2 factorial experiment in which the first factor, the survey protocol, varied the modes initially offered (either web alone or both web and paper) as well as the order in which they are offered in subsequent follow-up contacts (either concurrently or sequentially). This factor was crossed with a second factor offering various promised incentive amounts and response options (a $5 prepaid incentive with a promise of either $10 or $20 for response). Our research sought to evaluate the differences between protocols/incentive treatments relative to their costs, response rates, respondent choice of web versus paper response, and consistency with benchmark estimates in other large-scale surveys. Specifically, we introduce a new protocol referred to as choice+ (i.e., choice plus) that presents both web and paper response options concurrently at the initial request, but promises a bonus ($10) incentive for choosing the web option. Results presented are from the 2015 Residential Energy Consumption Survey National Pilot study—an experimental component of the main Residential Energy Consumption Survey sponsored by the U.S. Energy Information Administration. 1. INTRODUCTION This paper investigates a number of alternative data collection strategies for conducting dual mode surveys using a mail delivery platform where one mode uses a paper questionnaire and the other uses a web questionnaire. Four data collection protocols were tested in combination with two incentive treatments in a 4×2 factorial design. One novel development in this research is an experimental protocol referred to as choice plus (or choice+) which presents both web and paper response options concurrently at the initial request, but promises a bonus incentive if the response is by web. In this way, choice+ attempts to “push” respondents to the web rather than merely offering it as an option. The four data collection protocols tested in this study are web only, web/paper, choice, and choice+, which are defined as follows: Web only—only the web response option is offered for all survey response invitations. Web/paper —the web response option is offered at the first invitation while both the web and paper response options are offered in all subsequent invitations. Choice—response by either paper or web questionnaire is requested by each survey response invitation. Choice+—response by either paper or web questionnaire is requested by each survey response invitation. However, an additional $10 bonus incentive is promised if the respondent responds by web rather than by paper. The four protocols were crossed with two incentive treatments: a low-incentive treatment that included a $5 prepaid incentive in the first questionnaire mailing with the promise of an additional $10 for responding to the survey request and a high-incentive treatment that also included a $5 prepaid incentive provided in the first questionnaire mailing but promised an additional $20 for responding to the survey request. This research seeks to evaluate the differences among the eight protocol/incentive combinations (treatments) relative to their costs, response rates, consistency with benchmark estimates in other large-scale surveys, and the respondent’s choice of web versus paper response. The results presented are from the 2015 Residential Energy Consumption Survey National Pilot study—an experimental component of the main Residential Energy Consumption Survey sponsored by the U.S. Energy Information Administration (EIA). The next section reviews the relevant literature on web/paper mixed mode surveys, particularly regarding research on incentivizing response by the web versus paper. This section motivates the design of the experiment and provides some of the theory underpinning the decisions respondents make in choosing to respond to a survey request and their choice of mode. 2. BACKGROUND 2.1 Literature Review Since the 1990s, the Internet has become ubiquitous in many countries, particularly the United States. Perrin and Duggan (2015) report that 84 percent of American adults use the Internet, and the U.S. Census estimates that 81 percent of U.S. households now have Internet access (U.S. Census Bureau 2016). The Internet is a vital tool for communication, entertainment, and business, and, in recent years, an important vehicle for collecting data in federal surveys. The web is an attractive data collection mode in terms of cost and timeliness of results. Indeed, web survey methodology has matured at a time when traditional survey modes such as telephone and face-to-face interviewing are encountering greater cost and quality challenges. The advantages and disadvantages of the web have been documented extensively in the survey literature (see, for example, Couper 2008, 2011; de Leeuw and Hox 2011; Stern, Bilgen and Dillman 2014). Advantages include relative cost efficiency, the absence of interviewer effects, and ability to use sophisticated questionnaire instrumentation such as complex branching, hypertext instructions, drop-down menus, and visual aids. Disadvantages include the lack of a comprehensive frame, poor household coverage, limitations for selecting random person samples, and response rates that are low—roughly on par with telephone surveys (Harter, Battaglia, Buskirk, Dillman, English, et al. 2016; Link et al. 2008). Although uses of the web as the sole mode for conducting surveys are quite limited, there are significant advantages of including the web as a component of a mixed mode survey (de Leeuw and Hox 2011; Dillman and Edwards 2016). Paper questionnaires seem to be an ideal companion for web surveys because they too can operate on a mail-delivery platform. The paper mode mitigates the problem of Internet household undercoverage and offers some of the same benefits of self-administration as web questionnaires, although with some added costs for printing, shipping, and data processing. That respondents prefer paper response over web response (ceteris paribus) has been well-documented in the literature (see, for example, meta-analyses conducted by Shih and Fan 2008; Manfreda, Lozar, Berzelak, Haas, and Vehovar 2008). However, given the aforementioned advantages of web over paper, web response is the preferred mode of survey researchers. Pushing respondents to the web in spite of their preference is an important challenge and research area in the recent literature (Zimmer, Biemer, Kott, and Berry 2015; Dillman, Smyth, and Christian 2014). To maximize the benefits of paper and web dual mode designs, effective protocols are essential (Griffin, Fischer, and Morgan 2001). For these reasons, there has been considerable research on mixed mode surveys that combine paper and web to increase response rates and maximize the benefits offered by both. Invariably, the studies have allowed response by either paper or web, while strongly encouraging the latter mode through the use of incentives and the order in which the two modes are offered. For example, Messer and Dillman (2011) tested various dual mode designs that offered only one option in the initial contact before presenting a choice of either paper or web response in the nonresponse follow-up contacts. They found that offering web followed subsequently by paper resulted in much higher web response than paper followed by web. In fact, for the latter sequence, very few respondents used web. In addition, paper/web provided more demographically representative samples than the web/paper sequential protocol as the paper stage of the web/paper protocol seemed to attract a very different type of respondent than the web stage. The paper respondents tended to be older, less educated, and less likely to have Internet in the household than the earlier web respondents. Of particular interest in our research is the comparison of concurrent or “mode choice” designs with the aforementioned sequential and single mode alternatives. Medway and Fulton (2012) compared response rates for paper-alone protocols versus those that offered a choice of web and paper modes to respondents at the initial survey request. They found that paper alone resulted in higher response rates than the concurrent design (see, also, the review by Dykema, Jones, Piche’, and Stevenson 2013). The authors cite Schwartz’s (2004) explanation that suggests that, weighing the benefits and burdens of each mode, respondents may perceive either mode option as less appealing than if they were offered the mode by itself. Likewise, Millar and Dillman (2011), in experiments on undergraduate students where both email addresses and Internet access were available, compared response rates for single mode, concurrent dual mode, and sequential dual mode designs. They found that a sequential web/paper design resulted in response rates that were similar to paper only and that both rates were higher than those for the concurrent design. They also determined that a prepaid cash incentive enclosed with the initial request followed by an immediate email contact tended to increase the proportion choosing web. However, these findings are somewhat inconsistent with Matthews, Davis, Tancreto, Zelenak, and Ruiter (2012) who, in a test on the American Community Survey, found no significant differences in response rates between the mode choice and the concurrent designs. However, the sequential design was effective in encouraging more web response than the concurrent design, which is consistent with Millar and Dillman (2011). Regarding incentives, small, prepaid monetary incentives have consistently been shown to increase mail survey response rates (Church 1993; Jobber, Saunders, and Mitchell 2004; Singer and Ye 2013). Sample members seem to be more willing to respond to the survey request because compliance is seen as the repayment of a gift or favor. More recently, Avdeyeva and Matland (2013) in a general population survey in Russia found that an unconditional (or pre-) incentive of 50 rubles produced higher response rates than a conditional (or promised) incentive of 300 rubles. However, offering both types of incentives produced the highest response rate of any of the alternatives they tested. Finally, the novel approach tested in this paper is to incentivize response by web in a concurrent design by offering a bonus cash incentive for web response in the initial survey request. For the Baccalaureate and Beyond Survey, Wine, Cominole, Heuer and Riccobono (2006) found that offering an incentive for completing the web instrument in a concurrent design, referred to as choice+ in this paper, increased web completion rates by 4 percentage points (p < 0.05) over the simple choice (i.e., no bonus incentive) design. Subsequently, this result was replicated to an even greater effect for the National Post-secondary Student Aid Study. Although the populations for these studies are college students and not general populations, they do suggest some potential for the choice+ design to increase the proportion of respondents choosing web. Thus, the recent literature suggests that the use of both unconditional and conditional incentives will result in higher response rates than either used alone. Further, if maximizing web response is also desired, the use of web/paper is superior to paper/web. However, what has never been tested in general population surveys (to our knowledge) is using both the web and paper modes concurrently with a bonus incentive to push respondents to the web (i.e., choice+). That protocol is tested in the current research along with the web/paper sequential protocol and two other protocols (viz., choice and web only) that address one or more of the objectives described in the next section. 2.2 Study Objectives Our interest in finding the most suitable mixed mode design was driven by the goals of the 2015 Residential Energy Consumption Survey (RECS) Household Pilots. The RECS is a periodic survey of households that collects energy characteristics, energy usage patterns, and household demographics. The survey has been conducted since 1978, with the most recent completed iteration in 2009. Traditionally, the RECS is an in-person survey using computer-assisted personal interviewing (CAPI) for data collection. The foundation of the multi-stage, area probability sampling design relies on the USPS Computerized Delivery Sequence File (CDSF), but is supplemented by labor-intensive listing and rostering of housing units in some areas. EIA conducted a series of three pilot tests to determine the feasibility, cost-effectiveness, time efficiency, and response validity of the RECS Household Survey using a mixture of web and paper questionnaires delivered by mail. The first pilot test was conducted in coordination with the Joint Program in Survey Methodology (JPSM) at the University of Maryland. This test of an RECS-like household energy questionnaire was managed by staff and students in the JPSM practicum course. Web and paper instruments were offered to a nationwide sample of households using small, monetary incentives as tokens of appreciation. This project was not branded as an official EIA survey. Building on lessons learned from the JPSM study, EIA collaborated with IMG-Crown Energy Services and RTI International on the second pilot test, an EIA-branded RECS data collection conducted in five U.S. cities. The design for this “RECS Cities Pilot” varied the length of the questionnaire (i.e., minutes to complete) as well as the mode of response (paper versus web) in a factorial design. Most sampled households were offered a web invitation first, then paper during nonresponse follow-up. A subsample of households was offered paper first based on a low Internet access propensity score (Zimmer et al. 2015). The third test of self-administered modes was the RECS National Pilot, which was conducted from October 2015 to February 2016. This final test was intended to inform EIA of the best survey design for the future RECS program—a design that will produce precise and accurate estimates of household energy use for the target population. The National Pilot web and mail instruments were designed to be completed in 30 minutes, a design choice determined by the Cities Pilot length experiment. The target population for the National Pilot is all housing units (HUs) occupied as primary HUs in the United States. Vacant homes, seasonal HUs, and group quarters, such as dormitories, nursing homes, prisons, and military barracks, are excluded from the study. However, HUs on military installations are included. A total of 9,650 housing units were included in the National Pilot sample. The National Pilot tests the key design components that are under consideration and need to be considered for future self-administered RECS Household Surveys. Specifically, we sought to achieve the following four objectives: maximize the overall response rate as well as the response rates for key housing unit domains; achieve a balanced (selection weighted1) distribution of respondents based on agreement with benchmark distributions from the American Community Survey (ACS); maximize the respondents’ use of the web questionnaire response rather than paper response to reduce the costs for printing questionnaires, return postage, and the labor associated with mail receipting, data entry, and data review, and eliminate certain types of item nonresponse, out-of-range responses, branching errors, and inconsistent responses; and further reduce costs by encouraging response (by either web or paper modes) earlier in the data collection period, which would avoid repeated mailings and their associated costs. In addition to testing a single mode (web) design to concurrent (or choice) and sequential (web first then paper) designs, we were interested to see if we could effectively push respondents to complete the web survey by offering an additional promised incentive for completion by web rather than paper when two modes were offered concurrently—the so-called choice+ protocol. The details of the design and experiments follow in the next section. 3. EXPERIMENTAL DESIGN As previously noted, data collection for the National Pilot took place from October 2015 to February 2016. The sample consisted of an equal probability (epsem) address-based sample of 9,650 addresses selected from 200 primary sampling units in a two-stage cluster design. The sampling frame was essentially all residential addresses on the USPS Computerized Delivery Sequence File. To simplify data collection, drop-point addresses were excluded from the frame; however, because they account for only 1 percent of all addresses nationally, their exclusion should not have appreciable effects on the results. A number of quality indicators were continuously monitored via RTI’s Adaptive Total Design system (see Murphy, Biemer, and Berry in press). 3.1 General Protocol for All Treatments All sample members were sent a $5 prepaid cash incentive coupled with an additional promised cash incentive that varied by treatment as described in the next section. Consistent with the standard practice, a multiple-mailing contact strategy was used to maximize response rates, with an appropriate amount of time between mailings to allow sample members to complete their questionnaires. Here are the timeline and mailings that were used for all sample members. 3.1.1 Phase 1 of data collection Day 1: a postcard was mailed to each sample member notifying them that a survey invitation letter is on its way to them that would include $5 cash. Day 4: the first invitation letter was mailed with the cash incentive. Depending on the treatment condition, the invitation letter included both the survey URL with login instructions and a paper questionnaire. Other letters contained only the URL. In some cases, sample members who received both the URL and the paper questionnaire were offered a promised incentive of an additional $10 for completing the web questionnaire. These variations are summarized in table 1. Table 1. Sample Size Allocation to Treatments Protocol Incentive ($5 prepaid) Total + $10 (promised) + $20 (promised) Web only 1,207 1,206 2,413 Web/paper 1,206 1,206 2,412 Choice 1,207 1,206 2,413 Choice+ 1,206 1,206 2,412 Total 4,826 4,824 9,650 Protocol Incentive ($5 prepaid) Total + $10 (promised) + $20 (promised) Web only 1,207 1,206 2,413 Web/paper 1,206 1,206 2,412 Choice 1,207 1,206 2,413 Choice+ 1,206 1,206 2,412 Total 4,826 4,824 9,650 Table 1. Sample Size Allocation to Treatments Protocol Incentive ($5 prepaid) Total + $10 (promised) + $20 (promised) Web only 1,207 1,206 2,413 Web/paper 1,206 1,206 2,412 Choice 1,207 1,206 2,413 Choice+ 1,206 1,206 2,412 Total 4,826 4,824 9,650 Protocol Incentive ($5 prepaid) Total + $10 (promised) + $20 (promised) Web only 1,207 1,206 2,413 Web/paper 1,206 1,206 2,412 Choice 1,207 1,206 2,413 Choice+ 1,206 1,206 2,412 Total 4,826 4,824 9,650 Day 9: all sample members receive a bifold postcard reminder with the URL and web survey ID inside the fold for privacy. Day 24: a second invitation letter with an enclosed study brochure was sent via UPS mail innovations (UPS packaging, USPS delivery) repeating the offer in the first invitation but with a stronger plea to participate. The brochure described the study, answered common questions, and provided information on sponsorship and endorsements of the survey. For one protocol (web/paper described below), a paper questionnaire was enclosed for the first time as an alternative participation mode. Day 29: a second bifold reminder postcard similar to the first reminder. Day 49: the third and final letter for phase 1 included an invitation to participate and was sent to all nonrespondents via UPS with the “driver release” option. 3.2.2 Phase 2 of data collection Day 84: phase 2 extended nonresponse follow-up (NRFU) was initiated in which all respondents were mailed an abbreviated (two-page) paper questionnaire with a final appeal to participate to this extent in the RECS Pilot. The next section describes the experimental variations that were embedded in the general strategy to test two factors: protocol design for phase 1 and promised incentive amounts for phase 1 and phase 2. 3.2 Experimental Treatments 3.2.1 Phase 1 of data collection The phase 1 sample of 9,650 addresses were allocated approximately equally to eight treatments. These treatments, which are shown in table 1 with their sample allocations, were formed by crossing two factors—protocols with four levels and incentives with two levels—in a 4×2 factorial design. The equal sample allocation provided roughly equal power for all treatment comparisons. The low-incentive treatment included a $5 prepaid incentive provided in the first questionnaire mailing and a promise of an additional $10 for responding to the survey request. The high incentive treatment included a $5 prepaid incentive provided in the first questionnaire mailing and a promise of an additional $20 for responding to the survey request. As described in section 1, the four treatments were web only, web/paper, choice, and choice+. Table 2 summarizes the eight treatments tested in phase 1 of the survey. Table 2. Phase 1 Protocol and Incentive Treatments Protocol Promised incentive Response options offered 1st invitation (1st class mail) 2nd invitation (1st class mail) 3rd invitation (USPS driver release) Web only $10 Web only option $5 prepaid enclosed $10 promised Web only option Repeat $10 promised Web only option Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web only option Repeat $20 promised Web only option Repeat $20 promised Web/paper $10 Web only option $5 prepaid enclosed $10 promised Web or paper options Repeat $10 promised Web or paper options Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice $10 Web and paper options $5 prepaid enclosed $10 promised Web and paper options Repeat $10 promised Web and paper options Repeat $10 promised $20 Web and paper options $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice+ $10 (paper) or $20 (web) Web and paper options $5 prepaid $10 promised (paper) $20 promised (web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) $20 (paper) or $30 (web) Web and paper options $5 prepaid enclosed $20 promised (paper) $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Protocol Promised incentive Response options offered 1st invitation (1st class mail) 2nd invitation (1st class mail) 3rd invitation (USPS driver release) Web only $10 Web only option $5 prepaid enclosed $10 promised Web only option Repeat $10 promised Web only option Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web only option Repeat $20 promised Web only option Repeat $20 promised Web/paper $10 Web only option $5 prepaid enclosed $10 promised Web or paper options Repeat $10 promised Web or paper options Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice $10 Web and paper options $5 prepaid enclosed $10 promised Web and paper options Repeat $10 promised Web and paper options Repeat $10 promised $20 Web and paper options $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice+ $10 (paper) or $20 (web) Web and paper options $5 prepaid $10 promised (paper) $20 promised (web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) $20 (paper) or $30 (web) Web and paper options $5 prepaid enclosed $20 promised (paper) $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Table 2. Phase 1 Protocol and Incentive Treatments Protocol Promised incentive Response options offered 1st invitation (1st class mail) 2nd invitation (1st class mail) 3rd invitation (USPS driver release) Web only $10 Web only option $5 prepaid enclosed $10 promised Web only option Repeat $10 promised Web only option Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web only option Repeat $20 promised Web only option Repeat $20 promised Web/paper $10 Web only option $5 prepaid enclosed $10 promised Web or paper options Repeat $10 promised Web or paper options Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice $10 Web and paper options $5 prepaid enclosed $10 promised Web and paper options Repeat $10 promised Web and paper options Repeat $10 promised $20 Web and paper options $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice+ $10 (paper) or $20 (web) Web and paper options $5 prepaid $10 promised (paper) $20 promised (web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) $20 (paper) or $30 (web) Web and paper options $5 prepaid enclosed $20 promised (paper) $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Protocol Promised incentive Response options offered 1st invitation (1st class mail) 2nd invitation (1st class mail) 3rd invitation (USPS driver release) Web only $10 Web only option $5 prepaid enclosed $10 promised Web only option Repeat $10 promised Web only option Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web only option Repeat $20 promised Web only option Repeat $20 promised Web/paper $10 Web only option $5 prepaid enclosed $10 promised Web or paper options Repeat $10 promised Web or paper options Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice $10 Web and paper options $5 prepaid enclosed $10 promised Web and paper options Repeat $10 promised Web and paper options Repeat $10 promised $20 Web and paper options $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice+ $10 (paper) or $20 (web) Web and paper options $5 prepaid $10 promised (paper) $20 promised (web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) $20 (paper) or $30 (web) Web and paper options $5 prepaid enclosed $20 promised (paper) $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) 3.2.2 Phase 2 of data collection In addition to the phase 1 experiment, a follow-up survey with an abbreviated paper version (phase 2) was sent to the phase 1 nonrespondents. In phase 2, half of the nonrespondents (referred to as the xNRFU+ group) received an additional $10 payment for completing the abbreviated questionnaire, and the other half (referred to as the xNRFU group) received no additional payment. The sample sizes allocated to these two phase 2 treatments are shown in table 3. Table 3. Sample Size Allocation for Extended Nonresponse Follow-up Treatments NRFU (2-page paper questionnaire) No. xNRFU: phase 1 promised incentive, no additional incentive 2,935 xNRFU+: phase 1 promised incentive + $10 extra promised incentive 2,926 Total 5,861 NRFU (2-page paper questionnaire) No. xNRFU: phase 1 promised incentive, no additional incentive 2,935 xNRFU+: phase 1 promised incentive + $10 extra promised incentive 2,926 Total 5,861 Table 3. Sample Size Allocation for Extended Nonresponse Follow-up Treatments NRFU (2-page paper questionnaire) No. xNRFU: phase 1 promised incentive, no additional incentive 2,935 xNRFU+: phase 1 promised incentive + $10 extra promised incentive 2,926 Total 5,861 NRFU (2-page paper questionnaire) No. xNRFU: phase 1 promised incentive, no additional incentive 2,935 xNRFU+: phase 1 promised incentive + $10 extra promised incentive 2,926 Total 5,861 While the focus of this paper is on the phase 1 results, we will include a few of the main results from the phase 2 experiment primarily for completeness. However, a paper reporting these and other results is forthcoming. 4. RESULTS As a means of assessing quality in near real-time, daily submission rates were tracked for each protocol. Submission rates will differ from response rates because the former include responding units that are ultimately deemed ineligible based upon their responses or are otherwise excluded, for example, as result of excessive item nonresponse. Such exclusions were minimal in the Pilot. Additionally, response rates are often weighted for their selection probabilities, but, again, because of the epsem design, there were no important differences between weighted and unweighted response rates. Therefore, the final submission rates in figure 1 are very close to the final response rates. Figure 1. View largeDownload slide Submission Rates for the Four Protocols and Phase 2 by Data Collection Day. Figure 1. View largeDownload slide Submission Rates for the Four Protocols and Phase 2 by Data Collection Day. Figure 1 shows the daily submission rates by protocol (combining the two incentive treatments) and by day of data collection. The choice+ protocol maintained the highest submission rate throughout the collection period. The web only protocol performed on par with the other protocols during the initial contact periods, but lagged behind in the middle and late stages of collection. Note that the effect of sending out the paper questionnaires at day 24 for the web/paper protocol is evidenced by the slight upward bump in the submission rate around day 30. The final phase 1 (unweighted) response rates by treatment (protocol and incentive) are shown in table 4. Response rates were computed using the AAPOR RR3 where the parameter “e” was calculated using the latent class analysis approach of Biemer, Murphy, and Kott (2016). Table 4. Response Rates by Treatment—Phase 1 Protocol Incentives ($5 prepaid) Mean + $10 (promised) + $20 (promised) Web only 31.99 38.86 35.43 Web/paper 38.59 42.28 40.44 Choice 38.80 43.41 41.12 Choice+ 42.78 44.99 43.89 Mean 38.03 42.38 40.21 Protocol Incentives ($5 prepaid) Mean + $10 (promised) + $20 (promised) Web only 31.99 38.86 35.43 Web/paper 38.59 42.28 40.44 Choice 38.80 43.41 41.12 Choice+ 42.78 44.99 43.89 Mean 38.03 42.38 40.21 Table 4. Response Rates by Treatment—Phase 1 Protocol Incentives ($5 prepaid) Mean + $10 (promised) + $20 (promised) Web only 31.99 38.86 35.43 Web/paper 38.59 42.28 40.44 Choice 38.80 43.41 41.12 Choice+ 42.78 44.99 43.89 Mean 38.03 42.38 40.21 Protocol Incentives ($5 prepaid) Mean + $10 (promised) + $20 (promised) Web only 31.99 38.86 35.43 Web/paper 38.59 42.28 40.44 Choice 38.80 43.41 41.12 Choice+ 42.78 44.99 43.89 Mean 38.03 42.38 40.21 The web only and web/paper response rates in table 4 are consistent with findings in the recent literature. For example, over a number of studies in several states, Dillman, Smyth, and Christian (2014) found that response rates for web/paper ranged from 31 percent to 55 percent with a mean response rate of 44 percent (p. 430). Surveys that included incentives were typically in the higher range compared with those that did not. National surveys using a single mode web protocol are uncommon in the literature. However, Couper (2008) reported that single mode web surveys of the general population typically will deliver response rates in a fairly wide range depending upon the sponsor, incentive amount, and survey topic. The response rates in table 4 for the web only mode are perhaps higher than average but are nonetheless consistent with his findings. Tukey’s honest significant difference (HSD) test was performed to test the significance of response rate differences for all possible pairs. This conservative test ensures that the actual type I error rates are close to their nominal levels for comparing all 28 pairs of treatments. The results are summarized in table 5, which shows the differences for all pairs and their statistical significance. The HSD critical values were 6.2 and 7.2 for α = 0.05 (indicated in the table by a single asterisk) and α = 0.01 (indicated by a double asterisk), respectively. Table 5. Difference in Phase 1 Response Rates Web/paper/low Choice/low Choice+/low Web only/high Web/paper/high Choice/high Choice+/high Web only/low −6.60* −6.81* −10.79** −6.87* −10.29** −11.42** −13.00** Web/paper/low −0.21 −4.19 −0.27 −3.69 −4.82 −6.4* Choice/low −3.98 −0.06 −3.48 −4.61 −6.19 Choice+/low 3.92 0.5 −0.63 −2.21 Web only/high −3.42 −4.55 −6.13 Web/paper/high −1.13 −2.71 Choice/high −1.58 Web/paper/low Choice/low Choice+/low Web only/high Web/paper/high Choice/high Choice+/high Web only/low −6.60* −6.81* −10.79** −6.87* −10.29** −11.42** −13.00** Web/paper/low −0.21 −4.19 −0.27 −3.69 −4.82 −6.4* Choice/low −3.98 −0.06 −3.48 −4.61 −6.19 Choice+/low 3.92 0.5 −0.63 −2.21 Web only/high −3.42 −4.55 −6.13 Web/paper/high −1.13 −2.71 Choice/high −1.58 Note.— Entry is the difference between the row treatment response rate and the column treatment response rate. * HSD is significant at α = 0.05. ** HSD is significant at α = 0.01. Table 5. Difference in Phase 1 Response Rates Web/paper/low Choice/low Choice+/low Web only/high Web/paper/high Choice/high Choice+/high Web only/low −6.60* −6.81* −10.79** −6.87* −10.29** −11.42** −13.00** Web/paper/low −0.21 −4.19 −0.27 −3.69 −4.82 −6.4* Choice/low −3.98 −0.06 −3.48 −4.61 −6.19 Choice+/low 3.92 0.5 −0.63 −2.21 Web only/high −3.42 −4.55 −6.13 Web/paper/high −1.13 −2.71 Choice/high −1.58 Web/paper/low Choice/low Choice+/low Web only/high Web/paper/high Choice/high Choice+/high Web only/low −6.60* −6.81* −10.79** −6.87* −10.29** −11.42** −13.00** Web/paper/low −0.21 −4.19 −0.27 −3.69 −4.82 −6.4* Choice/low −3.98 −0.06 −3.48 −4.61 −6.19 Choice+/low 3.92 0.5 −0.63 −2.21 Web only/high −3.42 −4.55 −6.13 Web/paper/high −1.13 −2.71 Choice/high −1.58 Note.— Entry is the difference between the row treatment response rate and the column treatment response rate. * HSD is significant at α = 0.05. ** HSD is significant at α = 0.01. As shown in table 4, the choice+ protocol achieved the highest response rate overall and the highest response rate at both incentive levels. Choice+ was also the only protocol with a response rate higher than 40 percent at the low incentive level. The web only protocol response rates were the lowest at each incentive level. We did not find statistically significant differences between the web/paper and choice response rates, and overall the choice response was actually higher (41 percent) than web/paper (40 percent). This finding runs counter to the notion that a concurrent choice will reduce response rates, but is consistent with findings from tests in another recent large-scale, general population, government-sponsored survey (Matthews et al. 2012). When comparing response rates of each of the eight treatments against one another, as shown in table 5, we found that the web only/low treatment was significantly lower than the other seven treatments. The web/paper/low treatment was also significantly lower than the choice+/high treatment. The choice+/low response rate was also greater than two of the high-incentive groups. Future research might experiment with different incentive levels to determine if the difference between choice+ and other protocols is even greater at lower incentive levels (say, $5). The response rate was 4.4 percentage points higher for the higher incentive than that of the lower incentive. The effect was greatest for the web only protocol, where the difference was almost 7 percentage points and significantly greater than the other protocols. The effect of the higher promised incentive amount for the three other protocols did not differ significantly, and the differences averaged about 3.5 percentage points. However, further analysis reveals that the higher incentive substantially and significantly increased response rates for multifamily households for these protocols, although it is not clear why. As widely reported in the literature, when respondents are given a choice of responding by paper or web, they overwhelmingly choose paper. Our experiment confirmed this result as shown in figure 2. More than 72 percent of the sample opted for paper under the choice protocol. However, this effect is reversed for the choice+ protocol, with about 64 percent choosing to respond by web. Note that the rate of web response for the choice+ protocol is nearly the same as the corresponding rate for the web/paper protocol, but at a higher overall response rate. Figure 2. View largeDownload slide Mode of Response by Treatment. Figure 2. View largeDownload slide Mode of Response by Treatment. We examined how submission mode varied by age, urbanicity, and income and found some striking differences. As expected, submissions by web were significantly greater among the younger age groups (age 18–44 years), higher-income households (at least $60,000), and urban households. To compare total costs separately by treatment, we totaled up all relevant fixed and variable costs of materials, labor, printing, incentives, postage for the mailing the questionnaires, and follow-up contacts, as well as the costs of keying2 the paper and electronic questionnaires. Table 6 shows these results as a ratio of the total cost to the average total costs across all eight treatments. As expected, the web only protocol was the least expensive option, with only about 92 percent of the average cost. The total costs for the web/paper protocol were also below average. The choice+ protocol was only 2 percent above the average and about 4 percent higher than web/paper. However, for the low-incentive treatment, the costs for these two protocols only differ by 2 percent. The most expensive treatment was choice using the higher incentive. Table 6. Relative Cost per Completed Interview by Treatment Protocol Incentives ($5 prepaid) Average + $10 (promised) + $20 (promised) Web only 0.93 0.92 0.92 Web/paper 0.95 1.00 0.98 Choice 1.04 1.09 1.07 Choice+ 0.97 1.07 1.02 Average 0.98 1.02 1.00 Protocol Incentives ($5 prepaid) Average + $10 (promised) + $20 (promised) Web only 0.93 0.92 0.92 Web/paper 0.95 1.00 0.98 Choice 1.04 1.09 1.07 Choice+ 0.97 1.07 1.02 Average 0.98 1.02 1.00 Table 6. Relative Cost per Completed Interview by Treatment Protocol Incentives ($5 prepaid) Average + $10 (promised) + $20 (promised) Web only 0.93 0.92 0.92 Web/paper 0.95 1.00 0.98 Choice 1.04 1.09 1.07 Choice+ 0.97 1.07 1.02 Average 0.98 1.02 1.00 Protocol Incentives ($5 prepaid) Average + $10 (promised) + $20 (promised) Web only 0.93 0.92 0.92 Web/paper 0.95 1.00 0.98 Choice 1.04 1.09 1.07 Choice+ 0.97 1.07 1.02 Average 0.98 1.02 1.00 Clerical editing costs were not included in the cost comparisons because most of these activities were conducted by the government project sponsor (EIA), not RTI. However, a greater percentage of paper questionnaires and items within paper questionnaires failed edits compared to web submissions. Thus, protocols resulting in more paper questionnaire submissions (for example, the choice protocol) would incur disproportionately more costs, staff burden, and quality issues due to editing than would protocols with higher web submissions. The higher incentive added 5 percent to the web/paper and choice relative costs and 10 percent to the choice+ relative cost. Surprisingly, there is a very small difference in relative costs between the two incentive treatments for the web only protocol. This is owed to both the higher response rate and more timely response using the higher incentive that reduced the number of follow-up mailings required for the protocol. It should be noted that mailing costs constitute a higher percentage of total costs for web only than for other protocols that also incur substantial keying costs. Finally, to complete the phase 1 analysis, we consider the representativity of the responding sample as measured by comparing the distributions of National Pilot responses with the corresponding weighted 2014 ACS national distributions for 12 variables that are common to both surveys. For this analysis, we treat the ACS data as the gold standard; thus, differences between the National Pilot and the ACS are interpreted as indications of bias in the National Pilot. Because the variables considered in this analysis were categorical, we chose the dissimilarity index (d) as the indicator of unrepresentativeness, where d=0.5∑k|pNP,k−pACS,k| and where pNP,k and pACS,k are the National Pilot and ACS proportions in the kth category, respectively. This measure can be interpreted as the proportion of observations that would need to change categories in the National Pilot to achieve perfect agreement with the ACS. In that sense, 1-d may be regarded as the agreement rate between the two surveys. A value of d≤0.10 is usually regarded as “good” agreement, and d≤0.05 is regarded as “very good” agreement. The main objective of the representativity analysis is to determine which of the four test protocols best represents the target population in the responding sample prior to postsurvey adjustments. A protocol producing a small value of d for some variable, say V, is said to be “representative” with respect to V—that is, the protocol generates a respondent sample that is close to the population distribution for V. Likewise, a protocol producing a large value of d generates a respondent sample that is less representative with respect to V. To the extent that V is related to other substantive variables in the survey, d can be taken as an indicator of the risk of nonresponse bias in the substantive variables. Note that large values of d for a protocol do not necessarily suggest that the protocol will produce biased estimates because the respondent sample will ultimately be adjusted for nonresponse and coverage, which that will presumably eliminate the differences between the National Pilot and the ACS that led to the large values of d. However, the dissimilarity analysis does suggest the extent to which postsurvey weighting adjustments must be relied upon to mitigate the bias risks. In addition, postsurvey weighting adjustments are not always successful for reducing bias and may increase the standard errors of estimates because of increased weight variation. Thus, protocols with smaller dissimilarity indexes are preferable. Finally, it is important to note that even a protocol that produces perfectly representative samples may not generate zero values of d for at least three reasons. First, the 2014 ACS is not an ideal gold standard because the National Pilot was conducted more than a year later at the end of 2015. Changes in the target population in the intervening period will increase d. Second, both the National Pilot and the ACS are subject to sampling error, which has not been taken into account in the analysis. Third, there are minor differences in the question wordings, the definition of an eligible housing unit, and target populations for the two surveys. As an example of the latter, as previously noted, the National Pilot sample excluded drop points. Nevertheless, we believe the analysis is useful for revealing differences in representativity among the four protocols. Figure 3 shows the values of d expressed as a percentage for the following 12 variables: Figure 3. View largeDownload slide Dissimilarity Indexes for Selected Questionnaire Items Using the 2014 ACS as the Gold Standard. Proportion of cases that would need to switch categories for perfect agreement with the ACS—10 to 5 percent dissimilarity is considered to be “good” agreement; 5 percent or less is considered “very good.” Figure 3. View largeDownload slide Dissimilarity Indexes for Selected Questionnaire Items Using the 2014 ACS as the Gold Standard. Proportion of cases that would need to switch categories for perfect agreement with the ACS—10 to 5 percent dissimilarity is considered to be “good” agreement; 5 percent or less is considered “very good.” Respondent Hispanic (Hisp)—respondent is Hispanic or Latino Own home (Own)—housing unit owned by respondent or someone in household—not rented for money or occupied without payment of rent Refrigerator (Fridge)—at least one refrigerator is used in the housing unit Internet access (Access)—Internet access available in the housing unit Race (Race)—respondent’s race: white only, black only, American Indian/Alaskan Native only, Asian only, or other (includes two or more races) Year made (Year)—year housing unit was built: before 1950, 1950–1959, 1960–1969, 1970–1979, 1980–1989, 1990–1999, 2000–2009, 2010–2015 Household income (Income)—annual gross household income for the last year: less than $20,000; $20,000–39,999; $40,000–59,999; $60,000–79,999; $80,000–99,999; $100,00–119,999; $120,000–139,999; $140,000 or more Main heating fuel (Fuel)—of those who heat home, main space heating fuel: electricity, natural gas from underground pipes, propane (bottled gas), fuel oil, wood, and other Household size (HHSize)—number of household members including respondent: 1, 2, 3, 4 or more Age3 (Age)—age of respondent: <30, 30–44, 45–59, 60–74, 75+ Bedrooms (Bdrms)—number of bedrooms: 0, 1, 2, 3, 4, 5 or more Housing unit type (HUType)—type of housing unit: mobile home, single-family house detached from any other house, single-family house attached to one or more other houses (e.g., duplex, row house, townhome), apartment in a building with two to four units, apartment in a building with five or more units Figure 3 shows higher dissimilarity indices for Own, Year, HHSize, and HUType. Further analysis indicates possible underrepresentation for renters, older housing units, single-person households, and apartments, which is generally consistent with the literature (see, for example, Dillman, Smyth, and Christian 2014). Note that the largest value of d in the figure occurs for Access under the web only protocol and is about 13 percent. This is not surprising because persons who do not have Internet access have a lower propensity to respond via the Internet and would, thus, be underrepresented by web only. As shown in the last column of table 8, the average value of d across all 12 variables ranges between 4.77 percent (for choice+/high incentive) to 6.36 percent (web only/low incentive). It should be noted that d drops to 5.30 percent for web only/low if the variable Access is not considered. Thus, the dissimilarity indexes are in the “good” to “very good” range for all treatment conditions. With regard to the comparison between web/paper and choice+, the latter protocol produces eight (out of 12) values of d larger than 5 percent while the former protocol produces only six. However, web/paper has higher values of d; for example, three variables exceed 7 for web/paper while only two exceed 7 for choice+. Table 8. Summary of Key Metrics by Treatment for Phase 1 Protocol Response rate (%) Proportion of respondents completing by web (%) Relative cost Dissimilarity Relative to ACS (average d) $10 promised incentive  Web only 31.99 100.00 0.93 6.36  Web/paper 38.59 64.01 0.95 5.07  Choice 38.80 27.56 1.04 5.04  Choice+ 42.78 63.92 0.97 5.84 $20 promised incentive  Web only 38.86 100.00 0.92 5.61  Web/paper 42.28 70.99 1.00 4.93  Choice 43.41 27.42 1.09 5.13  Choice+ 44.99 64.50 1.07 4.77 Protocol Response rate (%) Proportion of respondents completing by web (%) Relative cost Dissimilarity Relative to ACS (average d) $10 promised incentive  Web only 31.99 100.00 0.93 6.36  Web/paper 38.59 64.01 0.95 5.07  Choice 38.80 27.56 1.04 5.04  Choice+ 42.78 63.92 0.97 5.84 $20 promised incentive  Web only 38.86 100.00 0.92 5.61  Web/paper 42.28 70.99 1.00 4.93  Choice 43.41 27.42 1.09 5.13  Choice+ 44.99 64.50 1.07 4.77 Table 8. Summary of Key Metrics by Treatment for Phase 1 Protocol Response rate (%) Proportion of respondents completing by web (%) Relative cost Dissimilarity Relative to ACS (average d) $10 promised incentive  Web only 31.99 100.00 0.93 6.36  Web/paper 38.59 64.01 0.95 5.07  Choice 38.80 27.56 1.04 5.04  Choice+ 42.78 63.92 0.97 5.84 $20 promised incentive  Web only 38.86 100.00 0.92 5.61  Web/paper 42.28 70.99 1.00 4.93  Choice 43.41 27.42 1.09 5.13  Choice+ 44.99 64.50 1.07 4.77 Protocol Response rate (%) Proportion of respondents completing by web (%) Relative cost Dissimilarity Relative to ACS (average d) $10 promised incentive  Web only 31.99 100.00 0.93 6.36  Web/paper 38.59 64.01 0.95 5.07  Choice 38.80 27.56 1.04 5.04  Choice+ 42.78 63.92 0.97 5.84 $20 promised incentive  Web only 38.86 100.00 0.92 5.61  Web/paper 42.28 70.99 1.00 4.93  Choice 43.41 27.42 1.09 5.13  Choice+ 44.99 64.50 1.07 4.77 Finally, as noted in section 3.2.2, the main data collection (referred to as phase 1) was followed by an NRFU phase (referred to as phase 2). Here we provide a few results and lessons learned from phase 2. As shown in table 7, approximately 25 percent of the phase 1 nonrespondents responded in the phase 2 xNRFU. The extra $10 promised incentive added about 5 points to this response rate, which was highly significant. Table 7. Response Rates by Treatment—Phase 2 xNRFU (2 page paper questionnaire) No extra incentive (control) 22.39 $10 extra promised incentive 27.58*** Mean 25.08 xNRFU (2 page paper questionnaire) No extra incentive (control) 22.39 $10 extra promised incentive 27.58*** Mean 25.08 *** Highly significantly different from control (p < 0.001). Table 7. Response Rates by Treatment—Phase 2 xNRFU (2 page paper questionnaire) No extra incentive (control) 22.39 $10 extra promised incentive 27.58*** Mean 25.08 xNRFU (2 page paper questionnaire) No extra incentive (control) 22.39 $10 extra promised incentive 27.58*** Mean 25.08 *** Highly significantly different from control (p < 0.001). As mentioned in section 3.2.2, a more complete examination of the phase 2 results will be considered in a subsequent paper. That paper will examine the characteristics of phase 2 respondents compared with phase 1 respondents; effects of treatments on NRFU response; effects of NRFU response on sample representativity; effects of NRFU weighting on nonresponse bias and variance; and costs versus benefits of xNRFU+ versus xNRFU. 5. SUMMARY AND CONCLUSIONS Declining response rates and reduced cost efficiency of face-to-face data collection motivated this research to examine several self-administered data collection approaches as alternatives to a traditional CAPI survey. The National Pilot study employed eight contact strategies across a sample of 9,650 U.S. households to identify the protocol that provided the best combination of adequate response rate, high proportion of respondents using the web survey option, low cost, and good respondent sample representativity. Two of those strategies, the choice+ treatments, used a bonus incentive to motivate web response over paper. With regard to response rate, the web only treatment was uniformly lower than all other protocols. Choice+ attained the highest response rate overall and for both incentive treatments. In addition, choice+ maintained the highest submission rate for essentially the entire data collection period. The extra $10 promised incentive increased the response rate overall by 4.4 percentage points, with the greatest increase for web only (6.9 percentage points). The proportion of respondents responding by web for the choice protocol was about 27.5 percent. This was increased to more than 64 percent for the choice+ protocol. This rate of web returns was about equal to the web/paper protocol for the low incentive and about 5 points lower than web/paper for the high incentive. While web is the preferred self-administered mode due to cost and data quality benefits, many studies must still use paper modes to reach those who prefer that mode or who cannot respond via the Internet. Regarding cost, across both incentive levels, the cost of choice+ was only 2 percent above the average cost and about 4 percent higher than web/paper. Surprisingly, choice+ cost 5 percent less than choice despite the additional cost of a $10 promised bonus incentive for web submission. The higher incentive treatment increased choice+ cost by 10 percent over the low-incentive treatment. The web only treatments were the least expensive treatments in this study due to the lack of costs for printing, receipting, and keying paper forms. Another surprising result was the decrease in web only cost when the incentive was increased $10; the relative cost per completed interview was 0.93 in the low group vs 0.92 in the high group. This result may be relevant for any survey being conducted solely via the web. In addition, the choice+ costs were not greater than the choice group despite the additional $10 incentive cost for web respondents. Even more importantly, but not surprising, all eight treatment costs were significantly lower than previous collections for the RECS via CAPI interviews. Taken together, as in table 8, one can review the results of the strategies along multiple dimensions to make a decision about which to consider “best.” For our purposes, the choice+ protocol, a concurrent dual mode method with extra incentive for the web response, was best suited to address the range of considerations. Choice+ had the highest response rate at both low and high incentive levels and maintained the highest submission rate throughout data collection. This is an indication that this strategy may be more robust than the traditional concurrent option (i.e., choice) or the sequential web/paper method. The choice+ group was also far better than the choice protocol and at least as good as the web/paper at eliciting response via the web. It would be interesting to see if this web response effect holds true at lower incentive levels (for example, $5 prepaid + $5 promised or $0 prepaid + $10 promised). While self-administration and the contact strategies described in this paper provide promising results, it should be noted that the response rates for all eight treatments were much lower than the 80 percent response traditionally attained in prior rounds of RECS that used CAPI methods. This raises concerns about nonresponse bias and coverage error. Although we were not able to compare National Pilot data in parallel with a contemporaneous CAPI study, comparisons with ACS benchmarks provide some evidence of respondent representativity. Those results were generally positive. Most comparisons with 12 ACS variables showed “good” or “very good” representativity across the four protocols, although comparisons indicate that respondents are less likely to be renters and apartment dwellers. Representativity for web/paper and choice+ was quite comparable. However, all treatments demonstrated good representativity overall, with an average d in the range 4.77 to 6.39 (table 8). As one might expect, households without Internet access in the web only protocol were the most underrepresented population. Finally, although it was not the focus of this paper, the phase 2 results presented here suggest that nonresponse follow-up using an abbreviated questionnaire can add substantially to the information provided by single phase data collection. It remains to evaluate the value added by these data, which is the topic of a subsequent paper. In all, these results suggest that, in spite of their lower response rates, mixed mode mail surveys can serve as a viable platform for traditionally CAPI data collections such as the RECS. All treatments, with the exception of the web only/low group (32 percent), achieved a response rate between 38 percent and 45 percent, attained acceptable respondent representativity, and delivered low-cost, completed questionnaires. Further cost efficiencies accrue for mail surveys because, unlike CAPI surveys, mail survey samples need not be clustered to minimize travel costs. These cost savings can allow for program flexibilities that are limited by the cost, precision, and timing constraints of CAPI surveys. Our analysis indicates that the choice+ self-administered strategy in particular is an effective method for simultaneously increasing both web and overall survey response. Further research might replicate the choice+ experiment and conduct additional experiments with variations on these strategies (i.e., lower choice+ incentives for web response). Additional considerations, such as differential measurement error by strategy, are also important to examine and weight against the other factors in selecting the most appropriate data collection protocol. Footnotes 1 Selection weighted means that the units are weighted by the inverse of their selection probabilities only; i.e., no weighting adjustments for nonresponse and/or noncoverage are applied. 2 In some surveys, using scanned forms (for example, via TeleForm) could save data entry costs and thus change the cost comparisons. For this Pilot study, scanned forms were considered but were deemed cost-ineffective. 3 The ACS reports the age of the “householder” or the “head of the household” whereas the National Pilot reports the age of the person responding to the survey. This would explain at least part of the dissimiliarity between the National Pilot and the ACS for Age. References Avdeyeva O. A. , Matland R. E. ( 2013 ), “An Experimental Test of Mail Surveys as a Tool for Social Inquiry in Russia,” International Journal of Public Opinion Research , 25 2 , 173 – 194 . Biemer P. , Murphy J. , Kott P. ( 2016 ), “ Estimating mail or web survey eligibility for undeliverable addresses: a latent class analysis approach,” Proceedings of the ASA Survey Methods Research Section and Presented at Joint Statistical Meetings of the ASA, Chicago, IL . pp. 1166 – 1172 . Church A. H. ( 1993 ), “ Estimating the Effect of Incentives on Mail Survey Response Rates: A Meta-analysis ,” Public Opinion Quarterly , 57 , 62 – 79 . Google Scholar CrossRef Search ADS Couper M. P. ( 2008 ), Designing Effective Web Surveys , New York : Cambridge University Press . Google Scholar CrossRef Search ADS Couper M. P. ( 2011 ), “ The Future of Modes of Data Collection ,” Public Opinion Quarterly , 75 , 889 – 908 . Google Scholar CrossRef Search ADS de Leeuw E. , Hox J. ( 2011 ), “Internet Surveys as Part of a Mixed Mode Design,” in Social and Behavioral Research and the Internet , eds. Das M. , Ester P. , Kaczmirek L. , pp. 45 – 76, New York : Taylor and Francis . Dillman D. , Edwards M. ( 2016 ), “Designing a Mixed Mode Survey,” in The Sage Handbook of Survey Methodology , eds. Wolf C. , Joye D. , Smith T.W. , Fu Y. , pp. 255 – 268, London : Sage Reference . Google Scholar CrossRef Search ADS Dillman D. , Smyth J. , Christian L. M. ( 2014 ), Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method ( 4th ed.) , Hoboken, NJ : John Wiley & Sons . Dykema J. , Jones N. , Piche T. , Stevenson J. ( 2013 ), “ Surveying Clinicians by Web: Current Issues in Design and Administration ,” Evaluation and the Health Professions , 36 , 352 – 381 . Google Scholar CrossRef Search ADS PubMed Griffin D. H. , Fischer D. P. , Morgan M. T. ( 2001 ), “Testing an Internet Response Option for the American Community Survey,” paper presented at the Annual Conference of the American Association of Public Opinion Research, Montreal, Quebec, Canada. Harter R. , Battaglia M. P. , Buskirk T. D. , Dillman D. A. , English N. et al. , ( 2016 ), “AAPOR Report: Address-Based Sampling,” Available at http://www.aapor.org/Education-Resources/Reports/Address-based-Sampling.aspx. Jobber D. , Saunders J. , Mitchell V. W. ( 2004 ), “ Prepaid Monetary Incentive Effects on Mail Survey Response ,” Journal of Business Research , 57 , 347 – 350 . Google Scholar CrossRef Search ADS Link M. W. , M. P. Battaglia , M. R. Frankel , L. Osborn , A. H. Mokdad et al. , ( 2008 ), A comparison of address-based sampling (ABS) versus random-digit dialing (RDD) for general population surveys. Public Opinion Quarterly , 72 1 , 6 – 27 . Manfreda K. , Lozar M. B. , Berzelak J. , Haas I. , Vehovar V. ( 2008 ), “ Web Surveys Versus Other Survey Modes: A Meta-Analysis Comparing Response Rates ,” International Journal of Market Research , 50 , 79 – 104 . Google Scholar CrossRef Search ADS Matthews B. , Davis M. , Tancreto J. , Zelenak M. F. , Ruiter M. ( 2012 ), “2011 American Community Survey Internet Tests: Results from Second Test in November 2011,” American Community Survey Research and Evaluation Program, #ACS12-RER-21, May 14, 2012. Medway R. , Fulton J. ( 2012 ), “ When More Gets You Less: A Meta-Analysis of the Effect of Concurrent Web Options on Mail Survey Response Rates, ” Public Opinion Quarterly , 76 , 733 – 746 . Google Scholar CrossRef Search ADS Messer B. L. , Dillman D. A. ( 2011 ), “ Surveying the General Public over the Internet Using Address-Based Sampling and Mail Contact Procedures ,” Public Opinion Quarterly , 75 , 429 – 457 . Google Scholar CrossRef Search ADS Millar M. M. , Dillman D. A. ( 2011 ), “ Improving Response to Web and Mixed-Mode Surveys ,” Public Opinion Quarterly , 75 , 249 – 269 . Google Scholar CrossRef Search ADS Murphy J. , Biemer P. , Berry C. (in press), “ Visualizing Real-Time Metrics Critical to Quality in a Mixed-Mode Self-Administered Survey ,” Journal of Official Statistics . Perrin A. , Duggan M. ( 2015 ), “Americans’ Internet Access: 2000-2015,” Pew Research Center, Available at: http://www.pewinternet.org/files/2015/06/2015-06-26_internet-usage-across-demographics-discover_FINAL.pdf. Schwartz B. ( 2004 ), The Paradox of Choice: Why More Is Less , New York : Harper Perennial . Shih T. H. , X. Fan ( 2008 ), Comparing response rates from web and mail surveys: A meta-analysis. Field methods , 20 3 , 249 – 271 . Singer E. , Ye C. ( 2013 ), “ The Use and Effects of Incentives in Surveys ,” The Annals of the American Academy of Political and Social Science , 645 , 112 – 141 . Google Scholar CrossRef Search ADS Stern M. J. , Bilgen I. , Dillman D. A. ( 2014 ), “ The State of Survey Methodology Challenges, Dilemmaas and New Frontiers in the Era of the Tailored Design ,” Field Methods , 26 , 284 – 301 . Google Scholar CrossRef Search ADS U.S. Census Bureau ( 2016 ), “2015 American Community Survey 1-Year Estimates,” Available at https://www.census.gov/programs-surveys/acs/news/data-releases/2015/release.html. Wine J. , Cominole M. , Heuer R. , Riccobono J. ( 2006 ), “Challenges of Designing and Implementing Multimode Instruments,” paper presented at Second International Conference on Telephone Survey Methodology, Miami, FL. Zimmer S. , Biemer P. , Kott P. , Berry C. ( 2015 ), “Testing a Model-Directed, Mixed Mode Protocol in the RECS Pilot Study,” Proceedings of the 2015 Federal Committee on Statistical Methodology Research Conference, Available at https://fcsm.sites.usa.gov/files/2016/03/G2_Zimmer_2015FCSM.pdf. © The Author 2017. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Survey Statistics and Methodology Oxford University Press

Using Bonus Monetary Incentives to Encourage Web Response in Mixed-Mode Household Surveys

Loading next page...
 
/lp/ou_press/using-bonus-monetary-incentives-to-encourage-web-response-in-mixed-nNBRxWihwQ
Publisher
Oxford University Press
Copyright
© The Author 2017. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please email: journals.permissions@oup.com
ISSN
2325-0984
eISSN
2325-0992
D.O.I.
10.1093/jssam/smx015
Publisher site
See Article on Publisher Site

Abstract

Abstract This paper builds on prior research on alternative protocols for conducting mixed-mode surveys by mail where respondents are asked to complete either a web questionnaire or a paper questionnaire. We present the results from a 4 × 2 factorial experiment in which the first factor, the survey protocol, varied the modes initially offered (either web alone or both web and paper) as well as the order in which they are offered in subsequent follow-up contacts (either concurrently or sequentially). This factor was crossed with a second factor offering various promised incentive amounts and response options (a $5 prepaid incentive with a promise of either $10 or $20 for response). Our research sought to evaluate the differences between protocols/incentive treatments relative to their costs, response rates, respondent choice of web versus paper response, and consistency with benchmark estimates in other large-scale surveys. Specifically, we introduce a new protocol referred to as choice+ (i.e., choice plus) that presents both web and paper response options concurrently at the initial request, but promises a bonus ($10) incentive for choosing the web option. Results presented are from the 2015 Residential Energy Consumption Survey National Pilot study—an experimental component of the main Residential Energy Consumption Survey sponsored by the U.S. Energy Information Administration. 1. INTRODUCTION This paper investigates a number of alternative data collection strategies for conducting dual mode surveys using a mail delivery platform where one mode uses a paper questionnaire and the other uses a web questionnaire. Four data collection protocols were tested in combination with two incentive treatments in a 4×2 factorial design. One novel development in this research is an experimental protocol referred to as choice plus (or choice+) which presents both web and paper response options concurrently at the initial request, but promises a bonus incentive if the response is by web. In this way, choice+ attempts to “push” respondents to the web rather than merely offering it as an option. The four data collection protocols tested in this study are web only, web/paper, choice, and choice+, which are defined as follows: Web only—only the web response option is offered for all survey response invitations. Web/paper —the web response option is offered at the first invitation while both the web and paper response options are offered in all subsequent invitations. Choice—response by either paper or web questionnaire is requested by each survey response invitation. Choice+—response by either paper or web questionnaire is requested by each survey response invitation. However, an additional $10 bonus incentive is promised if the respondent responds by web rather than by paper. The four protocols were crossed with two incentive treatments: a low-incentive treatment that included a $5 prepaid incentive in the first questionnaire mailing with the promise of an additional $10 for responding to the survey request and a high-incentive treatment that also included a $5 prepaid incentive provided in the first questionnaire mailing but promised an additional $20 for responding to the survey request. This research seeks to evaluate the differences among the eight protocol/incentive combinations (treatments) relative to their costs, response rates, consistency with benchmark estimates in other large-scale surveys, and the respondent’s choice of web versus paper response. The results presented are from the 2015 Residential Energy Consumption Survey National Pilot study—an experimental component of the main Residential Energy Consumption Survey sponsored by the U.S. Energy Information Administration (EIA). The next section reviews the relevant literature on web/paper mixed mode surveys, particularly regarding research on incentivizing response by the web versus paper. This section motivates the design of the experiment and provides some of the theory underpinning the decisions respondents make in choosing to respond to a survey request and their choice of mode. 2. BACKGROUND 2.1 Literature Review Since the 1990s, the Internet has become ubiquitous in many countries, particularly the United States. Perrin and Duggan (2015) report that 84 percent of American adults use the Internet, and the U.S. Census estimates that 81 percent of U.S. households now have Internet access (U.S. Census Bureau 2016). The Internet is a vital tool for communication, entertainment, and business, and, in recent years, an important vehicle for collecting data in federal surveys. The web is an attractive data collection mode in terms of cost and timeliness of results. Indeed, web survey methodology has matured at a time when traditional survey modes such as telephone and face-to-face interviewing are encountering greater cost and quality challenges. The advantages and disadvantages of the web have been documented extensively in the survey literature (see, for example, Couper 2008, 2011; de Leeuw and Hox 2011; Stern, Bilgen and Dillman 2014). Advantages include relative cost efficiency, the absence of interviewer effects, and ability to use sophisticated questionnaire instrumentation such as complex branching, hypertext instructions, drop-down menus, and visual aids. Disadvantages include the lack of a comprehensive frame, poor household coverage, limitations for selecting random person samples, and response rates that are low—roughly on par with telephone surveys (Harter, Battaglia, Buskirk, Dillman, English, et al. 2016; Link et al. 2008). Although uses of the web as the sole mode for conducting surveys are quite limited, there are significant advantages of including the web as a component of a mixed mode survey (de Leeuw and Hox 2011; Dillman and Edwards 2016). Paper questionnaires seem to be an ideal companion for web surveys because they too can operate on a mail-delivery platform. The paper mode mitigates the problem of Internet household undercoverage and offers some of the same benefits of self-administration as web questionnaires, although with some added costs for printing, shipping, and data processing. That respondents prefer paper response over web response (ceteris paribus) has been well-documented in the literature (see, for example, meta-analyses conducted by Shih and Fan 2008; Manfreda, Lozar, Berzelak, Haas, and Vehovar 2008). However, given the aforementioned advantages of web over paper, web response is the preferred mode of survey researchers. Pushing respondents to the web in spite of their preference is an important challenge and research area in the recent literature (Zimmer, Biemer, Kott, and Berry 2015; Dillman, Smyth, and Christian 2014). To maximize the benefits of paper and web dual mode designs, effective protocols are essential (Griffin, Fischer, and Morgan 2001). For these reasons, there has been considerable research on mixed mode surveys that combine paper and web to increase response rates and maximize the benefits offered by both. Invariably, the studies have allowed response by either paper or web, while strongly encouraging the latter mode through the use of incentives and the order in which the two modes are offered. For example, Messer and Dillman (2011) tested various dual mode designs that offered only one option in the initial contact before presenting a choice of either paper or web response in the nonresponse follow-up contacts. They found that offering web followed subsequently by paper resulted in much higher web response than paper followed by web. In fact, for the latter sequence, very few respondents used web. In addition, paper/web provided more demographically representative samples than the web/paper sequential protocol as the paper stage of the web/paper protocol seemed to attract a very different type of respondent than the web stage. The paper respondents tended to be older, less educated, and less likely to have Internet in the household than the earlier web respondents. Of particular interest in our research is the comparison of concurrent or “mode choice” designs with the aforementioned sequential and single mode alternatives. Medway and Fulton (2012) compared response rates for paper-alone protocols versus those that offered a choice of web and paper modes to respondents at the initial survey request. They found that paper alone resulted in higher response rates than the concurrent design (see, also, the review by Dykema, Jones, Piche’, and Stevenson 2013). The authors cite Schwartz’s (2004) explanation that suggests that, weighing the benefits and burdens of each mode, respondents may perceive either mode option as less appealing than if they were offered the mode by itself. Likewise, Millar and Dillman (2011), in experiments on undergraduate students where both email addresses and Internet access were available, compared response rates for single mode, concurrent dual mode, and sequential dual mode designs. They found that a sequential web/paper design resulted in response rates that were similar to paper only and that both rates were higher than those for the concurrent design. They also determined that a prepaid cash incentive enclosed with the initial request followed by an immediate email contact tended to increase the proportion choosing web. However, these findings are somewhat inconsistent with Matthews, Davis, Tancreto, Zelenak, and Ruiter (2012) who, in a test on the American Community Survey, found no significant differences in response rates between the mode choice and the concurrent designs. However, the sequential design was effective in encouraging more web response than the concurrent design, which is consistent with Millar and Dillman (2011). Regarding incentives, small, prepaid monetary incentives have consistently been shown to increase mail survey response rates (Church 1993; Jobber, Saunders, and Mitchell 2004; Singer and Ye 2013). Sample members seem to be more willing to respond to the survey request because compliance is seen as the repayment of a gift or favor. More recently, Avdeyeva and Matland (2013) in a general population survey in Russia found that an unconditional (or pre-) incentive of 50 rubles produced higher response rates than a conditional (or promised) incentive of 300 rubles. However, offering both types of incentives produced the highest response rate of any of the alternatives they tested. Finally, the novel approach tested in this paper is to incentivize response by web in a concurrent design by offering a bonus cash incentive for web response in the initial survey request. For the Baccalaureate and Beyond Survey, Wine, Cominole, Heuer and Riccobono (2006) found that offering an incentive for completing the web instrument in a concurrent design, referred to as choice+ in this paper, increased web completion rates by 4 percentage points (p < 0.05) over the simple choice (i.e., no bonus incentive) design. Subsequently, this result was replicated to an even greater effect for the National Post-secondary Student Aid Study. Although the populations for these studies are college students and not general populations, they do suggest some potential for the choice+ design to increase the proportion of respondents choosing web. Thus, the recent literature suggests that the use of both unconditional and conditional incentives will result in higher response rates than either used alone. Further, if maximizing web response is also desired, the use of web/paper is superior to paper/web. However, what has never been tested in general population surveys (to our knowledge) is using both the web and paper modes concurrently with a bonus incentive to push respondents to the web (i.e., choice+). That protocol is tested in the current research along with the web/paper sequential protocol and two other protocols (viz., choice and web only) that address one or more of the objectives described in the next section. 2.2 Study Objectives Our interest in finding the most suitable mixed mode design was driven by the goals of the 2015 Residential Energy Consumption Survey (RECS) Household Pilots. The RECS is a periodic survey of households that collects energy characteristics, energy usage patterns, and household demographics. The survey has been conducted since 1978, with the most recent completed iteration in 2009. Traditionally, the RECS is an in-person survey using computer-assisted personal interviewing (CAPI) for data collection. The foundation of the multi-stage, area probability sampling design relies on the USPS Computerized Delivery Sequence File (CDSF), but is supplemented by labor-intensive listing and rostering of housing units in some areas. EIA conducted a series of three pilot tests to determine the feasibility, cost-effectiveness, time efficiency, and response validity of the RECS Household Survey using a mixture of web and paper questionnaires delivered by mail. The first pilot test was conducted in coordination with the Joint Program in Survey Methodology (JPSM) at the University of Maryland. This test of an RECS-like household energy questionnaire was managed by staff and students in the JPSM practicum course. Web and paper instruments were offered to a nationwide sample of households using small, monetary incentives as tokens of appreciation. This project was not branded as an official EIA survey. Building on lessons learned from the JPSM study, EIA collaborated with IMG-Crown Energy Services and RTI International on the second pilot test, an EIA-branded RECS data collection conducted in five U.S. cities. The design for this “RECS Cities Pilot” varied the length of the questionnaire (i.e., minutes to complete) as well as the mode of response (paper versus web) in a factorial design. Most sampled households were offered a web invitation first, then paper during nonresponse follow-up. A subsample of households was offered paper first based on a low Internet access propensity score (Zimmer et al. 2015). The third test of self-administered modes was the RECS National Pilot, which was conducted from October 2015 to February 2016. This final test was intended to inform EIA of the best survey design for the future RECS program—a design that will produce precise and accurate estimates of household energy use for the target population. The National Pilot web and mail instruments were designed to be completed in 30 minutes, a design choice determined by the Cities Pilot length experiment. The target population for the National Pilot is all housing units (HUs) occupied as primary HUs in the United States. Vacant homes, seasonal HUs, and group quarters, such as dormitories, nursing homes, prisons, and military barracks, are excluded from the study. However, HUs on military installations are included. A total of 9,650 housing units were included in the National Pilot sample. The National Pilot tests the key design components that are under consideration and need to be considered for future self-administered RECS Household Surveys. Specifically, we sought to achieve the following four objectives: maximize the overall response rate as well as the response rates for key housing unit domains; achieve a balanced (selection weighted1) distribution of respondents based on agreement with benchmark distributions from the American Community Survey (ACS); maximize the respondents’ use of the web questionnaire response rather than paper response to reduce the costs for printing questionnaires, return postage, and the labor associated with mail receipting, data entry, and data review, and eliminate certain types of item nonresponse, out-of-range responses, branching errors, and inconsistent responses; and further reduce costs by encouraging response (by either web or paper modes) earlier in the data collection period, which would avoid repeated mailings and their associated costs. In addition to testing a single mode (web) design to concurrent (or choice) and sequential (web first then paper) designs, we were interested to see if we could effectively push respondents to complete the web survey by offering an additional promised incentive for completion by web rather than paper when two modes were offered concurrently—the so-called choice+ protocol. The details of the design and experiments follow in the next section. 3. EXPERIMENTAL DESIGN As previously noted, data collection for the National Pilot took place from October 2015 to February 2016. The sample consisted of an equal probability (epsem) address-based sample of 9,650 addresses selected from 200 primary sampling units in a two-stage cluster design. The sampling frame was essentially all residential addresses on the USPS Computerized Delivery Sequence File. To simplify data collection, drop-point addresses were excluded from the frame; however, because they account for only 1 percent of all addresses nationally, their exclusion should not have appreciable effects on the results. A number of quality indicators were continuously monitored via RTI’s Adaptive Total Design system (see Murphy, Biemer, and Berry in press). 3.1 General Protocol for All Treatments All sample members were sent a $5 prepaid cash incentive coupled with an additional promised cash incentive that varied by treatment as described in the next section. Consistent with the standard practice, a multiple-mailing contact strategy was used to maximize response rates, with an appropriate amount of time between mailings to allow sample members to complete their questionnaires. Here are the timeline and mailings that were used for all sample members. 3.1.1 Phase 1 of data collection Day 1: a postcard was mailed to each sample member notifying them that a survey invitation letter is on its way to them that would include $5 cash. Day 4: the first invitation letter was mailed with the cash incentive. Depending on the treatment condition, the invitation letter included both the survey URL with login instructions and a paper questionnaire. Other letters contained only the URL. In some cases, sample members who received both the URL and the paper questionnaire were offered a promised incentive of an additional $10 for completing the web questionnaire. These variations are summarized in table 1. Table 1. Sample Size Allocation to Treatments Protocol Incentive ($5 prepaid) Total + $10 (promised) + $20 (promised) Web only 1,207 1,206 2,413 Web/paper 1,206 1,206 2,412 Choice 1,207 1,206 2,413 Choice+ 1,206 1,206 2,412 Total 4,826 4,824 9,650 Protocol Incentive ($5 prepaid) Total + $10 (promised) + $20 (promised) Web only 1,207 1,206 2,413 Web/paper 1,206 1,206 2,412 Choice 1,207 1,206 2,413 Choice+ 1,206 1,206 2,412 Total 4,826 4,824 9,650 Table 1. Sample Size Allocation to Treatments Protocol Incentive ($5 prepaid) Total + $10 (promised) + $20 (promised) Web only 1,207 1,206 2,413 Web/paper 1,206 1,206 2,412 Choice 1,207 1,206 2,413 Choice+ 1,206 1,206 2,412 Total 4,826 4,824 9,650 Protocol Incentive ($5 prepaid) Total + $10 (promised) + $20 (promised) Web only 1,207 1,206 2,413 Web/paper 1,206 1,206 2,412 Choice 1,207 1,206 2,413 Choice+ 1,206 1,206 2,412 Total 4,826 4,824 9,650 Day 9: all sample members receive a bifold postcard reminder with the URL and web survey ID inside the fold for privacy. Day 24: a second invitation letter with an enclosed study brochure was sent via UPS mail innovations (UPS packaging, USPS delivery) repeating the offer in the first invitation but with a stronger plea to participate. The brochure described the study, answered common questions, and provided information on sponsorship and endorsements of the survey. For one protocol (web/paper described below), a paper questionnaire was enclosed for the first time as an alternative participation mode. Day 29: a second bifold reminder postcard similar to the first reminder. Day 49: the third and final letter for phase 1 included an invitation to participate and was sent to all nonrespondents via UPS with the “driver release” option. 3.2.2 Phase 2 of data collection Day 84: phase 2 extended nonresponse follow-up (NRFU) was initiated in which all respondents were mailed an abbreviated (two-page) paper questionnaire with a final appeal to participate to this extent in the RECS Pilot. The next section describes the experimental variations that were embedded in the general strategy to test two factors: protocol design for phase 1 and promised incentive amounts for phase 1 and phase 2. 3.2 Experimental Treatments 3.2.1 Phase 1 of data collection The phase 1 sample of 9,650 addresses were allocated approximately equally to eight treatments. These treatments, which are shown in table 1 with their sample allocations, were formed by crossing two factors—protocols with four levels and incentives with two levels—in a 4×2 factorial design. The equal sample allocation provided roughly equal power for all treatment comparisons. The low-incentive treatment included a $5 prepaid incentive provided in the first questionnaire mailing and a promise of an additional $10 for responding to the survey request. The high incentive treatment included a $5 prepaid incentive provided in the first questionnaire mailing and a promise of an additional $20 for responding to the survey request. As described in section 1, the four treatments were web only, web/paper, choice, and choice+. Table 2 summarizes the eight treatments tested in phase 1 of the survey. Table 2. Phase 1 Protocol and Incentive Treatments Protocol Promised incentive Response options offered 1st invitation (1st class mail) 2nd invitation (1st class mail) 3rd invitation (USPS driver release) Web only $10 Web only option $5 prepaid enclosed $10 promised Web only option Repeat $10 promised Web only option Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web only option Repeat $20 promised Web only option Repeat $20 promised Web/paper $10 Web only option $5 prepaid enclosed $10 promised Web or paper options Repeat $10 promised Web or paper options Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice $10 Web and paper options $5 prepaid enclosed $10 promised Web and paper options Repeat $10 promised Web and paper options Repeat $10 promised $20 Web and paper options $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice+ $10 (paper) or $20 (web) Web and paper options $5 prepaid $10 promised (paper) $20 promised (web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) $20 (paper) or $30 (web) Web and paper options $5 prepaid enclosed $20 promised (paper) $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Protocol Promised incentive Response options offered 1st invitation (1st class mail) 2nd invitation (1st class mail) 3rd invitation (USPS driver release) Web only $10 Web only option $5 prepaid enclosed $10 promised Web only option Repeat $10 promised Web only option Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web only option Repeat $20 promised Web only option Repeat $20 promised Web/paper $10 Web only option $5 prepaid enclosed $10 promised Web or paper options Repeat $10 promised Web or paper options Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice $10 Web and paper options $5 prepaid enclosed $10 promised Web and paper options Repeat $10 promised Web and paper options Repeat $10 promised $20 Web and paper options $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice+ $10 (paper) or $20 (web) Web and paper options $5 prepaid $10 promised (paper) $20 promised (web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) $20 (paper) or $30 (web) Web and paper options $5 prepaid enclosed $20 promised (paper) $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Table 2. Phase 1 Protocol and Incentive Treatments Protocol Promised incentive Response options offered 1st invitation (1st class mail) 2nd invitation (1st class mail) 3rd invitation (USPS driver release) Web only $10 Web only option $5 prepaid enclosed $10 promised Web only option Repeat $10 promised Web only option Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web only option Repeat $20 promised Web only option Repeat $20 promised Web/paper $10 Web only option $5 prepaid enclosed $10 promised Web or paper options Repeat $10 promised Web or paper options Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice $10 Web and paper options $5 prepaid enclosed $10 promised Web and paper options Repeat $10 promised Web and paper options Repeat $10 promised $20 Web and paper options $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice+ $10 (paper) or $20 (web) Web and paper options $5 prepaid $10 promised (paper) $20 promised (web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) $20 (paper) or $30 (web) Web and paper options $5 prepaid enclosed $20 promised (paper) $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Protocol Promised incentive Response options offered 1st invitation (1st class mail) 2nd invitation (1st class mail) 3rd invitation (USPS driver release) Web only $10 Web only option $5 prepaid enclosed $10 promised Web only option Repeat $10 promised Web only option Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web only option Repeat $20 promised Web only option Repeat $20 promised Web/paper $10 Web only option $5 prepaid enclosed $10 promised Web or paper options Repeat $10 promised Web or paper options Repeat $10 promised $20 Web only option $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice $10 Web and paper options $5 prepaid enclosed $10 promised Web and paper options Repeat $10 promised Web and paper options Repeat $10 promised $20 Web and paper options $5 prepaid enclosed $20 promised Web and paper options Repeat $20 promised Web and paper options Repeat $20 promised Choice+ $10 (paper) or $20 (web) Web and paper options $5 prepaid $10 promised (paper) $20 promised (web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) Web and paper options Repeat $10 promised (paper) Repeat $20 promised web) $20 (paper) or $30 (web) Web and paper options $5 prepaid enclosed $20 promised (paper) $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) Web and paper options Repeat $20 promised (paper) Repeat $30 promised (web) 3.2.2 Phase 2 of data collection In addition to the phase 1 experiment, a follow-up survey with an abbreviated paper version (phase 2) was sent to the phase 1 nonrespondents. In phase 2, half of the nonrespondents (referred to as the xNRFU+ group) received an additional $10 payment for completing the abbreviated questionnaire, and the other half (referred to as the xNRFU group) received no additional payment. The sample sizes allocated to these two phase 2 treatments are shown in table 3. Table 3. Sample Size Allocation for Extended Nonresponse Follow-up Treatments NRFU (2-page paper questionnaire) No. xNRFU: phase 1 promised incentive, no additional incentive 2,935 xNRFU+: phase 1 promised incentive + $10 extra promised incentive 2,926 Total 5,861 NRFU (2-page paper questionnaire) No. xNRFU: phase 1 promised incentive, no additional incentive 2,935 xNRFU+: phase 1 promised incentive + $10 extra promised incentive 2,926 Total 5,861 Table 3. Sample Size Allocation for Extended Nonresponse Follow-up Treatments NRFU (2-page paper questionnaire) No. xNRFU: phase 1 promised incentive, no additional incentive 2,935 xNRFU+: phase 1 promised incentive + $10 extra promised incentive 2,926 Total 5,861 NRFU (2-page paper questionnaire) No. xNRFU: phase 1 promised incentive, no additional incentive 2,935 xNRFU+: phase 1 promised incentive + $10 extra promised incentive 2,926 Total 5,861 While the focus of this paper is on the phase 1 results, we will include a few of the main results from the phase 2 experiment primarily for completeness. However, a paper reporting these and other results is forthcoming. 4. RESULTS As a means of assessing quality in near real-time, daily submission rates were tracked for each protocol. Submission rates will differ from response rates because the former include responding units that are ultimately deemed ineligible based upon their responses or are otherwise excluded, for example, as result of excessive item nonresponse. Such exclusions were minimal in the Pilot. Additionally, response rates are often weighted for their selection probabilities, but, again, because of the epsem design, there were no important differences between weighted and unweighted response rates. Therefore, the final submission rates in figure 1 are very close to the final response rates. Figure 1. View largeDownload slide Submission Rates for the Four Protocols and Phase 2 by Data Collection Day. Figure 1. View largeDownload slide Submission Rates for the Four Protocols and Phase 2 by Data Collection Day. Figure 1 shows the daily submission rates by protocol (combining the two incentive treatments) and by day of data collection. The choice+ protocol maintained the highest submission rate throughout the collection period. The web only protocol performed on par with the other protocols during the initial contact periods, but lagged behind in the middle and late stages of collection. Note that the effect of sending out the paper questionnaires at day 24 for the web/paper protocol is evidenced by the slight upward bump in the submission rate around day 30. The final phase 1 (unweighted) response rates by treatment (protocol and incentive) are shown in table 4. Response rates were computed using the AAPOR RR3 where the parameter “e” was calculated using the latent class analysis approach of Biemer, Murphy, and Kott (2016). Table 4. Response Rates by Treatment—Phase 1 Protocol Incentives ($5 prepaid) Mean + $10 (promised) + $20 (promised) Web only 31.99 38.86 35.43 Web/paper 38.59 42.28 40.44 Choice 38.80 43.41 41.12 Choice+ 42.78 44.99 43.89 Mean 38.03 42.38 40.21 Protocol Incentives ($5 prepaid) Mean + $10 (promised) + $20 (promised) Web only 31.99 38.86 35.43 Web/paper 38.59 42.28 40.44 Choice 38.80 43.41 41.12 Choice+ 42.78 44.99 43.89 Mean 38.03 42.38 40.21 Table 4. Response Rates by Treatment—Phase 1 Protocol Incentives ($5 prepaid) Mean + $10 (promised) + $20 (promised) Web only 31.99 38.86 35.43 Web/paper 38.59 42.28 40.44 Choice 38.80 43.41 41.12 Choice+ 42.78 44.99 43.89 Mean 38.03 42.38 40.21 Protocol Incentives ($5 prepaid) Mean + $10 (promised) + $20 (promised) Web only 31.99 38.86 35.43 Web/paper 38.59 42.28 40.44 Choice 38.80 43.41 41.12 Choice+ 42.78 44.99 43.89 Mean 38.03 42.38 40.21 The web only and web/paper response rates in table 4 are consistent with findings in the recent literature. For example, over a number of studies in several states, Dillman, Smyth, and Christian (2014) found that response rates for web/paper ranged from 31 percent to 55 percent with a mean response rate of 44 percent (p. 430). Surveys that included incentives were typically in the higher range compared with those that did not. National surveys using a single mode web protocol are uncommon in the literature. However, Couper (2008) reported that single mode web surveys of the general population typically will deliver response rates in a fairly wide range depending upon the sponsor, incentive amount, and survey topic. The response rates in table 4 for the web only mode are perhaps higher than average but are nonetheless consistent with his findings. Tukey’s honest significant difference (HSD) test was performed to test the significance of response rate differences for all possible pairs. This conservative test ensures that the actual type I error rates are close to their nominal levels for comparing all 28 pairs of treatments. The results are summarized in table 5, which shows the differences for all pairs and their statistical significance. The HSD critical values were 6.2 and 7.2 for α = 0.05 (indicated in the table by a single asterisk) and α = 0.01 (indicated by a double asterisk), respectively. Table 5. Difference in Phase 1 Response Rates Web/paper/low Choice/low Choice+/low Web only/high Web/paper/high Choice/high Choice+/high Web only/low −6.60* −6.81* −10.79** −6.87* −10.29** −11.42** −13.00** Web/paper/low −0.21 −4.19 −0.27 −3.69 −4.82 −6.4* Choice/low −3.98 −0.06 −3.48 −4.61 −6.19 Choice+/low 3.92 0.5 −0.63 −2.21 Web only/high −3.42 −4.55 −6.13 Web/paper/high −1.13 −2.71 Choice/high −1.58 Web/paper/low Choice/low Choice+/low Web only/high Web/paper/high Choice/high Choice+/high Web only/low −6.60* −6.81* −10.79** −6.87* −10.29** −11.42** −13.00** Web/paper/low −0.21 −4.19 −0.27 −3.69 −4.82 −6.4* Choice/low −3.98 −0.06 −3.48 −4.61 −6.19 Choice+/low 3.92 0.5 −0.63 −2.21 Web only/high −3.42 −4.55 −6.13 Web/paper/high −1.13 −2.71 Choice/high −1.58 Note.— Entry is the difference between the row treatment response rate and the column treatment response rate. * HSD is significant at α = 0.05. ** HSD is significant at α = 0.01. Table 5. Difference in Phase 1 Response Rates Web/paper/low Choice/low Choice+/low Web only/high Web/paper/high Choice/high Choice+/high Web only/low −6.60* −6.81* −10.79** −6.87* −10.29** −11.42** −13.00** Web/paper/low −0.21 −4.19 −0.27 −3.69 −4.82 −6.4* Choice/low −3.98 −0.06 −3.48 −4.61 −6.19 Choice+/low 3.92 0.5 −0.63 −2.21 Web only/high −3.42 −4.55 −6.13 Web/paper/high −1.13 −2.71 Choice/high −1.58 Web/paper/low Choice/low Choice+/low Web only/high Web/paper/high Choice/high Choice+/high Web only/low −6.60* −6.81* −10.79** −6.87* −10.29** −11.42** −13.00** Web/paper/low −0.21 −4.19 −0.27 −3.69 −4.82 −6.4* Choice/low −3.98 −0.06 −3.48 −4.61 −6.19 Choice+/low 3.92 0.5 −0.63 −2.21 Web only/high −3.42 −4.55 −6.13 Web/paper/high −1.13 −2.71 Choice/high −1.58 Note.— Entry is the difference between the row treatment response rate and the column treatment response rate. * HSD is significant at α = 0.05. ** HSD is significant at α = 0.01. As shown in table 4, the choice+ protocol achieved the highest response rate overall and the highest response rate at both incentive levels. Choice+ was also the only protocol with a response rate higher than 40 percent at the low incentive level. The web only protocol response rates were the lowest at each incentive level. We did not find statistically significant differences between the web/paper and choice response rates, and overall the choice response was actually higher (41 percent) than web/paper (40 percent). This finding runs counter to the notion that a concurrent choice will reduce response rates, but is consistent with findings from tests in another recent large-scale, general population, government-sponsored survey (Matthews et al. 2012). When comparing response rates of each of the eight treatments against one another, as shown in table 5, we found that the web only/low treatment was significantly lower than the other seven treatments. The web/paper/low treatment was also significantly lower than the choice+/high treatment. The choice+/low response rate was also greater than two of the high-incentive groups. Future research might experiment with different incentive levels to determine if the difference between choice+ and other protocols is even greater at lower incentive levels (say, $5). The response rate was 4.4 percentage points higher for the higher incentive than that of the lower incentive. The effect was greatest for the web only protocol, where the difference was almost 7 percentage points and significantly greater than the other protocols. The effect of the higher promised incentive amount for the three other protocols did not differ significantly, and the differences averaged about 3.5 percentage points. However, further analysis reveals that the higher incentive substantially and significantly increased response rates for multifamily households for these protocols, although it is not clear why. As widely reported in the literature, when respondents are given a choice of responding by paper or web, they overwhelmingly choose paper. Our experiment confirmed this result as shown in figure 2. More than 72 percent of the sample opted for paper under the choice protocol. However, this effect is reversed for the choice+ protocol, with about 64 percent choosing to respond by web. Note that the rate of web response for the choice+ protocol is nearly the same as the corresponding rate for the web/paper protocol, but at a higher overall response rate. Figure 2. View largeDownload slide Mode of Response by Treatment. Figure 2. View largeDownload slide Mode of Response by Treatment. We examined how submission mode varied by age, urbanicity, and income and found some striking differences. As expected, submissions by web were significantly greater among the younger age groups (age 18–44 years), higher-income households (at least $60,000), and urban households. To compare total costs separately by treatment, we totaled up all relevant fixed and variable costs of materials, labor, printing, incentives, postage for the mailing the questionnaires, and follow-up contacts, as well as the costs of keying2 the paper and electronic questionnaires. Table 6 shows these results as a ratio of the total cost to the average total costs across all eight treatments. As expected, the web only protocol was the least expensive option, with only about 92 percent of the average cost. The total costs for the web/paper protocol were also below average. The choice+ protocol was only 2 percent above the average and about 4 percent higher than web/paper. However, for the low-incentive treatment, the costs for these two protocols only differ by 2 percent. The most expensive treatment was choice using the higher incentive. Table 6. Relative Cost per Completed Interview by Treatment Protocol Incentives ($5 prepaid) Average + $10 (promised) + $20 (promised) Web only 0.93 0.92 0.92 Web/paper 0.95 1.00 0.98 Choice 1.04 1.09 1.07 Choice+ 0.97 1.07 1.02 Average 0.98 1.02 1.00 Protocol Incentives ($5 prepaid) Average + $10 (promised) + $20 (promised) Web only 0.93 0.92 0.92 Web/paper 0.95 1.00 0.98 Choice 1.04 1.09 1.07 Choice+ 0.97 1.07 1.02 Average 0.98 1.02 1.00 Table 6. Relative Cost per Completed Interview by Treatment Protocol Incentives ($5 prepaid) Average + $10 (promised) + $20 (promised) Web only 0.93 0.92 0.92 Web/paper 0.95 1.00 0.98 Choice 1.04 1.09 1.07 Choice+ 0.97 1.07 1.02 Average 0.98 1.02 1.00 Protocol Incentives ($5 prepaid) Average + $10 (promised) + $20 (promised) Web only 0.93 0.92 0.92 Web/paper 0.95 1.00 0.98 Choice 1.04 1.09 1.07 Choice+ 0.97 1.07 1.02 Average 0.98 1.02 1.00 Clerical editing costs were not included in the cost comparisons because most of these activities were conducted by the government project sponsor (EIA), not RTI. However, a greater percentage of paper questionnaires and items within paper questionnaires failed edits compared to web submissions. Thus, protocols resulting in more paper questionnaire submissions (for example, the choice protocol) would incur disproportionately more costs, staff burden, and quality issues due to editing than would protocols with higher web submissions. The higher incentive added 5 percent to the web/paper and choice relative costs and 10 percent to the choice+ relative cost. Surprisingly, there is a very small difference in relative costs between the two incentive treatments for the web only protocol. This is owed to both the higher response rate and more timely response using the higher incentive that reduced the number of follow-up mailings required for the protocol. It should be noted that mailing costs constitute a higher percentage of total costs for web only than for other protocols that also incur substantial keying costs. Finally, to complete the phase 1 analysis, we consider the representativity of the responding sample as measured by comparing the distributions of National Pilot responses with the corresponding weighted 2014 ACS national distributions for 12 variables that are common to both surveys. For this analysis, we treat the ACS data as the gold standard; thus, differences between the National Pilot and the ACS are interpreted as indications of bias in the National Pilot. Because the variables considered in this analysis were categorical, we chose the dissimilarity index (d) as the indicator of unrepresentativeness, where d=0.5∑k|pNP,k−pACS,k| and where pNP,k and pACS,k are the National Pilot and ACS proportions in the kth category, respectively. This measure can be interpreted as the proportion of observations that would need to change categories in the National Pilot to achieve perfect agreement with the ACS. In that sense, 1-d may be regarded as the agreement rate between the two surveys. A value of d≤0.10 is usually regarded as “good” agreement, and d≤0.05 is regarded as “very good” agreement. The main objective of the representativity analysis is to determine which of the four test protocols best represents the target population in the responding sample prior to postsurvey adjustments. A protocol producing a small value of d for some variable, say V, is said to be “representative” with respect to V—that is, the protocol generates a respondent sample that is close to the population distribution for V. Likewise, a protocol producing a large value of d generates a respondent sample that is less representative with respect to V. To the extent that V is related to other substantive variables in the survey, d can be taken as an indicator of the risk of nonresponse bias in the substantive variables. Note that large values of d for a protocol do not necessarily suggest that the protocol will produce biased estimates because the respondent sample will ultimately be adjusted for nonresponse and coverage, which that will presumably eliminate the differences between the National Pilot and the ACS that led to the large values of d. However, the dissimilarity analysis does suggest the extent to which postsurvey weighting adjustments must be relied upon to mitigate the bias risks. In addition, postsurvey weighting adjustments are not always successful for reducing bias and may increase the standard errors of estimates because of increased weight variation. Thus, protocols with smaller dissimilarity indexes are preferable. Finally, it is important to note that even a protocol that produces perfectly representative samples may not generate zero values of d for at least three reasons. First, the 2014 ACS is not an ideal gold standard because the National Pilot was conducted more than a year later at the end of 2015. Changes in the target population in the intervening period will increase d. Second, both the National Pilot and the ACS are subject to sampling error, which has not been taken into account in the analysis. Third, there are minor differences in the question wordings, the definition of an eligible housing unit, and target populations for the two surveys. As an example of the latter, as previously noted, the National Pilot sample excluded drop points. Nevertheless, we believe the analysis is useful for revealing differences in representativity among the four protocols. Figure 3 shows the values of d expressed as a percentage for the following 12 variables: Figure 3. View largeDownload slide Dissimilarity Indexes for Selected Questionnaire Items Using the 2014 ACS as the Gold Standard. Proportion of cases that would need to switch categories for perfect agreement with the ACS—10 to 5 percent dissimilarity is considered to be “good” agreement; 5 percent or less is considered “very good.” Figure 3. View largeDownload slide Dissimilarity Indexes for Selected Questionnaire Items Using the 2014 ACS as the Gold Standard. Proportion of cases that would need to switch categories for perfect agreement with the ACS—10 to 5 percent dissimilarity is considered to be “good” agreement; 5 percent or less is considered “very good.” Respondent Hispanic (Hisp)—respondent is Hispanic or Latino Own home (Own)—housing unit owned by respondent or someone in household—not rented for money or occupied without payment of rent Refrigerator (Fridge)—at least one refrigerator is used in the housing unit Internet access (Access)—Internet access available in the housing unit Race (Race)—respondent’s race: white only, black only, American Indian/Alaskan Native only, Asian only, or other (includes two or more races) Year made (Year)—year housing unit was built: before 1950, 1950–1959, 1960–1969, 1970–1979, 1980–1989, 1990–1999, 2000–2009, 2010–2015 Household income (Income)—annual gross household income for the last year: less than $20,000; $20,000–39,999; $40,000–59,999; $60,000–79,999; $80,000–99,999; $100,00–119,999; $120,000–139,999; $140,000 or more Main heating fuel (Fuel)—of those who heat home, main space heating fuel: electricity, natural gas from underground pipes, propane (bottled gas), fuel oil, wood, and other Household size (HHSize)—number of household members including respondent: 1, 2, 3, 4 or more Age3 (Age)—age of respondent: <30, 30–44, 45–59, 60–74, 75+ Bedrooms (Bdrms)—number of bedrooms: 0, 1, 2, 3, 4, 5 or more Housing unit type (HUType)—type of housing unit: mobile home, single-family house detached from any other house, single-family house attached to one or more other houses (e.g., duplex, row house, townhome), apartment in a building with two to four units, apartment in a building with five or more units Figure 3 shows higher dissimilarity indices for Own, Year, HHSize, and HUType. Further analysis indicates possible underrepresentation for renters, older housing units, single-person households, and apartments, which is generally consistent with the literature (see, for example, Dillman, Smyth, and Christian 2014). Note that the largest value of d in the figure occurs for Access under the web only protocol and is about 13 percent. This is not surprising because persons who do not have Internet access have a lower propensity to respond via the Internet and would, thus, be underrepresented by web only. As shown in the last column of table 8, the average value of d across all 12 variables ranges between 4.77 percent (for choice+/high incentive) to 6.36 percent (web only/low incentive). It should be noted that d drops to 5.30 percent for web only/low if the variable Access is not considered. Thus, the dissimilarity indexes are in the “good” to “very good” range for all treatment conditions. With regard to the comparison between web/paper and choice+, the latter protocol produces eight (out of 12) values of d larger than 5 percent while the former protocol produces only six. However, web/paper has higher values of d; for example, three variables exceed 7 for web/paper while only two exceed 7 for choice+. Table 8. Summary of Key Metrics by Treatment for Phase 1 Protocol Response rate (%) Proportion of respondents completing by web (%) Relative cost Dissimilarity Relative to ACS (average d) $10 promised incentive  Web only 31.99 100.00 0.93 6.36  Web/paper 38.59 64.01 0.95 5.07  Choice 38.80 27.56 1.04 5.04  Choice+ 42.78 63.92 0.97 5.84 $20 promised incentive  Web only 38.86 100.00 0.92 5.61  Web/paper 42.28 70.99 1.00 4.93  Choice 43.41 27.42 1.09 5.13  Choice+ 44.99 64.50 1.07 4.77 Protocol Response rate (%) Proportion of respondents completing by web (%) Relative cost Dissimilarity Relative to ACS (average d) $10 promised incentive  Web only 31.99 100.00 0.93 6.36  Web/paper 38.59 64.01 0.95 5.07  Choice 38.80 27.56 1.04 5.04  Choice+ 42.78 63.92 0.97 5.84 $20 promised incentive  Web only 38.86 100.00 0.92 5.61  Web/paper 42.28 70.99 1.00 4.93  Choice 43.41 27.42 1.09 5.13  Choice+ 44.99 64.50 1.07 4.77 Table 8. Summary of Key Metrics by Treatment for Phase 1 Protocol Response rate (%) Proportion of respondents completing by web (%) Relative cost Dissimilarity Relative to ACS (average d) $10 promised incentive  Web only 31.99 100.00 0.93 6.36  Web/paper 38.59 64.01 0.95 5.07  Choice 38.80 27.56 1.04 5.04  Choice+ 42.78 63.92 0.97 5.84 $20 promised incentive  Web only 38.86 100.00 0.92 5.61  Web/paper 42.28 70.99 1.00 4.93  Choice 43.41 27.42 1.09 5.13  Choice+ 44.99 64.50 1.07 4.77 Protocol Response rate (%) Proportion of respondents completing by web (%) Relative cost Dissimilarity Relative to ACS (average d) $10 promised incentive  Web only 31.99 100.00 0.93 6.36  Web/paper 38.59 64.01 0.95 5.07  Choice 38.80 27.56 1.04 5.04  Choice+ 42.78 63.92 0.97 5.84 $20 promised incentive  Web only 38.86 100.00 0.92 5.61  Web/paper 42.28 70.99 1.00 4.93  Choice 43.41 27.42 1.09 5.13  Choice+ 44.99 64.50 1.07 4.77 Finally, as noted in section 3.2.2, the main data collection (referred to as phase 1) was followed by an NRFU phase (referred to as phase 2). Here we provide a few results and lessons learned from phase 2. As shown in table 7, approximately 25 percent of the phase 1 nonrespondents responded in the phase 2 xNRFU. The extra $10 promised incentive added about 5 points to this response rate, which was highly significant. Table 7. Response Rates by Treatment—Phase 2 xNRFU (2 page paper questionnaire) No extra incentive (control) 22.39 $10 extra promised incentive 27.58*** Mean 25.08 xNRFU (2 page paper questionnaire) No extra incentive (control) 22.39 $10 extra promised incentive 27.58*** Mean 25.08 *** Highly significantly different from control (p < 0.001). Table 7. Response Rates by Treatment—Phase 2 xNRFU (2 page paper questionnaire) No extra incentive (control) 22.39 $10 extra promised incentive 27.58*** Mean 25.08 xNRFU (2 page paper questionnaire) No extra incentive (control) 22.39 $10 extra promised incentive 27.58*** Mean 25.08 *** Highly significantly different from control (p < 0.001). As mentioned in section 3.2.2, a more complete examination of the phase 2 results will be considered in a subsequent paper. That paper will examine the characteristics of phase 2 respondents compared with phase 1 respondents; effects of treatments on NRFU response; effects of NRFU response on sample representativity; effects of NRFU weighting on nonresponse bias and variance; and costs versus benefits of xNRFU+ versus xNRFU. 5. SUMMARY AND CONCLUSIONS Declining response rates and reduced cost efficiency of face-to-face data collection motivated this research to examine several self-administered data collection approaches as alternatives to a traditional CAPI survey. The National Pilot study employed eight contact strategies across a sample of 9,650 U.S. households to identify the protocol that provided the best combination of adequate response rate, high proportion of respondents using the web survey option, low cost, and good respondent sample representativity. Two of those strategies, the choice+ treatments, used a bonus incentive to motivate web response over paper. With regard to response rate, the web only treatment was uniformly lower than all other protocols. Choice+ attained the highest response rate overall and for both incentive treatments. In addition, choice+ maintained the highest submission rate for essentially the entire data collection period. The extra $10 promised incentive increased the response rate overall by 4.4 percentage points, with the greatest increase for web only (6.9 percentage points). The proportion of respondents responding by web for the choice protocol was about 27.5 percent. This was increased to more than 64 percent for the choice+ protocol. This rate of web returns was about equal to the web/paper protocol for the low incentive and about 5 points lower than web/paper for the high incentive. While web is the preferred self-administered mode due to cost and data quality benefits, many studies must still use paper modes to reach those who prefer that mode or who cannot respond via the Internet. Regarding cost, across both incentive levels, the cost of choice+ was only 2 percent above the average cost and about 4 percent higher than web/paper. Surprisingly, choice+ cost 5 percent less than choice despite the additional cost of a $10 promised bonus incentive for web submission. The higher incentive treatment increased choice+ cost by 10 percent over the low-incentive treatment. The web only treatments were the least expensive treatments in this study due to the lack of costs for printing, receipting, and keying paper forms. Another surprising result was the decrease in web only cost when the incentive was increased $10; the relative cost per completed interview was 0.93 in the low group vs 0.92 in the high group. This result may be relevant for any survey being conducted solely via the web. In addition, the choice+ costs were not greater than the choice group despite the additional $10 incentive cost for web respondents. Even more importantly, but not surprising, all eight treatment costs were significantly lower than previous collections for the RECS via CAPI interviews. Taken together, as in table 8, one can review the results of the strategies along multiple dimensions to make a decision about which to consider “best.” For our purposes, the choice+ protocol, a concurrent dual mode method with extra incentive for the web response, was best suited to address the range of considerations. Choice+ had the highest response rate at both low and high incentive levels and maintained the highest submission rate throughout data collection. This is an indication that this strategy may be more robust than the traditional concurrent option (i.e., choice) or the sequential web/paper method. The choice+ group was also far better than the choice protocol and at least as good as the web/paper at eliciting response via the web. It would be interesting to see if this web response effect holds true at lower incentive levels (for example, $5 prepaid + $5 promised or $0 prepaid + $10 promised). While self-administration and the contact strategies described in this paper provide promising results, it should be noted that the response rates for all eight treatments were much lower than the 80 percent response traditionally attained in prior rounds of RECS that used CAPI methods. This raises concerns about nonresponse bias and coverage error. Although we were not able to compare National Pilot data in parallel with a contemporaneous CAPI study, comparisons with ACS benchmarks provide some evidence of respondent representativity. Those results were generally positive. Most comparisons with 12 ACS variables showed “good” or “very good” representativity across the four protocols, although comparisons indicate that respondents are less likely to be renters and apartment dwellers. Representativity for web/paper and choice+ was quite comparable. However, all treatments demonstrated good representativity overall, with an average d in the range 4.77 to 6.39 (table 8). As one might expect, households without Internet access in the web only protocol were the most underrepresented population. Finally, although it was not the focus of this paper, the phase 2 results presented here suggest that nonresponse follow-up using an abbreviated questionnaire can add substantially to the information provided by single phase data collection. It remains to evaluate the value added by these data, which is the topic of a subsequent paper. In all, these results suggest that, in spite of their lower response rates, mixed mode mail surveys can serve as a viable platform for traditionally CAPI data collections such as the RECS. All treatments, with the exception of the web only/low group (32 percent), achieved a response rate between 38 percent and 45 percent, attained acceptable respondent representativity, and delivered low-cost, completed questionnaires. Further cost efficiencies accrue for mail surveys because, unlike CAPI surveys, mail survey samples need not be clustered to minimize travel costs. These cost savings can allow for program flexibilities that are limited by the cost, precision, and timing constraints of CAPI surveys. Our analysis indicates that the choice+ self-administered strategy in particular is an effective method for simultaneously increasing both web and overall survey response. Further research might replicate the choice+ experiment and conduct additional experiments with variations on these strategies (i.e., lower choice+ incentives for web response). Additional considerations, such as differential measurement error by strategy, are also important to examine and weight against the other factors in selecting the most appropriate data collection protocol. Footnotes 1 Selection weighted means that the units are weighted by the inverse of their selection probabilities only; i.e., no weighting adjustments for nonresponse and/or noncoverage are applied. 2 In some surveys, using scanned forms (for example, via TeleForm) could save data entry costs and thus change the cost comparisons. For this Pilot study, scanned forms were considered but were deemed cost-ineffective. 3 The ACS reports the age of the “householder” or the “head of the household” whereas the National Pilot reports the age of the person responding to the survey. This would explain at least part of the dissimiliarity between the National Pilot and the ACS for Age. References Avdeyeva O. A. , Matland R. E. ( 2013 ), “An Experimental Test of Mail Surveys as a Tool for Social Inquiry in Russia,” International Journal of Public Opinion Research , 25 2 , 173 – 194 . Biemer P. , Murphy J. , Kott P. ( 2016 ), “ Estimating mail or web survey eligibility for undeliverable addresses: a latent class analysis approach,” Proceedings of the ASA Survey Methods Research Section and Presented at Joint Statistical Meetings of the ASA, Chicago, IL . pp. 1166 – 1172 . Church A. H. ( 1993 ), “ Estimating the Effect of Incentives on Mail Survey Response Rates: A Meta-analysis ,” Public Opinion Quarterly , 57 , 62 – 79 . Google Scholar CrossRef Search ADS Couper M. P. ( 2008 ), Designing Effective Web Surveys , New York : Cambridge University Press . Google Scholar CrossRef Search ADS Couper M. P. ( 2011 ), “ The Future of Modes of Data Collection ,” Public Opinion Quarterly , 75 , 889 – 908 . Google Scholar CrossRef Search ADS de Leeuw E. , Hox J. ( 2011 ), “Internet Surveys as Part of a Mixed Mode Design,” in Social and Behavioral Research and the Internet , eds. Das M. , Ester P. , Kaczmirek L. , pp. 45 – 76, New York : Taylor and Francis . Dillman D. , Edwards M. ( 2016 ), “Designing a Mixed Mode Survey,” in The Sage Handbook of Survey Methodology , eds. Wolf C. , Joye D. , Smith T.W. , Fu Y. , pp. 255 – 268, London : Sage Reference . Google Scholar CrossRef Search ADS Dillman D. , Smyth J. , Christian L. M. ( 2014 ), Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method ( 4th ed.) , Hoboken, NJ : John Wiley & Sons . Dykema J. , Jones N. , Piche T. , Stevenson J. ( 2013 ), “ Surveying Clinicians by Web: Current Issues in Design and Administration ,” Evaluation and the Health Professions , 36 , 352 – 381 . Google Scholar CrossRef Search ADS PubMed Griffin D. H. , Fischer D. P. , Morgan M. T. ( 2001 ), “Testing an Internet Response Option for the American Community Survey,” paper presented at the Annual Conference of the American Association of Public Opinion Research, Montreal, Quebec, Canada. Harter R. , Battaglia M. P. , Buskirk T. D. , Dillman D. A. , English N. et al. , ( 2016 ), “AAPOR Report: Address-Based Sampling,” Available at http://www.aapor.org/Education-Resources/Reports/Address-based-Sampling.aspx. Jobber D. , Saunders J. , Mitchell V. W. ( 2004 ), “ Prepaid Monetary Incentive Effects on Mail Survey Response ,” Journal of Business Research , 57 , 347 – 350 . Google Scholar CrossRef Search ADS Link M. W. , M. P. Battaglia , M. R. Frankel , L. Osborn , A. H. Mokdad et al. , ( 2008 ), A comparison of address-based sampling (ABS) versus random-digit dialing (RDD) for general population surveys. Public Opinion Quarterly , 72 1 , 6 – 27 . Manfreda K. , Lozar M. B. , Berzelak J. , Haas I. , Vehovar V. ( 2008 ), “ Web Surveys Versus Other Survey Modes: A Meta-Analysis Comparing Response Rates ,” International Journal of Market Research , 50 , 79 – 104 . Google Scholar CrossRef Search ADS Matthews B. , Davis M. , Tancreto J. , Zelenak M. F. , Ruiter M. ( 2012 ), “2011 American Community Survey Internet Tests: Results from Second Test in November 2011,” American Community Survey Research and Evaluation Program, #ACS12-RER-21, May 14, 2012. Medway R. , Fulton J. ( 2012 ), “ When More Gets You Less: A Meta-Analysis of the Effect of Concurrent Web Options on Mail Survey Response Rates, ” Public Opinion Quarterly , 76 , 733 – 746 . Google Scholar CrossRef Search ADS Messer B. L. , Dillman D. A. ( 2011 ), “ Surveying the General Public over the Internet Using Address-Based Sampling and Mail Contact Procedures ,” Public Opinion Quarterly , 75 , 429 – 457 . Google Scholar CrossRef Search ADS Millar M. M. , Dillman D. A. ( 2011 ), “ Improving Response to Web and Mixed-Mode Surveys ,” Public Opinion Quarterly , 75 , 249 – 269 . Google Scholar CrossRef Search ADS Murphy J. , Biemer P. , Berry C. (in press), “ Visualizing Real-Time Metrics Critical to Quality in a Mixed-Mode Self-Administered Survey ,” Journal of Official Statistics . Perrin A. , Duggan M. ( 2015 ), “Americans’ Internet Access: 2000-2015,” Pew Research Center, Available at: http://www.pewinternet.org/files/2015/06/2015-06-26_internet-usage-across-demographics-discover_FINAL.pdf. Schwartz B. ( 2004 ), The Paradox of Choice: Why More Is Less , New York : Harper Perennial . Shih T. H. , X. Fan ( 2008 ), Comparing response rates from web and mail surveys: A meta-analysis. Field methods , 20 3 , 249 – 271 . Singer E. , Ye C. ( 2013 ), “ The Use and Effects of Incentives in Surveys ,” The Annals of the American Academy of Political and Social Science , 645 , 112 – 141 . Google Scholar CrossRef Search ADS Stern M. J. , Bilgen I. , Dillman D. A. ( 2014 ), “ The State of Survey Methodology Challenges, Dilemmaas and New Frontiers in the Era of the Tailored Design ,” Field Methods , 26 , 284 – 301 . Google Scholar CrossRef Search ADS U.S. Census Bureau ( 2016 ), “2015 American Community Survey 1-Year Estimates,” Available at https://www.census.gov/programs-surveys/acs/news/data-releases/2015/release.html. Wine J. , Cominole M. , Heuer R. , Riccobono J. ( 2006 ), “Challenges of Designing and Implementing Multimode Instruments,” paper presented at Second International Conference on Telephone Survey Methodology, Miami, FL. Zimmer S. , Biemer P. , Kott P. , Berry C. ( 2015 ), “Testing a Model-Directed, Mixed Mode Protocol in the RECS Pilot Study,” Proceedings of the 2015 Federal Committee on Statistical Methodology Research Conference, Available at https://fcsm.sites.usa.gov/files/2016/03/G2_Zimmer_2015FCSM.pdf. © The Author 2017. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

Journal of Survey Statistics and MethodologyOxford University Press

Published: Jun 30, 2017

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off