Testing the Effects of Envelope Features on Survey Response in a Telephone Survey Advance Letter Mailing Experiment

Testing the Effects of Envelope Features on Survey Response in a Telephone Survey Advance Letter... Abstract There has been considerable research in the past three decades about the value of sending an advance letter to respondents prior to making the formal request for cooperation in a survey. This literature has shown that advance letters have a positive effect on survey response rates, but the literature has almost entirely focused on what is enclosed inside the envelope. In contrast, little attention has been given to the importance of the envelope/mailer that is used. However, whatever is enclosed inside the envelope can have no impact on the recipient unless the envelope is opened. We report on a large national experiment that tested the effects of two aspects of the advance envelope of a survey that sampled telephone numbers and gathered data via telephone: (1) envelope color and (2) the first line of the addressee. Our study was conducted as part of the CDC’s 2014 National Immunization Survey (NIS). In the landline RDD sample, all the advance letters were randomly assigned to one of nine conditions using a 3 × 3 factorial design. This experiment tested the effects on the telephone survey response of whether the envelope was white, tan, or light blue and whether the envelope was addressed to the name of the person matched to the address, to “Resident,” or to “Parent/Guardian.” It was found that a tan envelope addressed to “Parent/Guardian” had a highly significant impact, achieving an approximately 35% higher eligibility rate compared to the control condition (white envelope addressed with the name that was matched to the address). Making envelope features more responsive to the potential participants did not raise the cost of the mailings, but it did have meaningful cost-beneficial effects for the NIS. There is research on the effects of advance contacts on survey response rates—in particular, the use of advance letters in landline telephone surveys where addresses can be matched to a large portion of phone numbers. But this research provides sparse information about how to deploy advance contact techniques with sampled households so as to achieve the most cost-effective outcomes (Groves and Snowden, 1987; Camburn, Lavrakas, Battaglia, Massey, and Wright, 1996; de Leeuw, Callegaro, Hox, Korendijk, and Lensvelt-Mulder, 2007; Edwards, Roberts, Clarke, Diguiseppi, Wentz, et al., 2009). In reviewing this literature, there appear to be three domains of factors associated with the advance mailings that could be used to improve survey response: Factors related to the likelihood that the advance mailer will be opened. These factors are related to the envelope and the mailing service used to deliver it, such as the postage used on the envelope, the sender that is listed in the return address, graphics on the envelope, or the size of the envelope. Table 1 shows factors that fall into this domain. Table 1. Factors Related to the Likelihood the Advance Mailer will be Opened Factors Examples Type of delivery service/class USPS 1st Class, USPS Priority Mail, Fed-Ex, UPS Type of mailer Padded envelope, box, transparent window allowing some contexts to be seen Mailer size Small letter, standard letter, 6" x 9", 9" x 12" Mailer color White, business tan, pastel Addressee on mailer Name of resident, current resident, parent, head of household Return address of sender University, government agency, survey organization, corporate sponsor Graphics on mailer Cartoon art, scenic image, celebrity picture Branding logo on mailer Organizational branding, branding of the survey Topical phrasing/teaser on mailer “Childhood vaccinations,” “Crime prevention awareness,” “Educational attainment,” “U.S. foreign policy” Endorsement on mailer Elected official, famous athlete, music celebrity, television celebrity Font style Times Roman, Arial; Courier, Verdana Font size 10, 11, 12, 14 Factors Examples Type of delivery service/class USPS 1st Class, USPS Priority Mail, Fed-Ex, UPS Type of mailer Padded envelope, box, transparent window allowing some contexts to be seen Mailer size Small letter, standard letter, 6" x 9", 9" x 12" Mailer color White, business tan, pastel Addressee on mailer Name of resident, current resident, parent, head of household Return address of sender University, government agency, survey organization, corporate sponsor Graphics on mailer Cartoon art, scenic image, celebrity picture Branding logo on mailer Organizational branding, branding of the survey Topical phrasing/teaser on mailer “Childhood vaccinations,” “Crime prevention awareness,” “Educational attainment,” “U.S. foreign policy” Endorsement on mailer Elected official, famous athlete, music celebrity, television celebrity Font style Times Roman, Arial; Courier, Verdana Font size 10, 11, 12, 14 Table 1. Factors Related to the Likelihood the Advance Mailer will be Opened Factors Examples Type of delivery service/class USPS 1st Class, USPS Priority Mail, Fed-Ex, UPS Type of mailer Padded envelope, box, transparent window allowing some contexts to be seen Mailer size Small letter, standard letter, 6" x 9", 9" x 12" Mailer color White, business tan, pastel Addressee on mailer Name of resident, current resident, parent, head of household Return address of sender University, government agency, survey organization, corporate sponsor Graphics on mailer Cartoon art, scenic image, celebrity picture Branding logo on mailer Organizational branding, branding of the survey Topical phrasing/teaser on mailer “Childhood vaccinations,” “Crime prevention awareness,” “Educational attainment,” “U.S. foreign policy” Endorsement on mailer Elected official, famous athlete, music celebrity, television celebrity Font style Times Roman, Arial; Courier, Verdana Font size 10, 11, 12, 14 Factors Examples Type of delivery service/class USPS 1st Class, USPS Priority Mail, Fed-Ex, UPS Type of mailer Padded envelope, box, transparent window allowing some contexts to be seen Mailer size Small letter, standard letter, 6" x 9", 9" x 12" Mailer color White, business tan, pastel Addressee on mailer Name of resident, current resident, parent, head of household Return address of sender University, government agency, survey organization, corporate sponsor Graphics on mailer Cartoon art, scenic image, celebrity picture Branding logo on mailer Organizational branding, branding of the survey Topical phrasing/teaser on mailer “Childhood vaccinations,” “Crime prevention awareness,” “Educational attainment,” “U.S. foreign policy” Endorsement on mailer Elected official, famous athlete, music celebrity, television celebrity Font style Times Roman, Arial; Courier, Verdana Font size 10, 11, 12, 14 Factors related to the likelihood that the contents of the mailer will be read and recalled. These factors are related to the substance and formatting of the letter itself (e.g., the information conveyed in the letter about the purpose and confidentiality of the study, the letterhead on which the letter is printed, font size, type and color of the text, and the credentials of the sender and signatory) and what else may be included inside the envelope (e.g., a noncontingent incentive, a brochure, or an FAQ insert). Table 2 shows factors that fall into this domain. Table 2. Factors Related to the Likelihood the Contents of the Mailer will Be Read and Recalled Factors Examples Letterhead University, government, corporate Branding logo on page Organizational branding, branding of the survey Endorsement on page Elected official, famous athlete, music celebrity, television celebrity, subject matter expert Font style of text Times-Roman, Arial; Courier, Verdana Font size of text 10, 11, 12, 14 Salutation/Greetings None, generic, personalized Reading level of text 12th grade, 8th grade, 5th grade Length of text More than four paragraphs or more than 300 words, 3–4 paragraphs, two paragraphs, one paragraph Text about benefits to respondent Enclosed noncontingent incentive, contingent incentive, altruism, enjoyment Text about the sponsor Name of sponsor, history of sponsor, stature of sponsor, sector of sponsor Text about the purpose of study Societal or other benefits that will be result from the study, e.g., better health, lower crime, higher employment, cleaner environment Text about eligibility criteria Age, education, health status, low income, employment status Text about questionnaire length and respondent burden Estimated time for completion, other burdens (e.g., information look-ups) Text about privacy protections Promised anonymity, promised confidentiality, survey organization’s history of keeping data secure Text about contact info Survey organization’s website, email, phone number, mailing address Text about sampling How addresses, phone numbers, names were selected, why form of selection was used Text about data collection mode(s) In-person, telephone, mail, internet, mixed-mode data collection Text requesting that others in household be told of study Request person reading letter to tell everyone in household or tell all adults Nature of the signature Pre-printed, hand-signed, blue ink, black ink, other colored ink Signatory’s affiliation University, government, corporate, more than one Credentials of signatory MD, MA, PhD, LLD Sex of signatory Female, Male, Ambiguous Ethnicity of signatory Asian, African, Middle Eastern, Hispanic, East European, Scandinavian Language of text English, Spanish, both Use of informational inserts FAQs, organizational brochure, historical information about study Factors Examples Letterhead University, government, corporate Branding logo on page Organizational branding, branding of the survey Endorsement on page Elected official, famous athlete, music celebrity, television celebrity, subject matter expert Font style of text Times-Roman, Arial; Courier, Verdana Font size of text 10, 11, 12, 14 Salutation/Greetings None, generic, personalized Reading level of text 12th grade, 8th grade, 5th grade Length of text More than four paragraphs or more than 300 words, 3–4 paragraphs, two paragraphs, one paragraph Text about benefits to respondent Enclosed noncontingent incentive, contingent incentive, altruism, enjoyment Text about the sponsor Name of sponsor, history of sponsor, stature of sponsor, sector of sponsor Text about the purpose of study Societal or other benefits that will be result from the study, e.g., better health, lower crime, higher employment, cleaner environment Text about eligibility criteria Age, education, health status, low income, employment status Text about questionnaire length and respondent burden Estimated time for completion, other burdens (e.g., information look-ups) Text about privacy protections Promised anonymity, promised confidentiality, survey organization’s history of keeping data secure Text about contact info Survey organization’s website, email, phone number, mailing address Text about sampling How addresses, phone numbers, names were selected, why form of selection was used Text about data collection mode(s) In-person, telephone, mail, internet, mixed-mode data collection Text requesting that others in household be told of study Request person reading letter to tell everyone in household or tell all adults Nature of the signature Pre-printed, hand-signed, blue ink, black ink, other colored ink Signatory’s affiliation University, government, corporate, more than one Credentials of signatory MD, MA, PhD, LLD Sex of signatory Female, Male, Ambiguous Ethnicity of signatory Asian, African, Middle Eastern, Hispanic, East European, Scandinavian Language of text English, Spanish, both Use of informational inserts FAQs, organizational brochure, historical information about study Table 2. Factors Related to the Likelihood the Contents of the Mailer will Be Read and Recalled Factors Examples Letterhead University, government, corporate Branding logo on page Organizational branding, branding of the survey Endorsement on page Elected official, famous athlete, music celebrity, television celebrity, subject matter expert Font style of text Times-Roman, Arial; Courier, Verdana Font size of text 10, 11, 12, 14 Salutation/Greetings None, generic, personalized Reading level of text 12th grade, 8th grade, 5th grade Length of text More than four paragraphs or more than 300 words, 3–4 paragraphs, two paragraphs, one paragraph Text about benefits to respondent Enclosed noncontingent incentive, contingent incentive, altruism, enjoyment Text about the sponsor Name of sponsor, history of sponsor, stature of sponsor, sector of sponsor Text about the purpose of study Societal or other benefits that will be result from the study, e.g., better health, lower crime, higher employment, cleaner environment Text about eligibility criteria Age, education, health status, low income, employment status Text about questionnaire length and respondent burden Estimated time for completion, other burdens (e.g., information look-ups) Text about privacy protections Promised anonymity, promised confidentiality, survey organization’s history of keeping data secure Text about contact info Survey organization’s website, email, phone number, mailing address Text about sampling How addresses, phone numbers, names were selected, why form of selection was used Text about data collection mode(s) In-person, telephone, mail, internet, mixed-mode data collection Text requesting that others in household be told of study Request person reading letter to tell everyone in household or tell all adults Nature of the signature Pre-printed, hand-signed, blue ink, black ink, other colored ink Signatory’s affiliation University, government, corporate, more than one Credentials of signatory MD, MA, PhD, LLD Sex of signatory Female, Male, Ambiguous Ethnicity of signatory Asian, African, Middle Eastern, Hispanic, East European, Scandinavian Language of text English, Spanish, both Use of informational inserts FAQs, organizational brochure, historical information about study Factors Examples Letterhead University, government, corporate Branding logo on page Organizational branding, branding of the survey Endorsement on page Elected official, famous athlete, music celebrity, television celebrity, subject matter expert Font style of text Times-Roman, Arial; Courier, Verdana Font size of text 10, 11, 12, 14 Salutation/Greetings None, generic, personalized Reading level of text 12th grade, 8th grade, 5th grade Length of text More than four paragraphs or more than 300 words, 3–4 paragraphs, two paragraphs, one paragraph Text about benefits to respondent Enclosed noncontingent incentive, contingent incentive, altruism, enjoyment Text about the sponsor Name of sponsor, history of sponsor, stature of sponsor, sector of sponsor Text about the purpose of study Societal or other benefits that will be result from the study, e.g., better health, lower crime, higher employment, cleaner environment Text about eligibility criteria Age, education, health status, low income, employment status Text about questionnaire length and respondent burden Estimated time for completion, other burdens (e.g., information look-ups) Text about privacy protections Promised anonymity, promised confidentiality, survey organization’s history of keeping data secure Text about contact info Survey organization’s website, email, phone number, mailing address Text about sampling How addresses, phone numbers, names were selected, why form of selection was used Text about data collection mode(s) In-person, telephone, mail, internet, mixed-mode data collection Text requesting that others in household be told of study Request person reading letter to tell everyone in household or tell all adults Nature of the signature Pre-printed, hand-signed, blue ink, black ink, other colored ink Signatory’s affiliation University, government, corporate, more than one Credentials of signatory MD, MA, PhD, LLD Sex of signatory Female, Male, Ambiguous Ethnicity of signatory Asian, African, Middle Eastern, Hispanic, East European, Scandinavian Language of text English, Spanish, both Use of informational inserts FAQs, organizational brochure, historical information about study Interviewerfactorsrelated to the likelihood of respondent cooperation. In interviewer-administered surveys, factors are related to the interviewers who try to contact the households at addresses to which the letters were mailed and then try to gain cooperation from the households, such as the training that interviewers received about the importance of the letter or whether the introduction of the survey explicitly mentions an advance letter being sent. Table 3 shows factors that fall into this domain. Table 3. Interviewer Factors Related to the Likelihood of Respondent Cooperation Factors Examples Text in the intro of the CATI/CAPI script about the letter being sent Explicit mention in script that is required for interviewers to provide; mention in script that is optional for interviewers to provide Special interviewer training for how best to leverage the letter History of the use of survey advance letters; examples of how to mention the letter at various places in the interview; practice mentioning the letter Timing of the arrival of letter compared to when interviewer contact is made Interviewers’ finesse in deciding how salient the letter likely will be to the recipient depending on the number of days the envelope arrived prior to contact for data collection, which will vary throughout the field period Questionnaire items about the letter Did household receive letter, did anyone read letter, did everyone in the household get told about letter, does respondent remember anything about letter Valence that interviewers place on the letter that was sent Positive, neutral, negative Factors Examples Text in the intro of the CATI/CAPI script about the letter being sent Explicit mention in script that is required for interviewers to provide; mention in script that is optional for interviewers to provide Special interviewer training for how best to leverage the letter History of the use of survey advance letters; examples of how to mention the letter at various places in the interview; practice mentioning the letter Timing of the arrival of letter compared to when interviewer contact is made Interviewers’ finesse in deciding how salient the letter likely will be to the recipient depending on the number of days the envelope arrived prior to contact for data collection, which will vary throughout the field period Questionnaire items about the letter Did household receive letter, did anyone read letter, did everyone in the household get told about letter, does respondent remember anything about letter Valence that interviewers place on the letter that was sent Positive, neutral, negative Table 3. Interviewer Factors Related to the Likelihood of Respondent Cooperation Factors Examples Text in the intro of the CATI/CAPI script about the letter being sent Explicit mention in script that is required for interviewers to provide; mention in script that is optional for interviewers to provide Special interviewer training for how best to leverage the letter History of the use of survey advance letters; examples of how to mention the letter at various places in the interview; practice mentioning the letter Timing of the arrival of letter compared to when interviewer contact is made Interviewers’ finesse in deciding how salient the letter likely will be to the recipient depending on the number of days the envelope arrived prior to contact for data collection, which will vary throughout the field period Questionnaire items about the letter Did household receive letter, did anyone read letter, did everyone in the household get told about letter, does respondent remember anything about letter Valence that interviewers place on the letter that was sent Positive, neutral, negative Factors Examples Text in the intro of the CATI/CAPI script about the letter being sent Explicit mention in script that is required for interviewers to provide; mention in script that is optional for interviewers to provide Special interviewer training for how best to leverage the letter History of the use of survey advance letters; examples of how to mention the letter at various places in the interview; practice mentioning the letter Timing of the arrival of letter compared to when interviewer contact is made Interviewers’ finesse in deciding how salient the letter likely will be to the recipient depending on the number of days the envelope arrived prior to contact for data collection, which will vary throughout the field period Questionnaire items about the letter Did household receive letter, did anyone read letter, did everyone in the household get told about letter, does respondent remember anything about letter Valence that interviewers place on the letter that was sent Positive, neutral, negative The first domain of factors (Table 1) is important because the contents of the letter (including the possibility of finding a noncontingent incentive inside the envelope) cannot have any effect on survey response unless the mailer is opened. Thus, the first goals in trying to successfully deploy advance letters are (a) to get the envelope delivered to the correct address and (b) to get people to open the envelope that is delivered to them. Relatively little has been reported about factors that are related to the envelope and the mail delivery service used to deliver it and how these factors affect the propensity of a recipient opening it. This dearth of knowledge is readily apparent when one considers what de Leeuw et al. (2007) did not report in their otherwise extensive meta-evaluation of the effects of advanced letters (i.e., they included no factors from this envelope-domain in their meta-evaluation). In addition, the dearth in research about envelope factors associated with advance letters that are used in telephone surveys is also apparent from an extremely comprehensive review of the entire mail survey methodological literature conducted by Edwards et al. (2009); (this review was not limited to methodological research on advance letters in surveys). These scholars identified 481 published experiments of mail survey factors through 2008.1 Twelve percent of these studies (n = 57) addressed factors associated with the mailer that was used. Furthermore, Edwards et al. (2009) identified 110 separate factors that were tested in this body of literature, and only ten of the factors were associated with the mailer that was used. These mailer-related factors in mail surveying included using (a) hand-written labels, (b) tan envelopes, (c) stamps, (d) first class postage, (e) registered or certified delivery, (f) a larger than letter-size envelope, and (g) university sponsorship. All of these factors were found to be associated with a higher response rate in mail surveys. However, all the studies that assessed these factors in the Edwards et al. (2009) review focused on mail survey response, not telephone survey response. Although it is plausible that factors associated with higher mail survey response rates would also be associated with more effective advance letter mailings for telephone surveys, none of the studies in the Edwards et al. (2009) review looked at the impact of these factors on response to telephone surveys. The factors in the second domain (Table 2) make up the preponderance of what past research on advance letters effects has tested, i.e., most studies have tested the content that was conveyed in the letter, the reading level at which that content was written, various aspects of the formatting of the letter, and/or what else was included inside the envelope besides the letter. de Leeuw et al. (2007) reported average effects of eleven percentage-point gains in cooperation, and eight percentage-point gains in response rates, linked primarily to the content of what the advance letter conveyed, including the likelihood that the advance letter served to legitimize the study. The third domain (Table 3) concerns how the interviewers actually “use” to their advantage the fact that an advance letter was mailed, and this has remained essentially untested. For example, de Leeuw et al. (2007) included only a gross factor from this domain in their meta-evaluation (i.e., whether some “special training” was provided to interviewer). Edwards et al. (2009) reported no results related to interviewer factors. Leverage Salience Theory (Groves, Singer and Corning, 2000) posits that interviewers need to effectively tailor the ways they try to gain cooperation from individual respondents, including how they make use of the fact that an advance letter was sent. Groves and Snowden (1987) reasoned, but did not test, that knowing an advance letter was sent to a household makes the interviewer more confident when contacting that household. A knowledge gap clearly exists as to (a) how interviewers try to leverage (i.e., influence the respondent) the fact that an advance letter has been sent and (b) which leveraging approaches, if any, yield the most positive outcomes in raising survey response. Instead, from anecdotal experience, many survey organizations appear to assume that their interviewers will “naturally” know how to take advantage of the fact that an advance letter has been sent to the household and assume that interviewers do not need any specific training on this, beyond informing their interviewers of what is included in the advance mailer/letter. We report the findings of a full factorial experiment within the 2014 National Immunization Survey (NIS) and NIS-Teen that tested two factors in the first domain: (1) the color of the mailer containing the advance letter and (2) what was printed for the addressee in the first line of the address to which the mailer was sent. We hypothesized that these envelope-related features would make a difference in survey response across the different conditions that were tested. We conducted this study to learn (1) how to gain higher response rates and cost-efficiencies in the NIS surveys and (2) whether modifying these two features of the advance letter envelope (#10 size) would achieve these goals. 1. METHODS 1.1 Background on the NIS and NIS-Teen The NIS is one of the largest dual-frame random-digit dialing (RDD) surveys in the world. It is sponsored by the US Centers for Disease Control and Prevention (CDC) and has been conducted each year since 1994 to gather data critical to understanding the level of childhood vaccination coverage throughout the United States. The primary purpose of the NIS is to provide state and national estimates of vaccination coverage of children 19–35 months old. Since 2006, the NIS-Teen has been conducted using a subset of the NIS sample to produce state and national estimates of vaccination coverage of adolescents 13–17 years old. Since 2005, the NIS and NIS-Teen have been conducted by NORC at the University of Chicago. The NIS and NIS-Teen currently sample households in all fifty states, the District of Columbia, and selected local areas. NIS and NIS-Teen data are collected in two phases: a telephone survey of parents and guardians of 19–35 month-old children and of parents and guardians of teenagers (13–17 years of age) is conducted, followed by a mail survey sent to the vaccination provider(s) of these children and teenagers to obtain their vaccination histories with consent from a parent or guardian. In the first phase, sampled landline and cell-phone numbers are sent through an automated process to remove a portion of the non-working and non-residential numbers prior to dialing. The remaining landline sample telephone numbers that could be matched to a mailing address are then sent an advance letter approximately ten days prior to dialing. The landline and cell-phone sample telephone numbers are then dialed, and telephone interviewers attempt to screen each contacted household to determine if an age-eligible child lives there. Once it is determined that a household contains an age-eligible child, interviewers try to obtain demographic information on the child/teenager, mother, and household, as well as some vaccination-related data about young children (19–35 months old) and teenagers (13–17 years old). The designated respondent in each eligible household is a parent or guardian of the age-eligible child or children who reside(s) in the household. In addition, the interviewer tries to gain the consent of the parent or guardian to contact the child’s or teenager’s vaccination providers and tries to obtain the providers’ contact information. If consent is granted, then in the second phase the vaccination providers are contacted to obtain the vaccination history of the child or teenager. 1.2 3 x 3 Factorial Experimental Design The experiment used a 3 x 3 full factorial design and was conducted from October 2014 to February 2015. The two factors that were tested in the experiment pertained to the standard (for the United States) “letter-sized” #10 envelope that was used to send an advance letter to the addresses that were matched to sampled landline telephone numbers. (Images of these envelopes [Figures 1–9] can be viewed via JSSAM’s Supplementary Materials online.) The following provides details of the conditions tested in the experiment. Color of the Envelope. Three conditions were tested for this factor: the current white envelope (control condition C1) a business “tan” envelope (treatment condition A1 a light-blue colored envelope (treatment condition B1) Apart from the color differences of the mailing envelopes, everything else about the envelopes was identical across the three different color conditions. Furthermore, all envelopes mailed in this experiment contained the exact same letter. It was hypothesized that the tan envelope (A1) and blue envelope (B1) would achieve higher levels of response than the white (control) envelope (C1). In the case of the tan envelope, it was reasoned that it would make the mailing appear to be more “official” and “professional” looking than the white control envelope and that this would increase response rates as it would help to convey a sense of importance to the NIS advance mailing (cf. Dillman, Smyth, and Christian, 2009; Edwards et al., 2009). In the case of the light blue envelope, it was reasoned that it would appeal more to parents and guardians than the control white envelope, as light blue is often thought of as a “baby color,” and that this would increase response rates. Addressee. The name of the recipient in the first line of the address to which the letter was mailed had three conditions tested: the actual name of the resident that was found by the address-to-phone number matching process (control condition C2), and if no name was found then “Current Resident” was used2 the word “Resident” (treatment condition A2), and the words “Parent/Guardian” (treatment condition B2). Apart from these differences in the first (top) line of the recipient-address field, everything else about the address field was held constant in the manner in which it was printed and formatted. As noted above, nothing was done to alter anything related to the letter that was enclosed in the envelope. None of the letters used a greeting for the recipient. Thus, there was no replication on the letter of what was written in the first line of the address on the envelope. It was hypothesized that “Resident” (A2) and “Parent/Guardian” (B2) as the addressee would achieve higher levels of response than the use of a name matched to the phone number (control, C2). In the case of “Resident,” it was reasoned that it would avoid the problem where the matched name was not the correct name for anyone in the household, thus causing the letter to be entirely disregarded by the current resident(s). In the case of “Parent/Guardian,” it was reasoned (a) that by avoiding the use of a name, it would also avoid the problem where the matched name was not the correct name for anyone in the household and (b) that it would have a special “personalized” appeal (cf. Dillman, Lesser, Mason, Carlson, Willits et al., 2007; Dillman et al., 2009; Edwards et al., 2009) to the specific target population of the NIS—i.e., parents and guardians of young children and teenagers. 1.3 Sampling Issues Each case in the Quarter 4 (Q4) 2014 NIS landline sample was randomly assigned to (1) one of the three envelope color conditions (white, tan, or blue) and (2) independently, to one of the three addressee conditions (matched name, “Resident,” “Parent/Guardian”). In this way, 1/9th of the cases were flagged to belong to each of the nine combinations for envelope color and addressee. After cases were assigned to one of the nine combinations, the flags were examined for orthogonality (i.e., independence) with respect to each other and to other variables on the sample control file. For operational reasons, the flagging was done on the entire landline sample at the time of sample selection for Q4/2014. Following standard NIS procedures, the sample was divided into replicates, and prior to each replicate being released for data collection, it was sent through Marketing System Group’s IDplus pre-screening process. Numbers not pre-screened out were subjected to the Targus (now part of Neustar) address-matching protocol. Because the initial sample had been randomly assigned to one of the nine experimental combinations, approximately 1/9th of the sample with matched addresses in each replicate fell into each of the nine experimental combinations.3 In total, 485,694 NIS advance letters were mailed in Q4/2014, with approximately 54,000 cases in each of the nine treatment combinations. 1.4 Dependent Measures There were four primary dependent variables of interest, each of which was a binary (0/1) variable: Screening: if the telephone number was successfully screened for eligibility Eligibility: if the telephone number was found to reach an age-eligible household Completion: if the telephone number led to a completed interview Consent: if the telephone number led to consent being given to contact the young child’s or teenager’s vaccination provider(s) 1.5. Additional Data About the Advance Letters During the NIS telephone screener, after first being told, “A letter from the Department of Health and Human Services describing the National Immunization Survey may have been sent to your home recently,” each parent or guardian that indicated there was a young child in the household was asked a series of Yes/No questions about the advance mailing.4 Those questions were worded as follows: Did your household receive this letter? (5,266 were asked this question) Did someone in your home open that letter? (1,285 were asked this question) Did someone in your home read the letter inside the envelope? (1,124 were asked this question) Do you recall anything that was written in that letter? (1,024 were asked this question) These questions were used to provide further insight into the “impact” the design of the envelope may have had on the household. The choice of these questions was based on the logic that to be effective in stimulating response from the designated respondent at the sampled telephone number, the advance letter envelope needed to be received at the household, opened by one of the residents of the household, and read by at least one of the residents. The last factor being tested was if the parent or guardian that was being interviewed would recall something about the letter. 1.6 NIS Outcome Rates for the Landline Sample Mailed an Advance Letter Of the 485,694 NIS sampled phone numbers that were matched to addresses and sent advance letters, 52.4 percent (n = 254,722) were resolved (i.e., it was determined whether or not the phone number was a working residential number). Of those that were resolved, 64.1 percent (n = 163,231) were working residential numbers. Of those working residential numbers, 92.8 percent (n = 151,526) completed the screener. Of those households that completed the screener, 1.2 percent (n = 1,755) were eligible for the NIS interview. Of those households that were identified as eligible, 83.5 percent (n = 1,466) completed the interview. Of those that completed the interview, 70.5 percent (n = 1,034) of the parents/guardians gave consent to contact their child’s vaccination provider. The overall response rate, AAPOR RR3 (AAPOR, 2016), is the number of completed interviews divided by the number of eligible units in the sample. Because at the end of data collection some landline phone numbers remained unresolved and some of the landline numbers that were resolved as households remained unscreened, to compute a response rate it was necessary to estimate the number of eligible units among the unresolved phone numbers and among the unscreened households; that is, it is necessary to choose values for e1 and e2 in the relevant AAPOR response rate formula: AAPORRR3ResponseRate=CompletedInterviewsIdentifiedEligibles+e1·e2·Unresolved+e2·Unscreened In this formula, e1 is the assumed rate of working residential numbers among the unresolved numbers, and e2 is the assumed eligibility rate among unscreened households. The NIS uses proportional allocation assumptions, i.e., the CASRO method (CASRO, 1982), to set e1 and e2. That is, e1 was set equal to the observed working residential number rate among resolved landline numbers (64.1 percent), and e2 was set equal to the observed eligibility rate among the screened households (1.2 percent) in the landline sample. Under these assumptions, the response rate can be written as the product of the resolution rate (52.4 percent above), the screener completion rate (92.8 percent), and the interview completion rate (83.5 percent), yielding an estimated response rate for those landline numbers that were sent advance letters of 40.6 percent. Because it is easier to resolve a non-working phone number than it is to resolve a working residential phone number, the proportional allocation (CASRO) method may assume too few working residential numbers among the unresolved numbers; thus, it may set too low a value for the e1 proportion. The most conservative approach to estimating e1would be to assume that all of the unresolved numbers are working residential numbers (i.e., e1=1); under this assumption, the estimated response rate for the landline sample that were mailed advance letters becomes 32.1 percent. It is highly unlikely that all of the unresolved numbers are working residential numbers, so this alternative is undoubtedly too low (i.e., overly pessimistic), but by using this much lower rate to anchor one end of the response rate range, the true response rate among those sent an advance letter is likely to be between 32 percent and 41 percent. 1.7 NIS-Teen Outcome Rates for the Landline Sample Mailed an Advance Letter Of the 485,694 landline phone numbers sampled for the NIS which were mailed an advance letter, 213,772 were part of the NIS-Teen sample. Of these, 52.6 percent (n = 112,368) were resolved (i.e., it was determined whether or not the phone number was a working residential number), and of those that were resolved, 63.7 percent (n = 71,562) were working residential numbers. Of those working residential numbers, 87.9 percent (n = 62,936) completed the NIS-Teen screener, and of those that completed the NIS-Teen screener, 5.9 percent (n = 3,713) were eligible for the NIS-Teen interview. Of those households that were identified as eligible for the NIS-Teen interview, 84.5 percent (n = 3,138) completed the interview, and of those that completed the NIS-Teen interview, 64.2 percent (n = 2,016) of the parents/guardians gave consent to contact their child’s vaccination provider. The CASRO response rate for the NIS-Teen landline sample cases mailed an advance letter was 39.1 percent. If we take a more conservative approach and assume all of the unresolved numbers are working residential numbers (i.e., e1=1), the estimated response rate for the NIS-Teen landline sample that were mailed advance letters becomes 30.8 percent, so the true NIS-Teen response rate among those sent an advance letter is likely to be between 31 percent and 39 percent. 1.8 Analytics As noted above, the process of identifying eligible households and obtaining parental or guardian consent to contact vaccination providers on the NIS and NIS-Teen can be broken down into the following steps: completing the eligibility screener; identifying eligible households; completing the interview; and obtaining consent to contact vaccination providers. To examine the impact of advance letter envelope color and addressee experimental treatments on the NIS and NIS-Teen, logistic regression was used to model conditional outcomes, i.e., each outcome measure as a function of the treatment variables among the set of cases eligible for that step in the NIS process. As such, we specifically examined the following: screener completion rate among all cases mailed an advance letter; eligibility rate among advance letter cases that completed the screener; interview completion rate among advance letter cases that were identified as eligible; and rate of consent to contact vaccination providers among advance letter cases that completed the interview. In this way, if there was a treatment effect on the outcome measures, we could identify the step(s) in the NIS process (screener completion, eligibility, interview completion, and/or consent) in which a treatment was having an impact. Therefore, when a statistically significant treatment effect was observed for a particular outcome, we examined the outcome rate (e.g., the screener yield rate, the eligibility rate among screener completes, the interview completion rate among identified eligible households, or the consent rate among completed interviews) for that step of the process for each envelope color and addressee combination in order to identify which treatment(s) impacted the outcome and estimate effect sizes. For each outcome measure, we fit a logistic regression model relating that outcome to the color and the addressee treatments assigned to the case. The independent variables that were controlled in the experiment were the main effect of the envelope color treatment, the main effect of the addressee treatment, and the interaction effect defined by the combination of the envelope color and addressee. The models were fit in SAS using proc logistic, with the three-level envelope color variable and the three-level addressee variable treated as classification variables using the “class” statement in the SAS procedure. 2. RESULTS 2.1 Asking Respondents about the Advance Letter As noted above as part of the NIS screener, respondents reporting that the household contained a young child were asked a series of questions about the advance letter. Parents and guardians first were asked if they were aware whether their household received the letter, and 25 percent replied affirmatively that their household had received it (1,295/5,266). Those who reported that their household received the advance letter were asked if anyone had opened the letter. Nearly nine in ten (88 percent) reported affirmatively that the letter had been opened (1,126/1,285). Those reporting that the letter had been opened were asked if anyone had read the contents of the letter. More than nine in ten (92 percent) reported affirmatively that the letter had been read (1,038/1,124). Finally, the parent or guardian was asked whether she or he recalled any of the contents of the letter, and seven in ten (70 percent) reported affirmatively that they did recall some of the contents (729/1,034). Of note, none of the experimental conditions (envelope colors or different addressees) were found to have any effect on the self-reported answers to the four advance letter questions, and the only demographic factor found to correlate with the awareness variables was the number of adults in the household. Here, as would be expected5, the greater the number of adults in the household, the less likely the respondent was to report awareness of the letter being received (p < 0.001); 33 percent of households with one or two adults in residence reported that the letter had been received versus 21 percent of households with three or more adults in residence. 2.2 Experimental Effects on NIS Survey Outcomes Table 4 shows the results from the logistic regression analyses that tested the two main effects and the interaction effect for each of the four dependent variables. There were no statistically significant main effects or interaction effects associated with three of the four outcomes: the screening of households for the presence of young children, completing questionnaires, or gaining consent to contact vaccination providers. For eligibility among screened households, there was a statistically significant main effect for the addressee factor (p < 0.03) whereby the use of “Parent/Guardian” as the addressee led to a higher rate of finding that an eligible child lived in the household compared to the other two addressees that were tested. There was a marginally significant interaction effect of address and color (p < 0.08), whereby the “Parent/Guardian” addressee coupled with the tan envelope showed the highest eligibility rate.6 Table 4. Logistic Regression Results (Main Effects and Interaction Effect) for Each Dependent Variable Wald Chi-Square Degrees of Freedom p SCREENED, among dialed n = 485,694 dialed  Main Effect for Envelope Color 2.89 2 >0.10  Main Effect for Addressee 0.38 2 >0.10  Interaction Effect of Color and Addressee 2.92 4 >0.10 ELIGIBLE, among screened n = 151,526 screened  Main Effect for Envelope Color 0.63 2 >0.10  Main Effect for Addressee 7.50 2 <0.03  Interaction Effect of Color and Addressee 8.44 4 <0.08 COMPLETE, among eligible n = 1,755 eligible  Main Effect for Envelope Color 0.39 2 >0.10  Main Effect for Addressee 1.18 2 >0.10  Interaction Effect of Color and Addressee 1.35 4 >0.10 CONSENT, among completed n = 1,466 completed  Main Effect for Envelope Color 1.72 2 >0.10  Main Effect for Addressee 0.22 2 >0.10  Interaction Effect of Color and Addressee 2.51 4 >0.10 Wald Chi-Square Degrees of Freedom p SCREENED, among dialed n = 485,694 dialed  Main Effect for Envelope Color 2.89 2 >0.10  Main Effect for Addressee 0.38 2 >0.10  Interaction Effect of Color and Addressee 2.92 4 >0.10 ELIGIBLE, among screened n = 151,526 screened  Main Effect for Envelope Color 0.63 2 >0.10  Main Effect for Addressee 7.50 2 <0.03  Interaction Effect of Color and Addressee 8.44 4 <0.08 COMPLETE, among eligible n = 1,755 eligible  Main Effect for Envelope Color 0.39 2 >0.10  Main Effect for Addressee 1.18 2 >0.10  Interaction Effect of Color and Addressee 1.35 4 >0.10 CONSENT, among completed n = 1,466 completed  Main Effect for Envelope Color 1.72 2 >0.10  Main Effect for Addressee 0.22 2 >0.10  Interaction Effect of Color and Addressee 2.51 4 >0.10 Table 4. Logistic Regression Results (Main Effects and Interaction Effect) for Each Dependent Variable Wald Chi-Square Degrees of Freedom p SCREENED, among dialed n = 485,694 dialed  Main Effect for Envelope Color 2.89 2 >0.10  Main Effect for Addressee 0.38 2 >0.10  Interaction Effect of Color and Addressee 2.92 4 >0.10 ELIGIBLE, among screened n = 151,526 screened  Main Effect for Envelope Color 0.63 2 >0.10  Main Effect for Addressee 7.50 2 <0.03  Interaction Effect of Color and Addressee 8.44 4 <0.08 COMPLETE, among eligible n = 1,755 eligible  Main Effect for Envelope Color 0.39 2 >0.10  Main Effect for Addressee 1.18 2 >0.10  Interaction Effect of Color and Addressee 1.35 4 >0.10 CONSENT, among completed n = 1,466 completed  Main Effect for Envelope Color 1.72 2 >0.10  Main Effect for Addressee 0.22 2 >0.10  Interaction Effect of Color and Addressee 2.51 4 >0.10 Wald Chi-Square Degrees of Freedom p SCREENED, among dialed n = 485,694 dialed  Main Effect for Envelope Color 2.89 2 >0.10  Main Effect for Addressee 0.38 2 >0.10  Interaction Effect of Color and Addressee 2.92 4 >0.10 ELIGIBLE, among screened n = 151,526 screened  Main Effect for Envelope Color 0.63 2 >0.10  Main Effect for Addressee 7.50 2 <0.03  Interaction Effect of Color and Addressee 8.44 4 <0.08 COMPLETE, among eligible n = 1,755 eligible  Main Effect for Envelope Color 0.39 2 >0.10  Main Effect for Addressee 1.18 2 >0.10  Interaction Effect of Color and Addressee 1.35 4 >0.10 CONSENT, among completed n = 1,466 completed  Main Effect for Envelope Color 1.72 2 >0.10  Main Effect for Addressee 0.22 2 >0.10  Interaction Effect of Color and Addressee 2.51 4 >0.10 Table 5 presents the counts and data collection outcome rates (percentages) for each of the nine treatment combinations. As shown in the table, the interaction effect associated with the color and addressee on eligibility is due to the tan envelope (condition A1) with “Parent/Guardian” (condition B2) as the addressee; this combination of treatments (A1B2) had an eligibility rate of 1.46 percent. This rate is 35.2 percent higher ((1.46/1.08) - 1= 0.352) than the eligibility rate of the control condition (C1C2) of 1.08 percent; i.e., the white envelope with matched addressee’s name. Thus, the eligibility rate for the tan envelope with “Parent/Guardian” in the addressee line was found to exceed the control condition by more than one-third. Because the eligibility rate was so much higher among households mailed a tan envelope addressed to “Parent/Guardian,” 25 percent less interviewer time was required to achieve a completion with an eligible household among those landline phone numbers mailed a tan envelope addressed to “Parent/Guardian” compared with those landline phone numbers mailed a white envelope addressed to the name of the person found for that address in the matching process.7 Table 5. Counts and Percentages of Data Collection Outcomes by Experimental Group Envelope Color Addressee Condition Number Dialed Screened, Among Dialed Eligible, Among Screened Complete, Among Eligible Consent, Among Complete White Name C1C2 53,926 16,897 183 152 113 31.3% 1.08% 83.1% 74.3% Resident C1A2 53,996 17,017 193 167 122 31.5% 1.13% 86.5% 73.1% Parent/Guardian C1B2 54,078 16,880 208 173 120 31.2% 1.23% 83.2% 69.4% Tan Name A1C2 53,604 16,792 181 148 101 31.3% 1.08% 81.8% 68.2% Resident A1A2 53,928 16,678 175 147 104 30.9% 1.05% 84.0% 70.7% Parent/Guardian A1B2 54,099 16,874 246 207 138 31.2% 1.46% 84.1% 66.7% Blue Name B1C2 54,111 16,833 185 151 104 31.1% 1.10% 81.6% 68.9% Resident B1A2 54,251 16,845 198 163 115 31.1% 1.18% 82.3% 70.6% Parent/Guardian B1B2 53,701 16,710 186 158 117 31.1% 1.11% 84.9% 74.1% Envelope Color Addressee Condition Number Dialed Screened, Among Dialed Eligible, Among Screened Complete, Among Eligible Consent, Among Complete White Name C1C2 53,926 16,897 183 152 113 31.3% 1.08% 83.1% 74.3% Resident C1A2 53,996 17,017 193 167 122 31.5% 1.13% 86.5% 73.1% Parent/Guardian C1B2 54,078 16,880 208 173 120 31.2% 1.23% 83.2% 69.4% Tan Name A1C2 53,604 16,792 181 148 101 31.3% 1.08% 81.8% 68.2% Resident A1A2 53,928 16,678 175 147 104 30.9% 1.05% 84.0% 70.7% Parent/Guardian A1B2 54,099 16,874 246 207 138 31.2% 1.46% 84.1% 66.7% Blue Name B1C2 54,111 16,833 185 151 104 31.1% 1.10% 81.6% 68.9% Resident B1A2 54,251 16,845 198 163 115 31.1% 1.18% 82.3% 70.6% Parent/Guardian B1B2 53,701 16,710 186 158 117 31.1% 1.11% 84.9% 74.1% Table 5. Counts and Percentages of Data Collection Outcomes by Experimental Group Envelope Color Addressee Condition Number Dialed Screened, Among Dialed Eligible, Among Screened Complete, Among Eligible Consent, Among Complete White Name C1C2 53,926 16,897 183 152 113 31.3% 1.08% 83.1% 74.3% Resident C1A2 53,996 17,017 193 167 122 31.5% 1.13% 86.5% 73.1% Parent/Guardian C1B2 54,078 16,880 208 173 120 31.2% 1.23% 83.2% 69.4% Tan Name A1C2 53,604 16,792 181 148 101 31.3% 1.08% 81.8% 68.2% Resident A1A2 53,928 16,678 175 147 104 30.9% 1.05% 84.0% 70.7% Parent/Guardian A1B2 54,099 16,874 246 207 138 31.2% 1.46% 84.1% 66.7% Blue Name B1C2 54,111 16,833 185 151 104 31.1% 1.10% 81.6% 68.9% Resident B1A2 54,251 16,845 198 163 115 31.1% 1.18% 82.3% 70.6% Parent/Guardian B1B2 53,701 16,710 186 158 117 31.1% 1.11% 84.9% 74.1% Envelope Color Addressee Condition Number Dialed Screened, Among Dialed Eligible, Among Screened Complete, Among Eligible Consent, Among Complete White Name C1C2 53,926 16,897 183 152 113 31.3% 1.08% 83.1% 74.3% Resident C1A2 53,996 17,017 193 167 122 31.5% 1.13% 86.5% 73.1% Parent/Guardian C1B2 54,078 16,880 208 173 120 31.2% 1.23% 83.2% 69.4% Tan Name A1C2 53,604 16,792 181 148 101 31.3% 1.08% 81.8% 68.2% Resident A1A2 53,928 16,678 175 147 104 30.9% 1.05% 84.0% 70.7% Parent/Guardian A1B2 54,099 16,874 246 207 138 31.2% 1.46% 84.1% 66.7% Blue Name B1C2 54,111 16,833 185 151 104 31.1% 1.10% 81.6% 68.9% Resident B1A2 54,251 16,845 198 163 115 31.1% 1.18% 82.3% 70.6% Parent/Guardian B1B2 53,701 16,710 186 158 117 31.1% 1.11% 84.9% 74.1% 2.3 Experimental Effects on NIS-Teen Survey Outcomes The same analyses were conducted for each of our four dependent variables as they related to (a) screening households for teenagers in residence, (b) finding households that contained eligible teenagers, (c) completing interviews about a teenage resident’s vaccination history, and (d) gaining consent to contact a teenager’s vaccination provider(s). However, unlike what was found for NIS outcomes, there were no significant or marginally significant effects associated with the envelope color or the addressee on any of the dependent variables for the NIS-Teen; in particular, eligibility rates for households that completed the NIS-Teen screener were not significantly different across the envelope color or addressee conditions. 3. DISCUSSION We found that the color of the envelope and the nature of the addressee can affect the rate at which sampled households that are mailed an advance letter are subsequently identified as being eligible for the telephone survey8. This in turn translated into proportionally more telephone interviews being completed while fielding the same amount of sample and, therefore, less interviewer time per completed interview. In particular, 25 percent fewer interviewer hours were spent per completed interview, and the observed eligibility rate was 35 percent higher among cases mailed a tan envelope addressed to “Parent/Guardian” (condition A1B2) when compared to cases mailed a white envelope addressed to the name that was matched to the address (i.e., our control condition, C1C2). These effect size estimates are very large, and in our opinion, it is unlikely that the true effect sizes are of these magnitudes. However, these results do suggest that the envelope features can impact the observed eligibility rate and cost per completed interview. Based on the findings from our study, starting in Quarter 4 of 2015, the advance letters for the NIS and NIS-Teen have been sent in tan envelopes addressed to “Parent/Guardian.” There has been a fair amount of past research conducted on the effects of advance letters on survey outcome rates (cf. de Leeuw et al., 2007). However, almost all of this experimentation has focused on the nature of the letter and on the contents of the envelope, such as a noncontingent incentive or a frequently asked questions insert. In contrast, little has been reported about the features of the envelope/mailer that conveys the advance letter and how those features may affect survey outcomes. What has been reported suggests that response rates may be raised by using (a) hand-written labels, (b) tan envelopes, (c) stamps, (d) first class postage, (e) registered or certified delivery, (f) a larger than letter-size envelope, and (g) university sponsorship for the mailers that are sent to sampled addresses (Edwards et al., 2009); although, those findings are for mail surveys in general and not specifically for advance letters used in a telephone survey. Nonetheless, unless an envelope is opened, its contents cannot have an effect on the intended recipient(s). Thus, we believe that it is prudent for survey researchers to give more consideration to what will motivate sampled respondents to open the advance letter that is mailed to them. We conducted a very large national experimental testing two envelope features that we reasoned would affect whether or not our advance letter mailing would be opened and read. The factors we tested were the nature of the addressee to whom the envelope was sent and the color of the envelope. Our findings support the notion that features associated with the envelope that is used for an advance letter can make a meaningful difference in whether or not that respondent subsequently completes the survey task when later contacted by an interviewer. The specific way that we operationalized our test of the addressee line was particular to the NIS and NIS-Teen, i.e., gathering information about children/teenagers from their parent or guardian. However, the change in the envelope did not have the same impact across both surveys. Our findings with parents and guardians of young children were not replicated by our findings with parents and guardians of teenagers, and this is discussed below in more detail. This suggests the need to tailor the addressee line to the particular study (cf. Dillman et al. 2007). For example, in a survey about satisfaction with recent technology purchases using a customer list as the sampling frame, one could envision advance letters addressed to “Recent Technology Purchaser.” Or for a survey about the political preferences of those without any party affiliation using a registered voters list for the sampling frame, one could envision an advance letter addressed to “Independent-minded Voter.” In contrast, for a survey that samples the general population without any screening, our findings about the nature of the addressee line on an advance letter envelope would not seem to have any relevance, although we could imagine that a researcher might consider using something such as “Concerned Citizen,” depending on the subject matter of the questionnaire. Recently, Lavrakas, Dirksz, Lusskin, and Ponce (2016) reported that addressing envelopes to “Residente Actual/Current Resident” raised the response rate in a mail survey of high-density Hispanic areas by 14 percent (p < 0.02) compared to using “Current Resident.” Our finding that the color of the advance letter envelope made a meaningful difference in survey outcomes suggests that this too should be something about which researchers think carefully. Using a non-white envelope for an advance contact mailing may have little or no cost implications for one’s survey budget. We suspect that in an age when the public receives so much junk mail, sending an advance letter in a more official-looking tan envelope may distinguish a mailing from typical junk mail that gets readily tossed without anyone opening it (cf. Dillman et al., 2009). As noted in Table 1, there are many envelope features that researchers could actively consider in designing the envelope used for sending an advance contact communication to make it more likely that it will be opened and its contents be read. In thinking further about the results we have reported, one might ask, “Why were the effects for parents and guardians of young children limited to the eligibility rate, and why were those results not observed for parent and guardians of teenagers?” The envelopes that were used in our experiment had a return address of “Department of Health and Human Services, Public Health Service, Centers for Disease Control, National Center for Health Statistics” and were marked with “OFFICIAL BUSINESS”9. We reason that these envelopes, when combined with a “Parent/Guardian” addressee, may have been more salient to a parent or guardian of a young child than to the parent of a teenager. We suspect that most parents and guardians are especially concerned about the health of their young children, as these children are generally most vulnerable to infectious diseases. If it is true that the parents and guardians of young children are more aware of and concerned about health issues related to their children than are parents of teenagers, then it may explain why an envelope from an official government health agency that is addressed to “Parent/Guardian” is more salient to the parents and guardians of young children than it is to the parents and guardians of teenagers. Our study was limited to households with landline telephone numbers that could be matched to a mailing address. Whether our results would apply to households with a cell phone number that might be matched to a mailing address is uncertain, but we have no reason to suspect that they would not. Furthermore, although the matching of addresses to cell phone numbers in the United States is now not a productive endeavor, the rate of successfully matching addresses to US cell phone numbers will likely continue to grow over time. Additionally, only respondents reporting the presence of a young child in the household were asked if they had received the letter, and therefore, we could not examine the relationship between reporting that the letter had been received and the screener completion rate or the eligibility rate among those that completed the screener. Moreover, only one-fourth (25 percent) of those respondents with a young child who were asked reported that their household had received the letter.10 There are at least five reasons that this would be expected to happen, and each deserves much more investigation by survey researchers who are thinking about using advance letters. First, an advance letter may never reach the intended recipient because the address is incorrect. This suggests that researchers need to know much more about the reliability of the addresses they use for their advance mailings, especially whenever they are not using an address-based sampling design. Second, an advance letter may be accurately addressed but never be delivered to that address (or be delivered too late to have any effect) by the delivery service that is being used. It is likely that poor delivery service is not a geographically random occurrence. Instead, it may be that certain “types” of locales (e.g., lower income versus upper income; rural versus suburban versus urban), for many reasons, are subject to different quality of mail delivery. If this is the case, and if an advance letter stimulates positive survey response (as the literature shows that it often does; de Leeuw et al. 2007), then differential nonresponse is likely resulting from the variation in the quality of the local mail delivery service. This is another factor that survey researchers need to understand better, because it is possible that using an advance letter under these conditions is counterproductive to efforts to reduce total error in a particular survey. Third, an advance letter may be delivered to the correct address but may never be opened. The research we have reported here speaks to this issue. However, many more factors associated with the envelope need to be studied using unconfounded experiments for researchers to be able to have more confidence in choosing design features for the envelopes that they use in their advance mailings (cf. Edwards et al., 2009). Fourth, an advance letter may be opened, but it may never be read or its contents may not effectively motivate the reader to be more likely to comply with the subsequent survey task. de Leeuw et al. (2007) demonstrated that there are many features associated with the contents of an advance letter that have meaningful positive effects on survey response. But in comparing our Table 2 with the features studied in the de Leeuw et al. meta-analysis, it is clear that there are a great many features of advance letter mailings that may affect survey response that have not been adequately tested. As such, a great deal of additional research of factors in this domain is called for. Fifth, someone at the household who reads a superbly crafted advance letter may not be the person who subsequently is contacted by an interviewer in a telephone or in-person survey or who subsequently receives and opens a questionnaire packet for a mail survey. Unless the person who was exposed to the advance letter informs others in her/his household about the contents of the advance letter and about the details of the forthcoming survey, the effort to create and send the advance letter may be entirely wasted. Related to this, we are unaware of any testing of the effect of explicitly asking the reader of an advance letter to actively share its content with others (e.g., all other adults) in her or his household. This is easy to test, and it deserves to be tested. The envelope features of the advance mailings in this study did not raise the cost of the mailings, and as shown in our findings, they had meaningful and cost-beneficial effects for a given survey. Despite the large amount of past experimentation into the effects of advance letters, there are many features of such mailings, including those related to the envelope and its delivery that the field of survey research has yet to test adequately. Supplementary Materials Supplementary materials are available online at https://academic.oup.com/jssam. Footnotes 1 We conducted an extensive search of the research literature on mailer/envelope effects from 2009 through 2016; in particular, we searched survey research methods archives and health research methods archives, including each issue of Public Opinion Quarterly and the American Journal of Epidemiology. We also searched all AAPOR conference programs during this time period. In the course of this search, we were surprised to find only one study that addressed envelope features and mail survey response rates (Dykema, Jaques, Cyffka, Assad, Hammers, et al., 2015) and none that addressed advance letters used with telephone surveys. 2 Two percent (2.24%) of the envelopes in this condition had “Current Resident” used as the addressee, whereas the other 97.76% had the name that was matched to the address as the addressee. 3 The address-match rates among the telephone numbers in each condition were very similar across the nine experimental groups, ranging from 40.8 percent to 41.1 percent, but they were not identical due to sample matching variation. 4 The NIS screener first asked respondents for the number of children age 12 months to 4 years in the household. Those that indicated the presence of such a child were then asked the series of questions about the advance mailing. Following these questions, the respondent was asked for the dates of birth of those children age 12 months to 4 years in the household to determine which, if any, were 19 to 35 months old, i.e., eligible for the NIS. In this way, the questions about the advance mailing were asked of all households containing a child age 12 months to 4 years (not just those eligible for the NIS), but were not asked of households that did not have a child age 12 months to 4 years. 5 This would be expected because the greater the number of adults in the household, the lower the probability that the parent or guardian being interviewed would have been the adult to have opened the letter or to have been told by another adult that a letter had been received informing the household of a forthcoming telephone survey about vaccinations of young children and teenagers. 6 A household either has eligible children or it does not, so the “Parent/Guardian” addressee cannot be causing more households to truly be eligible; however, the “Parent/Guardian” addressee line may have caused more parents and guardians with eligible children to (a) open the envelope, (b) decide to begin the questionnaire, and (c) be willing to report that an eligible child lived in their household, when compared with the case for the other two addressee line conditions, resulting in a higher observed eligibility rate for the “Parent/Guardian” treatment. 7 The interviewer time per completion was computed as the sum of the durations of all calls made to all cases in each treatment combination divided by the total number of interviews completed in that treatment combination. 8 Please note that the causal mechanisms at work that produced this observed outcome here are uncertain; please see footnote 6 for more discussion of this issue. 9 Images of the envelopes can be found at the JSSAM Supplementary Material website. 10 Because only those respondents reporting that a child between the ages of 12 months and 4 years lived in the household were asked whether they had received the letter, we do not know what percentage of households without a young child would have reported receiving the letter. References AAPOR ( 2016 ), Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys ( 9th ed. ). Deerfield, IL : American Association for Public Opinion Research . http://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf. Camburn D. , Lavrakas P. J. , Battaglia M. , Massey J. , Wright R. ( 1996 ), “Using Advance Respondent Letters in Random-Digit-Dialing Telephone Surveys,” American Statistical Association 1995 Proceedings: Section on Survey Research Methods , 969 – 974 . CASRO ( 1982 ), On the Definition of Response Rates. Available at, http://c.ymcdn.com/sites/www.casro.org/resource/resmgr/docs/casro_on_definitions_of_resp.pdf?hhSearchTerms=%22response+and+rates%22. de Leeuw E. , Callegaro M. , Hox J. , Korendijk E. , Lensvelt-Mulder G. ( 2007 ), “ The Influence of Advance Letters on Response in Telephone Surveys: A Meta-Analysis ,” Public Opinion Quarterly , 71 , 413 – 443 . Google Scholar CrossRef Search ADS Dillman D. A. , Lesser V. , Mason R. , Carlson J. , Willits F. , Robertson R. , Burke B. ( 2007 ), “ Personalization of Mail Surveys for General Public and Populations with a Group Identity: Results from Nine Studies ,” Rural Sociology , 72 , 632 – 646 . Google Scholar CrossRef Search ADS Dillman D. A. , Smyth J. D. , Christian L. M. ( 2009 ), Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method , ( 3rd ed.). Hoboken, NJ : John Wiley & Sons . Dykema J. , Jaques K. , Cyffka K. , Assad N. , Hammers R. G. , Elver K. , Malecki K. C. , Stevenson J. ( 2015 ), “ Effects of Sequential Prepaid Incentives and Envelope Messaging in Mail Surveys ,” Public Opinion Quarterly , 79 , 906 – 931 . Google Scholar CrossRef Search ADS Edwards P. J. , Roberts I. , Clarke M. J. , Diguiseppi C. , Wentz R. , Kwan I. , Cooper R. , Felix L. M. , Pratap S. ( 2009 ), “ Methods to Increase Response to Postal and Electronic Questionnaires ,” Cochrane Database of Systematic Reviews , 3 , Article Number MR000008. Groves R. M. , Singer E. , Corning A. ( 2000 ), “ Leverage Salience Theory of Survey Participation ,” Public Opinion Quarterly , 64 , 299 – 308 . Google Scholar CrossRef Search ADS PubMed Groves R. M. , Snowden C. ( 1987 ), “The Effects of Advance Letters on Response Rates in Linked Telephone Surveys.” Proceedings of the Section on Survey Research Methods of the American Statistical Association , pp. 633 – 638 . Lavrakas P. J. , Dirksz G. , Lusskin L. , Ponce B. ( 2016 ), “ Experimenting with the Addressee Line in a Mail Survey of Hispanic Households ,” Survey Practice , 9 , http://www.surveypractice.org/index.php/SurveyPractice/article/view/380/pdf_80. © The Author 2017. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Survey Statistics and Methodology Oxford University Press

Testing the Effects of Envelope Features on Survey Response in a Telephone Survey Advance Letter Mailing Experiment

Loading next page...
 
/lp/ou_press/testing-the-effects-of-envelope-features-on-survey-response-in-a-UHHG5D3Jvl
Publisher
Oxford University Press
Copyright
© The Author 2017. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please email: journals.permissions@oup.com
ISSN
2325-0984
eISSN
2325-0992
D.O.I.
10.1093/jssam/smx023
Publisher site
See Article on Publisher Site

Abstract

Abstract There has been considerable research in the past three decades about the value of sending an advance letter to respondents prior to making the formal request for cooperation in a survey. This literature has shown that advance letters have a positive effect on survey response rates, but the literature has almost entirely focused on what is enclosed inside the envelope. In contrast, little attention has been given to the importance of the envelope/mailer that is used. However, whatever is enclosed inside the envelope can have no impact on the recipient unless the envelope is opened. We report on a large national experiment that tested the effects of two aspects of the advance envelope of a survey that sampled telephone numbers and gathered data via telephone: (1) envelope color and (2) the first line of the addressee. Our study was conducted as part of the CDC’s 2014 National Immunization Survey (NIS). In the landline RDD sample, all the advance letters were randomly assigned to one of nine conditions using a 3 × 3 factorial design. This experiment tested the effects on the telephone survey response of whether the envelope was white, tan, or light blue and whether the envelope was addressed to the name of the person matched to the address, to “Resident,” or to “Parent/Guardian.” It was found that a tan envelope addressed to “Parent/Guardian” had a highly significant impact, achieving an approximately 35% higher eligibility rate compared to the control condition (white envelope addressed with the name that was matched to the address). Making envelope features more responsive to the potential participants did not raise the cost of the mailings, but it did have meaningful cost-beneficial effects for the NIS. There is research on the effects of advance contacts on survey response rates—in particular, the use of advance letters in landline telephone surveys where addresses can be matched to a large portion of phone numbers. But this research provides sparse information about how to deploy advance contact techniques with sampled households so as to achieve the most cost-effective outcomes (Groves and Snowden, 1987; Camburn, Lavrakas, Battaglia, Massey, and Wright, 1996; de Leeuw, Callegaro, Hox, Korendijk, and Lensvelt-Mulder, 2007; Edwards, Roberts, Clarke, Diguiseppi, Wentz, et al., 2009). In reviewing this literature, there appear to be three domains of factors associated with the advance mailings that could be used to improve survey response: Factors related to the likelihood that the advance mailer will be opened. These factors are related to the envelope and the mailing service used to deliver it, such as the postage used on the envelope, the sender that is listed in the return address, graphics on the envelope, or the size of the envelope. Table 1 shows factors that fall into this domain. Table 1. Factors Related to the Likelihood the Advance Mailer will be Opened Factors Examples Type of delivery service/class USPS 1st Class, USPS Priority Mail, Fed-Ex, UPS Type of mailer Padded envelope, box, transparent window allowing some contexts to be seen Mailer size Small letter, standard letter, 6" x 9", 9" x 12" Mailer color White, business tan, pastel Addressee on mailer Name of resident, current resident, parent, head of household Return address of sender University, government agency, survey organization, corporate sponsor Graphics on mailer Cartoon art, scenic image, celebrity picture Branding logo on mailer Organizational branding, branding of the survey Topical phrasing/teaser on mailer “Childhood vaccinations,” “Crime prevention awareness,” “Educational attainment,” “U.S. foreign policy” Endorsement on mailer Elected official, famous athlete, music celebrity, television celebrity Font style Times Roman, Arial; Courier, Verdana Font size 10, 11, 12, 14 Factors Examples Type of delivery service/class USPS 1st Class, USPS Priority Mail, Fed-Ex, UPS Type of mailer Padded envelope, box, transparent window allowing some contexts to be seen Mailer size Small letter, standard letter, 6" x 9", 9" x 12" Mailer color White, business tan, pastel Addressee on mailer Name of resident, current resident, parent, head of household Return address of sender University, government agency, survey organization, corporate sponsor Graphics on mailer Cartoon art, scenic image, celebrity picture Branding logo on mailer Organizational branding, branding of the survey Topical phrasing/teaser on mailer “Childhood vaccinations,” “Crime prevention awareness,” “Educational attainment,” “U.S. foreign policy” Endorsement on mailer Elected official, famous athlete, music celebrity, television celebrity Font style Times Roman, Arial; Courier, Verdana Font size 10, 11, 12, 14 Table 1. Factors Related to the Likelihood the Advance Mailer will be Opened Factors Examples Type of delivery service/class USPS 1st Class, USPS Priority Mail, Fed-Ex, UPS Type of mailer Padded envelope, box, transparent window allowing some contexts to be seen Mailer size Small letter, standard letter, 6" x 9", 9" x 12" Mailer color White, business tan, pastel Addressee on mailer Name of resident, current resident, parent, head of household Return address of sender University, government agency, survey organization, corporate sponsor Graphics on mailer Cartoon art, scenic image, celebrity picture Branding logo on mailer Organizational branding, branding of the survey Topical phrasing/teaser on mailer “Childhood vaccinations,” “Crime prevention awareness,” “Educational attainment,” “U.S. foreign policy” Endorsement on mailer Elected official, famous athlete, music celebrity, television celebrity Font style Times Roman, Arial; Courier, Verdana Font size 10, 11, 12, 14 Factors Examples Type of delivery service/class USPS 1st Class, USPS Priority Mail, Fed-Ex, UPS Type of mailer Padded envelope, box, transparent window allowing some contexts to be seen Mailer size Small letter, standard letter, 6" x 9", 9" x 12" Mailer color White, business tan, pastel Addressee on mailer Name of resident, current resident, parent, head of household Return address of sender University, government agency, survey organization, corporate sponsor Graphics on mailer Cartoon art, scenic image, celebrity picture Branding logo on mailer Organizational branding, branding of the survey Topical phrasing/teaser on mailer “Childhood vaccinations,” “Crime prevention awareness,” “Educational attainment,” “U.S. foreign policy” Endorsement on mailer Elected official, famous athlete, music celebrity, television celebrity Font style Times Roman, Arial; Courier, Verdana Font size 10, 11, 12, 14 Factors related to the likelihood that the contents of the mailer will be read and recalled. These factors are related to the substance and formatting of the letter itself (e.g., the information conveyed in the letter about the purpose and confidentiality of the study, the letterhead on which the letter is printed, font size, type and color of the text, and the credentials of the sender and signatory) and what else may be included inside the envelope (e.g., a noncontingent incentive, a brochure, or an FAQ insert). Table 2 shows factors that fall into this domain. Table 2. Factors Related to the Likelihood the Contents of the Mailer will Be Read and Recalled Factors Examples Letterhead University, government, corporate Branding logo on page Organizational branding, branding of the survey Endorsement on page Elected official, famous athlete, music celebrity, television celebrity, subject matter expert Font style of text Times-Roman, Arial; Courier, Verdana Font size of text 10, 11, 12, 14 Salutation/Greetings None, generic, personalized Reading level of text 12th grade, 8th grade, 5th grade Length of text More than four paragraphs or more than 300 words, 3–4 paragraphs, two paragraphs, one paragraph Text about benefits to respondent Enclosed noncontingent incentive, contingent incentive, altruism, enjoyment Text about the sponsor Name of sponsor, history of sponsor, stature of sponsor, sector of sponsor Text about the purpose of study Societal or other benefits that will be result from the study, e.g., better health, lower crime, higher employment, cleaner environment Text about eligibility criteria Age, education, health status, low income, employment status Text about questionnaire length and respondent burden Estimated time for completion, other burdens (e.g., information look-ups) Text about privacy protections Promised anonymity, promised confidentiality, survey organization’s history of keeping data secure Text about contact info Survey organization’s website, email, phone number, mailing address Text about sampling How addresses, phone numbers, names were selected, why form of selection was used Text about data collection mode(s) In-person, telephone, mail, internet, mixed-mode data collection Text requesting that others in household be told of study Request person reading letter to tell everyone in household or tell all adults Nature of the signature Pre-printed, hand-signed, blue ink, black ink, other colored ink Signatory’s affiliation University, government, corporate, more than one Credentials of signatory MD, MA, PhD, LLD Sex of signatory Female, Male, Ambiguous Ethnicity of signatory Asian, African, Middle Eastern, Hispanic, East European, Scandinavian Language of text English, Spanish, both Use of informational inserts FAQs, organizational brochure, historical information about study Factors Examples Letterhead University, government, corporate Branding logo on page Organizational branding, branding of the survey Endorsement on page Elected official, famous athlete, music celebrity, television celebrity, subject matter expert Font style of text Times-Roman, Arial; Courier, Verdana Font size of text 10, 11, 12, 14 Salutation/Greetings None, generic, personalized Reading level of text 12th grade, 8th grade, 5th grade Length of text More than four paragraphs or more than 300 words, 3–4 paragraphs, two paragraphs, one paragraph Text about benefits to respondent Enclosed noncontingent incentive, contingent incentive, altruism, enjoyment Text about the sponsor Name of sponsor, history of sponsor, stature of sponsor, sector of sponsor Text about the purpose of study Societal or other benefits that will be result from the study, e.g., better health, lower crime, higher employment, cleaner environment Text about eligibility criteria Age, education, health status, low income, employment status Text about questionnaire length and respondent burden Estimated time for completion, other burdens (e.g., information look-ups) Text about privacy protections Promised anonymity, promised confidentiality, survey organization’s history of keeping data secure Text about contact info Survey organization’s website, email, phone number, mailing address Text about sampling How addresses, phone numbers, names were selected, why form of selection was used Text about data collection mode(s) In-person, telephone, mail, internet, mixed-mode data collection Text requesting that others in household be told of study Request person reading letter to tell everyone in household or tell all adults Nature of the signature Pre-printed, hand-signed, blue ink, black ink, other colored ink Signatory’s affiliation University, government, corporate, more than one Credentials of signatory MD, MA, PhD, LLD Sex of signatory Female, Male, Ambiguous Ethnicity of signatory Asian, African, Middle Eastern, Hispanic, East European, Scandinavian Language of text English, Spanish, both Use of informational inserts FAQs, organizational brochure, historical information about study Table 2. Factors Related to the Likelihood the Contents of the Mailer will Be Read and Recalled Factors Examples Letterhead University, government, corporate Branding logo on page Organizational branding, branding of the survey Endorsement on page Elected official, famous athlete, music celebrity, television celebrity, subject matter expert Font style of text Times-Roman, Arial; Courier, Verdana Font size of text 10, 11, 12, 14 Salutation/Greetings None, generic, personalized Reading level of text 12th grade, 8th grade, 5th grade Length of text More than four paragraphs or more than 300 words, 3–4 paragraphs, two paragraphs, one paragraph Text about benefits to respondent Enclosed noncontingent incentive, contingent incentive, altruism, enjoyment Text about the sponsor Name of sponsor, history of sponsor, stature of sponsor, sector of sponsor Text about the purpose of study Societal or other benefits that will be result from the study, e.g., better health, lower crime, higher employment, cleaner environment Text about eligibility criteria Age, education, health status, low income, employment status Text about questionnaire length and respondent burden Estimated time for completion, other burdens (e.g., information look-ups) Text about privacy protections Promised anonymity, promised confidentiality, survey organization’s history of keeping data secure Text about contact info Survey organization’s website, email, phone number, mailing address Text about sampling How addresses, phone numbers, names were selected, why form of selection was used Text about data collection mode(s) In-person, telephone, mail, internet, mixed-mode data collection Text requesting that others in household be told of study Request person reading letter to tell everyone in household or tell all adults Nature of the signature Pre-printed, hand-signed, blue ink, black ink, other colored ink Signatory’s affiliation University, government, corporate, more than one Credentials of signatory MD, MA, PhD, LLD Sex of signatory Female, Male, Ambiguous Ethnicity of signatory Asian, African, Middle Eastern, Hispanic, East European, Scandinavian Language of text English, Spanish, both Use of informational inserts FAQs, organizational brochure, historical information about study Factors Examples Letterhead University, government, corporate Branding logo on page Organizational branding, branding of the survey Endorsement on page Elected official, famous athlete, music celebrity, television celebrity, subject matter expert Font style of text Times-Roman, Arial; Courier, Verdana Font size of text 10, 11, 12, 14 Salutation/Greetings None, generic, personalized Reading level of text 12th grade, 8th grade, 5th grade Length of text More than four paragraphs or more than 300 words, 3–4 paragraphs, two paragraphs, one paragraph Text about benefits to respondent Enclosed noncontingent incentive, contingent incentive, altruism, enjoyment Text about the sponsor Name of sponsor, history of sponsor, stature of sponsor, sector of sponsor Text about the purpose of study Societal or other benefits that will be result from the study, e.g., better health, lower crime, higher employment, cleaner environment Text about eligibility criteria Age, education, health status, low income, employment status Text about questionnaire length and respondent burden Estimated time for completion, other burdens (e.g., information look-ups) Text about privacy protections Promised anonymity, promised confidentiality, survey organization’s history of keeping data secure Text about contact info Survey organization’s website, email, phone number, mailing address Text about sampling How addresses, phone numbers, names were selected, why form of selection was used Text about data collection mode(s) In-person, telephone, mail, internet, mixed-mode data collection Text requesting that others in household be told of study Request person reading letter to tell everyone in household or tell all adults Nature of the signature Pre-printed, hand-signed, blue ink, black ink, other colored ink Signatory’s affiliation University, government, corporate, more than one Credentials of signatory MD, MA, PhD, LLD Sex of signatory Female, Male, Ambiguous Ethnicity of signatory Asian, African, Middle Eastern, Hispanic, East European, Scandinavian Language of text English, Spanish, both Use of informational inserts FAQs, organizational brochure, historical information about study Interviewerfactorsrelated to the likelihood of respondent cooperation. In interviewer-administered surveys, factors are related to the interviewers who try to contact the households at addresses to which the letters were mailed and then try to gain cooperation from the households, such as the training that interviewers received about the importance of the letter or whether the introduction of the survey explicitly mentions an advance letter being sent. Table 3 shows factors that fall into this domain. Table 3. Interviewer Factors Related to the Likelihood of Respondent Cooperation Factors Examples Text in the intro of the CATI/CAPI script about the letter being sent Explicit mention in script that is required for interviewers to provide; mention in script that is optional for interviewers to provide Special interviewer training for how best to leverage the letter History of the use of survey advance letters; examples of how to mention the letter at various places in the interview; practice mentioning the letter Timing of the arrival of letter compared to when interviewer contact is made Interviewers’ finesse in deciding how salient the letter likely will be to the recipient depending on the number of days the envelope arrived prior to contact for data collection, which will vary throughout the field period Questionnaire items about the letter Did household receive letter, did anyone read letter, did everyone in the household get told about letter, does respondent remember anything about letter Valence that interviewers place on the letter that was sent Positive, neutral, negative Factors Examples Text in the intro of the CATI/CAPI script about the letter being sent Explicit mention in script that is required for interviewers to provide; mention in script that is optional for interviewers to provide Special interviewer training for how best to leverage the letter History of the use of survey advance letters; examples of how to mention the letter at various places in the interview; practice mentioning the letter Timing of the arrival of letter compared to when interviewer contact is made Interviewers’ finesse in deciding how salient the letter likely will be to the recipient depending on the number of days the envelope arrived prior to contact for data collection, which will vary throughout the field period Questionnaire items about the letter Did household receive letter, did anyone read letter, did everyone in the household get told about letter, does respondent remember anything about letter Valence that interviewers place on the letter that was sent Positive, neutral, negative Table 3. Interviewer Factors Related to the Likelihood of Respondent Cooperation Factors Examples Text in the intro of the CATI/CAPI script about the letter being sent Explicit mention in script that is required for interviewers to provide; mention in script that is optional for interviewers to provide Special interviewer training for how best to leverage the letter History of the use of survey advance letters; examples of how to mention the letter at various places in the interview; practice mentioning the letter Timing of the arrival of letter compared to when interviewer contact is made Interviewers’ finesse in deciding how salient the letter likely will be to the recipient depending on the number of days the envelope arrived prior to contact for data collection, which will vary throughout the field period Questionnaire items about the letter Did household receive letter, did anyone read letter, did everyone in the household get told about letter, does respondent remember anything about letter Valence that interviewers place on the letter that was sent Positive, neutral, negative Factors Examples Text in the intro of the CATI/CAPI script about the letter being sent Explicit mention in script that is required for interviewers to provide; mention in script that is optional for interviewers to provide Special interviewer training for how best to leverage the letter History of the use of survey advance letters; examples of how to mention the letter at various places in the interview; practice mentioning the letter Timing of the arrival of letter compared to when interviewer contact is made Interviewers’ finesse in deciding how salient the letter likely will be to the recipient depending on the number of days the envelope arrived prior to contact for data collection, which will vary throughout the field period Questionnaire items about the letter Did household receive letter, did anyone read letter, did everyone in the household get told about letter, does respondent remember anything about letter Valence that interviewers place on the letter that was sent Positive, neutral, negative The first domain of factors (Table 1) is important because the contents of the letter (including the possibility of finding a noncontingent incentive inside the envelope) cannot have any effect on survey response unless the mailer is opened. Thus, the first goals in trying to successfully deploy advance letters are (a) to get the envelope delivered to the correct address and (b) to get people to open the envelope that is delivered to them. Relatively little has been reported about factors that are related to the envelope and the mail delivery service used to deliver it and how these factors affect the propensity of a recipient opening it. This dearth of knowledge is readily apparent when one considers what de Leeuw et al. (2007) did not report in their otherwise extensive meta-evaluation of the effects of advanced letters (i.e., they included no factors from this envelope-domain in their meta-evaluation). In addition, the dearth in research about envelope factors associated with advance letters that are used in telephone surveys is also apparent from an extremely comprehensive review of the entire mail survey methodological literature conducted by Edwards et al. (2009); (this review was not limited to methodological research on advance letters in surveys). These scholars identified 481 published experiments of mail survey factors through 2008.1 Twelve percent of these studies (n = 57) addressed factors associated with the mailer that was used. Furthermore, Edwards et al. (2009) identified 110 separate factors that were tested in this body of literature, and only ten of the factors were associated with the mailer that was used. These mailer-related factors in mail surveying included using (a) hand-written labels, (b) tan envelopes, (c) stamps, (d) first class postage, (e) registered or certified delivery, (f) a larger than letter-size envelope, and (g) university sponsorship. All of these factors were found to be associated with a higher response rate in mail surveys. However, all the studies that assessed these factors in the Edwards et al. (2009) review focused on mail survey response, not telephone survey response. Although it is plausible that factors associated with higher mail survey response rates would also be associated with more effective advance letter mailings for telephone surveys, none of the studies in the Edwards et al. (2009) review looked at the impact of these factors on response to telephone surveys. The factors in the second domain (Table 2) make up the preponderance of what past research on advance letters effects has tested, i.e., most studies have tested the content that was conveyed in the letter, the reading level at which that content was written, various aspects of the formatting of the letter, and/or what else was included inside the envelope besides the letter. de Leeuw et al. (2007) reported average effects of eleven percentage-point gains in cooperation, and eight percentage-point gains in response rates, linked primarily to the content of what the advance letter conveyed, including the likelihood that the advance letter served to legitimize the study. The third domain (Table 3) concerns how the interviewers actually “use” to their advantage the fact that an advance letter was mailed, and this has remained essentially untested. For example, de Leeuw et al. (2007) included only a gross factor from this domain in their meta-evaluation (i.e., whether some “special training” was provided to interviewer). Edwards et al. (2009) reported no results related to interviewer factors. Leverage Salience Theory (Groves, Singer and Corning, 2000) posits that interviewers need to effectively tailor the ways they try to gain cooperation from individual respondents, including how they make use of the fact that an advance letter was sent. Groves and Snowden (1987) reasoned, but did not test, that knowing an advance letter was sent to a household makes the interviewer more confident when contacting that household. A knowledge gap clearly exists as to (a) how interviewers try to leverage (i.e., influence the respondent) the fact that an advance letter has been sent and (b) which leveraging approaches, if any, yield the most positive outcomes in raising survey response. Instead, from anecdotal experience, many survey organizations appear to assume that their interviewers will “naturally” know how to take advantage of the fact that an advance letter has been sent to the household and assume that interviewers do not need any specific training on this, beyond informing their interviewers of what is included in the advance mailer/letter. We report the findings of a full factorial experiment within the 2014 National Immunization Survey (NIS) and NIS-Teen that tested two factors in the first domain: (1) the color of the mailer containing the advance letter and (2) what was printed for the addressee in the first line of the address to which the mailer was sent. We hypothesized that these envelope-related features would make a difference in survey response across the different conditions that were tested. We conducted this study to learn (1) how to gain higher response rates and cost-efficiencies in the NIS surveys and (2) whether modifying these two features of the advance letter envelope (#10 size) would achieve these goals. 1. METHODS 1.1 Background on the NIS and NIS-Teen The NIS is one of the largest dual-frame random-digit dialing (RDD) surveys in the world. It is sponsored by the US Centers for Disease Control and Prevention (CDC) and has been conducted each year since 1994 to gather data critical to understanding the level of childhood vaccination coverage throughout the United States. The primary purpose of the NIS is to provide state and national estimates of vaccination coverage of children 19–35 months old. Since 2006, the NIS-Teen has been conducted using a subset of the NIS sample to produce state and national estimates of vaccination coverage of adolescents 13–17 years old. Since 2005, the NIS and NIS-Teen have been conducted by NORC at the University of Chicago. The NIS and NIS-Teen currently sample households in all fifty states, the District of Columbia, and selected local areas. NIS and NIS-Teen data are collected in two phases: a telephone survey of parents and guardians of 19–35 month-old children and of parents and guardians of teenagers (13–17 years of age) is conducted, followed by a mail survey sent to the vaccination provider(s) of these children and teenagers to obtain their vaccination histories with consent from a parent or guardian. In the first phase, sampled landline and cell-phone numbers are sent through an automated process to remove a portion of the non-working and non-residential numbers prior to dialing. The remaining landline sample telephone numbers that could be matched to a mailing address are then sent an advance letter approximately ten days prior to dialing. The landline and cell-phone sample telephone numbers are then dialed, and telephone interviewers attempt to screen each contacted household to determine if an age-eligible child lives there. Once it is determined that a household contains an age-eligible child, interviewers try to obtain demographic information on the child/teenager, mother, and household, as well as some vaccination-related data about young children (19–35 months old) and teenagers (13–17 years old). The designated respondent in each eligible household is a parent or guardian of the age-eligible child or children who reside(s) in the household. In addition, the interviewer tries to gain the consent of the parent or guardian to contact the child’s or teenager’s vaccination providers and tries to obtain the providers’ contact information. If consent is granted, then in the second phase the vaccination providers are contacted to obtain the vaccination history of the child or teenager. 1.2 3 x 3 Factorial Experimental Design The experiment used a 3 x 3 full factorial design and was conducted from October 2014 to February 2015. The two factors that were tested in the experiment pertained to the standard (for the United States) “letter-sized” #10 envelope that was used to send an advance letter to the addresses that were matched to sampled landline telephone numbers. (Images of these envelopes [Figures 1–9] can be viewed via JSSAM’s Supplementary Materials online.) The following provides details of the conditions tested in the experiment. Color of the Envelope. Three conditions were tested for this factor: the current white envelope (control condition C1) a business “tan” envelope (treatment condition A1 a light-blue colored envelope (treatment condition B1) Apart from the color differences of the mailing envelopes, everything else about the envelopes was identical across the three different color conditions. Furthermore, all envelopes mailed in this experiment contained the exact same letter. It was hypothesized that the tan envelope (A1) and blue envelope (B1) would achieve higher levels of response than the white (control) envelope (C1). In the case of the tan envelope, it was reasoned that it would make the mailing appear to be more “official” and “professional” looking than the white control envelope and that this would increase response rates as it would help to convey a sense of importance to the NIS advance mailing (cf. Dillman, Smyth, and Christian, 2009; Edwards et al., 2009). In the case of the light blue envelope, it was reasoned that it would appeal more to parents and guardians than the control white envelope, as light blue is often thought of as a “baby color,” and that this would increase response rates. Addressee. The name of the recipient in the first line of the address to which the letter was mailed had three conditions tested: the actual name of the resident that was found by the address-to-phone number matching process (control condition C2), and if no name was found then “Current Resident” was used2 the word “Resident” (treatment condition A2), and the words “Parent/Guardian” (treatment condition B2). Apart from these differences in the first (top) line of the recipient-address field, everything else about the address field was held constant in the manner in which it was printed and formatted. As noted above, nothing was done to alter anything related to the letter that was enclosed in the envelope. None of the letters used a greeting for the recipient. Thus, there was no replication on the letter of what was written in the first line of the address on the envelope. It was hypothesized that “Resident” (A2) and “Parent/Guardian” (B2) as the addressee would achieve higher levels of response than the use of a name matched to the phone number (control, C2). In the case of “Resident,” it was reasoned that it would avoid the problem where the matched name was not the correct name for anyone in the household, thus causing the letter to be entirely disregarded by the current resident(s). In the case of “Parent/Guardian,” it was reasoned (a) that by avoiding the use of a name, it would also avoid the problem where the matched name was not the correct name for anyone in the household and (b) that it would have a special “personalized” appeal (cf. Dillman, Lesser, Mason, Carlson, Willits et al., 2007; Dillman et al., 2009; Edwards et al., 2009) to the specific target population of the NIS—i.e., parents and guardians of young children and teenagers. 1.3 Sampling Issues Each case in the Quarter 4 (Q4) 2014 NIS landline sample was randomly assigned to (1) one of the three envelope color conditions (white, tan, or blue) and (2) independently, to one of the three addressee conditions (matched name, “Resident,” “Parent/Guardian”). In this way, 1/9th of the cases were flagged to belong to each of the nine combinations for envelope color and addressee. After cases were assigned to one of the nine combinations, the flags were examined for orthogonality (i.e., independence) with respect to each other and to other variables on the sample control file. For operational reasons, the flagging was done on the entire landline sample at the time of sample selection for Q4/2014. Following standard NIS procedures, the sample was divided into replicates, and prior to each replicate being released for data collection, it was sent through Marketing System Group’s IDplus pre-screening process. Numbers not pre-screened out were subjected to the Targus (now part of Neustar) address-matching protocol. Because the initial sample had been randomly assigned to one of the nine experimental combinations, approximately 1/9th of the sample with matched addresses in each replicate fell into each of the nine experimental combinations.3 In total, 485,694 NIS advance letters were mailed in Q4/2014, with approximately 54,000 cases in each of the nine treatment combinations. 1.4 Dependent Measures There were four primary dependent variables of interest, each of which was a binary (0/1) variable: Screening: if the telephone number was successfully screened for eligibility Eligibility: if the telephone number was found to reach an age-eligible household Completion: if the telephone number led to a completed interview Consent: if the telephone number led to consent being given to contact the young child’s or teenager’s vaccination provider(s) 1.5. Additional Data About the Advance Letters During the NIS telephone screener, after first being told, “A letter from the Department of Health and Human Services describing the National Immunization Survey may have been sent to your home recently,” each parent or guardian that indicated there was a young child in the household was asked a series of Yes/No questions about the advance mailing.4 Those questions were worded as follows: Did your household receive this letter? (5,266 were asked this question) Did someone in your home open that letter? (1,285 were asked this question) Did someone in your home read the letter inside the envelope? (1,124 were asked this question) Do you recall anything that was written in that letter? (1,024 were asked this question) These questions were used to provide further insight into the “impact” the design of the envelope may have had on the household. The choice of these questions was based on the logic that to be effective in stimulating response from the designated respondent at the sampled telephone number, the advance letter envelope needed to be received at the household, opened by one of the residents of the household, and read by at least one of the residents. The last factor being tested was if the parent or guardian that was being interviewed would recall something about the letter. 1.6 NIS Outcome Rates for the Landline Sample Mailed an Advance Letter Of the 485,694 NIS sampled phone numbers that were matched to addresses and sent advance letters, 52.4 percent (n = 254,722) were resolved (i.e., it was determined whether or not the phone number was a working residential number). Of those that were resolved, 64.1 percent (n = 163,231) were working residential numbers. Of those working residential numbers, 92.8 percent (n = 151,526) completed the screener. Of those households that completed the screener, 1.2 percent (n = 1,755) were eligible for the NIS interview. Of those households that were identified as eligible, 83.5 percent (n = 1,466) completed the interview. Of those that completed the interview, 70.5 percent (n = 1,034) of the parents/guardians gave consent to contact their child’s vaccination provider. The overall response rate, AAPOR RR3 (AAPOR, 2016), is the number of completed interviews divided by the number of eligible units in the sample. Because at the end of data collection some landline phone numbers remained unresolved and some of the landline numbers that were resolved as households remained unscreened, to compute a response rate it was necessary to estimate the number of eligible units among the unresolved phone numbers and among the unscreened households; that is, it is necessary to choose values for e1 and e2 in the relevant AAPOR response rate formula: AAPORRR3ResponseRate=CompletedInterviewsIdentifiedEligibles+e1·e2·Unresolved+e2·Unscreened In this formula, e1 is the assumed rate of working residential numbers among the unresolved numbers, and e2 is the assumed eligibility rate among unscreened households. The NIS uses proportional allocation assumptions, i.e., the CASRO method (CASRO, 1982), to set e1 and e2. That is, e1 was set equal to the observed working residential number rate among resolved landline numbers (64.1 percent), and e2 was set equal to the observed eligibility rate among the screened households (1.2 percent) in the landline sample. Under these assumptions, the response rate can be written as the product of the resolution rate (52.4 percent above), the screener completion rate (92.8 percent), and the interview completion rate (83.5 percent), yielding an estimated response rate for those landline numbers that were sent advance letters of 40.6 percent. Because it is easier to resolve a non-working phone number than it is to resolve a working residential phone number, the proportional allocation (CASRO) method may assume too few working residential numbers among the unresolved numbers; thus, it may set too low a value for the e1 proportion. The most conservative approach to estimating e1would be to assume that all of the unresolved numbers are working residential numbers (i.e., e1=1); under this assumption, the estimated response rate for the landline sample that were mailed advance letters becomes 32.1 percent. It is highly unlikely that all of the unresolved numbers are working residential numbers, so this alternative is undoubtedly too low (i.e., overly pessimistic), but by using this much lower rate to anchor one end of the response rate range, the true response rate among those sent an advance letter is likely to be between 32 percent and 41 percent. 1.7 NIS-Teen Outcome Rates for the Landline Sample Mailed an Advance Letter Of the 485,694 landline phone numbers sampled for the NIS which were mailed an advance letter, 213,772 were part of the NIS-Teen sample. Of these, 52.6 percent (n = 112,368) were resolved (i.e., it was determined whether or not the phone number was a working residential number), and of those that were resolved, 63.7 percent (n = 71,562) were working residential numbers. Of those working residential numbers, 87.9 percent (n = 62,936) completed the NIS-Teen screener, and of those that completed the NIS-Teen screener, 5.9 percent (n = 3,713) were eligible for the NIS-Teen interview. Of those households that were identified as eligible for the NIS-Teen interview, 84.5 percent (n = 3,138) completed the interview, and of those that completed the NIS-Teen interview, 64.2 percent (n = 2,016) of the parents/guardians gave consent to contact their child’s vaccination provider. The CASRO response rate for the NIS-Teen landline sample cases mailed an advance letter was 39.1 percent. If we take a more conservative approach and assume all of the unresolved numbers are working residential numbers (i.e., e1=1), the estimated response rate for the NIS-Teen landline sample that were mailed advance letters becomes 30.8 percent, so the true NIS-Teen response rate among those sent an advance letter is likely to be between 31 percent and 39 percent. 1.8 Analytics As noted above, the process of identifying eligible households and obtaining parental or guardian consent to contact vaccination providers on the NIS and NIS-Teen can be broken down into the following steps: completing the eligibility screener; identifying eligible households; completing the interview; and obtaining consent to contact vaccination providers. To examine the impact of advance letter envelope color and addressee experimental treatments on the NIS and NIS-Teen, logistic regression was used to model conditional outcomes, i.e., each outcome measure as a function of the treatment variables among the set of cases eligible for that step in the NIS process. As such, we specifically examined the following: screener completion rate among all cases mailed an advance letter; eligibility rate among advance letter cases that completed the screener; interview completion rate among advance letter cases that were identified as eligible; and rate of consent to contact vaccination providers among advance letter cases that completed the interview. In this way, if there was a treatment effect on the outcome measures, we could identify the step(s) in the NIS process (screener completion, eligibility, interview completion, and/or consent) in which a treatment was having an impact. Therefore, when a statistically significant treatment effect was observed for a particular outcome, we examined the outcome rate (e.g., the screener yield rate, the eligibility rate among screener completes, the interview completion rate among identified eligible households, or the consent rate among completed interviews) for that step of the process for each envelope color and addressee combination in order to identify which treatment(s) impacted the outcome and estimate effect sizes. For each outcome measure, we fit a logistic regression model relating that outcome to the color and the addressee treatments assigned to the case. The independent variables that were controlled in the experiment were the main effect of the envelope color treatment, the main effect of the addressee treatment, and the interaction effect defined by the combination of the envelope color and addressee. The models were fit in SAS using proc logistic, with the three-level envelope color variable and the three-level addressee variable treated as classification variables using the “class” statement in the SAS procedure. 2. RESULTS 2.1 Asking Respondents about the Advance Letter As noted above as part of the NIS screener, respondents reporting that the household contained a young child were asked a series of questions about the advance letter. Parents and guardians first were asked if they were aware whether their household received the letter, and 25 percent replied affirmatively that their household had received it (1,295/5,266). Those who reported that their household received the advance letter were asked if anyone had opened the letter. Nearly nine in ten (88 percent) reported affirmatively that the letter had been opened (1,126/1,285). Those reporting that the letter had been opened were asked if anyone had read the contents of the letter. More than nine in ten (92 percent) reported affirmatively that the letter had been read (1,038/1,124). Finally, the parent or guardian was asked whether she or he recalled any of the contents of the letter, and seven in ten (70 percent) reported affirmatively that they did recall some of the contents (729/1,034). Of note, none of the experimental conditions (envelope colors or different addressees) were found to have any effect on the self-reported answers to the four advance letter questions, and the only demographic factor found to correlate with the awareness variables was the number of adults in the household. Here, as would be expected5, the greater the number of adults in the household, the less likely the respondent was to report awareness of the letter being received (p < 0.001); 33 percent of households with one or two adults in residence reported that the letter had been received versus 21 percent of households with three or more adults in residence. 2.2 Experimental Effects on NIS Survey Outcomes Table 4 shows the results from the logistic regression analyses that tested the two main effects and the interaction effect for each of the four dependent variables. There were no statistically significant main effects or interaction effects associated with three of the four outcomes: the screening of households for the presence of young children, completing questionnaires, or gaining consent to contact vaccination providers. For eligibility among screened households, there was a statistically significant main effect for the addressee factor (p < 0.03) whereby the use of “Parent/Guardian” as the addressee led to a higher rate of finding that an eligible child lived in the household compared to the other two addressees that were tested. There was a marginally significant interaction effect of address and color (p < 0.08), whereby the “Parent/Guardian” addressee coupled with the tan envelope showed the highest eligibility rate.6 Table 4. Logistic Regression Results (Main Effects and Interaction Effect) for Each Dependent Variable Wald Chi-Square Degrees of Freedom p SCREENED, among dialed n = 485,694 dialed  Main Effect for Envelope Color 2.89 2 >0.10  Main Effect for Addressee 0.38 2 >0.10  Interaction Effect of Color and Addressee 2.92 4 >0.10 ELIGIBLE, among screened n = 151,526 screened  Main Effect for Envelope Color 0.63 2 >0.10  Main Effect for Addressee 7.50 2 <0.03  Interaction Effect of Color and Addressee 8.44 4 <0.08 COMPLETE, among eligible n = 1,755 eligible  Main Effect for Envelope Color 0.39 2 >0.10  Main Effect for Addressee 1.18 2 >0.10  Interaction Effect of Color and Addressee 1.35 4 >0.10 CONSENT, among completed n = 1,466 completed  Main Effect for Envelope Color 1.72 2 >0.10  Main Effect for Addressee 0.22 2 >0.10  Interaction Effect of Color and Addressee 2.51 4 >0.10 Wald Chi-Square Degrees of Freedom p SCREENED, among dialed n = 485,694 dialed  Main Effect for Envelope Color 2.89 2 >0.10  Main Effect for Addressee 0.38 2 >0.10  Interaction Effect of Color and Addressee 2.92 4 >0.10 ELIGIBLE, among screened n = 151,526 screened  Main Effect for Envelope Color 0.63 2 >0.10  Main Effect for Addressee 7.50 2 <0.03  Interaction Effect of Color and Addressee 8.44 4 <0.08 COMPLETE, among eligible n = 1,755 eligible  Main Effect for Envelope Color 0.39 2 >0.10  Main Effect for Addressee 1.18 2 >0.10  Interaction Effect of Color and Addressee 1.35 4 >0.10 CONSENT, among completed n = 1,466 completed  Main Effect for Envelope Color 1.72 2 >0.10  Main Effect for Addressee 0.22 2 >0.10  Interaction Effect of Color and Addressee 2.51 4 >0.10 Table 4. Logistic Regression Results (Main Effects and Interaction Effect) for Each Dependent Variable Wald Chi-Square Degrees of Freedom p SCREENED, among dialed n = 485,694 dialed  Main Effect for Envelope Color 2.89 2 >0.10  Main Effect for Addressee 0.38 2 >0.10  Interaction Effect of Color and Addressee 2.92 4 >0.10 ELIGIBLE, among screened n = 151,526 screened  Main Effect for Envelope Color 0.63 2 >0.10  Main Effect for Addressee 7.50 2 <0.03  Interaction Effect of Color and Addressee 8.44 4 <0.08 COMPLETE, among eligible n = 1,755 eligible  Main Effect for Envelope Color 0.39 2 >0.10  Main Effect for Addressee 1.18 2 >0.10  Interaction Effect of Color and Addressee 1.35 4 >0.10 CONSENT, among completed n = 1,466 completed  Main Effect for Envelope Color 1.72 2 >0.10  Main Effect for Addressee 0.22 2 >0.10  Interaction Effect of Color and Addressee 2.51 4 >0.10 Wald Chi-Square Degrees of Freedom p SCREENED, among dialed n = 485,694 dialed  Main Effect for Envelope Color 2.89 2 >0.10  Main Effect for Addressee 0.38 2 >0.10  Interaction Effect of Color and Addressee 2.92 4 >0.10 ELIGIBLE, among screened n = 151,526 screened  Main Effect for Envelope Color 0.63 2 >0.10  Main Effect for Addressee 7.50 2 <0.03  Interaction Effect of Color and Addressee 8.44 4 <0.08 COMPLETE, among eligible n = 1,755 eligible  Main Effect for Envelope Color 0.39 2 >0.10  Main Effect for Addressee 1.18 2 >0.10  Interaction Effect of Color and Addressee 1.35 4 >0.10 CONSENT, among completed n = 1,466 completed  Main Effect for Envelope Color 1.72 2 >0.10  Main Effect for Addressee 0.22 2 >0.10  Interaction Effect of Color and Addressee 2.51 4 >0.10 Table 5 presents the counts and data collection outcome rates (percentages) for each of the nine treatment combinations. As shown in the table, the interaction effect associated with the color and addressee on eligibility is due to the tan envelope (condition A1) with “Parent/Guardian” (condition B2) as the addressee; this combination of treatments (A1B2) had an eligibility rate of 1.46 percent. This rate is 35.2 percent higher ((1.46/1.08) - 1= 0.352) than the eligibility rate of the control condition (C1C2) of 1.08 percent; i.e., the white envelope with matched addressee’s name. Thus, the eligibility rate for the tan envelope with “Parent/Guardian” in the addressee line was found to exceed the control condition by more than one-third. Because the eligibility rate was so much higher among households mailed a tan envelope addressed to “Parent/Guardian,” 25 percent less interviewer time was required to achieve a completion with an eligible household among those landline phone numbers mailed a tan envelope addressed to “Parent/Guardian” compared with those landline phone numbers mailed a white envelope addressed to the name of the person found for that address in the matching process.7 Table 5. Counts and Percentages of Data Collection Outcomes by Experimental Group Envelope Color Addressee Condition Number Dialed Screened, Among Dialed Eligible, Among Screened Complete, Among Eligible Consent, Among Complete White Name C1C2 53,926 16,897 183 152 113 31.3% 1.08% 83.1% 74.3% Resident C1A2 53,996 17,017 193 167 122 31.5% 1.13% 86.5% 73.1% Parent/Guardian C1B2 54,078 16,880 208 173 120 31.2% 1.23% 83.2% 69.4% Tan Name A1C2 53,604 16,792 181 148 101 31.3% 1.08% 81.8% 68.2% Resident A1A2 53,928 16,678 175 147 104 30.9% 1.05% 84.0% 70.7% Parent/Guardian A1B2 54,099 16,874 246 207 138 31.2% 1.46% 84.1% 66.7% Blue Name B1C2 54,111 16,833 185 151 104 31.1% 1.10% 81.6% 68.9% Resident B1A2 54,251 16,845 198 163 115 31.1% 1.18% 82.3% 70.6% Parent/Guardian B1B2 53,701 16,710 186 158 117 31.1% 1.11% 84.9% 74.1% Envelope Color Addressee Condition Number Dialed Screened, Among Dialed Eligible, Among Screened Complete, Among Eligible Consent, Among Complete White Name C1C2 53,926 16,897 183 152 113 31.3% 1.08% 83.1% 74.3% Resident C1A2 53,996 17,017 193 167 122 31.5% 1.13% 86.5% 73.1% Parent/Guardian C1B2 54,078 16,880 208 173 120 31.2% 1.23% 83.2% 69.4% Tan Name A1C2 53,604 16,792 181 148 101 31.3% 1.08% 81.8% 68.2% Resident A1A2 53,928 16,678 175 147 104 30.9% 1.05% 84.0% 70.7% Parent/Guardian A1B2 54,099 16,874 246 207 138 31.2% 1.46% 84.1% 66.7% Blue Name B1C2 54,111 16,833 185 151 104 31.1% 1.10% 81.6% 68.9% Resident B1A2 54,251 16,845 198 163 115 31.1% 1.18% 82.3% 70.6% Parent/Guardian B1B2 53,701 16,710 186 158 117 31.1% 1.11% 84.9% 74.1% Table 5. Counts and Percentages of Data Collection Outcomes by Experimental Group Envelope Color Addressee Condition Number Dialed Screened, Among Dialed Eligible, Among Screened Complete, Among Eligible Consent, Among Complete White Name C1C2 53,926 16,897 183 152 113 31.3% 1.08% 83.1% 74.3% Resident C1A2 53,996 17,017 193 167 122 31.5% 1.13% 86.5% 73.1% Parent/Guardian C1B2 54,078 16,880 208 173 120 31.2% 1.23% 83.2% 69.4% Tan Name A1C2 53,604 16,792 181 148 101 31.3% 1.08% 81.8% 68.2% Resident A1A2 53,928 16,678 175 147 104 30.9% 1.05% 84.0% 70.7% Parent/Guardian A1B2 54,099 16,874 246 207 138 31.2% 1.46% 84.1% 66.7% Blue Name B1C2 54,111 16,833 185 151 104 31.1% 1.10% 81.6% 68.9% Resident B1A2 54,251 16,845 198 163 115 31.1% 1.18% 82.3% 70.6% Parent/Guardian B1B2 53,701 16,710 186 158 117 31.1% 1.11% 84.9% 74.1% Envelope Color Addressee Condition Number Dialed Screened, Among Dialed Eligible, Among Screened Complete, Among Eligible Consent, Among Complete White Name C1C2 53,926 16,897 183 152 113 31.3% 1.08% 83.1% 74.3% Resident C1A2 53,996 17,017 193 167 122 31.5% 1.13% 86.5% 73.1% Parent/Guardian C1B2 54,078 16,880 208 173 120 31.2% 1.23% 83.2% 69.4% Tan Name A1C2 53,604 16,792 181 148 101 31.3% 1.08% 81.8% 68.2% Resident A1A2 53,928 16,678 175 147 104 30.9% 1.05% 84.0% 70.7% Parent/Guardian A1B2 54,099 16,874 246 207 138 31.2% 1.46% 84.1% 66.7% Blue Name B1C2 54,111 16,833 185 151 104 31.1% 1.10% 81.6% 68.9% Resident B1A2 54,251 16,845 198 163 115 31.1% 1.18% 82.3% 70.6% Parent/Guardian B1B2 53,701 16,710 186 158 117 31.1% 1.11% 84.9% 74.1% 2.3 Experimental Effects on NIS-Teen Survey Outcomes The same analyses were conducted for each of our four dependent variables as they related to (a) screening households for teenagers in residence, (b) finding households that contained eligible teenagers, (c) completing interviews about a teenage resident’s vaccination history, and (d) gaining consent to contact a teenager’s vaccination provider(s). However, unlike what was found for NIS outcomes, there were no significant or marginally significant effects associated with the envelope color or the addressee on any of the dependent variables for the NIS-Teen; in particular, eligibility rates for households that completed the NIS-Teen screener were not significantly different across the envelope color or addressee conditions. 3. DISCUSSION We found that the color of the envelope and the nature of the addressee can affect the rate at which sampled households that are mailed an advance letter are subsequently identified as being eligible for the telephone survey8. This in turn translated into proportionally more telephone interviews being completed while fielding the same amount of sample and, therefore, less interviewer time per completed interview. In particular, 25 percent fewer interviewer hours were spent per completed interview, and the observed eligibility rate was 35 percent higher among cases mailed a tan envelope addressed to “Parent/Guardian” (condition A1B2) when compared to cases mailed a white envelope addressed to the name that was matched to the address (i.e., our control condition, C1C2). These effect size estimates are very large, and in our opinion, it is unlikely that the true effect sizes are of these magnitudes. However, these results do suggest that the envelope features can impact the observed eligibility rate and cost per completed interview. Based on the findings from our study, starting in Quarter 4 of 2015, the advance letters for the NIS and NIS-Teen have been sent in tan envelopes addressed to “Parent/Guardian.” There has been a fair amount of past research conducted on the effects of advance letters on survey outcome rates (cf. de Leeuw et al., 2007). However, almost all of this experimentation has focused on the nature of the letter and on the contents of the envelope, such as a noncontingent incentive or a frequently asked questions insert. In contrast, little has been reported about the features of the envelope/mailer that conveys the advance letter and how those features may affect survey outcomes. What has been reported suggests that response rates may be raised by using (a) hand-written labels, (b) tan envelopes, (c) stamps, (d) first class postage, (e) registered or certified delivery, (f) a larger than letter-size envelope, and (g) university sponsorship for the mailers that are sent to sampled addresses (Edwards et al., 2009); although, those findings are for mail surveys in general and not specifically for advance letters used in a telephone survey. Nonetheless, unless an envelope is opened, its contents cannot have an effect on the intended recipient(s). Thus, we believe that it is prudent for survey researchers to give more consideration to what will motivate sampled respondents to open the advance letter that is mailed to them. We conducted a very large national experimental testing two envelope features that we reasoned would affect whether or not our advance letter mailing would be opened and read. The factors we tested were the nature of the addressee to whom the envelope was sent and the color of the envelope. Our findings support the notion that features associated with the envelope that is used for an advance letter can make a meaningful difference in whether or not that respondent subsequently completes the survey task when later contacted by an interviewer. The specific way that we operationalized our test of the addressee line was particular to the NIS and NIS-Teen, i.e., gathering information about children/teenagers from their parent or guardian. However, the change in the envelope did not have the same impact across both surveys. Our findings with parents and guardians of young children were not replicated by our findings with parents and guardians of teenagers, and this is discussed below in more detail. This suggests the need to tailor the addressee line to the particular study (cf. Dillman et al. 2007). For example, in a survey about satisfaction with recent technology purchases using a customer list as the sampling frame, one could envision advance letters addressed to “Recent Technology Purchaser.” Or for a survey about the political preferences of those without any party affiliation using a registered voters list for the sampling frame, one could envision an advance letter addressed to “Independent-minded Voter.” In contrast, for a survey that samples the general population without any screening, our findings about the nature of the addressee line on an advance letter envelope would not seem to have any relevance, although we could imagine that a researcher might consider using something such as “Concerned Citizen,” depending on the subject matter of the questionnaire. Recently, Lavrakas, Dirksz, Lusskin, and Ponce (2016) reported that addressing envelopes to “Residente Actual/Current Resident” raised the response rate in a mail survey of high-density Hispanic areas by 14 percent (p < 0.02) compared to using “Current Resident.” Our finding that the color of the advance letter envelope made a meaningful difference in survey outcomes suggests that this too should be something about which researchers think carefully. Using a non-white envelope for an advance contact mailing may have little or no cost implications for one’s survey budget. We suspect that in an age when the public receives so much junk mail, sending an advance letter in a more official-looking tan envelope may distinguish a mailing from typical junk mail that gets readily tossed without anyone opening it (cf. Dillman et al., 2009). As noted in Table 1, there are many envelope features that researchers could actively consider in designing the envelope used for sending an advance contact communication to make it more likely that it will be opened and its contents be read. In thinking further about the results we have reported, one might ask, “Why were the effects for parents and guardians of young children limited to the eligibility rate, and why were those results not observed for parent and guardians of teenagers?” The envelopes that were used in our experiment had a return address of “Department of Health and Human Services, Public Health Service, Centers for Disease Control, National Center for Health Statistics” and were marked with “OFFICIAL BUSINESS”9. We reason that these envelopes, when combined with a “Parent/Guardian” addressee, may have been more salient to a parent or guardian of a young child than to the parent of a teenager. We suspect that most parents and guardians are especially concerned about the health of their young children, as these children are generally most vulnerable to infectious diseases. If it is true that the parents and guardians of young children are more aware of and concerned about health issues related to their children than are parents of teenagers, then it may explain why an envelope from an official government health agency that is addressed to “Parent/Guardian” is more salient to the parents and guardians of young children than it is to the parents and guardians of teenagers. Our study was limited to households with landline telephone numbers that could be matched to a mailing address. Whether our results would apply to households with a cell phone number that might be matched to a mailing address is uncertain, but we have no reason to suspect that they would not. Furthermore, although the matching of addresses to cell phone numbers in the United States is now not a productive endeavor, the rate of successfully matching addresses to US cell phone numbers will likely continue to grow over time. Additionally, only respondents reporting the presence of a young child in the household were asked if they had received the letter, and therefore, we could not examine the relationship between reporting that the letter had been received and the screener completion rate or the eligibility rate among those that completed the screener. Moreover, only one-fourth (25 percent) of those respondents with a young child who were asked reported that their household had received the letter.10 There are at least five reasons that this would be expected to happen, and each deserves much more investigation by survey researchers who are thinking about using advance letters. First, an advance letter may never reach the intended recipient because the address is incorrect. This suggests that researchers need to know much more about the reliability of the addresses they use for their advance mailings, especially whenever they are not using an address-based sampling design. Second, an advance letter may be accurately addressed but never be delivered to that address (or be delivered too late to have any effect) by the delivery service that is being used. It is likely that poor delivery service is not a geographically random occurrence. Instead, it may be that certain “types” of locales (e.g., lower income versus upper income; rural versus suburban versus urban), for many reasons, are subject to different quality of mail delivery. If this is the case, and if an advance letter stimulates positive survey response (as the literature shows that it often does; de Leeuw et al. 2007), then differential nonresponse is likely resulting from the variation in the quality of the local mail delivery service. This is another factor that survey researchers need to understand better, because it is possible that using an advance letter under these conditions is counterproductive to efforts to reduce total error in a particular survey. Third, an advance letter may be delivered to the correct address but may never be opened. The research we have reported here speaks to this issue. However, many more factors associated with the envelope need to be studied using unconfounded experiments for researchers to be able to have more confidence in choosing design features for the envelopes that they use in their advance mailings (cf. Edwards et al., 2009). Fourth, an advance letter may be opened, but it may never be read or its contents may not effectively motivate the reader to be more likely to comply with the subsequent survey task. de Leeuw et al. (2007) demonstrated that there are many features associated with the contents of an advance letter that have meaningful positive effects on survey response. But in comparing our Table 2 with the features studied in the de Leeuw et al. meta-analysis, it is clear that there are a great many features of advance letter mailings that may affect survey response that have not been adequately tested. As such, a great deal of additional research of factors in this domain is called for. Fifth, someone at the household who reads a superbly crafted advance letter may not be the person who subsequently is contacted by an interviewer in a telephone or in-person survey or who subsequently receives and opens a questionnaire packet for a mail survey. Unless the person who was exposed to the advance letter informs others in her/his household about the contents of the advance letter and about the details of the forthcoming survey, the effort to create and send the advance letter may be entirely wasted. Related to this, we are unaware of any testing of the effect of explicitly asking the reader of an advance letter to actively share its content with others (e.g., all other adults) in her or his household. This is easy to test, and it deserves to be tested. The envelope features of the advance mailings in this study did not raise the cost of the mailings, and as shown in our findings, they had meaningful and cost-beneficial effects for a given survey. Despite the large amount of past experimentation into the effects of advance letters, there are many features of such mailings, including those related to the envelope and its delivery that the field of survey research has yet to test adequately. Supplementary Materials Supplementary materials are available online at https://academic.oup.com/jssam. Footnotes 1 We conducted an extensive search of the research literature on mailer/envelope effects from 2009 through 2016; in particular, we searched survey research methods archives and health research methods archives, including each issue of Public Opinion Quarterly and the American Journal of Epidemiology. We also searched all AAPOR conference programs during this time period. In the course of this search, we were surprised to find only one study that addressed envelope features and mail survey response rates (Dykema, Jaques, Cyffka, Assad, Hammers, et al., 2015) and none that addressed advance letters used with telephone surveys. 2 Two percent (2.24%) of the envelopes in this condition had “Current Resident” used as the addressee, whereas the other 97.76% had the name that was matched to the address as the addressee. 3 The address-match rates among the telephone numbers in each condition were very similar across the nine experimental groups, ranging from 40.8 percent to 41.1 percent, but they were not identical due to sample matching variation. 4 The NIS screener first asked respondents for the number of children age 12 months to 4 years in the household. Those that indicated the presence of such a child were then asked the series of questions about the advance mailing. Following these questions, the respondent was asked for the dates of birth of those children age 12 months to 4 years in the household to determine which, if any, were 19 to 35 months old, i.e., eligible for the NIS. In this way, the questions about the advance mailing were asked of all households containing a child age 12 months to 4 years (not just those eligible for the NIS), but were not asked of households that did not have a child age 12 months to 4 years. 5 This would be expected because the greater the number of adults in the household, the lower the probability that the parent or guardian being interviewed would have been the adult to have opened the letter or to have been told by another adult that a letter had been received informing the household of a forthcoming telephone survey about vaccinations of young children and teenagers. 6 A household either has eligible children or it does not, so the “Parent/Guardian” addressee cannot be causing more households to truly be eligible; however, the “Parent/Guardian” addressee line may have caused more parents and guardians with eligible children to (a) open the envelope, (b) decide to begin the questionnaire, and (c) be willing to report that an eligible child lived in their household, when compared with the case for the other two addressee line conditions, resulting in a higher observed eligibility rate for the “Parent/Guardian” treatment. 7 The interviewer time per completion was computed as the sum of the durations of all calls made to all cases in each treatment combination divided by the total number of interviews completed in that treatment combination. 8 Please note that the causal mechanisms at work that produced this observed outcome here are uncertain; please see footnote 6 for more discussion of this issue. 9 Images of the envelopes can be found at the JSSAM Supplementary Material website. 10 Because only those respondents reporting that a child between the ages of 12 months and 4 years lived in the household were asked whether they had received the letter, we do not know what percentage of households without a young child would have reported receiving the letter. References AAPOR ( 2016 ), Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys ( 9th ed. ). Deerfield, IL : American Association for Public Opinion Research . http://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf. Camburn D. , Lavrakas P. J. , Battaglia M. , Massey J. , Wright R. ( 1996 ), “Using Advance Respondent Letters in Random-Digit-Dialing Telephone Surveys,” American Statistical Association 1995 Proceedings: Section on Survey Research Methods , 969 – 974 . CASRO ( 1982 ), On the Definition of Response Rates. Available at, http://c.ymcdn.com/sites/www.casro.org/resource/resmgr/docs/casro_on_definitions_of_resp.pdf?hhSearchTerms=%22response+and+rates%22. de Leeuw E. , Callegaro M. , Hox J. , Korendijk E. , Lensvelt-Mulder G. ( 2007 ), “ The Influence of Advance Letters on Response in Telephone Surveys: A Meta-Analysis ,” Public Opinion Quarterly , 71 , 413 – 443 . Google Scholar CrossRef Search ADS Dillman D. A. , Lesser V. , Mason R. , Carlson J. , Willits F. , Robertson R. , Burke B. ( 2007 ), “ Personalization of Mail Surveys for General Public and Populations with a Group Identity: Results from Nine Studies ,” Rural Sociology , 72 , 632 – 646 . Google Scholar CrossRef Search ADS Dillman D. A. , Smyth J. D. , Christian L. M. ( 2009 ), Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method , ( 3rd ed.). Hoboken, NJ : John Wiley & Sons . Dykema J. , Jaques K. , Cyffka K. , Assad N. , Hammers R. G. , Elver K. , Malecki K. C. , Stevenson J. ( 2015 ), “ Effects of Sequential Prepaid Incentives and Envelope Messaging in Mail Surveys ,” Public Opinion Quarterly , 79 , 906 – 931 . Google Scholar CrossRef Search ADS Edwards P. J. , Roberts I. , Clarke M. J. , Diguiseppi C. , Wentz R. , Kwan I. , Cooper R. , Felix L. M. , Pratap S. ( 2009 ), “ Methods to Increase Response to Postal and Electronic Questionnaires ,” Cochrane Database of Systematic Reviews , 3 , Article Number MR000008. Groves R. M. , Singer E. , Corning A. ( 2000 ), “ Leverage Salience Theory of Survey Participation ,” Public Opinion Quarterly , 64 , 299 – 308 . Google Scholar CrossRef Search ADS PubMed Groves R. M. , Snowden C. ( 1987 ), “The Effects of Advance Letters on Response Rates in Linked Telephone Surveys.” Proceedings of the Section on Survey Research Methods of the American Statistical Association , pp. 633 – 638 . Lavrakas P. J. , Dirksz G. , Lusskin L. , Ponce B. ( 2016 ), “ Experimenting with the Addressee Line in a Mail Survey of Hispanic Households ,” Survey Practice , 9 , http://www.surveypractice.org/index.php/SurveyPractice/article/view/380/pdf_80. © The Author 2017. Published by Oxford University Press on behalf of the American Association for Public Opinion Research. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

Journal of Survey Statistics and MethodologyOxford University Press

Published: Sep 18, 2017

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off