How Representative Are SMS Surveys in Africa? Experimental Evidence From Four Countries

How Representative Are SMS Surveys in Africa? Experimental Evidence From Four Countries Abstract Short message service (SMS) surveys are fast, inexpensive, and increasingly common. However, there is limited research on this new mode. Using an experimental design, we conducted general population SMS surveys in Ghana, Kenya, Nigeria, and Uganda. This article (1) reports the levels and components of nonresponse; (2) evaluates the representativeness of SMS surveys relative to benchmark data; and (3) explores strategies to improve representativeness. We find that SMS surveys underrepresent women, older people, those with less education, and less technologically savvy people. Sending reminders improved representativeness, but offering shorter surveys or higher incentives did not. We conclude that presently, general population SMS surveys cannot replace face-to-face surveys. Our article provides practical guidance for survey designers and directions for future research. The rapid growth of mobile phones is transforming data collection in low- and middle-income countries. In sub-Saharan Africa, mobile phone subscriptions nearly doubled between 2010 and 2015, from 200 to 386 million subscribers (GSMA, 2015). More people across Africa have access to mobile phone networks than to electricity and piped water (Mitullah, Samson, Wambua, & Balongo, 2016). Owing to this growth, mobile applications are flourishing, and people increasingly communicate via short message service (SMS), or text messaging. In an SMS survey, each question is sent as a separate SMS. Respondents reply by SMS with the number corresponding to their response, and then receive the next question via another SMS. In low- and middle-income countries, SMS surveys have grown in popularity in recent years because of their speed and low cost. For example, SMS is used for large-scale public opinion surveys (GeoPoll, 2015; UNICEF, 2015), rapid surveys during emergencies such as the 2014 Ebola crisis (World Health Organization, 2014), and public health surveillance (Centers for Disease Control and Prevention, 2015). The recent proliferation of SMS surveys parallels the growth of Web surveys in the late 1990s (Couper, 2000). Because SMS and Web are self-administered through a device, they offer enormous cost and time savings, but also present challenges with coverage and nonresponse. The remarkable growth of SMS surveys for public opinion and other uses in low- and middle-income countries has outpaced scientific investigation of the validity and best practices of this new mode. There is more research on SMS surveys in developed countries (Conrad, Schober, Antoun, Hupp, & Yan, 2017; De Bruijne & Wijnant, 2014; Hoe & Grunwald, 2015; Steeh, Buskirk, & Callegaro, 2007). For SMS surveys in lower-income countries, there is good reason to anticipate significant coverage errors (because of lack of mobile phone access and frame errors) and nonresponse errors (because of illiteracy, concern about costs, distrust of surveys, difficulty charging devices). Some of these errors are like any self-administered mode, whereas others are unique to SMS. However, there is little research on the representativeness of SMS surveys compared with benchmark face-to-face (FTF) surveys (Gibson et al., 2017). Further, there is scant experimental research on how to improve the representativeness of SMS developing country samples through strategies like incentives, questionnaire length, and reminders. Our research addresses the lack of research on SMS survey representativeness and best practices in design. We seek to evaluate the quality of SMS survey samples and provide practical guidance about optimizing SMS surveys in developing countries. Using an experimental design, we conducted SMS surveys of the general population (age 18–64 years) in Kenya, Ghana, Nigeria, and Uganda. We ask the following research questions: RQ1: How representative are SMS surveys? This question has three subparts: 1a. What are the levels and components of nonresponse to SMS surveys? 1b. How representative are SMS population-based survey samples? 1c. How much does incomplete access to mobile phones affect representativeness? RQ2: How can survey design improve the representativeness of SMS surveys? Background: Coverage and Nonresponse in SMS Surveys In this section, we discuss how coverage and nonresponse errors can affect the representativeness of SMS surveys. Coverage Coverage refers to the extent to which the sampling frame overlaps with the target population. Undercoverage occurs when units in the target population are omitted from the frame. If many units are nonrandomly missing from the frame, coverage errors can create bias (Eckman, 2015). We anticipate three main types of undercoverage errors for SMS surveys. First, many SMS surveys interview mobile phone users but draw inferences to the general population. Across Africa, however, 16% of adults lack access to a mobile phone (Mitullah & Kamau, 2013); phone access is more common among people who are educated, male, and those who speak English (Pew Research Center, 2015). This undercoverage could bias survey statistics, just as lack of universal Internet access can bias Web surveys (Bethlehem, 2010; Eckman, 2015), and lack of complete phone access can bias telephone surveys (Peytchev, Carley-Baxter, & Black, 2011). In the African context, it is unclear whether undercoverage will affect survey statistics. For example, if mobile phone access is high enough, and differences between those with and without phone phones are not large, then this undercoverage may not meaningfully affect survey statistics. Second, the method of frame construction can also introduce coverage errors. For random digit dial (RDD) frames, all or nearly all numbers in the target population should be available for selection. For list frames, vendors typically build frames by establishing relationships with mobile network operators. This method provides vendors with access to known active phone numbers—improving efficiency during data collection. However, vendors may not have relationships with all mobile network operators in a country, leading to undercoverage. These errors can impact sample representativeness because the subscriber bases differ across mobile network operators in terms of socioeconomic status, geography, age, and other factors. Third, regulations in some countries prevent sending people SMS invitations without prior consent. This is the case with Uganda in our study (and many other developed countries; see American Association for Public Opinion Research, 2010). Thus, only individuals who have volunteered for surveys in the past are included in the frame, impacting sample representativeness. In contrast, it is possible to send SMS invitations without prior consent in other countries (Kenya, Nigeria, and Ghana in our study). Nonresponse Like other self-administered modes (e.g., Web, mail), response rates to SMS surveys are expected to be lower than interviewer-administered modes. Without an interviewer to convey the survey’s legitimacy and importance of responding, individuals may find it easy to ignore the survey request. Like Web surveys, people may not respond because of discomfort with technology (Ganesan, Prashant, &Jhunjhunwala, 2012; Hoe & Grunwald, 2015) or concerns that the survey is illegitimate or “spam.” Further, people weaker literacy skills cannot participate. In 2014, the illiteracy rate was 41% across Africa (UNESCO, 2014), and illiterate people are more likely to be poor, female, and rural (Leo, Morello, Mellon, Peixoto, & Davenport, 2015). The nature of SMS surveys in lower-income countries also presents other challenges. SMS messages are generally limited to 160 characters, which leaves little room to craft a comprehensive, persuasive survey request. SMS surveys cannot include visual cues (e.g., logos or graphics) to bolster response, as in Web surveys (Fan & Yan, 2010). In addition, people may be concerned about incurring data charges on their phone, even if an incentive is offered. Infrastructure is also a challenge. People in developing countries may not receive SMS survey invitations because of poor mobile network connections, particularly in rural areas (Ganesan et al., 2012). The lack of access to affordable, regular electricity means that many people keep their phones off and only turn on their phones when needed (Carmer & Boots, 2013; Dillon, 2012). When the phone is turned on, the invitation may be mixed in with other messages or the data collection period may be over. Another source of nonresponse concerns access to a shared phone. Sharing mobile phones is common in Africa (Pew Research Center, 2015). Shared phones may lead to nonresponse errors and bias if certain types of people are more likely to have access to phone. For example, a husband may be more likely to possess a mobile phone that he shares with his wife, reducing the possibility that the wife will receive and answer the SMS survey. Previous Research on SMS Survey Representativeness Coverage and nonresponse errors can impact the representativeness of SMS general population surveys. In South Africa, two comparisons between SMS and FTF surveys suggest that SMS surveys may overrepresent educated, affluent, and younger people (Broich, 2015; Lombaard & Richman, 2015). Similar patterns have been found when recontacting previous FTF survey respondents via SMS (Ballivian, Azevedo, & Durbin, 2015) and telephone (Mahfoud, Ghandour, Ghandour, Mokdad, & Sibai, 2015)—as well as cross-sectional interactive voice response (IVR) surveys (Leo et al., 2015). Building on this research, we provide a more comprehensive view by studying four countries and using a richer set of variables for comparison. We also investigate the impact of one type of undercoverage error: the lack of universal mobile phone access. Many people lack phones (in our data, 26% in Nigeria, 25% in Uganda, 21% in Ghana, and 8% in Kenya), and because demographics may be correlated with phone access, the lack of phone access may impact representativeness of SMS surveys. Researchers have just started exploring ways to improve the representativeness of SMS surveys. Johnson (2016) reported that time of day for the survey invitation did not affect response rates. Most other research has focused on incentives. Typically, incentives are mobile phone airtime that is automatically applied to the respondent’s mobile phone account after the survey. While a nominal incentive is important for response, there is little evidence about the optimal incentive level for SMS surveys. Johnson (2016) found no difference in response rates between a 0.50 and 1.00 USD in airtime among users of a family planning SMS service in Kenya. This finding echoes research from computer-assisted telephone interviewing (CATI) surveys in developing countries (Ballivian et al., 2015) and Web surveys in developed countries (Fan & Yan, 2010), where nominal increases in incentives do not meaningfully improve data quality. Two studies found that entering respondents in a lottery raised response rates more than airtime (Broich, 2015; Leo et al., 2015), though Johnson (2016) found no difference. Notably, these studies focus exclusively on response rates, and do not investigate whether incentives impact sample representativeness. Our study also considers sample composition. We also explore two other ways to improve the representativeness of SMS surveys: survey length and reminders. Offering a shorter survey—for example, 8 questions instead of 16—may encourage more individuals to respond, particularly lower socioeconomic status people who may be more concerned about data charges and power on their phone. If this is true, then a shorter survey may result in more representative samples. Further, sending reminders is another less explored way of reducing nonresponse error. Reminders are standard protocols for increasing response rates and improving representativeness in Web surveys (Fan & Yan, 2010). Some SMS surveys use reminders, but many SMS surveys do not because of the emphasis on speed. People with more sporadic phone access or time to complete a survey—such as less educated or rural people—may be more likely to participate if they receive a reminder. Data and Methods This section describes the SMS surveys we conducted, the benchmark FTF data we use to evaluate the representativeness of the SMS surveys, and the analysis. SMS Survey Data We conducted SMS surveys in Ghana, Kenya, Nigeria, and Uganda, working with GeoPoll as our sample and data collection vendor. This section describes the sample design, questionnaire, and data collection for the SMS surveys. We also elaborate on the Kenya survey, which included several experiments. Sample design GeoPoll obtains lists of active mobile phone numbers through established relationships with mobile phone network operators in each country. These lists are updated periodically to add newly activated phone numbers and to delete inactive numbers. Using these frames, we drew random samples of mobile phone numbers in each country. We stratified the samples by mobile network operator and geography (province in Kenya, region in Ghana and Uganda, geopolitical zone in Nigeria). The stratification used proportional allocation, yielding a self-weighting sample of phone numbers. There was no within-phone sampling because of challenges in sampling users of the same phone. We return to this issue in the “Discussion” section. GeoPoll’s method of constructing a frame through mobile network operators has the benefit of excluding numbers that were never assigned or not in service. But this method can also lead to coverage errors. At the time of our study, GeoPoll lacked agreements with some mobile network operators in Ghana. Further, even if GeoPoll had an agreement with a network operator, the sample may have undercoverage (new numbers were activated but were not yet in the frame) and overcoverage (inactive numbers or USB modems). Table 1 shows the composition of mobile network operators between completed interviews in our SMS survey and external data from government communication authorities. In Kenya, there were only minor differences. In Ghana, however, 90% of our achieved sample was from the MTN network, but MTN comprises only 46% of the overall market. Discrepancies also existed in Nigeria and Uganda. Discrepancies between data sources reflect both coverage and nonresponse errors in our SMS surveys, and affect sample representativeness because mobile network operators have different sociodemographic subscriber bases (Broich, 2015). Table 1 Comparison of Mobile Network Operator in SMS Survey Data and Mobile Market Share (Percentages)   Kenya   Ghana   Nigeria   Uganda     Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   n  2,960  …  2,277  …  2,392  …  2,068  …  Safaricom  77.5  67.0  …  …  …  …  …  …  Telkom (Orange)  7.0  11.2  …  …  …  …  1.5  3.5  Equitel (Finserve)  0  2.4  …  …  …  …  …  …  Airtel  13.4  19.4  4.4  13.0  15.8  21  89.3  38.6  MTN  …  …  90.0  46.3  58.4  42  5.7  51.3  Vodafone  …  …  2.3  22.4  …  …  …  …  Tigo  …  …  1.0  13.6  …  …  …  …  Expresso  …  …  0.5  0.4  …  …  …  …  Uganda Telkom  …  …  …  …  …  …  0  5.2  Sure Telkom  …  …  …  …  …  …  0  0.6  Globacom  …  …  0.5  4.3  20.0  21  …  …  Etisalat  …  …  …  …  4.5  16  …  …  Yu  1.0  0  …  …  …  …  …  …  Visa  …  …  …  …  0.5  0  …  …  UTL  …  …  …  …  …  …  1.8  …  Other  0.6  0  …  …  …  …  0.9  0.8  Do not know  0.4  …  1.1    1.0  0  0.9  …  Total  100.0  100.0  100.0  100.0  100.0  100.0  100.0  100.0    Kenya   Ghana   Nigeria   Uganda     Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   n  2,960  …  2,277  …  2,392  …  2,068  …  Safaricom  77.5  67.0  …  …  …  …  …  …  Telkom (Orange)  7.0  11.2  …  …  …  …  1.5  3.5  Equitel (Finserve)  0  2.4  …  …  …  …  …  …  Airtel  13.4  19.4  4.4  13.0  15.8  21  89.3  38.6  MTN  …  …  90.0  46.3  58.4  42  5.7  51.3  Vodafone  …  …  2.3  22.4  …  …  …  …  Tigo  …  …  1.0  13.6  …  …  …  …  Expresso  …  …  0.5  0.4  …  …  …  …  Uganda Telkom  …  …  …  …  …  …  0  5.2  Sure Telkom  …  …  …  …  …  …  0  0.6  Globacom  …  …  0.5  4.3  20.0  21  …  …  Etisalat  …  …  …  …  4.5  16  …  …  Yu  1.0  0  …  …  …  …  …  …  Visa  …  …  …  …  0.5  0  …  …  UTL  …  …  …  …  …  …  1.8  …  Other  0.6  0  …  …  …  …  0.9  0.8  Do not know  0.4  …  1.1    1.0  0  0.9  …  Total  100.0  100.0  100.0  100.0  100.0  100.0  100.0  100.0  Notes. SMS data are unweighted. The target population for the SMS surveys is age 18–64 years. SMS = short message service. External data sources: Kenya: Fourth Quarter Sector Statistics Report for the Financial Year 2014/2015 (April–June 2015), Communications Authority of Kenya. Ghana: Mobile voice subscription trends for August 2015, National Communications Authority. Uganda: Mobile network access for MVNOs, Market Assessment, January 2015, Uganda Communications Commission. Nigeria: Market share of mobile operators, September 2015, Nigerian Communications Commission. Table 1 Comparison of Mobile Network Operator in SMS Survey Data and Mobile Market Share (Percentages)   Kenya   Ghana   Nigeria   Uganda     Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   n  2,960  …  2,277  …  2,392  …  2,068  …  Safaricom  77.5  67.0  …  …  …  …  …  …  Telkom (Orange)  7.0  11.2  …  …  …  …  1.5  3.5  Equitel (Finserve)  0  2.4  …  …  …  …  …  …  Airtel  13.4  19.4  4.4  13.0  15.8  21  89.3  38.6  MTN  …  …  90.0  46.3  58.4  42  5.7  51.3  Vodafone  …  …  2.3  22.4  …  …  …  …  Tigo  …  …  1.0  13.6  …  …  …  …  Expresso  …  …  0.5  0.4  …  …  …  …  Uganda Telkom  …  …  …  …  …  …  0  5.2  Sure Telkom  …  …  …  …  …  …  0  0.6  Globacom  …  …  0.5  4.3  20.0  21  …  …  Etisalat  …  …  …  …  4.5  16  …  …  Yu  1.0  0  …  …  …  …  …  …  Visa  …  …  …  …  0.5  0  …  …  UTL  …  …  …  …  …  …  1.8  …  Other  0.6  0  …  …  …  …  0.9  0.8  Do not know  0.4  …  1.1    1.0  0  0.9  …  Total  100.0  100.0  100.0  100.0  100.0  100.0  100.0  100.0    Kenya   Ghana   Nigeria   Uganda     Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   n  2,960  …  2,277  …  2,392  …  2,068  …  Safaricom  77.5  67.0  …  …  …  …  …  …  Telkom (Orange)  7.0  11.2  …  …  …  …  1.5  3.5  Equitel (Finserve)  0  2.4  …  …  …  …  …  …  Airtel  13.4  19.4  4.4  13.0  15.8  21  89.3  38.6  MTN  …  …  90.0  46.3  58.4  42  5.7  51.3  Vodafone  …  …  2.3  22.4  …  …  …  …  Tigo  …  …  1.0  13.6  …  …  …  …  Expresso  …  …  0.5  0.4  …  …  …  …  Uganda Telkom  …  …  …  …  …  …  0  5.2  Sure Telkom  …  …  …  …  …  …  0  0.6  Globacom  …  …  0.5  4.3  20.0  21  …  …  Etisalat  …  …  …  …  4.5  16  …  …  Yu  1.0  0  …  …  …  …  …  …  Visa  …  …  …  …  0.5  0  …  …  UTL  …  …  …  …  …  …  1.8  …  Other  0.6  0  …  …  …  …  0.9  0.8  Do not know  0.4  …  1.1    1.0  0  0.9  …  Total  100.0  100.0  100.0  100.0  100.0  100.0  100.0  100.0  Notes. SMS data are unweighted. The target population for the SMS surveys is age 18–64 years. SMS = short message service. External data sources: Kenya: Fourth Quarter Sector Statistics Report for the Financial Year 2014/2015 (April–June 2015), Communications Authority of Kenya. Ghana: Mobile voice subscription trends for August 2015, National Communications Authority. Uganda: Mobile network access for MVNOs, Market Assessment, January 2015, Uganda Communications Commission. Nigeria: Market share of mobile operators, September 2015, Nigerian Communications Commission. Another source of undercoverage stems from government regulations in Uganda that prevent survey organizations from sending SMS invitations to people who have not previous opted into a GeoPoll survey. Therefore, our sample in Uganda is an opt-in sample: 100% had previously participated in GeoPoll surveys before in Uganda. This contrasts with previous participation rates of 1.5% in Ghana, 0.3% in Nigeria, and 17.3% in Kenya. Questionnaire The core SMS survey consisted of 16 questions about demographics, socioeconomic status, and technology. Each survey question was sent as a separate SMS to the respondent. The respondent selected an answer from a closed ended list by entering a number associated with the response, and then the next question was sent via SMS. To ensure comparability with the benchmark data, we used questions and response options from the FTF data, with minor adaptations for mode. See Online Supplement for question wording. Experimental design in Kenya For each sampled case, we randomly assigned survey length (8 vs. 16 questions) and incentive amount (the standard 0.5 USD incentive vs. 1.25 USD). We also randomized whether “don’t know” was offered as a response option; we report this experiment in a different article (in process at time of publication). Data collection We began collecting data in Kenya first, starting with a pilot study to test our procedures (n = 457 completed interviews). The pilot test did not reveal any problems, so we merged the pilot test data with the main study for most analyses. We collected data in Kenya in late November and early December of 2015, and the other countries in late November 2015. Sample members received an SMS that introduced the study, specified the number of questions, and offered an incentive in the form of mobile phone airtime. In Kenya, the incentive was randomly assigned to be either 0.50 USD or 1.25 USD. In Ghana, Nigeria, and Uganda, the incentive was 0.50 USD. The survey was offered in the major languages for each country (English and Swahili in Kenya; English and Twi in Ghana; English, Hausa, Igbo, and Yoruba in Nigeria; and English and Luganda in Uganda). Respondents selected their language on the first screen of the survey. Individuals <18 years or >64 years of age were screened out as ineligible. Eligible respondents then answered survey questions and received mobile phone airtime for completing. GeoPoll sent up to three follow-up reminders to nonrespondents. A small proportion of cases (<0.5%) were dropped because of concerns about data quality. Benchmark Data We compare SMS survey respondents with respondents from three benchmark FTF sources. We compare the SMS and FTF data across three groups of variables: demographics (age and gender), socioeconomics and technology (education, employment, multiple SIM card, shared SIM card, difficulty charging phone, aware of Internet, use Internet), and housing (shelter type and roof). For the age and gender comparisons, we compare SMS data with Census Data in Kenya (2009), Nigeria (2006), Ghana (2010), and United Nations (UN) population data for Uganda (2010). For the socioeconomic and technology comparisons, we analyze the “Technology Adoption Surveys,” four computer-assisted personal interviewing surveys in Kenya (n = 3,364), Ghana (n = 3,113), Nigeria (n = 3,042), and Uganda (n = 3,075) conducted in 2014–2015. Response rates were 64% in Kenya, 54% in Ghana, 64% in Nigeria, and 77% in Uganda. The survey is based on an area probability sample of households created through geographic information systems technology. Housing structure (roof and shelter type) is a commonly used indicator of socioeconomic status in Africa. For these comparisons, we analyze FTF Afrobarometer surveys from Kenya, Ghana, Nigeria, and Uganda. In each country, 2,400 interviews were completed. These surveys were based on nationally representative samples of the adult population age ≥18 years, using a random walk procedure to sample households. Each country used paper-and-pencil interviewing, and data were collected in 2012 (and 2011–2012 in Uganda). Response rates were 73% in Kenya, 73% in Ghana, 90% in Nigeria, and 87% in Uganda. The Afrobarometer is one of the main data sources about political attitudes and behavior in Africa and is used widely by academics, government, and donors (see http://www.afrobarometer.org/.). Analysis Our analysis proceeds in three stages. The first stage investigates Research Question 1a: What are the levels and components (e.g., refusal, no answer, breakoff) of nonresponse to SMS surveys in each country? We present response rates and case dispositions for each country. We explore one component of nonresponse in greater detail: breakoff. In this analysis, we present breakoff rates for each country and question-level breakoff rates to understand when respondents stop answering questions. We compare the demographic characteristics of breakoff cases from completed cases to understand how breakoff can impact sample representativeness. The second stage investigates the representativeness of SMS surveys (Research Question 1b) and how much lack of mobile phone access affects representativeness (Research Question 1c) by comparing SMS and FTF respondents. For comparisons by age and gender, we compare SMS surveys to Census (Ghana, Kenya, Nigeria) or UN data (Uganda) through univariate distributions. Based on this analysis, we created poststratification weights to align the SMS survey samples to age and gender population totals from Census or UN data. This allows us to evaluate SMS sample representativeness in subsequent analysis after adjusting for age and gender. See “Results” section for more information on weight construction, sizes of weights, and other information about weighting. We compare the weighted SMS data to 11 variables about sociodemographics and technology (Technology Adoption Surveys) and housing structure (Afrobarometer.) These variables are all categorical or binary. For each variable, we present percentage point differences between the SMS data and the entire sample from the FTF survey:   PercentagefromSMSsurvey−PercentagefromFTFsurvey. (1) We show the percentage point differences graphically. In the graphic, we highlight percentage point differences that are significantly different from 0 (defined as having nonoverlapping confidence intervals) using stars. Full tables that contain point estimates and confidence intervals for the SMS and FTF estimates are available in the Online Supplement. We treat the FTF survey as the benchmark data source, and interpret departures from the FTF survey as evidence of errors in the SMS survey. Positive numbers mean the SMS survey overestimates the percentage; negative numbers mean the SMS survey underestimates the percentage. To investigate the impact of undercoverage because of mobile phone access, we compare SMS and FTF data, but restricting the FTF data to respondents with a mobile phone:   PercentagefromSMSsurvey–PercentagefromFTFsurvey(basedonFTFrespondentswithphones). (2) This comparison eliminates the component of coverage error because of an individual’s mobile phone access—although other types of coverage error (because of mobile network operator, opt-in), and nonresponse may still be present. If indeed undercoverage because of incomplete access to mobile phones affects representativeness, then the differences between SMS and FTF should be attenuated once we restrict the FTF survey data to individuals with mobile phones. In the third stage, we explore Research Question 2: How can design improve the representativeness of SMS surveys? We explore three strategies: (1) increasing the incentive amount, (2) offering a shorter survey, and (3) sending reminders to nonrespondents. This analysis is limited to the main study in Kenya, which used an experimental design for incentive and survey length and more robust reminders. (It excludes data from the Kenya pilot test, which did not include experiments.) We test whether incentive and survey length affect response rate and sample composition. For reminders, we test whether the characteristics of individuals who completed the interview on the first invitation differ from those that completed after receiving a reminder invitation. Results Describing Levels and Components of Nonresponse In Table 2, we show response rates and case dispositions for each country. Response rates were particularly low in Ghana (0.62%) and Nigeria (0.28%), but higher in Kenya (12%) and Uganda (14%). The cross-country differences stem in part from the frame composition. In all countries, response rates were higher among people who had previously participated in GeoPoll surveys compared with nonparticipants (see bottom two rows of Table 2). Previous participants comprised a large share of the frame in Kenya (17%) and Uganda (100%), compared with Nigeria and Ghana, where previous participants comprised <2% of the frame. Table 2 Response Rates and Case Dispositions, by Country   Kenya  Ghana  Nigeria  Uganda  Number of sampled cases  24,954  394,181  1,124,759  14,686  Number of completed interviews  2,960  2,277  2,392  2,068  Case dispositions (of entire sample)   No answer (%)  83.98  92.13  75.36  81.06   Completed (%)  11.86  0.58  0.21  14.08   Breakoff (%)  1.75  0.36  0.19  3.62   Rejected—quality (%)  0.36  0.02  0.01  0.22   Refused (%)  0.48  0.05  0.02  0.25   Undeliverable (%)  1.14  6.78  24.16  0.0   Ineligible (because of age) (%)  0.53  0.08  0.05  0.78   Total (%)  100  100  100  100  Response rate (%)   Entire sample (%)  12.06  0.62  0.28  14.19   Previously participated in  GeoPoll surveys (%)  45.87  15.46  28.45  14.19      Never participated in GeoPoll  surveys (%)  4.83  0.36  0.19  n/a    Kenya  Ghana  Nigeria  Uganda  Number of sampled cases  24,954  394,181  1,124,759  14,686  Number of completed interviews  2,960  2,277  2,392  2,068  Case dispositions (of entire sample)   No answer (%)  83.98  92.13  75.36  81.06   Completed (%)  11.86  0.58  0.21  14.08   Breakoff (%)  1.75  0.36  0.19  3.62   Rejected—quality (%)  0.36  0.02  0.01  0.22   Refused (%)  0.48  0.05  0.02  0.25   Undeliverable (%)  1.14  6.78  24.16  0.0   Ineligible (because of age) (%)  0.53  0.08  0.05  0.78   Total (%)  100  100  100  100  Response rate (%)   Entire sample (%)  12.06  0.62  0.28  14.19   Previously participated in  GeoPoll surveys (%)  45.87  15.46  28.45  14.19      Never participated in GeoPoll  surveys (%)  4.83  0.36  0.19  n/a  Note. Calculations based on American Association of Public Opinion Research (AAPOR) Response Rate #1. The target population is age 18–64 years. Table 2 Response Rates and Case Dispositions, by Country   Kenya  Ghana  Nigeria  Uganda  Number of sampled cases  24,954  394,181  1,124,759  14,686  Number of completed interviews  2,960  2,277  2,392  2,068  Case dispositions (of entire sample)   No answer (%)  83.98  92.13  75.36  81.06   Completed (%)  11.86  0.58  0.21  14.08   Breakoff (%)  1.75  0.36  0.19  3.62   Rejected—quality (%)  0.36  0.02  0.01  0.22   Refused (%)  0.48  0.05  0.02  0.25   Undeliverable (%)  1.14  6.78  24.16  0.0   Ineligible (because of age) (%)  0.53  0.08  0.05  0.78   Total (%)  100  100  100  100  Response rate (%)   Entire sample (%)  12.06  0.62  0.28  14.19   Previously participated in  GeoPoll surveys (%)  45.87  15.46  28.45  14.19      Never participated in GeoPoll  surveys (%)  4.83  0.36  0.19  n/a    Kenya  Ghana  Nigeria  Uganda  Number of sampled cases  24,954  394,181  1,124,759  14,686  Number of completed interviews  2,960  2,277  2,392  2,068  Case dispositions (of entire sample)   No answer (%)  83.98  92.13  75.36  81.06   Completed (%)  11.86  0.58  0.21  14.08   Breakoff (%)  1.75  0.36  0.19  3.62   Rejected—quality (%)  0.36  0.02  0.01  0.22   Refused (%)  0.48  0.05  0.02  0.25   Undeliverable (%)  1.14  6.78  24.16  0.0   Ineligible (because of age) (%)  0.53  0.08  0.05  0.78   Total (%)  100  100  100  100  Response rate (%)   Entire sample (%)  12.06  0.62  0.28  14.19   Previously participated in  GeoPoll surveys (%)  45.87  15.46  28.45  14.19      Never participated in GeoPoll  surveys (%)  4.83  0.36  0.19  n/a  Note. Calculations based on American Association of Public Opinion Research (AAPOR) Response Rate #1. The target population is age 18–64 years. Most nonresponse was because of respondents not answering. Refusal rates were low, with <1% of the sample explicitly refusing to participate by responding “STOP” via SMS. Breakoff rates were high: as a proportion of respondents who agreed to participate, breakoff rates were 38% in Ghana, 13% in Kenya, 46% in Nigeria, 20% in Uganda. These results are similar to a SMS survey in South Africa (Broich, 2015)—but lower than Johnson (2016) and Hoe and Grunwald (2015), which had breakoff rates of approximately 50% (authors' calculations.) There is some evidence that these high breakoff rates may impact sample representativeness. Among respondents who answered the age question, younger people were more likely to complete than older people (p < .01 in each country). The difference in age between breakoff and completed cases was 3.2 years in Ghana (age 30.1 vs. 26.9 years; p < .01), 1.8 years in Kenya (age 30.3 vs. 28.5 years; p < .01), 1.0 years in Nigeria (age 29.1 vs. 28.2 years; p < .01), and 1.5 years in Uganda (age 27.0 vs. 25.5 years; p < .01). While these differences are statistically significant, they are modest in size, suggesting that breakoff has minor effects on survey representativeness. Figure 1 shows question-level breakoff rates for each question. Among those who agreed to participate in the survey, breakoff rates for the first question (age) were 25% in Nigeria, 21% in Ghana, 13% in Kenya, and 11% in Uganda. Question-level breakoff rates declined to between 3 and 7% for the second question. Most subsequent questions had breakoff rates of <3%. Figure 1 View largeDownload slide Question-level breakoff rate Figure 1 View largeDownload slide Question-level breakoff rate The Representativeness of SMS Surveys To investigate the representativeness of SMS surveys, we compared the SMS completed interviews with 11 variables from three benchmark data sources. Age and gender Figure 2 shows the joint distribution of age and gender for the SMS and benchmark data. (See Online Supplement for exact percentages.) The figure includes 95% confidence intervals for the SMS data, but no confidence intervals for the benchmark data because they come from Censuses. In all countries, SMS overrepresents younger men. In Kenya, for example, men 18–24 years old comprise 30% of SMS respondents but were only 14% in Census data. SMS underrepresents women in most countries, even among the younger age groups. One notable exception is Kenya, where the SMS and Census data have similar representation of women 18–24 years old. But in all countries, differences among older women are stark. In Nigeria, for example, women 45–64 years old are 1.3% of SMS respondents versus 7.6% in Census data. Our analysis cannot identify the specific error sources that produce differences between SMS and benchmark data; for example, we cannot say whether coverage error or versus nonresponse errors lead to the underrepresentation of older women. This analysis just describes the differences, without attempting to attribute the difference to specific error sources. Figure 2 View largeDownload slide Comparison of age and gender between SMS and benchmark data (95% confidence interval for SMS). Note. SMS = short message service Figure 2 View largeDownload slide Comparison of age and gender between SMS and benchmark data (95% confidence interval for SMS). Note. SMS = short message service Given these age and gender differences, we created poststratification weights that align the SMS data to the joint distribution of age and gender from Census and UN data. These weights adjust for both coverage and nonresponse. Weights were derived by dividing the percentage from the Census/UN data by the corresponding percentage from the SMS data. For example, the weight for men 18–29 years old in Kenya was 0.48 (14.1/29.5%). No other adjustments to the weights were made. Weights ranged from 0.39 to 6.39 in Ghana, 0.48 to 2.87 in Kenya, 0.47 to 5.99 in Nigeria, and 0.50 to 13.82 in Uganda. The remainder of the analysis in this section compares weighted data from the SMS surveys to FTF surveys. This analysis allow us to ask: After weighting the SMS data by age and gender, how representative are SMS surveys? Socioeconomic and technology variables In Figure 3, we show percentage point differences between SMS and FTF data. See the “Methods” section for a description of how these differences are calculated. Positive numbers indicate that SMS overestimates the variable; negative numbers indicate that SMS underestimates the variable. Owing to the large number of comparisons, we show data from Ghana as an example; full results, including confidence intervals, for all countries are available in the Online Supplement. Figure 3 View largeDownload slide Percentage point differences between SMS and FTF benchmark data in Ghana (stars represent significant differences from 0). Note. FTF = face-to-face; SMS = short message service Figure 3 View largeDownload slide Percentage point differences between SMS and FTF benchmark data in Ghana (stars represent significant differences from 0). Note. FTF = face-to-face; SMS = short message service The results for education are striking. The SMS survey underrepresents people with a primary education or less by 62 percentage points in Ghana and overrepresents those with more education. When we restrict the FTF survey to individuals with a mobile phone, the differences between SMS and FTF attenuate only slightly. This is because mobile penetration is high in these countries; there are few differences between “FTF with Mobile” and “FTF Entire Sample.” These patterns suggest that coverage errors because of mobile phone access explain only a small portion of the gap between SMS and FTF surveys. Similar patterns for education exist in Kenya, Nigeria, and Uganda (see Online Supplement). The overrepresentation of educated people is reflected in the languages SMS respondents chose: in all countries, over 90% selected English. In the Technology Adoption FTF surveys, English was used 51% of the time in Kenya, 34% in Uganda, 27% in Ghana, and 19% in Nigeria. In this survey, educated people were more likely to be interviewed in English. SMS surveys also overrepresent people who are employed full-time or part-time and underrepresent people who are self-employed. Part of these differences is because of employed people being more likely to have access to mobile phones. Interestingly, SMS surveys overrepresent the unemployed—perhaps because of the financial incentive for participation. The results for modern shelter and finished roof (which indicate higher socioeconomic status) are mixed. In Ghana, SMS overrepresents people with modern shelter, but there are no differences for roof. In Kenya, Nigeria, and Uganda, there are positive differences in some countries but negative differences in others. This pattern is consistent with the results from a World Bank panel survey, which used SMS to follow-up with respondents who were initially recruited FTF (Ballivian, Azevedo, and Durbin, 2015). In that study, SMS surveys understated poverty relative to FTF surveys, but results for household infrastructure were more mixed. Compared with the FTF data, SMS respondents are less likely to share a SIM card in Ghana and other countries. In all countries except Kenya, SMS respondents underrepresents people who have difficulty charging their phones, potentially reflecting nonresponse because of a lack of electricity. And in all countries, SMS overrepresents Internet users. As before, restricting the FTF sample to those with mobile phones only slightly attenuates differences between SMS and FTF data. Improving the Representativeness of SMS Surveys The results in Figures 2 and 3 suggest that SMS surveys underrepresent women and people who are older, less educated, and less technologically savvy. Given these findings, what can survey designers do to increase the participation of these underrepresented groups? In this section, we analyze the experimental data from Kenya to test three strategies. Incentives and survey length The standard incentive (0.5 USD) had the same response rate as the higher incentive (1.25 USD), both 13% [χ2 (1) = 0.9; p = .33]. Response rates were similar for the 8 and 16 question surveys (13 vs. 12%). Although statistically significant [χ2 (1) = 8.3; p < .01], this test is based on a large sample size (n = 19,622). We do not interpret this result as a meaningful difference. A supplementary analysis showed there was no statistically significant interaction between incentive and survey length (z = 0.09; p = .93). The composition of respondents was similar, regardless of incentive or survey length. Full tables are available in the Online Supplement. Reminders Owing to the emphasis on speed, some SMS surveys do not send reminder invitations to follow up with nonrespondents. To investigate whether reminders can improve sample representativeness, we compared the composition of respondents who completed the survey in Kenya after the first invitation (initial respondents in Column A of Table 3) and those who completed after receiving a reminder (reminder respondents in Column B). The table also includes percentage point differences (Column C) and p-value from chi-square tests (Column D). Table 3 Sample Composition of Completed Interviews, by Reminder (Kenya) Variable  A. Completed on initial invitation (%)  B. Completed after reminder (%)  C. Percentage point difference (%)  D. p-value from chi-square test    (n = 2,563)  (n = 397)  (A−B)  Female  34  38  −4  .18  Age (years)      18–24  48  38  10  <.01      25–34  31  33  −2      35–44  13  17  −4      45–64  9  12  −3  Education      Primary or less  17  23  −6  <.01      Secondary  45  44  1      Postsecondary  38  33  5  Employed full-time  15  12  3  .28  Modern shelter  63  59  4  .23  Modern roof  66  67  2  .68  More than one SIM card  62  57  5  .03  Anyone else use this SIM card  30  31  1  .47  Difficulty charging phone  43  42  1  .79  Aware of Internet  77  80  3  .41  Use Internet (among aware)  80  74  6  .09  Variable  A. Completed on initial invitation (%)  B. Completed after reminder (%)  C. Percentage point difference (%)  D. p-value from chi-square test    (n = 2,563)  (n = 397)  (A−B)  Female  34  38  −4  .18  Age (years)      18–24  48  38  10  <.01      25–34  31  33  −2      35–44  13  17  −4      45–64  9  12  −3  Education      Primary or less  17  23  −6  <.01      Secondary  45  44  1      Postsecondary  38  33  5  Employed full-time  15  12  3  .28  Modern shelter  63  59  4  .23  Modern roof  66  67  2  .68  More than one SIM card  62  57  5  .03  Anyone else use this SIM card  30  31  1  .47  Difficulty charging phone  43  42  1  .79  Aware of Internet  77  80  3  .41  Use Internet (among aware)  80  74  6  .09  Note. The target population is age 18–64 years. Table 3 Sample Composition of Completed Interviews, by Reminder (Kenya) Variable  A. Completed on initial invitation (%)  B. Completed after reminder (%)  C. Percentage point difference (%)  D. p-value from chi-square test    (n = 2,563)  (n = 397)  (A−B)  Female  34  38  −4  .18  Age (years)      18–24  48  38  10  <.01      25–34  31  33  −2      35–44  13  17  −4      45–64  9  12  −3  Education      Primary or less  17  23  −6  <.01      Secondary  45  44  1      Postsecondary  38  33  5  Employed full-time  15  12  3  .28  Modern shelter  63  59  4  .23  Modern roof  66  67  2  .68  More than one SIM card  62  57  5  .03  Anyone else use this SIM card  30  31  1  .47  Difficulty charging phone  43  42  1  .79  Aware of Internet  77  80  3  .41  Use Internet (among aware)  80  74  6  .09  Variable  A. Completed on initial invitation (%)  B. Completed after reminder (%)  C. Percentage point difference (%)  D. p-value from chi-square test    (n = 2,563)  (n = 397)  (A−B)  Female  34  38  −4  .18  Age (years)      18–24  48  38  10  <.01      25–34  31  33  −2      35–44  13  17  −4      45–64  9  12  −3  Education      Primary or less  17  23  −6  <.01      Secondary  45  44  1      Postsecondary  38  33  5  Employed full-time  15  12  3  .28  Modern shelter  63  59  4  .23  Modern roof  66  67  2  .68  More than one SIM card  62  57  5  .03  Anyone else use this SIM card  30  31  1  .47  Difficulty charging phone  43  42  1  .79  Aware of Internet  77  80  3  .41  Use Internet (among aware)  80  74  6  .09  Note. The target population is age 18–64 years. Respondents who completed the survey after a reminder were older than those who completed on the first invitation [χ2 (3) = 17.0; p < .01]. Further, reminder respondents were more likely to have a primary school education or less compared with initial respondents (23 vs. 17%; χ2 (3) = 17.0; p < .01). These results suggest that reminders are one tool survey designers can use to increase the representation of underrepresented groups—specifically older people and less educated—to SMS surveys. One caveat, however, is that we did not conduct an experiment on reminders: it is possible that some of these differences may reflect the types of people who are simply late responders rather than the effect of a reminder per se. Discussion SMS is a low-cost survey mode that can collect data rapidly from large and geographically dispersed populations. For applications where speed and scale are paramount (e.g., rapid response during a public health emergency), SMS clearly offers numerous advantages. But are SMS surveys suitable for general population public opinion surveys? Our study in Ghana, Kenya, Nigeria, and Uganda shows that SMS surveys overrepresented younger, male, higher educated, and technologically oriented people. Many differences were large: in Nigeria, for example, SMS overrepresented the population with a postsecondary education by 50 percentage points. These results suggest that SMS cannot replace high-quality FTF surveys, at least in these countries during late 2015. When SMS surveys of the general population are needed because of cost and time constraints, we encourage researchers to take the representativeness of SMS surveys into account when interpreting the results. Will growing mobile phone penetration make SMS surveys more representative in the future? After all, growth in Internet access over time improved representativeness of Web surveys in Europe (Mohorko, de Leeuw, and Hox, 2013). For Ghana, Kenya, Nigeria, and Uganda, however, we suspect the answer is no. According to our results, only a small part of the difference between SMS and FTF surveys is because of lack of mobile phone access. This is because mobile phone access is already high in the countries we studied, and there are only small differences between people with and without phones. But in countries with lower mobile penetration, there may be greater differences between people with and without mobile phones. In these contexts, growing mobile penetration may indeed make SMS surveys more representative. Postsurvey adjustments (for age, gender, and other demographics) may improve SMS survey representativeness. But just like for Web surveys, weighting is unlikely to compensate completely for survey errors (Bethlehem, 2010; Peytchev, Carley-Baxter, & Black, 2011). Given large underrepresentation (especially for education), some weights will be large, reducing precision. In addition, there are likely factors that affect response to SMS survey that are not available in population control data (e.g., access to electricity). Further, at least in the present day, the full range of variables necessary for accurate adjustments is also unknown. Survey researchers also can tailor their survey designs to improve sample representativeness. In Kenya, sending reminders reduced nonresponse bias by increasing the share of less educated and older respondents. We recommend that population-based SMS surveys use reminders. Future research should use experiments to disentangle the effects from reminders from respondents to tend to respond late. Two other strategies we evaluated in Kenya—offering a shorter survey and a higher airtime incentive—were not successful in increasing the response rate meaningfully or in making the respondent pool more diverse. The silver lining, however, is that our experiment shows there may be some leeway with survey length: SMS surveys are often short (e.g., 10 questions), but our research suggests SMS surveys could potentially include more questions without compromising data quality. We encourage future research that explores an upper limit for question length in Kenya and other countries. Our study highlights two important aspects of sampling for general population SMS surveys. First, additional research is needed on vendor list frames. These frames can have coverage errors, but also can include ancillary data that RDD samples lack. Ancillary data can help researchers study more narrow target populations or can be used for sample stratification. Researchers will need to balance the utility of these ancillary data against possible coverage errors and panel conditioning effects. Second, future research is needed on multiple and shared phones. Many people have more than one SIM card (62% in our Kenya data) and share SIM cards (30% in our Kenya data). Multiple and shared SIM cards affect selection probabilities; failing to adjust for this variation could affect data quality (Labrique et al., 2017). Limitations This article seeks to describe the representativeness of SMS surveys by comparing SMS with FTF survey data. The lack of representative SMS data likely stems from a combination of undercoverage (e.g., vendor frame errors, incomplete phone access) and nonresponse errors (e.g., illiteracy, language coverage, poor phone networks, discomfort with technology, distrust of surveys, financial concerns). The chief limitation of our study is that we cannot point to the full range of errors that lead to unrepresentative SMS samples. For example, are coverage or nonresponse errors more important? What type of nonresponse errors creates the most bias? Also, does measurement error play a role in explaining differences between SMS and FTF surveys? We look to future research to investigate these important questions. Another limitation concerns our focus on Anglophone countries with high mobile penetration; our conclusions may not apply to other countries. We look forward to research on SMS surveys in a wider range of countries. Finally, future research is needed using RDD samples. This research is important for separating errors specific to GeoPoll’s list frames from errors that apply to the SMS mode more generally. Future Directions The explosion of mobile telephony offers researchers new ways to collect data quickly and inexpensively. SMS is the most widely used mobile phone survey mode. But other modes exist, such as IVR, CATI, mobile Web, and chatbots. These modes have strengths and weaknesses. For example, IVR can include illiterate people (unlike SMS), but still suffers from the same issues with nonresponse as SMS. Mobile Web avoids the 160-character limit of SMS, but requires smartphones to function well. The live interviewer in CATI may increase response rates and offer better measurement but is more expensive and introduces interviewer effects. We look forward to research that compares the representativeness, cost, and time of these various survey modes. These are the “early days” of mobile phone surveys in lower-income countries. Although our results suggest SMS surveys have serious limitations in the present day, this study is a snapshot during a dynamic period of change. We look forward to research tracking the evolution of SMS and other mobile phone survey modes. We also suspect that mixed mode approaches that combine SMS with other modes may be a promising way to improve survey representativeness. Supplementary Data Supplementary Data are available at IJPOR online. Charles Q. Lau (PhD, Sociology, University of California, Los Angeles) is a Survey Methodologist at RTI International. Ansie Lombaard (PhD, Research Methodology, University of Stellenbosch) is a Senior Global Innovation Director with the Kantar Insights Division, based in Cape Town, South Africa. Melissa Baker (BA, Social Anthropology, SOAS University of London) is the Kantar Public CEO of Social Research and Evaluation in Africa, based in Nairobi, Kenya. Joe Eyerman (PhD, Political Science, Florida State University) is a Center Director in RTI International's Survey Research Division. Lisa Thalji (MA, Psychology, Temple University) is Vice President of the Survey Research Division at RTI International. Acknowledgement The authors thank Stephanie Eckman, Ashley Amaya, Luis Crouch, Michelle Harrison, Anna Wetterberg, Mariano Sana, and Eric Johnson for valuable feedback. Any errors that remain are of authors. Funding RTI International and Kantar supported this work. References American Association for Public Opinion Research. ( 2010). New considerations for survey researchers when planning and conducting RDD telephone surveys in the U.S. with respondents reached via cell phone numbers. AAPOR Cell Phone Task Force. Retrieved from http://www.aapor.org/AAPOR_Main/media/MainSiteFiles/2010AAPORCellPhoneTFReport.pdf. Ballivian A., Azevedo J. P., Durbin W. 2015. Using mobile phones for high-frequency data collection. In Toninelli D., Pinter R., de Pedraza P. (Eds.), Mobile research methods: Opportunities and challenges of mobile research methodologies  (pp. 21– 39). London: Ubiquity Press. Retrieved from http://dx.doi.org/10.5334/bar.c. Bethlehem J. ( 2010). Selection bias in web surveys. International Statistical Review , 78, 161– 188. doi: 10.1111/j.1751-5823.2010.00112.x. Google Scholar CrossRef Search ADS   Broich C. ( 2015). Offline data collection in Sub-Saharan Africa using SMS surveys: lessons learned. Paper presented at the meeting of the American Association for Public Opinion Research, Hollywood, FL. Carmer L., Boots M. ( 2013). Barriers and solutions in using M4D: Connecting directly to citizens for scalable impact. VOTO Mobile White paper. Retrieved from https://www.votomobile.org/files/M4D_Barriers_Whitepaper.pdf. Centers for Disease Control and Prevention. ( 2015). Data for health initiative. Retrieved from http://www.cdc.gov/globalhealth/healthprotection/data.html. Conrad F., Schober M., Antoun C., Hupp A., Yan H. ( 2017). Text interviews on mobile devices. In Biemer P., de Leeuw E., Eckman S., Edwards B., Kreuter F., Lyberg L., Tucker N. C., West B. (Eds.), Total survey error in practice . Hoboken, NJ: John Wiley & Sons, Inc. doi: 10.1002/9781119041702.ch14. Couper M. P. ( 2000). Web Surveys: A Review of Issues and Approaches. Public Opinion Quarterly , 64, 464– 494. Google Scholar CrossRef Search ADS PubMed  De Bruijne M., Wijnant A. ( 2014). Improving response rates and questionnaire design for mobile web surveys. Public Opinion Quarterly , 78, 951– 962. doi: 10.1093/poq/nfu046. Google Scholar CrossRef Search ADS   Dillon B. ( 2012). Using mobile phones to collect panel data in developing countries. Journal of International Development , 24, 518– 527. doi: 10.1002/jid.1771. Google Scholar CrossRef Search ADS   Eckman S. ( 2015). Does the inclusion of non-internet households in a web panel reduce coverage bias? Social Science Computer Review , 34, 41– 58. doi: 10.1177/0894439315572985. Google Scholar CrossRef Search ADS   Fan W., Yan Z. ( 2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior , 26, 132– 139. doi: 10.1016/j.chb.2009.10.015. Google Scholar CrossRef Search ADS   Ganesan M., Prashant S., Jhunjhunwala A. ( 2012). A review on the challenges in implementing mobile phone based data collection in developing countries. Journal of Health Informatics in Developing Countries , 6, 366– 374. GeoPoll ( 2015). Perceptions of South Africa’s state of the nation: TNS. Retrieved from: https://research.geopoll.com/case-studies/tns-state-of-the-nation-south-africa.html. Gibson D. G., Pereira A., Farrenkopf B. A., Labrique A. B., Pariyo G. W., Hyder A. A. ( 2017). Mobile phone surveys for collecting population-level estimates in low- and middle-income countries: A literature review. Journal of Medical Internet Research , 19, e139. doi: 10.2196/jmir.7428. Google Scholar CrossRef Search ADS PubMed  GSMA. ( 2015). The mobile economy: Sub-Saharan Africa 2015. Retrieved from https://gsmaintelligence.com/research/?file=721eb3d4b80a36451202d0473b3c4a63&download. Hoe N., Grunwald H. ( 2015). The role of automated SMS text messaging in survey research. Survey Practice , 8( 6), 1– 15. Google Scholar CrossRef Search ADS   Johnson D. ( 2016). Collecting data from mHealth users via SMS surveys: A case study in Kenya. Survey Practice , 9( 1), 1– 8. Google Scholar CrossRef Search ADS   Labrique A., Blynn E., Ahmed S., Gibson D., Pariyo G., Hyder A. A. ( 2017). Health surveys using mobile phones in developing countries: Automated active strata monitoring and other statistical considerations for improving precision and reducing biases. Journal of Medical Internet Research , 19, e121. doi: 10.2196/jmir.7329. Google Scholar CrossRef Search ADS PubMed  Lombaard A., Richman M. ( 2015). Mobile research in emerging markets: Taking the step into the world of probability sampling. Paper presented at the meeting of the World Association for Public Opinion Research, Buenos Aires, Argentina. Leo B., Morello R., Mellon J., Peixoto T., Davenport S. ( 2015). Do mobile phone surveys work in poor countries? Washington, DC: Center for Global Development Working Paper 398. Mahfoud Z., Ghandour L., Ghandour B., Mokdad A., Sibai A. M. ( 2015). Cell phone and face-to-face interview responses in population-based surveys: How do they compare? Field Methods , 27, 39– 54. doi: 10.1177/1525822X14540084. Google Scholar CrossRef Search ADS   Mitullah W., Kamau P. ( 2013). The partnership of free speech & good governance in Africa. Afrobarometer Policy Brief No. #3. Retrieved from http://afrobarometer.org/sites/default/files/publications/Briefing%20paper/ab_r5_policypaperno3.pdf. Mitullah W. V., Samson R., Wambua P. M., Balongo S. ( 2016). Building on Progress: Infrastructure development still a major challenge in Africa. Afrobarometer Dispatch No. 69. Retrieved from http://afrobarometer.org/sites/default/files/publications/Dispatches/ab_r6_dispatchno69_infrastructure_remains_challenge_en.pdf. Mohorko A., de Leeuw E., Hox J. ( 2013). Internet coverage and coverage bias in Europe: Developments across countries and time. Journal of Official Statistics , 29, 609– 622. doi: 10.2478/jos-2013-0042. Google Scholar CrossRef Search ADS   Peytchev A., Carley-Baxter L., Black M. ( 2011). Multiple sources of nonobservation error in telephone surveys: Coverage and nonresponse. Sociological Methods and Research , 40, 138– 168. doi: 10.1177/0049124110392547. Google Scholar CrossRef Search ADS   Pew Research Center. ( 2015). Cell phones in Africa: Communication lifeline. Pew Research Center. Retrieved from http://www.pewglobal.org/2015/04/15/cell-phones-in-africa-communication-lifeline/. Steeh C., Buskirk T., Callegaro M. ( 2007). Using text messages in U.S. mobile phone surveys. Field Methods , 19, 59– 75. doi: 10.1177/1525822X06292852. Google Scholar CrossRef Search ADS   UNESCO. ( 2014). Adult and youth literacy, UIS Fact Sheet. UNESCO Institute for Statistics. Retrieved from http://unesdoc.unesco.org/images/0022/002295/229504e.pdf. UNICEF. ( 2015). UNICEF’s U-report social platform hits 1 million active users. UNICEF News Note, 16 July, 2015. Retrieved from http://www.unicef.org/media/media_82583.html. World Health Organization. ( 2014). Government of Senegal boots ebola awareness through SMS campaign. Retrieved from http://www.who.int/features/2014/senegal-ebola-sms/en/. © The Author(s) 2018. Published by Oxford University Press on behalf of The World Association for Public Opinion Research. All rights reserved. This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png International Journal of Public Opinion Research Oxford University Press

How Representative Are SMS Surveys in Africa? Experimental Evidence From Four Countries

Loading next page...
 
/lp/ou_press/how-representative-are-sms-surveys-in-africa-experimental-evidence-kn7jda16QQ
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press on behalf of The World Association for Public Opinion Research. All rights reserved.
ISSN
0954-2892
eISSN
1471-6909
D.O.I.
10.1093/ijpor/edy008
Publisher site
See Article on Publisher Site

Abstract

Abstract Short message service (SMS) surveys are fast, inexpensive, and increasingly common. However, there is limited research on this new mode. Using an experimental design, we conducted general population SMS surveys in Ghana, Kenya, Nigeria, and Uganda. This article (1) reports the levels and components of nonresponse; (2) evaluates the representativeness of SMS surveys relative to benchmark data; and (3) explores strategies to improve representativeness. We find that SMS surveys underrepresent women, older people, those with less education, and less technologically savvy people. Sending reminders improved representativeness, but offering shorter surveys or higher incentives did not. We conclude that presently, general population SMS surveys cannot replace face-to-face surveys. Our article provides practical guidance for survey designers and directions for future research. The rapid growth of mobile phones is transforming data collection in low- and middle-income countries. In sub-Saharan Africa, mobile phone subscriptions nearly doubled between 2010 and 2015, from 200 to 386 million subscribers (GSMA, 2015). More people across Africa have access to mobile phone networks than to electricity and piped water (Mitullah, Samson, Wambua, & Balongo, 2016). Owing to this growth, mobile applications are flourishing, and people increasingly communicate via short message service (SMS), or text messaging. In an SMS survey, each question is sent as a separate SMS. Respondents reply by SMS with the number corresponding to their response, and then receive the next question via another SMS. In low- and middle-income countries, SMS surveys have grown in popularity in recent years because of their speed and low cost. For example, SMS is used for large-scale public opinion surveys (GeoPoll, 2015; UNICEF, 2015), rapid surveys during emergencies such as the 2014 Ebola crisis (World Health Organization, 2014), and public health surveillance (Centers for Disease Control and Prevention, 2015). The recent proliferation of SMS surveys parallels the growth of Web surveys in the late 1990s (Couper, 2000). Because SMS and Web are self-administered through a device, they offer enormous cost and time savings, but also present challenges with coverage and nonresponse. The remarkable growth of SMS surveys for public opinion and other uses in low- and middle-income countries has outpaced scientific investigation of the validity and best practices of this new mode. There is more research on SMS surveys in developed countries (Conrad, Schober, Antoun, Hupp, & Yan, 2017; De Bruijne & Wijnant, 2014; Hoe & Grunwald, 2015; Steeh, Buskirk, & Callegaro, 2007). For SMS surveys in lower-income countries, there is good reason to anticipate significant coverage errors (because of lack of mobile phone access and frame errors) and nonresponse errors (because of illiteracy, concern about costs, distrust of surveys, difficulty charging devices). Some of these errors are like any self-administered mode, whereas others are unique to SMS. However, there is little research on the representativeness of SMS surveys compared with benchmark face-to-face (FTF) surveys (Gibson et al., 2017). Further, there is scant experimental research on how to improve the representativeness of SMS developing country samples through strategies like incentives, questionnaire length, and reminders. Our research addresses the lack of research on SMS survey representativeness and best practices in design. We seek to evaluate the quality of SMS survey samples and provide practical guidance about optimizing SMS surveys in developing countries. Using an experimental design, we conducted SMS surveys of the general population (age 18–64 years) in Kenya, Ghana, Nigeria, and Uganda. We ask the following research questions: RQ1: How representative are SMS surveys? This question has three subparts: 1a. What are the levels and components of nonresponse to SMS surveys? 1b. How representative are SMS population-based survey samples? 1c. How much does incomplete access to mobile phones affect representativeness? RQ2: How can survey design improve the representativeness of SMS surveys? Background: Coverage and Nonresponse in SMS Surveys In this section, we discuss how coverage and nonresponse errors can affect the representativeness of SMS surveys. Coverage Coverage refers to the extent to which the sampling frame overlaps with the target population. Undercoverage occurs when units in the target population are omitted from the frame. If many units are nonrandomly missing from the frame, coverage errors can create bias (Eckman, 2015). We anticipate three main types of undercoverage errors for SMS surveys. First, many SMS surveys interview mobile phone users but draw inferences to the general population. Across Africa, however, 16% of adults lack access to a mobile phone (Mitullah & Kamau, 2013); phone access is more common among people who are educated, male, and those who speak English (Pew Research Center, 2015). This undercoverage could bias survey statistics, just as lack of universal Internet access can bias Web surveys (Bethlehem, 2010; Eckman, 2015), and lack of complete phone access can bias telephone surveys (Peytchev, Carley-Baxter, & Black, 2011). In the African context, it is unclear whether undercoverage will affect survey statistics. For example, if mobile phone access is high enough, and differences between those with and without phone phones are not large, then this undercoverage may not meaningfully affect survey statistics. Second, the method of frame construction can also introduce coverage errors. For random digit dial (RDD) frames, all or nearly all numbers in the target population should be available for selection. For list frames, vendors typically build frames by establishing relationships with mobile network operators. This method provides vendors with access to known active phone numbers—improving efficiency during data collection. However, vendors may not have relationships with all mobile network operators in a country, leading to undercoverage. These errors can impact sample representativeness because the subscriber bases differ across mobile network operators in terms of socioeconomic status, geography, age, and other factors. Third, regulations in some countries prevent sending people SMS invitations without prior consent. This is the case with Uganda in our study (and many other developed countries; see American Association for Public Opinion Research, 2010). Thus, only individuals who have volunteered for surveys in the past are included in the frame, impacting sample representativeness. In contrast, it is possible to send SMS invitations without prior consent in other countries (Kenya, Nigeria, and Ghana in our study). Nonresponse Like other self-administered modes (e.g., Web, mail), response rates to SMS surveys are expected to be lower than interviewer-administered modes. Without an interviewer to convey the survey’s legitimacy and importance of responding, individuals may find it easy to ignore the survey request. Like Web surveys, people may not respond because of discomfort with technology (Ganesan, Prashant, &Jhunjhunwala, 2012; Hoe & Grunwald, 2015) or concerns that the survey is illegitimate or “spam.” Further, people weaker literacy skills cannot participate. In 2014, the illiteracy rate was 41% across Africa (UNESCO, 2014), and illiterate people are more likely to be poor, female, and rural (Leo, Morello, Mellon, Peixoto, & Davenport, 2015). The nature of SMS surveys in lower-income countries also presents other challenges. SMS messages are generally limited to 160 characters, which leaves little room to craft a comprehensive, persuasive survey request. SMS surveys cannot include visual cues (e.g., logos or graphics) to bolster response, as in Web surveys (Fan & Yan, 2010). In addition, people may be concerned about incurring data charges on their phone, even if an incentive is offered. Infrastructure is also a challenge. People in developing countries may not receive SMS survey invitations because of poor mobile network connections, particularly in rural areas (Ganesan et al., 2012). The lack of access to affordable, regular electricity means that many people keep their phones off and only turn on their phones when needed (Carmer & Boots, 2013; Dillon, 2012). When the phone is turned on, the invitation may be mixed in with other messages or the data collection period may be over. Another source of nonresponse concerns access to a shared phone. Sharing mobile phones is common in Africa (Pew Research Center, 2015). Shared phones may lead to nonresponse errors and bias if certain types of people are more likely to have access to phone. For example, a husband may be more likely to possess a mobile phone that he shares with his wife, reducing the possibility that the wife will receive and answer the SMS survey. Previous Research on SMS Survey Representativeness Coverage and nonresponse errors can impact the representativeness of SMS general population surveys. In South Africa, two comparisons between SMS and FTF surveys suggest that SMS surveys may overrepresent educated, affluent, and younger people (Broich, 2015; Lombaard & Richman, 2015). Similar patterns have been found when recontacting previous FTF survey respondents via SMS (Ballivian, Azevedo, & Durbin, 2015) and telephone (Mahfoud, Ghandour, Ghandour, Mokdad, & Sibai, 2015)—as well as cross-sectional interactive voice response (IVR) surveys (Leo et al., 2015). Building on this research, we provide a more comprehensive view by studying four countries and using a richer set of variables for comparison. We also investigate the impact of one type of undercoverage error: the lack of universal mobile phone access. Many people lack phones (in our data, 26% in Nigeria, 25% in Uganda, 21% in Ghana, and 8% in Kenya), and because demographics may be correlated with phone access, the lack of phone access may impact representativeness of SMS surveys. Researchers have just started exploring ways to improve the representativeness of SMS surveys. Johnson (2016) reported that time of day for the survey invitation did not affect response rates. Most other research has focused on incentives. Typically, incentives are mobile phone airtime that is automatically applied to the respondent’s mobile phone account after the survey. While a nominal incentive is important for response, there is little evidence about the optimal incentive level for SMS surveys. Johnson (2016) found no difference in response rates between a 0.50 and 1.00 USD in airtime among users of a family planning SMS service in Kenya. This finding echoes research from computer-assisted telephone interviewing (CATI) surveys in developing countries (Ballivian et al., 2015) and Web surveys in developed countries (Fan & Yan, 2010), where nominal increases in incentives do not meaningfully improve data quality. Two studies found that entering respondents in a lottery raised response rates more than airtime (Broich, 2015; Leo et al., 2015), though Johnson (2016) found no difference. Notably, these studies focus exclusively on response rates, and do not investigate whether incentives impact sample representativeness. Our study also considers sample composition. We also explore two other ways to improve the representativeness of SMS surveys: survey length and reminders. Offering a shorter survey—for example, 8 questions instead of 16—may encourage more individuals to respond, particularly lower socioeconomic status people who may be more concerned about data charges and power on their phone. If this is true, then a shorter survey may result in more representative samples. Further, sending reminders is another less explored way of reducing nonresponse error. Reminders are standard protocols for increasing response rates and improving representativeness in Web surveys (Fan & Yan, 2010). Some SMS surveys use reminders, but many SMS surveys do not because of the emphasis on speed. People with more sporadic phone access or time to complete a survey—such as less educated or rural people—may be more likely to participate if they receive a reminder. Data and Methods This section describes the SMS surveys we conducted, the benchmark FTF data we use to evaluate the representativeness of the SMS surveys, and the analysis. SMS Survey Data We conducted SMS surveys in Ghana, Kenya, Nigeria, and Uganda, working with GeoPoll as our sample and data collection vendor. This section describes the sample design, questionnaire, and data collection for the SMS surveys. We also elaborate on the Kenya survey, which included several experiments. Sample design GeoPoll obtains lists of active mobile phone numbers through established relationships with mobile phone network operators in each country. These lists are updated periodically to add newly activated phone numbers and to delete inactive numbers. Using these frames, we drew random samples of mobile phone numbers in each country. We stratified the samples by mobile network operator and geography (province in Kenya, region in Ghana and Uganda, geopolitical zone in Nigeria). The stratification used proportional allocation, yielding a self-weighting sample of phone numbers. There was no within-phone sampling because of challenges in sampling users of the same phone. We return to this issue in the “Discussion” section. GeoPoll’s method of constructing a frame through mobile network operators has the benefit of excluding numbers that were never assigned or not in service. But this method can also lead to coverage errors. At the time of our study, GeoPoll lacked agreements with some mobile network operators in Ghana. Further, even if GeoPoll had an agreement with a network operator, the sample may have undercoverage (new numbers were activated but were not yet in the frame) and overcoverage (inactive numbers or USB modems). Table 1 shows the composition of mobile network operators between completed interviews in our SMS survey and external data from government communication authorities. In Kenya, there were only minor differences. In Ghana, however, 90% of our achieved sample was from the MTN network, but MTN comprises only 46% of the overall market. Discrepancies also existed in Nigeria and Uganda. Discrepancies between data sources reflect both coverage and nonresponse errors in our SMS surveys, and affect sample representativeness because mobile network operators have different sociodemographic subscriber bases (Broich, 2015). Table 1 Comparison of Mobile Network Operator in SMS Survey Data and Mobile Market Share (Percentages)   Kenya   Ghana   Nigeria   Uganda     Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   n  2,960  …  2,277  …  2,392  …  2,068  …  Safaricom  77.5  67.0  …  …  …  …  …  …  Telkom (Orange)  7.0  11.2  …  …  …  …  1.5  3.5  Equitel (Finserve)  0  2.4  …  …  …  …  …  …  Airtel  13.4  19.4  4.4  13.0  15.8  21  89.3  38.6  MTN  …  …  90.0  46.3  58.4  42  5.7  51.3  Vodafone  …  …  2.3  22.4  …  …  …  …  Tigo  …  …  1.0  13.6  …  …  …  …  Expresso  …  …  0.5  0.4  …  …  …  …  Uganda Telkom  …  …  …  …  …  …  0  5.2  Sure Telkom  …  …  …  …  …  …  0  0.6  Globacom  …  …  0.5  4.3  20.0  21  …  …  Etisalat  …  …  …  …  4.5  16  …  …  Yu  1.0  0  …  …  …  …  …  …  Visa  …  …  …  …  0.5  0  …  …  UTL  …  …  …  …  …  …  1.8  …  Other  0.6  0  …  …  …  …  0.9  0.8  Do not know  0.4  …  1.1    1.0  0  0.9  …  Total  100.0  100.0  100.0  100.0  100.0  100.0  100.0  100.0    Kenya   Ghana   Nigeria   Uganda     Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   n  2,960  …  2,277  …  2,392  …  2,068  …  Safaricom  77.5  67.0  …  …  …  …  …  …  Telkom (Orange)  7.0  11.2  …  …  …  …  1.5  3.5  Equitel (Finserve)  0  2.4  …  …  …  …  …  …  Airtel  13.4  19.4  4.4  13.0  15.8  21  89.3  38.6  MTN  …  …  90.0  46.3  58.4  42  5.7  51.3  Vodafone  …  …  2.3  22.4  …  …  …  …  Tigo  …  …  1.0  13.6  …  …  …  …  Expresso  …  …  0.5  0.4  …  …  …  …  Uganda Telkom  …  …  …  …  …  …  0  5.2  Sure Telkom  …  …  …  …  …  …  0  0.6  Globacom  …  …  0.5  4.3  20.0  21  …  …  Etisalat  …  …  …  …  4.5  16  …  …  Yu  1.0  0  …  …  …  …  …  …  Visa  …  …  …  …  0.5  0  …  …  UTL  …  …  …  …  …  …  1.8  …  Other  0.6  0  …  …  …  …  0.9  0.8  Do not know  0.4  …  1.1    1.0  0  0.9  …  Total  100.0  100.0  100.0  100.0  100.0  100.0  100.0  100.0  Notes. SMS data are unweighted. The target population for the SMS surveys is age 18–64 years. SMS = short message service. External data sources: Kenya: Fourth Quarter Sector Statistics Report for the Financial Year 2014/2015 (April–June 2015), Communications Authority of Kenya. Ghana: Mobile voice subscription trends for August 2015, National Communications Authority. Uganda: Mobile network access for MVNOs, Market Assessment, January 2015, Uganda Communications Commission. Nigeria: Market share of mobile operators, September 2015, Nigerian Communications Commission. Table 1 Comparison of Mobile Network Operator in SMS Survey Data and Mobile Market Share (Percentages)   Kenya   Ghana   Nigeria   Uganda     Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   n  2,960  …  2,277  …  2,392  …  2,068  …  Safaricom  77.5  67.0  …  …  …  …  …  …  Telkom (Orange)  7.0  11.2  …  …  …  …  1.5  3.5  Equitel (Finserve)  0  2.4  …  …  …  …  …  …  Airtel  13.4  19.4  4.4  13.0  15.8  21  89.3  38.6  MTN  …  …  90.0  46.3  58.4  42  5.7  51.3  Vodafone  …  …  2.3  22.4  …  …  …  …  Tigo  …  …  1.0  13.6  …  …  …  …  Expresso  …  …  0.5  0.4  …  …  …  …  Uganda Telkom  …  …  …  …  …  …  0  5.2  Sure Telkom  …  …  …  …  …  …  0  0.6  Globacom  …  …  0.5  4.3  20.0  21  …  …  Etisalat  …  …  …  …  4.5  16  …  …  Yu  1.0  0  …  …  …  …  …  …  Visa  …  …  …  …  0.5  0  …  …  UTL  …  …  …  …  …  …  1.8  …  Other  0.6  0  …  …  …  …  0.9  0.8  Do not know  0.4  …  1.1    1.0  0  0.9  …  Total  100.0  100.0  100.0  100.0  100.0  100.0  100.0  100.0    Kenya   Ghana   Nigeria   Uganda     Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   Survey (%)   External (%)   n  2,960  …  2,277  …  2,392  …  2,068  …  Safaricom  77.5  67.0  …  …  …  …  …  …  Telkom (Orange)  7.0  11.2  …  …  …  …  1.5  3.5  Equitel (Finserve)  0  2.4  …  …  …  …  …  …  Airtel  13.4  19.4  4.4  13.0  15.8  21  89.3  38.6  MTN  …  …  90.0  46.3  58.4  42  5.7  51.3  Vodafone  …  …  2.3  22.4  …  …  …  …  Tigo  …  …  1.0  13.6  …  …  …  …  Expresso  …  …  0.5  0.4  …  …  …  …  Uganda Telkom  …  …  …  …  …  …  0  5.2  Sure Telkom  …  …  …  …  …  …  0  0.6  Globacom  …  …  0.5  4.3  20.0  21  …  …  Etisalat  …  …  …  …  4.5  16  …  …  Yu  1.0  0  …  …  …  …  …  …  Visa  …  …  …  …  0.5  0  …  …  UTL  …  …  …  …  …  …  1.8  …  Other  0.6  0  …  …  …  …  0.9  0.8  Do not know  0.4  …  1.1    1.0  0  0.9  …  Total  100.0  100.0  100.0  100.0  100.0  100.0  100.0  100.0  Notes. SMS data are unweighted. The target population for the SMS surveys is age 18–64 years. SMS = short message service. External data sources: Kenya: Fourth Quarter Sector Statistics Report for the Financial Year 2014/2015 (April–June 2015), Communications Authority of Kenya. Ghana: Mobile voice subscription trends for August 2015, National Communications Authority. Uganda: Mobile network access for MVNOs, Market Assessment, January 2015, Uganda Communications Commission. Nigeria: Market share of mobile operators, September 2015, Nigerian Communications Commission. Another source of undercoverage stems from government regulations in Uganda that prevent survey organizations from sending SMS invitations to people who have not previous opted into a GeoPoll survey. Therefore, our sample in Uganda is an opt-in sample: 100% had previously participated in GeoPoll surveys before in Uganda. This contrasts with previous participation rates of 1.5% in Ghana, 0.3% in Nigeria, and 17.3% in Kenya. Questionnaire The core SMS survey consisted of 16 questions about demographics, socioeconomic status, and technology. Each survey question was sent as a separate SMS to the respondent. The respondent selected an answer from a closed ended list by entering a number associated with the response, and then the next question was sent via SMS. To ensure comparability with the benchmark data, we used questions and response options from the FTF data, with minor adaptations for mode. See Online Supplement for question wording. Experimental design in Kenya For each sampled case, we randomly assigned survey length (8 vs. 16 questions) and incentive amount (the standard 0.5 USD incentive vs. 1.25 USD). We also randomized whether “don’t know” was offered as a response option; we report this experiment in a different article (in process at time of publication). Data collection We began collecting data in Kenya first, starting with a pilot study to test our procedures (n = 457 completed interviews). The pilot test did not reveal any problems, so we merged the pilot test data with the main study for most analyses. We collected data in Kenya in late November and early December of 2015, and the other countries in late November 2015. Sample members received an SMS that introduced the study, specified the number of questions, and offered an incentive in the form of mobile phone airtime. In Kenya, the incentive was randomly assigned to be either 0.50 USD or 1.25 USD. In Ghana, Nigeria, and Uganda, the incentive was 0.50 USD. The survey was offered in the major languages for each country (English and Swahili in Kenya; English and Twi in Ghana; English, Hausa, Igbo, and Yoruba in Nigeria; and English and Luganda in Uganda). Respondents selected their language on the first screen of the survey. Individuals <18 years or >64 years of age were screened out as ineligible. Eligible respondents then answered survey questions and received mobile phone airtime for completing. GeoPoll sent up to three follow-up reminders to nonrespondents. A small proportion of cases (<0.5%) were dropped because of concerns about data quality. Benchmark Data We compare SMS survey respondents with respondents from three benchmark FTF sources. We compare the SMS and FTF data across three groups of variables: demographics (age and gender), socioeconomics and technology (education, employment, multiple SIM card, shared SIM card, difficulty charging phone, aware of Internet, use Internet), and housing (shelter type and roof). For the age and gender comparisons, we compare SMS data with Census Data in Kenya (2009), Nigeria (2006), Ghana (2010), and United Nations (UN) population data for Uganda (2010). For the socioeconomic and technology comparisons, we analyze the “Technology Adoption Surveys,” four computer-assisted personal interviewing surveys in Kenya (n = 3,364), Ghana (n = 3,113), Nigeria (n = 3,042), and Uganda (n = 3,075) conducted in 2014–2015. Response rates were 64% in Kenya, 54% in Ghana, 64% in Nigeria, and 77% in Uganda. The survey is based on an area probability sample of households created through geographic information systems technology. Housing structure (roof and shelter type) is a commonly used indicator of socioeconomic status in Africa. For these comparisons, we analyze FTF Afrobarometer surveys from Kenya, Ghana, Nigeria, and Uganda. In each country, 2,400 interviews were completed. These surveys were based on nationally representative samples of the adult population age ≥18 years, using a random walk procedure to sample households. Each country used paper-and-pencil interviewing, and data were collected in 2012 (and 2011–2012 in Uganda). Response rates were 73% in Kenya, 73% in Ghana, 90% in Nigeria, and 87% in Uganda. The Afrobarometer is one of the main data sources about political attitudes and behavior in Africa and is used widely by academics, government, and donors (see http://www.afrobarometer.org/.). Analysis Our analysis proceeds in three stages. The first stage investigates Research Question 1a: What are the levels and components (e.g., refusal, no answer, breakoff) of nonresponse to SMS surveys in each country? We present response rates and case dispositions for each country. We explore one component of nonresponse in greater detail: breakoff. In this analysis, we present breakoff rates for each country and question-level breakoff rates to understand when respondents stop answering questions. We compare the demographic characteristics of breakoff cases from completed cases to understand how breakoff can impact sample representativeness. The second stage investigates the representativeness of SMS surveys (Research Question 1b) and how much lack of mobile phone access affects representativeness (Research Question 1c) by comparing SMS and FTF respondents. For comparisons by age and gender, we compare SMS surveys to Census (Ghana, Kenya, Nigeria) or UN data (Uganda) through univariate distributions. Based on this analysis, we created poststratification weights to align the SMS survey samples to age and gender population totals from Census or UN data. This allows us to evaluate SMS sample representativeness in subsequent analysis after adjusting for age and gender. See “Results” section for more information on weight construction, sizes of weights, and other information about weighting. We compare the weighted SMS data to 11 variables about sociodemographics and technology (Technology Adoption Surveys) and housing structure (Afrobarometer.) These variables are all categorical or binary. For each variable, we present percentage point differences between the SMS data and the entire sample from the FTF survey:   PercentagefromSMSsurvey−PercentagefromFTFsurvey. (1) We show the percentage point differences graphically. In the graphic, we highlight percentage point differences that are significantly different from 0 (defined as having nonoverlapping confidence intervals) using stars. Full tables that contain point estimates and confidence intervals for the SMS and FTF estimates are available in the Online Supplement. We treat the FTF survey as the benchmark data source, and interpret departures from the FTF survey as evidence of errors in the SMS survey. Positive numbers mean the SMS survey overestimates the percentage; negative numbers mean the SMS survey underestimates the percentage. To investigate the impact of undercoverage because of mobile phone access, we compare SMS and FTF data, but restricting the FTF data to respondents with a mobile phone:   PercentagefromSMSsurvey–PercentagefromFTFsurvey(basedonFTFrespondentswithphones). (2) This comparison eliminates the component of coverage error because of an individual’s mobile phone access—although other types of coverage error (because of mobile network operator, opt-in), and nonresponse may still be present. If indeed undercoverage because of incomplete access to mobile phones affects representativeness, then the differences between SMS and FTF should be attenuated once we restrict the FTF survey data to individuals with mobile phones. In the third stage, we explore Research Question 2: How can design improve the representativeness of SMS surveys? We explore three strategies: (1) increasing the incentive amount, (2) offering a shorter survey, and (3) sending reminders to nonrespondents. This analysis is limited to the main study in Kenya, which used an experimental design for incentive and survey length and more robust reminders. (It excludes data from the Kenya pilot test, which did not include experiments.) We test whether incentive and survey length affect response rate and sample composition. For reminders, we test whether the characteristics of individuals who completed the interview on the first invitation differ from those that completed after receiving a reminder invitation. Results Describing Levels and Components of Nonresponse In Table 2, we show response rates and case dispositions for each country. Response rates were particularly low in Ghana (0.62%) and Nigeria (0.28%), but higher in Kenya (12%) and Uganda (14%). The cross-country differences stem in part from the frame composition. In all countries, response rates were higher among people who had previously participated in GeoPoll surveys compared with nonparticipants (see bottom two rows of Table 2). Previous participants comprised a large share of the frame in Kenya (17%) and Uganda (100%), compared with Nigeria and Ghana, where previous participants comprised <2% of the frame. Table 2 Response Rates and Case Dispositions, by Country   Kenya  Ghana  Nigeria  Uganda  Number of sampled cases  24,954  394,181  1,124,759  14,686  Number of completed interviews  2,960  2,277  2,392  2,068  Case dispositions (of entire sample)   No answer (%)  83.98  92.13  75.36  81.06   Completed (%)  11.86  0.58  0.21  14.08   Breakoff (%)  1.75  0.36  0.19  3.62   Rejected—quality (%)  0.36  0.02  0.01  0.22   Refused (%)  0.48  0.05  0.02  0.25   Undeliverable (%)  1.14  6.78  24.16  0.0   Ineligible (because of age) (%)  0.53  0.08  0.05  0.78   Total (%)  100  100  100  100  Response rate (%)   Entire sample (%)  12.06  0.62  0.28  14.19   Previously participated in  GeoPoll surveys (%)  45.87  15.46  28.45  14.19      Never participated in GeoPoll  surveys (%)  4.83  0.36  0.19  n/a    Kenya  Ghana  Nigeria  Uganda  Number of sampled cases  24,954  394,181  1,124,759  14,686  Number of completed interviews  2,960  2,277  2,392  2,068  Case dispositions (of entire sample)   No answer (%)  83.98  92.13  75.36  81.06   Completed (%)  11.86  0.58  0.21  14.08   Breakoff (%)  1.75  0.36  0.19  3.62   Rejected—quality (%)  0.36  0.02  0.01  0.22   Refused (%)  0.48  0.05  0.02  0.25   Undeliverable (%)  1.14  6.78  24.16  0.0   Ineligible (because of age) (%)  0.53  0.08  0.05  0.78   Total (%)  100  100  100  100  Response rate (%)   Entire sample (%)  12.06  0.62  0.28  14.19   Previously participated in  GeoPoll surveys (%)  45.87  15.46  28.45  14.19      Never participated in GeoPoll  surveys (%)  4.83  0.36  0.19  n/a  Note. Calculations based on American Association of Public Opinion Research (AAPOR) Response Rate #1. The target population is age 18–64 years. Table 2 Response Rates and Case Dispositions, by Country   Kenya  Ghana  Nigeria  Uganda  Number of sampled cases  24,954  394,181  1,124,759  14,686  Number of completed interviews  2,960  2,277  2,392  2,068  Case dispositions (of entire sample)   No answer (%)  83.98  92.13  75.36  81.06   Completed (%)  11.86  0.58  0.21  14.08   Breakoff (%)  1.75  0.36  0.19  3.62   Rejected—quality (%)  0.36  0.02  0.01  0.22   Refused (%)  0.48  0.05  0.02  0.25   Undeliverable (%)  1.14  6.78  24.16  0.0   Ineligible (because of age) (%)  0.53  0.08  0.05  0.78   Total (%)  100  100  100  100  Response rate (%)   Entire sample (%)  12.06  0.62  0.28  14.19   Previously participated in  GeoPoll surveys (%)  45.87  15.46  28.45  14.19      Never participated in GeoPoll  surveys (%)  4.83  0.36  0.19  n/a    Kenya  Ghana  Nigeria  Uganda  Number of sampled cases  24,954  394,181  1,124,759  14,686  Number of completed interviews  2,960  2,277  2,392  2,068  Case dispositions (of entire sample)   No answer (%)  83.98  92.13  75.36  81.06   Completed (%)  11.86  0.58  0.21  14.08   Breakoff (%)  1.75  0.36  0.19  3.62   Rejected—quality (%)  0.36  0.02  0.01  0.22   Refused (%)  0.48  0.05  0.02  0.25   Undeliverable (%)  1.14  6.78  24.16  0.0   Ineligible (because of age) (%)  0.53  0.08  0.05  0.78   Total (%)  100  100  100  100  Response rate (%)   Entire sample (%)  12.06  0.62  0.28  14.19   Previously participated in  GeoPoll surveys (%)  45.87  15.46  28.45  14.19      Never participated in GeoPoll  surveys (%)  4.83  0.36  0.19  n/a  Note. Calculations based on American Association of Public Opinion Research (AAPOR) Response Rate #1. The target population is age 18–64 years. Most nonresponse was because of respondents not answering. Refusal rates were low, with <1% of the sample explicitly refusing to participate by responding “STOP” via SMS. Breakoff rates were high: as a proportion of respondents who agreed to participate, breakoff rates were 38% in Ghana, 13% in Kenya, 46% in Nigeria, 20% in Uganda. These results are similar to a SMS survey in South Africa (Broich, 2015)—but lower than Johnson (2016) and Hoe and Grunwald (2015), which had breakoff rates of approximately 50% (authors' calculations.) There is some evidence that these high breakoff rates may impact sample representativeness. Among respondents who answered the age question, younger people were more likely to complete than older people (p < .01 in each country). The difference in age between breakoff and completed cases was 3.2 years in Ghana (age 30.1 vs. 26.9 years; p < .01), 1.8 years in Kenya (age 30.3 vs. 28.5 years; p < .01), 1.0 years in Nigeria (age 29.1 vs. 28.2 years; p < .01), and 1.5 years in Uganda (age 27.0 vs. 25.5 years; p < .01). While these differences are statistically significant, they are modest in size, suggesting that breakoff has minor effects on survey representativeness. Figure 1 shows question-level breakoff rates for each question. Among those who agreed to participate in the survey, breakoff rates for the first question (age) were 25% in Nigeria, 21% in Ghana, 13% in Kenya, and 11% in Uganda. Question-level breakoff rates declined to between 3 and 7% for the second question. Most subsequent questions had breakoff rates of <3%. Figure 1 View largeDownload slide Question-level breakoff rate Figure 1 View largeDownload slide Question-level breakoff rate The Representativeness of SMS Surveys To investigate the representativeness of SMS surveys, we compared the SMS completed interviews with 11 variables from three benchmark data sources. Age and gender Figure 2 shows the joint distribution of age and gender for the SMS and benchmark data. (See Online Supplement for exact percentages.) The figure includes 95% confidence intervals for the SMS data, but no confidence intervals for the benchmark data because they come from Censuses. In all countries, SMS overrepresents younger men. In Kenya, for example, men 18–24 years old comprise 30% of SMS respondents but were only 14% in Census data. SMS underrepresents women in most countries, even among the younger age groups. One notable exception is Kenya, where the SMS and Census data have similar representation of women 18–24 years old. But in all countries, differences among older women are stark. In Nigeria, for example, women 45–64 years old are 1.3% of SMS respondents versus 7.6% in Census data. Our analysis cannot identify the specific error sources that produce differences between SMS and benchmark data; for example, we cannot say whether coverage error or versus nonresponse errors lead to the underrepresentation of older women. This analysis just describes the differences, without attempting to attribute the difference to specific error sources. Figure 2 View largeDownload slide Comparison of age and gender between SMS and benchmark data (95% confidence interval for SMS). Note. SMS = short message service Figure 2 View largeDownload slide Comparison of age and gender between SMS and benchmark data (95% confidence interval for SMS). Note. SMS = short message service Given these age and gender differences, we created poststratification weights that align the SMS data to the joint distribution of age and gender from Census and UN data. These weights adjust for both coverage and nonresponse. Weights were derived by dividing the percentage from the Census/UN data by the corresponding percentage from the SMS data. For example, the weight for men 18–29 years old in Kenya was 0.48 (14.1/29.5%). No other adjustments to the weights were made. Weights ranged from 0.39 to 6.39 in Ghana, 0.48 to 2.87 in Kenya, 0.47 to 5.99 in Nigeria, and 0.50 to 13.82 in Uganda. The remainder of the analysis in this section compares weighted data from the SMS surveys to FTF surveys. This analysis allow us to ask: After weighting the SMS data by age and gender, how representative are SMS surveys? Socioeconomic and technology variables In Figure 3, we show percentage point differences between SMS and FTF data. See the “Methods” section for a description of how these differences are calculated. Positive numbers indicate that SMS overestimates the variable; negative numbers indicate that SMS underestimates the variable. Owing to the large number of comparisons, we show data from Ghana as an example; full results, including confidence intervals, for all countries are available in the Online Supplement. Figure 3 View largeDownload slide Percentage point differences between SMS and FTF benchmark data in Ghana (stars represent significant differences from 0). Note. FTF = face-to-face; SMS = short message service Figure 3 View largeDownload slide Percentage point differences between SMS and FTF benchmark data in Ghana (stars represent significant differences from 0). Note. FTF = face-to-face; SMS = short message service The results for education are striking. The SMS survey underrepresents people with a primary education or less by 62 percentage points in Ghana and overrepresents those with more education. When we restrict the FTF survey to individuals with a mobile phone, the differences between SMS and FTF attenuate only slightly. This is because mobile penetration is high in these countries; there are few differences between “FTF with Mobile” and “FTF Entire Sample.” These patterns suggest that coverage errors because of mobile phone access explain only a small portion of the gap between SMS and FTF surveys. Similar patterns for education exist in Kenya, Nigeria, and Uganda (see Online Supplement). The overrepresentation of educated people is reflected in the languages SMS respondents chose: in all countries, over 90% selected English. In the Technology Adoption FTF surveys, English was used 51% of the time in Kenya, 34% in Uganda, 27% in Ghana, and 19% in Nigeria. In this survey, educated people were more likely to be interviewed in English. SMS surveys also overrepresent people who are employed full-time or part-time and underrepresent people who are self-employed. Part of these differences is because of employed people being more likely to have access to mobile phones. Interestingly, SMS surveys overrepresent the unemployed—perhaps because of the financial incentive for participation. The results for modern shelter and finished roof (which indicate higher socioeconomic status) are mixed. In Ghana, SMS overrepresents people with modern shelter, but there are no differences for roof. In Kenya, Nigeria, and Uganda, there are positive differences in some countries but negative differences in others. This pattern is consistent with the results from a World Bank panel survey, which used SMS to follow-up with respondents who were initially recruited FTF (Ballivian, Azevedo, and Durbin, 2015). In that study, SMS surveys understated poverty relative to FTF surveys, but results for household infrastructure were more mixed. Compared with the FTF data, SMS respondents are less likely to share a SIM card in Ghana and other countries. In all countries except Kenya, SMS respondents underrepresents people who have difficulty charging their phones, potentially reflecting nonresponse because of a lack of electricity. And in all countries, SMS overrepresents Internet users. As before, restricting the FTF sample to those with mobile phones only slightly attenuates differences between SMS and FTF data. Improving the Representativeness of SMS Surveys The results in Figures 2 and 3 suggest that SMS surveys underrepresent women and people who are older, less educated, and less technologically savvy. Given these findings, what can survey designers do to increase the participation of these underrepresented groups? In this section, we analyze the experimental data from Kenya to test three strategies. Incentives and survey length The standard incentive (0.5 USD) had the same response rate as the higher incentive (1.25 USD), both 13% [χ2 (1) = 0.9; p = .33]. Response rates were similar for the 8 and 16 question surveys (13 vs. 12%). Although statistically significant [χ2 (1) = 8.3; p < .01], this test is based on a large sample size (n = 19,622). We do not interpret this result as a meaningful difference. A supplementary analysis showed there was no statistically significant interaction between incentive and survey length (z = 0.09; p = .93). The composition of respondents was similar, regardless of incentive or survey length. Full tables are available in the Online Supplement. Reminders Owing to the emphasis on speed, some SMS surveys do not send reminder invitations to follow up with nonrespondents. To investigate whether reminders can improve sample representativeness, we compared the composition of respondents who completed the survey in Kenya after the first invitation (initial respondents in Column A of Table 3) and those who completed after receiving a reminder (reminder respondents in Column B). The table also includes percentage point differences (Column C) and p-value from chi-square tests (Column D). Table 3 Sample Composition of Completed Interviews, by Reminder (Kenya) Variable  A. Completed on initial invitation (%)  B. Completed after reminder (%)  C. Percentage point difference (%)  D. p-value from chi-square test    (n = 2,563)  (n = 397)  (A−B)  Female  34  38  −4  .18  Age (years)      18–24  48  38  10  <.01      25–34  31  33  −2      35–44  13  17  −4      45–64  9  12  −3  Education      Primary or less  17  23  −6  <.01      Secondary  45  44  1      Postsecondary  38  33  5  Employed full-time  15  12  3  .28  Modern shelter  63  59  4  .23  Modern roof  66  67  2  .68  More than one SIM card  62  57  5  .03  Anyone else use this SIM card  30  31  1  .47  Difficulty charging phone  43  42  1  .79  Aware of Internet  77  80  3  .41  Use Internet (among aware)  80  74  6  .09  Variable  A. Completed on initial invitation (%)  B. Completed after reminder (%)  C. Percentage point difference (%)  D. p-value from chi-square test    (n = 2,563)  (n = 397)  (A−B)  Female  34  38  −4  .18  Age (years)      18–24  48  38  10  <.01      25–34  31  33  −2      35–44  13  17  −4      45–64  9  12  −3  Education      Primary or less  17  23  −6  <.01      Secondary  45  44  1      Postsecondary  38  33  5  Employed full-time  15  12  3  .28  Modern shelter  63  59  4  .23  Modern roof  66  67  2  .68  More than one SIM card  62  57  5  .03  Anyone else use this SIM card  30  31  1  .47  Difficulty charging phone  43  42  1  .79  Aware of Internet  77  80  3  .41  Use Internet (among aware)  80  74  6  .09  Note. The target population is age 18–64 years. Table 3 Sample Composition of Completed Interviews, by Reminder (Kenya) Variable  A. Completed on initial invitation (%)  B. Completed after reminder (%)  C. Percentage point difference (%)  D. p-value from chi-square test    (n = 2,563)  (n = 397)  (A−B)  Female  34  38  −4  .18  Age (years)      18–24  48  38  10  <.01      25–34  31  33  −2      35–44  13  17  −4      45–64  9  12  −3  Education      Primary or less  17  23  −6  <.01      Secondary  45  44  1      Postsecondary  38  33  5  Employed full-time  15  12  3  .28  Modern shelter  63  59  4  .23  Modern roof  66  67  2  .68  More than one SIM card  62  57  5  .03  Anyone else use this SIM card  30  31  1  .47  Difficulty charging phone  43  42  1  .79  Aware of Internet  77  80  3  .41  Use Internet (among aware)  80  74  6  .09  Variable  A. Completed on initial invitation (%)  B. Completed after reminder (%)  C. Percentage point difference (%)  D. p-value from chi-square test    (n = 2,563)  (n = 397)  (A−B)  Female  34  38  −4  .18  Age (years)      18–24  48  38  10  <.01      25–34  31  33  −2      35–44  13  17  −4      45–64  9  12  −3  Education      Primary or less  17  23  −6  <.01      Secondary  45  44  1      Postsecondary  38  33  5  Employed full-time  15  12  3  .28  Modern shelter  63  59  4  .23  Modern roof  66  67  2  .68  More than one SIM card  62  57  5  .03  Anyone else use this SIM card  30  31  1  .47  Difficulty charging phone  43  42  1  .79  Aware of Internet  77  80  3  .41  Use Internet (among aware)  80  74  6  .09  Note. The target population is age 18–64 years. Respondents who completed the survey after a reminder were older than those who completed on the first invitation [χ2 (3) = 17.0; p < .01]. Further, reminder respondents were more likely to have a primary school education or less compared with initial respondents (23 vs. 17%; χ2 (3) = 17.0; p < .01). These results suggest that reminders are one tool survey designers can use to increase the representation of underrepresented groups—specifically older people and less educated—to SMS surveys. One caveat, however, is that we did not conduct an experiment on reminders: it is possible that some of these differences may reflect the types of people who are simply late responders rather than the effect of a reminder per se. Discussion SMS is a low-cost survey mode that can collect data rapidly from large and geographically dispersed populations. For applications where speed and scale are paramount (e.g., rapid response during a public health emergency), SMS clearly offers numerous advantages. But are SMS surveys suitable for general population public opinion surveys? Our study in Ghana, Kenya, Nigeria, and Uganda shows that SMS surveys overrepresented younger, male, higher educated, and technologically oriented people. Many differences were large: in Nigeria, for example, SMS overrepresented the population with a postsecondary education by 50 percentage points. These results suggest that SMS cannot replace high-quality FTF surveys, at least in these countries during late 2015. When SMS surveys of the general population are needed because of cost and time constraints, we encourage researchers to take the representativeness of SMS surveys into account when interpreting the results. Will growing mobile phone penetration make SMS surveys more representative in the future? After all, growth in Internet access over time improved representativeness of Web surveys in Europe (Mohorko, de Leeuw, and Hox, 2013). For Ghana, Kenya, Nigeria, and Uganda, however, we suspect the answer is no. According to our results, only a small part of the difference between SMS and FTF surveys is because of lack of mobile phone access. This is because mobile phone access is already high in the countries we studied, and there are only small differences between people with and without phones. But in countries with lower mobile penetration, there may be greater differences between people with and without mobile phones. In these contexts, growing mobile penetration may indeed make SMS surveys more representative. Postsurvey adjustments (for age, gender, and other demographics) may improve SMS survey representativeness. But just like for Web surveys, weighting is unlikely to compensate completely for survey errors (Bethlehem, 2010; Peytchev, Carley-Baxter, & Black, 2011). Given large underrepresentation (especially for education), some weights will be large, reducing precision. In addition, there are likely factors that affect response to SMS survey that are not available in population control data (e.g., access to electricity). Further, at least in the present day, the full range of variables necessary for accurate adjustments is also unknown. Survey researchers also can tailor their survey designs to improve sample representativeness. In Kenya, sending reminders reduced nonresponse bias by increasing the share of less educated and older respondents. We recommend that population-based SMS surveys use reminders. Future research should use experiments to disentangle the effects from reminders from respondents to tend to respond late. Two other strategies we evaluated in Kenya—offering a shorter survey and a higher airtime incentive—were not successful in increasing the response rate meaningfully or in making the respondent pool more diverse. The silver lining, however, is that our experiment shows there may be some leeway with survey length: SMS surveys are often short (e.g., 10 questions), but our research suggests SMS surveys could potentially include more questions without compromising data quality. We encourage future research that explores an upper limit for question length in Kenya and other countries. Our study highlights two important aspects of sampling for general population SMS surveys. First, additional research is needed on vendor list frames. These frames can have coverage errors, but also can include ancillary data that RDD samples lack. Ancillary data can help researchers study more narrow target populations or can be used for sample stratification. Researchers will need to balance the utility of these ancillary data against possible coverage errors and panel conditioning effects. Second, future research is needed on multiple and shared phones. Many people have more than one SIM card (62% in our Kenya data) and share SIM cards (30% in our Kenya data). Multiple and shared SIM cards affect selection probabilities; failing to adjust for this variation could affect data quality (Labrique et al., 2017). Limitations This article seeks to describe the representativeness of SMS surveys by comparing SMS with FTF survey data. The lack of representative SMS data likely stems from a combination of undercoverage (e.g., vendor frame errors, incomplete phone access) and nonresponse errors (e.g., illiteracy, language coverage, poor phone networks, discomfort with technology, distrust of surveys, financial concerns). The chief limitation of our study is that we cannot point to the full range of errors that lead to unrepresentative SMS samples. For example, are coverage or nonresponse errors more important? What type of nonresponse errors creates the most bias? Also, does measurement error play a role in explaining differences between SMS and FTF surveys? We look to future research to investigate these important questions. Another limitation concerns our focus on Anglophone countries with high mobile penetration; our conclusions may not apply to other countries. We look forward to research on SMS surveys in a wider range of countries. Finally, future research is needed using RDD samples. This research is important for separating errors specific to GeoPoll’s list frames from errors that apply to the SMS mode more generally. Future Directions The explosion of mobile telephony offers researchers new ways to collect data quickly and inexpensively. SMS is the most widely used mobile phone survey mode. But other modes exist, such as IVR, CATI, mobile Web, and chatbots. These modes have strengths and weaknesses. For example, IVR can include illiterate people (unlike SMS), but still suffers from the same issues with nonresponse as SMS. Mobile Web avoids the 160-character limit of SMS, but requires smartphones to function well. The live interviewer in CATI may increase response rates and offer better measurement but is more expensive and introduces interviewer effects. We look forward to research that compares the representativeness, cost, and time of these various survey modes. These are the “early days” of mobile phone surveys in lower-income countries. Although our results suggest SMS surveys have serious limitations in the present day, this study is a snapshot during a dynamic period of change. We look forward to research tracking the evolution of SMS and other mobile phone survey modes. We also suspect that mixed mode approaches that combine SMS with other modes may be a promising way to improve survey representativeness. Supplementary Data Supplementary Data are available at IJPOR online. Charles Q. Lau (PhD, Sociology, University of California, Los Angeles) is a Survey Methodologist at RTI International. Ansie Lombaard (PhD, Research Methodology, University of Stellenbosch) is a Senior Global Innovation Director with the Kantar Insights Division, based in Cape Town, South Africa. Melissa Baker (BA, Social Anthropology, SOAS University of London) is the Kantar Public CEO of Social Research and Evaluation in Africa, based in Nairobi, Kenya. Joe Eyerman (PhD, Political Science, Florida State University) is a Center Director in RTI International's Survey Research Division. Lisa Thalji (MA, Psychology, Temple University) is Vice President of the Survey Research Division at RTI International. Acknowledgement The authors thank Stephanie Eckman, Ashley Amaya, Luis Crouch, Michelle Harrison, Anna Wetterberg, Mariano Sana, and Eric Johnson for valuable feedback. Any errors that remain are of authors. Funding RTI International and Kantar supported this work. References American Association for Public Opinion Research. ( 2010). New considerations for survey researchers when planning and conducting RDD telephone surveys in the U.S. with respondents reached via cell phone numbers. AAPOR Cell Phone Task Force. Retrieved from http://www.aapor.org/AAPOR_Main/media/MainSiteFiles/2010AAPORCellPhoneTFReport.pdf. Ballivian A., Azevedo J. P., Durbin W. 2015. Using mobile phones for high-frequency data collection. In Toninelli D., Pinter R., de Pedraza P. (Eds.), Mobile research methods: Opportunities and challenges of mobile research methodologies  (pp. 21– 39). London: Ubiquity Press. Retrieved from http://dx.doi.org/10.5334/bar.c. Bethlehem J. ( 2010). Selection bias in web surveys. International Statistical Review , 78, 161– 188. doi: 10.1111/j.1751-5823.2010.00112.x. Google Scholar CrossRef Search ADS   Broich C. ( 2015). Offline data collection in Sub-Saharan Africa using SMS surveys: lessons learned. Paper presented at the meeting of the American Association for Public Opinion Research, Hollywood, FL. Carmer L., Boots M. ( 2013). Barriers and solutions in using M4D: Connecting directly to citizens for scalable impact. VOTO Mobile White paper. Retrieved from https://www.votomobile.org/files/M4D_Barriers_Whitepaper.pdf. Centers for Disease Control and Prevention. ( 2015). Data for health initiative. Retrieved from http://www.cdc.gov/globalhealth/healthprotection/data.html. Conrad F., Schober M., Antoun C., Hupp A., Yan H. ( 2017). Text interviews on mobile devices. In Biemer P., de Leeuw E., Eckman S., Edwards B., Kreuter F., Lyberg L., Tucker N. C., West B. (Eds.), Total survey error in practice . Hoboken, NJ: John Wiley & Sons, Inc. doi: 10.1002/9781119041702.ch14. Couper M. P. ( 2000). Web Surveys: A Review of Issues and Approaches. Public Opinion Quarterly , 64, 464– 494. Google Scholar CrossRef Search ADS PubMed  De Bruijne M., Wijnant A. ( 2014). Improving response rates and questionnaire design for mobile web surveys. Public Opinion Quarterly , 78, 951– 962. doi: 10.1093/poq/nfu046. Google Scholar CrossRef Search ADS   Dillon B. ( 2012). Using mobile phones to collect panel data in developing countries. Journal of International Development , 24, 518– 527. doi: 10.1002/jid.1771. Google Scholar CrossRef Search ADS   Eckman S. ( 2015). Does the inclusion of non-internet households in a web panel reduce coverage bias? Social Science Computer Review , 34, 41– 58. doi: 10.1177/0894439315572985. Google Scholar CrossRef Search ADS   Fan W., Yan Z. ( 2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior , 26, 132– 139. doi: 10.1016/j.chb.2009.10.015. Google Scholar CrossRef Search ADS   Ganesan M., Prashant S., Jhunjhunwala A. ( 2012). A review on the challenges in implementing mobile phone based data collection in developing countries. Journal of Health Informatics in Developing Countries , 6, 366– 374. GeoPoll ( 2015). Perceptions of South Africa’s state of the nation: TNS. Retrieved from: https://research.geopoll.com/case-studies/tns-state-of-the-nation-south-africa.html. Gibson D. G., Pereira A., Farrenkopf B. A., Labrique A. B., Pariyo G. W., Hyder A. A. ( 2017). Mobile phone surveys for collecting population-level estimates in low- and middle-income countries: A literature review. Journal of Medical Internet Research , 19, e139. doi: 10.2196/jmir.7428. Google Scholar CrossRef Search ADS PubMed  GSMA. ( 2015). The mobile economy: Sub-Saharan Africa 2015. Retrieved from https://gsmaintelligence.com/research/?file=721eb3d4b80a36451202d0473b3c4a63&download. Hoe N., Grunwald H. ( 2015). The role of automated SMS text messaging in survey research. Survey Practice , 8( 6), 1– 15. Google Scholar CrossRef Search ADS   Johnson D. ( 2016). Collecting data from mHealth users via SMS surveys: A case study in Kenya. Survey Practice , 9( 1), 1– 8. Google Scholar CrossRef Search ADS   Labrique A., Blynn E., Ahmed S., Gibson D., Pariyo G., Hyder A. A. ( 2017). Health surveys using mobile phones in developing countries: Automated active strata monitoring and other statistical considerations for improving precision and reducing biases. Journal of Medical Internet Research , 19, e121. doi: 10.2196/jmir.7329. Google Scholar CrossRef Search ADS PubMed  Lombaard A., Richman M. ( 2015). Mobile research in emerging markets: Taking the step into the world of probability sampling. Paper presented at the meeting of the World Association for Public Opinion Research, Buenos Aires, Argentina. Leo B., Morello R., Mellon J., Peixoto T., Davenport S. ( 2015). Do mobile phone surveys work in poor countries? Washington, DC: Center for Global Development Working Paper 398. Mahfoud Z., Ghandour L., Ghandour B., Mokdad A., Sibai A. M. ( 2015). Cell phone and face-to-face interview responses in population-based surveys: How do they compare? Field Methods , 27, 39– 54. doi: 10.1177/1525822X14540084. Google Scholar CrossRef Search ADS   Mitullah W., Kamau P. ( 2013). The partnership of free speech & good governance in Africa. Afrobarometer Policy Brief No. #3. Retrieved from http://afrobarometer.org/sites/default/files/publications/Briefing%20paper/ab_r5_policypaperno3.pdf. Mitullah W. V., Samson R., Wambua P. M., Balongo S. ( 2016). Building on Progress: Infrastructure development still a major challenge in Africa. Afrobarometer Dispatch No. 69. Retrieved from http://afrobarometer.org/sites/default/files/publications/Dispatches/ab_r6_dispatchno69_infrastructure_remains_challenge_en.pdf. Mohorko A., de Leeuw E., Hox J. ( 2013). Internet coverage and coverage bias in Europe: Developments across countries and time. Journal of Official Statistics , 29, 609– 622. doi: 10.2478/jos-2013-0042. Google Scholar CrossRef Search ADS   Peytchev A., Carley-Baxter L., Black M. ( 2011). Multiple sources of nonobservation error in telephone surveys: Coverage and nonresponse. Sociological Methods and Research , 40, 138– 168. doi: 10.1177/0049124110392547. Google Scholar CrossRef Search ADS   Pew Research Center. ( 2015). Cell phones in Africa: Communication lifeline. Pew Research Center. Retrieved from http://www.pewglobal.org/2015/04/15/cell-phones-in-africa-communication-lifeline/. Steeh C., Buskirk T., Callegaro M. ( 2007). Using text messages in U.S. mobile phone surveys. Field Methods , 19, 59– 75. doi: 10.1177/1525822X06292852. Google Scholar CrossRef Search ADS   UNESCO. ( 2014). Adult and youth literacy, UIS Fact Sheet. UNESCO Institute for Statistics. Retrieved from http://unesdoc.unesco.org/images/0022/002295/229504e.pdf. UNICEF. ( 2015). UNICEF’s U-report social platform hits 1 million active users. UNICEF News Note, 16 July, 2015. Retrieved from http://www.unicef.org/media/media_82583.html. World Health Organization. ( 2014). Government of Senegal boots ebola awareness through SMS campaign. Retrieved from http://www.who.int/features/2014/senegal-ebola-sms/en/. © The Author(s) 2018. Published by Oxford University Press on behalf of The World Association for Public Opinion Research. All rights reserved. This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

International Journal of Public Opinion ResearchOxford University Press

Published: Apr 13, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off