TY - JOUR
AU - Chandir,, Subhash
AB - Abstract Background Household surveys are an essential tool for vaccine coverage monitoring in developing countries, and the World Health Organization (WHO) Expanded Program on Immunization (EPI) cluster survey design has been a default choice for decades. In response to methodological limitations of the traditional EPI sampling, alternative methods have been proposed, based on modern statistical and geographical techniques. This study compared the coverage estimates and the time efficiency of the EPI sampling design and two alternative methods: the compact segment sampling and innovative grid-based geographical information system (GIS) sampling. Methods We conducted a series of equal-sized concurrent prospective vaccine coverage surveys in Karachi, Pakistan, from January to December 2016, using traditional EPI, compact segment and grid-based GIS sampling methods. Results No differences in vaccine coverage estimates were identified across sampling methods in the peri-urban setting; however, due to stronger clustering effects and correct incorporation of sampling weights, the compact segment [design effect (DEFF) = 2.03] and the grid-based GIS surveys (DEFF = 1.72) had higher design effects and, therefore, appeared to have lower statistical precision than the traditional EPI surveys (DEFF = 1.57). To achieve the same level of apparent precision, data collection activities in the compact segment surveys would require more than twice the implementation time needed compared with the traditional EPI surveys. Conclusions The precision of the EPI surveys appeared higher than that of the alternative methods because, under a questionable self-weighting assumption, the estimated design effect did not account for variable sampling weights. The compact segment and grid-based GIS methods were designed to improve randomness and representativeness of sampling households. Although these alternative methods did not result in coverage estimates that differed from the EPI survey results in the peri-urban setting, they have a lower risk of selection bias and therefore may be preferred. Immunization programmes, survey, probability sample, vaccine coverage, Pakistan, epidemiological methods Key Messages Compared with the traditional Expanded Program on Immunization (EPI) cluster survey sampling design, two alternative methods with known or approximated sampling probability, namely compact segment and grid-based geographical information system (GIS) sampling, produced similar vaccine coverage estimates in a series of equal-sized concurrent prospective surveys in a peri-urban setting in Karachi, Pakistan. Since the traditional EPI surveys did not correctly account for the sampling weights, the design effects were underestimated. The compact segment and the grid-based GIS surveys had shown higher design effects and, therefore require a larger sample size than the traditional EPI surveys for the same level of apparent precision. More methodologically rigorous sampling designs are preferred in future household surveys. Surveyors need to evaluate and be prepared for the potential increase in sample size requirements and implementation costs. Introduction Household surveys serve an important role in immunization programme monitoring activities, especially in low-income countries where immunization registries are often incomplete, unreliable or inaccessible.1 The World Health Organization (WHO) Expanded Program on Immunization (EPI) vaccine coverage cluster survey design has been a default choice for national and subnational vaccine coverage surveys since its development in the 1970s,2–4 and has been adapted for rapid assessment of other community health measures.5 The original WHO EPI survey manual6 established a practical protocol to estimate vaccination coverage among children from 12 to 23 months of age, with a ± 10% margin of error. It typically samples 30 clusters, selected among all clusters with probability proportional to size (PPS); within the clusters, seven children are selected using a well-known ‘spin-the-pen’ (‘random walk’) sampling method. In a 2018 revision of the manual,7,8 more adaptive methods were recommended for sample size determination and ‘random walk’ methods to meet the needs in various settings, particularly urban areas. As a result of the non-probability sampling design and the practice of assuming that the sample is self-weighting, traditional EPI surveys are frequently easier and require fewer resources to implement than conventional cluster surveys that require a complete listing of households within clusters. The traditional EPI sampling design has made rapid survey-based vaccine coverage monitoring feasible and widely implemented in low-income countries. However, the traditional EPI method has been criticized for various technical disadvantages, including validity issues of self-weighting in PPS due to the often inaccurate or incomplete cluster sampling frames, the lack of statistical rigour with non-probability ‘random walk’ sampling, which also makes the self-weighting assumption unverified, and the inevitable subjectivity of interviewers in household selection in stage two of the method.3,9–11 In response to an increasing need for survey data accuracy and precision, the 2018 revision of the WHO EPI survey manual has included recommendations on probability-based sampling, weighted analysis and centralized household selection (rather than by interviewers in the field).7,8 Previous studies have developed and evaluated various options to address certain disadvantages of the EPI sampling method without dramatically increasing the requirements of budget, time and staff expertise. Turner et al.12 and Milligan et al.13 developed a compact segment approach to create probability-based household samples and remove fieldwork subjectivity in selecting households and children. A comparative field study13 with small sample size showed no statistically different results between compact segment surveys and traditional EPI surveys. Luman et al.14 used a systematic random sampling (SystRS) approach to create a probability-based sample in rapid coverage surveys. With the help of geographical information system (GIS) tools, Grais et al.3 attempted to reduce fieldwork subjectivity by improving random selection of the first household. They randomly picked a Global Positioning System (GPS) point within each cluster boundary, or an x-y coordinate on a map grid, and used the household closest to the selected GPS point or coordinates as the starting point of the ‘random walk’. Unlike Grais, Lowther et al.15 completely abandoned the ‘random walk’ method, by enumerating all residential dwellings of selected clusters in the satellite imagery and randomly selecting dwellings to visit before going to the field. Myatt et al.5 illustrated a centric systematic area sampling method to avoid the PPS sampling of clusters when there is no reliable cluster sampling frame. In their approach, map grids were used to divide the study area into dozens of quadrats, i.e. non-overlapping squares of equal size, and one or more enumeration blocks at the centre of each quadrat was visited as the selected clusters. Finally, Galway et al.16 presented a solution for surveys in areas of security concern and with no reliable cluster sampling frame. Based on publicly available population distribution interpolation datasets, they performed national-level PPS sampling of clusters and applied a grid-based method to select the first household of the ‘random walk’ process. The objective of this study was to compare point estimates of coverage, precision and implementation time among three sampling designs: the traditional EPI method described in the 2005 WHO manual17 and two rapid household designs with known or approximated sampling probability. The first of the two alternative designs was the previously described compact segment approach.12,13 We selected this method because it theoretically promises a reduction in selection biases and requires no extra statistical expertise to implement. The second alternative design is an innovative grid-based GIS method with approximated sampling probability. We developed this method specifically for surveys in areas of security concern, with an intention to minimize fieldwork hours. Methods Study setting We conducted a series of concurrent prospective vaccine coverage surveys in Korangi town of the Sindh province in Pakistan, which was selected as the study site so that the three survey methods could be thoroughly tested in an insecure, peri-urban slum setting, which was considered challenging for vaccine coverage surveys. The local economy in Korangi is largely based on fishing and manufacturing. Its residents are often employed on a day-to-day basis and are currently experiencing the transition to an urban lifestyle—the population is growing rapidly, public transportation is improving incrementally and neighbourhoods are increasingly crowded within its 40.54 square kilometres.18 According to the Karachi Municipal Committee estimates of 2015, Korangi town had a total population of 1.1 million, including about 31 thousand infants (0–11 months of age). The population in Korangi town is ethnically diverse, with representation from all ethnic groups in Pakistan. The industrial area of Korangi town was excluded from our household cluster sampling frame. Ethical approval for all study procedures was obtained from the institutional review boards at Johns Hopkins Bloomberg School of Public Health and Interactive Research and Development in Karachi. All parents provided written informed consent. Vaccine coverage surveys Clustered vaccine coverage surveys were conducted in Korangi town, using three sampling methods: traditional EPI sampling, compact segment sampling and grid-based GIS sampling. To ensure sufficient sample size and reduce the effects of seasonality for cross-method comparison, each survey type was repeated in four rounds between January and December 2016, making a total of 12 surveys. In each round, surveys with different methods were conducted concurrently by separate field teams; the method used by each team was switched across rounds to reduce biases associated with field teams. Enumeration areas were allowed to be selected more than once for inclusion in different methods, but they could not be repeatedly surveyed with the same method across rounds. If a previously enrolled household was selected more than once with a different method, the household was identified through the recorded household address, residents’ recall of participation or interviewers’ recognition of the educational materials distributed by the previous interviewers. In case of overlapping enrolment across methods, interviews were not conducted twice but data from the original interview were used twice, once in each survey, for analysis. A summary of the three sampling methods, survey weights and assumptions is provided in Supplementary E, available as Supplementary data at IJE online. The sampling process for the EPI cluster surveys followed the 2005 WHO manual.17 In each study round, 30 of 387 enumeration blocks (clusters) were selected with probability proportional to the population size projection from the 1998 census. Each cluster contained a median projected population of 2535, ranging between 47 and 11 270. Three clusters with less than 150 residents were pre-excluded. Each selected cluster was divided into roughly equal-sized subdivisions, each containing 30–50 households. To help define the subdivisions, simple sketch maps of roads and landmarks were drawn in the field by field supervisors (Supplementary C, available as Supplementary data at IJE online). The size of the subdivisions was approximated based on observed household density, rather than precise household counting. After enumerating the subdivisions, we randomly selected one subdivision in the cluster and drew a second sketch map to list all households in the selected subdivision. Then, one of the enumerated households was randomly selected as the first to visit. Subsequently, the next closest household by distance was visited, until seven eligible children were enrolled from the cluster. In multi-family dwellings, the second household visited was the door nearest to the first. If two households were the same distance apart, the one on the right side was chosen. After all households on the floor were visited, the team randomly chose a direction, up or down the stairs, until the whole building was visited. Families sharing a kitchen and dining together were considered as being in the same household. All random selections were carried out by fieldworkers by entering the total number of subdivisions or clusters in a random number generator installed in interviewers’ mobile devices. In the compact segment12,13 surveys, clusters were randomly selected using the same PPS process but the method of defining and selecting subdivisions, which were called segments, was different. With the EPI sampling, the household listing was only performed in the selected subdivisions to find the initial household; however, in the compact segment surveys, households in the entire cluster were enumerated. A sketch map (Supplementary B, available as Supplementary data at IJE online) was drawn for each selected cluster to show streets, landmarks, dwellings and, in multi-family dwellings, the number of floors and the number of households per floor. Based on household counts, each cluster was divided into equal-sized segments containing 40–65 households and segments were numbered. One segment was selected using a random number generator from each cluster, and all households in the segment were visited for enrolment. The segment size was determined so that approximately seven enrolments were expected. The field teams usually spent an extra day or two before the survey in each cluster, to create the sketch map, enumerate the households and define the segment boundaries. With the grid-based GIS sampling, we did not rely on the census-based enumeration blocks to define clusters. Instead, the study area was divided into hundreds of non-overlapping squares of equal area, otherwise known as ‘quadrats’, by superimposing a grid (180 × 180 m) on a satellite image of Korangi (taken by WorldView-3 satellite sensor in November and December 2015). At the beginning of each round, 30 of 588 quadrats were selected by a random number generator. Quadrats at the periphery of the town were excluded if less than 50% of the quadrat area fell within the town boundaries. For each selected quadrat, all potential residential structures were manually marked on the satellite imagery (Supplementary D, available as Supplementary data at IJE online) and listed in sequence based on their distance to the geographical centre of the quadrat, with the closest listed first. ArcGIS 10.1 software was used to mark the imagery and calculate the distance from the centre. On the survey day, field staff located the geographical centre of each selected quadrat with printed satellite imagery, digital imagery preloaded in tablets and GPS devices. Household visits started at the dwelling closest to the centre and continued according to the predetermined distance until seven eligible children were enrolled. Interviewers visited sampled households on working days between 9 am and 4 pm. A household was eligible if at least one child between the age of 12 and 23 months lived in the household. If more than one eligible child were identified, the interviewer selected the youngest child. Informed consent was obtained from the mother or primary caregiver of the selected child before the interview. If the mother or primary caregiver was not present at the time of the visit, follow-up visits were conducted until a maximum of five visits were attempted. Because children enrolled during follow-up visits were not counted towards the seven-child target enrolment on EPI and grid-based GIS survey days, the actual enrolment in those clusters sometimes exceeded seven. Survey questions modified from the 2012 Pakistan Demographic and Health Surveys (DHS) survey instruments19 were used to document caregivers’ verbal recall of children’s immunization history. In addition home-based health records (HBR; i.e. vaccination cards) were requested, vaccinations recorded and cards photographed. The weight of the child was measured with portable toddler weight scales (Laica™; model PS3001) following the interview. Surveys and photographs were recorded with smartphones and tablets equipped with an Android application specifically designed for this study. A data quality coordinator reviewed HBR data input by interviewers. Daily hours spent on various fieldwork activities, including household sampling, interviews and travelling, were documented by the field supervisors. Sample size calculation The 2005 WHO reference manual17 recommended a 30 × 7 design, i.e. 30 clusters and seven observations in each cluster, for rapid vaccination coverage estimation with a desired precision of ±10% ⁠. To more accurately compare across survey methods, we applied the 30 × 7 design for all three methods. Each of the three survey types was repeated in four rounds, so that when data from the four rounds were combined, we could compare the third dose of pentavalent vaccine (Penta3) and the first dose of measles-containing vaccine (MCV1) coverage estimates across survey types with a significance level (α) of 5% (two-sided), a power 1-β of 80% and a detectable difference between any two methods of 8.5%. Statistical analysis Statistical analysis was conducted in Stata 13 (StataCorp LP, College Park, TX, USA). The WHO Vaccination Coverage Quality Indicators (VCQI; beta version 2.5) analytical package20 was used for vaccine coverage indicator estimation. By convention, the traditional EPI survey data were unweighted, but the compact segment and grid-based GIS survey data were weighted using the inverse probability of segment and dwelling selection, respectively. We used Wilson confidence intervals21–23 to quantify the uncertainty of vaccine coverage estimates. Categorical demographic covariates and coverage estimates were compared between sampling methods using Rao-Scott chi square tests, and continuous variables with non-normal distributions were compared using Wilcoxon rank sum tests. Data from children who were enrolled in more than one survey with different sampling methods were excluded from hypothesis tests to avoid correlation between measures. Weight-for-age Z-scores were calculated based on 2006 WHO child growth standards. Effective sample size was defined as the actual sample size divided by the design effect. Results Study population and characteristics Of the 16 497 households visited, 2444 (14.8%) children were eligible (Figure 1); 385 households were visited more than once before completing their survey. Among the 2424 (99.2%) eligible households that completed immunization questions, 126 (5.2%) were enrolled more than once in surveys with different sampling methods. The overlapping clusters are illustrated in Figure 2. Including overlapping households, the total sample size was 2550 which included 849 in EPI surveys, 844 in compact segment surveys and 857 in the grid-based GIS surveys. Across survey methods, children enrolled in the study shared similar demographic and health characteristics (Table 1). The proportion of girls enrolled in the compact segment surveys was slightly higher than in the grid-based GIS surveys. The EPI surveys enrolled fewer households in the wealthiest quintile than the compact segment survey (17.0% versus 23.1%, P-value = 0.022). Figure 1. View largeDownload slide Enrolment summary of all EPI, compact segment and grid-based GIS sampling surveys combined. Figure 1. View largeDownload slide Enrolment summary of all EPI, compact segment and grid-based GIS sampling surveys combined. Figure 2. View largeDownload slide Enrolment locations of surveys in Korangi town using three sampling methods. Figure 2. View largeDownload slide Enrolment locations of surveys in Korangi town using three sampling methods. Table 1. Sociodemographic and health characteristics of surveyed children and comparison among surveys Three methods combined Comparison by sampling methods (four rounds combined) Frequency (%a) / median [IQR] EPI surveys Compact segment surveys (CS) Grid-based GIS surveys P-valueb (EPI vs CS) P-valueb (EPI vs GIS) P-valueb (CS vs GIS) Number of children 2550 849 844 857 Child sex: female 1100 (42.8) 355 (42.0) 397 (46.2) 348 (40.3) 0.110 0.542 0.043 Child age in months at enrolment, mean 17.1 17.1 17.4 16.9 0.237 0.386 0.068 Child born in Korangi 2373 (92.9) 787 (92.8) 787 (93.1) 799 (92.9) 0.838 0.929 0.926 Father's education in years 8 (0, 10) 8 (0, 10) 8 (0, 10) 8 (0, 10) 0.638 0.069 0.159 Mother's education in years 8 (3, 10) 8 (2, 10) 8 (1, 10) 8 (0, 10) 0.953 0.103 0.121 Mother has a job, other than housewife 94 (3.7) 35 (4.1) 29 (3.3) 30 (3.7) 0.487 0.684 0.749 Mother owns a mobile phone 1687 (65.4) 557(65.7) 545 (64.0) 585 (66.6) 0.589 0.758 0.408 Institutional birth 2048 (79.4) 688 (81.1) 665 (77.7) 695 (79.4) 0.270 0.498 0.577 Breastfeeding duration (months) 14 (9, 18) 13 (8, 17) 14 (10, 18) 14 (10, 18) 0.468 0.487 0.519 Wealth index quintile 0.016 0.159 0.659  Poorest 492 (20.0) 149 (17.6) 166 (20.6) 177 (21.9) 0.257 0.075 0.671  Second 499 (20.0) 203 (23.9) 138 (16.9) 158 (19.3)  Third 493 (20.0) 175 (20.6) 163 (19.7) 155 (19.6)  Fourth 522 (20.0) 178 (21.0) 173 (19.8) 171 (19.6)  Wealthiest 539 (20.0) 144 (17.0) 204 (23.1) 191 (19.6) 0.022 0.268 0.211 Monthly household income in US dollars 135 (108, 181); 31.0% missing 135 (108, 163); 30.5% missing 135 (108, 181); 33.5% missing 135 (108, 163); 29.2% missing 0.484 0.476 0.493 Weight-for-age Z-score −1.46 −1.47 −1.46 −1.46 0.801 0.861 0.966 Three methods combined Comparison by sampling methods (four rounds combined) Frequency (%a) / median [IQR] EPI surveys Compact segment surveys (CS) Grid-based GIS surveys P-valueb (EPI vs CS) P-valueb (EPI vs GIS) P-valueb (CS vs GIS) Number of children 2550 849 844 857 Child sex: female 1100 (42.8) 355 (42.0) 397 (46.2) 348 (40.3) 0.110 0.542 0.043 Child age in months at enrolment, mean 17.1 17.1 17.4 16.9 0.237 0.386 0.068 Child born in Korangi 2373 (92.9) 787 (92.8) 787 (93.1) 799 (92.9) 0.838 0.929 0.926 Father's education in years 8 (0, 10) 8 (0, 10) 8 (0, 10) 8 (0, 10) 0.638 0.069 0.159 Mother's education in years 8 (3, 10) 8 (2, 10) 8 (1, 10) 8 (0, 10) 0.953 0.103 0.121 Mother has a job, other than housewife 94 (3.7) 35 (4.1) 29 (3.3) 30 (3.7) 0.487 0.684 0.749 Mother owns a mobile phone 1687 (65.4) 557(65.7) 545 (64.0) 585 (66.6) 0.589 0.758 0.408 Institutional birth 2048 (79.4) 688 (81.1) 665 (77.7) 695 (79.4) 0.270 0.498 0.577 Breastfeeding duration (months) 14 (9, 18) 13 (8, 17) 14 (10, 18) 14 (10, 18) 0.468 0.487 0.519 Wealth index quintile 0.016 0.159 0.659  Poorest 492 (20.0) 149 (17.6) 166 (20.6) 177 (21.9) 0.257 0.075 0.671  Second 499 (20.0) 203 (23.9) 138 (16.9) 158 (19.3)  Third 493 (20.0) 175 (20.6) 163 (19.7) 155 (19.6)  Fourth 522 (20.0) 178 (21.0) 173 (19.8) 171 (19.6)  Wealthiest 539 (20.0) 144 (17.0) 204 (23.1) 191 (19.6) 0.022 0.268 0.211 Monthly household income in US dollars 135 (108, 181); 31.0% missing 135 (108, 163); 30.5% missing 135 (108, 181); 33.5% missing 135 (108, 163); 29.2% missing 0.484 0.476 0.493 Weight-for-age Z-score −1.46 −1.47 −1.46 −1.46 0.801 0.861 0.966 IQR, interquartile range. P-value <0.05 are highlighted in bold. a Calculations of frequency were unweighted. Proportion estimates were weighted for compact segment and grid-based GIS surveys using the inverse probability of segment and dwelling selection, respectively; missing values were excluded before calculation of percentage. b Chi-square or Fisher's exact test for categorical variables; t test for normal continuous variables; Wilcoxon rank sum test for non-normal continuous variables. View Large Table 1. Sociodemographic and health characteristics of surveyed children and comparison among surveys Three methods combined Comparison by sampling methods (four rounds combined) Frequency (%a) / median [IQR] EPI surveys Compact segment surveys (CS) Grid-based GIS surveys P-valueb (EPI vs CS) P-valueb (EPI vs GIS) P-valueb (CS vs GIS) Number of children 2550 849 844 857 Child sex: female 1100 (42.8) 355 (42.0) 397 (46.2) 348 (40.3) 0.110 0.542 0.043 Child age in months at enrolment, mean 17.1 17.1 17.4 16.9 0.237 0.386 0.068 Child born in Korangi 2373 (92.9) 787 (92.8) 787 (93.1) 799 (92.9) 0.838 0.929 0.926 Father's education in years 8 (0, 10) 8 (0, 10) 8 (0, 10) 8 (0, 10) 0.638 0.069 0.159 Mother's education in years 8 (3, 10) 8 (2, 10) 8 (1, 10) 8 (0, 10) 0.953 0.103 0.121 Mother has a job, other than housewife 94 (3.7) 35 (4.1) 29 (3.3) 30 (3.7) 0.487 0.684 0.749 Mother owns a mobile phone 1687 (65.4) 557(65.7) 545 (64.0) 585 (66.6) 0.589 0.758 0.408 Institutional birth 2048 (79.4) 688 (81.1) 665 (77.7) 695 (79.4) 0.270 0.498 0.577 Breastfeeding duration (months) 14 (9, 18) 13 (8, 17) 14 (10, 18) 14 (10, 18) 0.468 0.487 0.519 Wealth index quintile 0.016 0.159 0.659  Poorest 492 (20.0) 149 (17.6) 166 (20.6) 177 (21.9) 0.257 0.075 0.671  Second 499 (20.0) 203 (23.9) 138 (16.9) 158 (19.3)  Third 493 (20.0) 175 (20.6) 163 (19.7) 155 (19.6)  Fourth 522 (20.0) 178 (21.0) 173 (19.8) 171 (19.6)  Wealthiest 539 (20.0) 144 (17.0) 204 (23.1) 191 (19.6) 0.022 0.268 0.211 Monthly household income in US dollars 135 (108, 181); 31.0% missing 135 (108, 163); 30.5% missing 135 (108, 181); 33.5% missing 135 (108, 163); 29.2% missing 0.484 0.476 0.493 Weight-for-age Z-score −1.46 −1.47 −1.46 −1.46 0.801 0.861 0.966 Three methods combined Comparison by sampling methods (four rounds combined) Frequency (%a) / median [IQR] EPI surveys Compact segment surveys (CS) Grid-based GIS surveys P-valueb (EPI vs CS) P-valueb (EPI vs GIS) P-valueb (CS vs GIS) Number of children 2550 849 844 857 Child sex: female 1100 (42.8) 355 (42.0) 397 (46.2) 348 (40.3) 0.110 0.542 0.043 Child age in months at enrolment, mean 17.1 17.1 17.4 16.9 0.237 0.386 0.068 Child born in Korangi 2373 (92.9) 787 (92.8) 787 (93.1) 799 (92.9) 0.838 0.929 0.926 Father's education in years 8 (0, 10) 8 (0, 10) 8 (0, 10) 8 (0, 10) 0.638 0.069 0.159 Mother's education in years 8 (3, 10) 8 (2, 10) 8 (1, 10) 8 (0, 10) 0.953 0.103 0.121 Mother has a job, other than housewife 94 (3.7) 35 (4.1) 29 (3.3) 30 (3.7) 0.487 0.684 0.749 Mother owns a mobile phone 1687 (65.4) 557(65.7) 545 (64.0) 585 (66.6) 0.589 0.758 0.408 Institutional birth 2048 (79.4) 688 (81.1) 665 (77.7) 695 (79.4) 0.270 0.498 0.577 Breastfeeding duration (months) 14 (9, 18) 13 (8, 17) 14 (10, 18) 14 (10, 18) 0.468 0.487 0.519 Wealth index quintile 0.016 0.159 0.659  Poorest 492 (20.0) 149 (17.6) 166 (20.6) 177 (21.9) 0.257 0.075 0.671  Second 499 (20.0) 203 (23.9) 138 (16.9) 158 (19.3)  Third 493 (20.0) 175 (20.6) 163 (19.7) 155 (19.6)  Fourth 522 (20.0) 178 (21.0) 173 (19.8) 171 (19.6)  Wealthiest 539 (20.0) 144 (17.0) 204 (23.1) 191 (19.6) 0.022 0.268 0.211 Monthly household income in US dollars 135 (108, 181); 31.0% missing 135 (108, 163); 30.5% missing 135 (108, 181); 33.5% missing 135 (108, 163); 29.2% missing 0.484 0.476 0.493 Weight-for-age Z-score −1.46 −1.47 −1.46 −1.46 0.801 0.861 0.966 IQR, interquartile range. P-value <0.05 are highlighted in bold. a Calculations of frequency were unweighted. Proportion estimates were weighted for compact segment and grid-based GIS surveys using the inverse probability of segment and dwelling selection, respectively; missing values were excluded before calculation of percentage. b Chi-square or Fisher's exact test for categorical variables; t test for normal continuous variables; Wilcoxon rank sum test for non-normal continuous variables. View Large Comparison of vaccine coverage estimates Key vaccine coverage indicators were estimated from the four combined rounds of data and compared across sampling methods (Table 2). Coverage of the first pentavalent vaccine dose (Penta1) was interpreted as an indicator of children’s access to routine immunization services; coverages of Penta3 and MCV1 were interpreted as demonstrating retention in the routine immunization programme at 14 weeks and 9 months of age, respectively. Coverage estimates based on vaccine receipt ascertained either by caregiver’s verbal recall or HBRs (Penta3 = 69.7% in EPI surveys) were considerably higher than the coverage estimates based on only HBR documentation (Penta3 = 47.5% in EPI surveys). Estimates for Penta3 and MCV1 using the revised WHO manual ‘valid vaccine coverage’ method, which includes the vaccine doses received in a valid time frame according to the EPI schedule, are presented in Table 2. Since the vaccination dates were only accurately ascertained from HBRs, valid vaccine coverage estimates (Penta3 = 38.0% in EPI surveys) were considerably lower than the crude estimates by HBRs or verbal history (Penta3 = 69.7% in EPI surveys). Table 2. Comparison of vaccine coverage estimates among surveys using EPI, compact segment and grid-based GIS sampling methods EPI surveys (N = 849) Compact segment surveys (N = 844) Grid-based GIS surveys (N = 857) Comparison by sampling methods Cov. (%) (95% CI)a SEa DEFF ICC Cov. (%) (95% CI) SE DEFF ICC Cov. (%) (95% CI) SE DEFF ICC P-valueb (EPI vs. CS) P-valueb (EPI vs. GIS) P-valueb (CS vs. GIS) Crude vaccine coverage by home-based records (HBRs) or verbal history = (sum of weights for respondents who received vaccine according to HBRs or verbal history) / (sum of weights for all respondents)  Penta1 88.2 (85.5, 90.5) 1.3 1.34 0.056 83.7 (79.4, 87.2) 2.0 2.44 0.236 88.5 (85.5, 90.9) 1.4 1.57 0.096 0.036 0.970 0.040  Penta3 69.7 (65.7, 73.5) 2.0 1.57 0.094 69.0 (64.4, 73.3) 2.3 2.03 0.168 70.8 (66.6, 74.6) 2.0 1.72 0.121 0.527 0.791 0.380  First dose of measles vaccine (MCV1) 75.3 (71.8, 78.5) 1.7 1.33 0.055 70.9 (66.1, 75.3) 2.4 2.28 0.210 74.4 (70.4, 77.9) 1.9 1.67 0.112 0.072 0.495 0.247 Crude vaccine coverage by HBRs = (sum of weights for respondents who received vaccine according to HBRs) / (sum of weights for all respondents)  Penta1 50.9 (46.8, 55.0) 2.1 1.48 0.079 49.5 (45.1, 53.8) 2.2 1.67 0.111 52.8 (47.7, 57.8) 2.6 2.33 0.223 0.662 0.684 0.427  Penta3 47.5 (43.3, 51.6) 2.1 1.53 0.088 44.4 (40.3, 48.5) 2.1 1.48 0.079 46.5 (41.6, 51.5) 2.5 2.20 0.201 0.245 0.631 0.569  MCV1 42.2 (38.3, 46.1) 2.0 1.41 0.067 41.0 (36.7, 45.4) 2.2 1.71 0.117 42.4 (37.2, 47.7) 2.7 2.55 0.259 0.748 0.837 0.938 Valid vaccine coverage = (sum of weights for respondents who received a valid vaccine dose according to HBRs)c/(sum of weights for all respondents)  Penta3 38.0 (34.0, 42.2) 2.1 1.58 0.096 33.9 (29.9, 38.2) 2.1 1.71 0.116 35.5 (31.0, 40.2) 2.4 2.08 0.181 0.115 0.294 0.657  MCV1 36.0 (32.2, 40.1) 2.0 1.50 0.083 32.3 (27.9, 37.0) 2.3 2.07 0.176 37.2 (32.2, 42.6) 2.7 2.61 0.270 0.145 0.935 0.237 Fully vaccinated by HBRs or verbal history = (sum of weights for respondents who received BCG, Penta3, OPV3 and MCV1 according to HBRs or verbal history)d/(sum of weights for all respondents) 51.5 (47.4, 55.5) 2.1 1.46 0.076 51.3 (46.9, 55.7) 2.3 1.73 0.121 54.4 (49.7, 59.1) 2.4 1.99 0.166 0.892 0.440 0.379 Not vaccinated by HBRs and verbal history = (sum of weights for respondents who received no BCG, Penta, OPV or MCV dose according to HBRs or verbal history)/(sum of weights for all respondents) 5.9 (4.3, 8.1) 1.0 1.44 0.072 10.0 (7.0, 14.1) 1.8 3.12 0.348 6.9 (5.1, 9.3) 1.1 1.51 0.086 0.034 0.376 0.130 Valid HBR retention = (sum of weights for respondents who show an HBR with at least one vaccination date on it) / (sum of weights for all respondents) 51.9 (47.9, 56.0) 2.1 1.47 0.077 48.8 (44.3, 53.3) 2.3 1.79 0.129 53.6 (48.7, 58.4) 2.5 2.10 0.184 0.281 0.766 0.200 EPI surveys (N = 849) Compact segment surveys (N = 844) Grid-based GIS surveys (N = 857) Comparison by sampling methods Cov. (%) (95% CI)a SEa DEFF ICC Cov. (%) (95% CI) SE DEFF ICC Cov. (%) (95% CI) SE DEFF ICC P-valueb (EPI vs. CS) P-valueb (EPI vs. GIS) P-valueb (CS vs. GIS) Crude vaccine coverage by home-based records (HBRs) or verbal history = (sum of weights for respondents who received vaccine according to HBRs or verbal history) / (sum of weights for all respondents)  Penta1 88.2 (85.5, 90.5) 1.3 1.34 0.056 83.7 (79.4, 87.2) 2.0 2.44 0.236 88.5 (85.5, 90.9) 1.4 1.57 0.096 0.036 0.970 0.040  Penta3 69.7 (65.7, 73.5) 2.0 1.57 0.094 69.0 (64.4, 73.3) 2.3 2.03 0.168 70.8 (66.6, 74.6) 2.0 1.72 0.121 0.527 0.791 0.380  First dose of measles vaccine (MCV1) 75.3 (71.8, 78.5) 1.7 1.33 0.055 70.9 (66.1, 75.3) 2.4 2.28 0.210 74.4 (70.4, 77.9) 1.9 1.67 0.112 0.072 0.495 0.247 Crude vaccine coverage by HBRs = (sum of weights for respondents who received vaccine according to HBRs) / (sum of weights for all respondents)  Penta1 50.9 (46.8, 55.0) 2.1 1.48 0.079 49.5 (45.1, 53.8) 2.2 1.67 0.111 52.8 (47.7, 57.8) 2.6 2.33 0.223 0.662 0.684 0.427  Penta3 47.5 (43.3, 51.6) 2.1 1.53 0.088 44.4 (40.3, 48.5) 2.1 1.48 0.079 46.5 (41.6, 51.5) 2.5 2.20 0.201 0.245 0.631 0.569  MCV1 42.2 (38.3, 46.1) 2.0 1.41 0.067 41.0 (36.7, 45.4) 2.2 1.71 0.117 42.4 (37.2, 47.7) 2.7 2.55 0.259 0.748 0.837 0.938 Valid vaccine coverage = (sum of weights for respondents who received a valid vaccine dose according to HBRs)c/(sum of weights for all respondents)  Penta3 38.0 (34.0, 42.2) 2.1 1.58 0.096 33.9 (29.9, 38.2) 2.1 1.71 0.116 35.5 (31.0, 40.2) 2.4 2.08 0.181 0.115 0.294 0.657  MCV1 36.0 (32.2, 40.1) 2.0 1.50 0.083 32.3 (27.9, 37.0) 2.3 2.07 0.176 37.2 (32.2, 42.6) 2.7 2.61 0.270 0.145 0.935 0.237 Fully vaccinated by HBRs or verbal history = (sum of weights for respondents who received BCG, Penta3, OPV3 and MCV1 according to HBRs or verbal history)d/(sum of weights for all respondents) 51.5 (47.4, 55.5) 2.1 1.46 0.076 51.3 (46.9, 55.7) 2.3 1.73 0.121 54.4 (49.7, 59.1) 2.4 1.99 0.166 0.892 0.440 0.379 Not vaccinated by HBRs and verbal history = (sum of weights for respondents who received no BCG, Penta, OPV or MCV dose according to HBRs or verbal history)/(sum of weights for all respondents) 5.9 (4.3, 8.1) 1.0 1.44 0.072 10.0 (7.0, 14.1) 1.8 3.12 0.348 6.9 (5.1, 9.3) 1.1 1.51 0.086 0.034 0.376 0.130 Valid HBR retention = (sum of weights for respondents who show an HBR with at least one vaccination date on it) / (sum of weights for all respondents) 51.9 (47.9, 56.0) 2.1 1.47 0.077 48.8 (44.3, 53.3) 2.3 1.79 0.129 53.6 (48.7, 58.4) 2.5 2.10 0.184 0.281 0.766 0.200 Cov. coverage; SE, standard error. P-value <0.05 are highlighted in bold. a Robust variance estimates and Wilson confidence intervals20-22 are reported to quantify the uncertainty of vaccine coverage estimates. b Coverage estimates were compared using Rao-Scott chi square tests; overlap sample between surveys with different sampling methods was excluded from hypothesis tests to avoid correlations. c Valid Penta3 dose was defined as pentavalent vaccination received after 14 weeks of age and at least 4 weeks after Penta2; valid MCV1 dose was defined as measles vaccination received after 9 months of age. d Full vaccination was defined as a child who received a birth dose of BCG vaccine, three doses of pentavalent vaccines (Penta), three doses of oral polio vaccines (OPV) (excluding the birth dose), and one dose of measles vaccine (MCV1). Routine immunization in Karachi also includes a birth dose of polio vaccine (OPV0), three doses of pneumococcal conjugate vaccines (PCV), an inactive polio vaccine (IPV) and a second dose of measles vaccine (MCV2). MCV2 was not included in the full vaccination definition because the age range of enrolled children (12–23 months) was not appropriate to assess the coverage of MCV2 scheduled at 15 months of age. Since MCV2 doses are often given between 15 and 18 months of age, restricting the sample to 19 months or older would result in a sample size too small to provide meaningful results. IPV was not included because the introduction of IPV was ongoing during the study time. PCV and OPV0 were not included because receipts of these doses could not be accurately ascertained in our study as some HBRs retrieved from Korangi were not designed to record these doses, i.e. did not have slots to record these doses. View Large Table 2. Comparison of vaccine coverage estimates among surveys using EPI, compact segment and grid-based GIS sampling methods EPI surveys (N = 849) Compact segment surveys (N = 844) Grid-based GIS surveys (N = 857) Comparison by sampling methods Cov. (%) (95% CI)a SEa DEFF ICC Cov. (%) (95% CI) SE DEFF ICC Cov. (%) (95% CI) SE DEFF ICC P-valueb (EPI vs. CS) P-valueb (EPI vs. GIS) P-valueb (CS vs. GIS) Crude vaccine coverage by home-based records (HBRs) or verbal history = (sum of weights for respondents who received vaccine according to HBRs or verbal history) / (sum of weights for all respondents)  Penta1 88.2 (85.5, 90.5) 1.3 1.34 0.056 83.7 (79.4, 87.2) 2.0 2.44 0.236 88.5 (85.5, 90.9) 1.4 1.57 0.096 0.036 0.970 0.040  Penta3 69.7 (65.7, 73.5) 2.0 1.57 0.094 69.0 (64.4, 73.3) 2.3 2.03 0.168 70.8 (66.6, 74.6) 2.0 1.72 0.121 0.527 0.791 0.380  First dose of measles vaccine (MCV1) 75.3 (71.8, 78.5) 1.7 1.33 0.055 70.9 (66.1, 75.3) 2.4 2.28 0.210 74.4 (70.4, 77.9) 1.9 1.67 0.112 0.072 0.495 0.247 Crude vaccine coverage by HBRs = (sum of weights for respondents who received vaccine according to HBRs) / (sum of weights for all respondents)  Penta1 50.9 (46.8, 55.0) 2.1 1.48 0.079 49.5 (45.1, 53.8) 2.2 1.67 0.111 52.8 (47.7, 57.8) 2.6 2.33 0.223 0.662 0.684 0.427  Penta3 47.5 (43.3, 51.6) 2.1 1.53 0.088 44.4 (40.3, 48.5) 2.1 1.48 0.079 46.5 (41.6, 51.5) 2.5 2.20 0.201 0.245 0.631 0.569  MCV1 42.2 (38.3, 46.1) 2.0 1.41 0.067 41.0 (36.7, 45.4) 2.2 1.71 0.117 42.4 (37.2, 47.7) 2.7 2.55 0.259 0.748 0.837 0.938 Valid vaccine coverage = (sum of weights for respondents who received a valid vaccine dose according to HBRs)c/(sum of weights for all respondents)  Penta3 38.0 (34.0, 42.2) 2.1 1.58 0.096 33.9 (29.9, 38.2) 2.1 1.71 0.116 35.5 (31.0, 40.2) 2.4 2.08 0.181 0.115 0.294 0.657  MCV1 36.0 (32.2, 40.1) 2.0 1.50 0.083 32.3 (27.9, 37.0) 2.3 2.07 0.176 37.2 (32.2, 42.6) 2.7 2.61 0.270 0.145 0.935 0.237 Fully vaccinated by HBRs or verbal history = (sum of weights for respondents who received BCG, Penta3, OPV3 and MCV1 according to HBRs or verbal history)d/(sum of weights for all respondents) 51.5 (47.4, 55.5) 2.1 1.46 0.076 51.3 (46.9, 55.7) 2.3 1.73 0.121 54.4 (49.7, 59.1) 2.4 1.99 0.166 0.892 0.440 0.379 Not vaccinated by HBRs and verbal history = (sum of weights for respondents who received no BCG, Penta, OPV or MCV dose according to HBRs or verbal history)/(sum of weights for all respondents) 5.9 (4.3, 8.1) 1.0 1.44 0.072 10.0 (7.0, 14.1) 1.8 3.12 0.348 6.9 (5.1, 9.3) 1.1 1.51 0.086 0.034 0.376 0.130 Valid HBR retention = (sum of weights for respondents who show an HBR with at least one vaccination date on it) / (sum of weights for all respondents) 51.9 (47.9, 56.0) 2.1 1.47 0.077 48.8 (44.3, 53.3) 2.3 1.79 0.129 53.6 (48.7, 58.4) 2.5 2.10 0.184 0.281 0.766 0.200 EPI surveys (N = 849) Compact segment surveys (N = 844) Grid-based GIS surveys (N = 857) Comparison by sampling methods Cov. (%) (95% CI)a SEa DEFF ICC Cov. (%) (95% CI) SE DEFF ICC Cov. (%) (95% CI) SE DEFF ICC P-valueb (EPI vs. CS) P-valueb (EPI vs. GIS) P-valueb (CS vs. GIS) Crude vaccine coverage by home-based records (HBRs) or verbal history = (sum of weights for respondents who received vaccine according to HBRs or verbal history) / (sum of weights for all respondents)  Penta1 88.2 (85.5, 90.5) 1.3 1.34 0.056 83.7 (79.4, 87.2) 2.0 2.44 0.236 88.5 (85.5, 90.9) 1.4 1.57 0.096 0.036 0.970 0.040  Penta3 69.7 (65.7, 73.5) 2.0 1.57 0.094 69.0 (64.4, 73.3) 2.3 2.03 0.168 70.8 (66.6, 74.6) 2.0 1.72 0.121 0.527 0.791 0.380  First dose of measles vaccine (MCV1) 75.3 (71.8, 78.5) 1.7 1.33 0.055 70.9 (66.1, 75.3) 2.4 2.28 0.210 74.4 (70.4, 77.9) 1.9 1.67 0.112 0.072 0.495 0.247 Crude vaccine coverage by HBRs = (sum of weights for respondents who received vaccine according to HBRs) / (sum of weights for all respondents)  Penta1 50.9 (46.8, 55.0) 2.1 1.48 0.079 49.5 (45.1, 53.8) 2.2 1.67 0.111 52.8 (47.7, 57.8) 2.6 2.33 0.223 0.662 0.684 0.427  Penta3 47.5 (43.3, 51.6) 2.1 1.53 0.088 44.4 (40.3, 48.5) 2.1 1.48 0.079 46.5 (41.6, 51.5) 2.5 2.20 0.201 0.245 0.631 0.569  MCV1 42.2 (38.3, 46.1) 2.0 1.41 0.067 41.0 (36.7, 45.4) 2.2 1.71 0.117 42.4 (37.2, 47.7) 2.7 2.55 0.259 0.748 0.837 0.938 Valid vaccine coverage = (sum of weights for respondents who received a valid vaccine dose according to HBRs)c/(sum of weights for all respondents)  Penta3 38.0 (34.0, 42.2) 2.1 1.58 0.096 33.9 (29.9, 38.2) 2.1 1.71 0.116 35.5 (31.0, 40.2) 2.4 2.08 0.181 0.115 0.294 0.657  MCV1 36.0 (32.2, 40.1) 2.0 1.50 0.083 32.3 (27.9, 37.0) 2.3 2.07 0.176 37.2 (32.2, 42.6) 2.7 2.61 0.270 0.145 0.935 0.237 Fully vaccinated by HBRs or verbal history = (sum of weights for respondents who received BCG, Penta3, OPV3 and MCV1 according to HBRs or verbal history)d/(sum of weights for all respondents) 51.5 (47.4, 55.5) 2.1 1.46 0.076 51.3 (46.9, 55.7) 2.3 1.73 0.121 54.4 (49.7, 59.1) 2.4 1.99 0.166 0.892 0.440 0.379 Not vaccinated by HBRs and verbal history = (sum of weights for respondents who received no BCG, Penta, OPV or MCV dose according to HBRs or verbal history)/(sum of weights for all respondents) 5.9 (4.3, 8.1) 1.0 1.44 0.072 10.0 (7.0, 14.1) 1.8 3.12 0.348 6.9 (5.1, 9.3) 1.1 1.51 0.086 0.034 0.376 0.130 Valid HBR retention = (sum of weights for respondents who show an HBR with at least one vaccination date on it) / (sum of weights for all respondents) 51.9 (47.9, 56.0) 2.1 1.47 0.077 48.8 (44.3, 53.3) 2.3 1.79 0.129 53.6 (48.7, 58.4) 2.5 2.10 0.184 0.281 0.766 0.200 Cov. coverage; SE, standard error. P-value <0.05 are highlighted in bold. a Robust variance estimates and Wilson confidence intervals20-22 are reported to quantify the uncertainty of vaccine coverage estimates. b Coverage estimates were compared using Rao-Scott chi square tests; overlap sample between surveys with different sampling methods was excluded from hypothesis tests to avoid correlations. c Valid Penta3 dose was defined as pentavalent vaccination received after 14 weeks of age and at least 4 weeks after Penta2; valid MCV1 dose was defined as measles vaccination received after 9 months of age. d Full vaccination was defined as a child who received a birth dose of BCG vaccine, three doses of pentavalent vaccines (Penta), three doses of oral polio vaccines (OPV) (excluding the birth dose), and one dose of measles vaccine (MCV1). Routine immunization in Karachi also includes a birth dose of polio vaccine (OPV0), three doses of pneumococcal conjugate vaccines (PCV), an inactive polio vaccine (IPV) and a second dose of measles vaccine (MCV2). MCV2 was not included in the full vaccination definition because the age range of enrolled children (12–23 months) was not appropriate to assess the coverage of MCV2 scheduled at 15 months of age. Since MCV2 doses are often given between 15 and 18 months of age, restricting the sample to 19 months or older would result in a sample size too small to provide meaningful results. IPV was not included because the introduction of IPV was ongoing during the study time. PCV and OPV0 were not included because receipts of these doses could not be accurately ascertained in our study as some HBRs retrieved from Korangi were not designed to record these doses, i.e. did not have slots to record these doses. View Large When vaccine coverage estimates were compared across the three sampling methods in Table 2, observed differences were all less than 6%. Since the study was only powered to identify potential coverage differences over 8.5%, we found no statistically significant differences for most coverage estimates. The only exception was that the crude Penta1 coverage by HBRs or verbal history in compact segment surveys was 5.4% age points [95% confidence interval (CI): (0.4, 10.4); P-value = 0.036] lower than in the EPI surveys and 5.3% age points [95% CI: (0.3, 10.4); P-value = 0.040] lower than in the grid-based GIS surveys. This finding was associated with an estimated 5.0% age points difference [95% CI: (0.4, 9.6); P-value = 0.034] of unvaccinated children between the EPI and compact segment survey participants. Despite these exceptions, we could not conclude that there was a systematic difference in coverage estimates by survey sampling method. Design effects were calculated for all coverage estimates and are compared across the three sampling methods in Table 2. Since the vaccine coverage estimates were similar across methods, larger design effects were associated with the larger error and lower precision of the coverage estimates. Among the estimates based on only HBR documented vaccine receipt, grid-based GIS surveys had the largest design effects and the EPI surveys had the smallest. Among the HBR or verbal history-based estimates, compact segment surveys had the largest design effects; EPI surveys still had the smallest. Comparison of survey implementation time and efficiency The implementation time of various field activities was summarized based on the daily fieldwork logs filled by survey teams (Table 3, Figure 3). For each EPI survey cluster, on average a field team consisting of a supervisor and three interviewers needed 262 min to complete enrolments and interviews. This included 50 min to travel to and from the field, 27 min to select the first household and 186 min to locate, screen, consent and interview households. For each household, the screening and consenting process took 7.7 min on average [median = 7; 25% percentile (p25) =5; p75 = 10], and an interview took 18.0 min (median = 16; p25 = 12; p75 = 20). In comparison, compact segment surveys and grid-based GIS surveys required similar travel time as the EPI method but less time to select the first household. This is because compact segment and grid-based GIS surveys did not include the EPI survey step of sketch mapping and within-subdivision household enumeration procedures. The full cluster detailed household mapping step in compact segment surveys nearly doubled overall implementation time as compared with the EPI survey method and required an extra day or more to complete, especially in large clusters. For grid-based GIS surveys, the household searching step required more time than the EPI survey method, because the interviewers frequently needed to walk long distances between dwellings in a predetermined sequence (see the satellite imagery in Supplementary D, available as Supplementary data at IJE online). Depending on the density and accessibility of dwellings, an extra 30 to 120 min approximately were needed for household searching in a typical quadrat compared with the convenient next-door visits in the EPI ‘random walk’ method. Figure 3. View largeDownload slide Comparison of work time and efficiency of EPI, compact segment and grid-based GIS sampling methods. Figure 3. View largeDownload slide Comparison of work time and efficiency of EPI, compact segment and grid-based GIS sampling methods. Table 3. Comparison of survey implementation time among surveys using EPI, compact segment and GIS/grid-based sampling methods EPI surveys Compact segment surveys GIS/grid-based surveys Median (p25, p75)a Mean Median (p25, p75)a Mean Median (p25, p75)a Mean Fieldwork timeb (minutes per cluster)  Travelling to and from field 48 (40, 60) 49.2 45 (37, 60) 47.6 50 (44, 60) 52.8  Mapping households in cluster mapsc 190 (170, 225) 232.4  Locating first householdd 20 (5, 40) 26.6 5 (2, 5) 5.7 5 (2, 5) 5.6  Household searching, enrolment and interview 150 (120, 200) 185.8 155 (125, 195) 179.1 203 (149, 321) 239.8  Total fieldwork time 261.6 464.8 298.2 Office work timee (minutes per cluster)  Defining and selecting segments ≈ 60  Producing satellite imagery printouts with household enumeration ≈ 30 Number of enrolmentsb per cluster 7 (7, 7) 6.9 6 (4, 9) 6.6 7 (7, 7) 6.5 Effective sample sizef per cluster  Crude Penta3 coverage by HBRs or verbal history 4.4 3.3 3.8  Crude MCV1 coverage by HBRs or verbal history 5.2 2.9 3.9  Crude Penta3 coverage by HBRs 4.5 4.5 3.0  Crude MCV1 coverage by HBRs 4.9 3.9 2.6 Implementation time (minutes) per effective sample  Crude Penta3 coverage by HBRs or verbal history 60 161 87  Crude MCV1 coverage by HBRs or verbal history 51 181 84  Crude Penta3 coverage by HBRs 58 118 110  Crude MCV1 coverage by HBRs 54 136 128 EPI surveys Compact segment surveys GIS/grid-based surveys Median (p25, p75)a Mean Median (p25, p75)a Mean Median (p25, p75)a Mean Fieldwork timeb (minutes per cluster)  Travelling to and from field 48 (40, 60) 49.2 45 (37, 60) 47.6 50 (44, 60) 52.8  Mapping households in cluster mapsc 190 (170, 225) 232.4  Locating first householdd 20 (5, 40) 26.6 5 (2, 5) 5.7 5 (2, 5) 5.6  Household searching, enrolment and interview 150 (120, 200) 185.8 155 (125, 195) 179.1 203 (149, 321) 239.8  Total fieldwork time 261.6 464.8 298.2 Office work timee (minutes per cluster)  Defining and selecting segments ≈ 60  Producing satellite imagery printouts with household enumeration ≈ 30 Number of enrolmentsb per cluster 7 (7, 7) 6.9 6 (4, 9) 6.6 7 (7, 7) 6.5 Effective sample sizef per cluster  Crude Penta3 coverage by HBRs or verbal history 4.4 3.3 3.8  Crude MCV1 coverage by HBRs or verbal history 5.2 2.9 3.9  Crude Penta3 coverage by HBRs 4.5 4.5 3.0  Crude MCV1 coverage by HBRs 4.9 3.9 2.6 Implementation time (minutes) per effective sample  Crude Penta3 coverage by HBRs or verbal history 60 161 87  Crude MCV1 coverage by HBRs or verbal history 51 181 84  Crude Penta3 coverage by HBRs 58 118 110  Crude MCV1 coverage by HBRs 54 136 128 a ‘p25’ stands for the 25% percentile; ‘p75’ stands for the 75% percentile. b If the mother, or primary caregiver, was not present at the time of the visit, follow-up visits were conducted until a maximum of five visits were attempted. Enrolments achieved in follow-up visits (N = 165) are excluded in the calculation of fieldwork time and efficiency. c Complete household mapping was only needed in compact segment surveys. The activity was conducted on a different day before survey, by field supervisors with the help of one or two interviewers. In large clusters, the mapping could take multiple days. d In EPI surveys, the process of locating the first household included: sketching a cluster map, roughly dividing the cluster into subdivisions and enumerating households in a selected subdivision. A team of one supervisor and three interviewers completed this process together, once arrived at the cluster on the morning of EPI survey day. e This category is only intended for activities that were implemented differently in the three sampling methods. Time spent on quality assurance, data inspection, training, piloting and statistical sampling preparation is excluded. Office work hours were not documented in logs, but approximates were provided here based on our experience. f Effective sample size = actual sample size/design effect (DEFF). View Large Table 3. Comparison of survey implementation time among surveys using EPI, compact segment and GIS/grid-based sampling methods EPI surveys Compact segment surveys GIS/grid-based surveys Median (p25, p75)a Mean Median (p25, p75)a Mean Median (p25, p75)a Mean Fieldwork timeb (minutes per cluster)  Travelling to and from field 48 (40, 60) 49.2 45 (37, 60) 47.6 50 (44, 60) 52.8  Mapping households in cluster mapsc 190 (170, 225) 232.4  Locating first householdd 20 (5, 40) 26.6 5 (2, 5) 5.7 5 (2, 5) 5.6  Household searching, enrolment and interview 150 (120, 200) 185.8 155 (125, 195) 179.1 203 (149, 321) 239.8  Total fieldwork time 261.6 464.8 298.2 Office work timee (minutes per cluster)  Defining and selecting segments ≈ 60  Producing satellite imagery printouts with household enumeration ≈ 30 Number of enrolmentsb per cluster 7 (7, 7) 6.9 6 (4, 9) 6.6 7 (7, 7) 6.5 Effective sample sizef per cluster  Crude Penta3 coverage by HBRs or verbal history 4.4 3.3 3.8  Crude MCV1 coverage by HBRs or verbal history 5.2 2.9 3.9  Crude Penta3 coverage by HBRs 4.5 4.5 3.0  Crude MCV1 coverage by HBRs 4.9 3.9 2.6 Implementation time (minutes) per effective sample  Crude Penta3 coverage by HBRs or verbal history 60 161 87  Crude MCV1 coverage by HBRs or verbal history 51 181 84  Crude Penta3 coverage by HBRs 58 118 110  Crude MCV1 coverage by HBRs 54 136 128 EPI surveys Compact segment surveys GIS/grid-based surveys Median (p25, p75)a Mean Median (p25, p75)a Mean Median (p25, p75)a Mean Fieldwork timeb (minutes per cluster)  Travelling to and from field 48 (40, 60) 49.2 45 (37, 60) 47.6 50 (44, 60) 52.8  Mapping households in cluster mapsc 190 (170, 225) 232.4  Locating first householdd 20 (5, 40) 26.6 5 (2, 5) 5.7 5 (2, 5) 5.6  Household searching, enrolment and interview 150 (120, 200) 185.8 155 (125, 195) 179.1 203 (149, 321) 239.8  Total fieldwork time 261.6 464.8 298.2 Office work timee (minutes per cluster)  Defining and selecting segments ≈ 60  Producing satellite imagery printouts with household enumeration ≈ 30 Number of enrolmentsb per cluster 7 (7, 7) 6.9 6 (4, 9) 6.6 7 (7, 7) 6.5 Effective sample sizef per cluster  Crude Penta3 coverage by HBRs or verbal history 4.4 3.3 3.8  Crude MCV1 coverage by HBRs or verbal history 5.2 2.9 3.9  Crude Penta3 coverage by HBRs 4.5 4.5 3.0  Crude MCV1 coverage by HBRs 4.9 3.9 2.6 Implementation time (minutes) per effective sample  Crude Penta3 coverage by HBRs or verbal history 60 161 87  Crude MCV1 coverage by HBRs or verbal history 51 181 84  Crude Penta3 coverage by HBRs 58 118 110  Crude MCV1 coverage by HBRs 54 136 128 a ‘p25’ stands for the 25% percentile; ‘p75’ stands for the 75% percentile. b If the mother, or primary caregiver, was not present at the time of the visit, follow-up visits were conducted until a maximum of five visits were attempted. Enrolments achieved in follow-up visits (N = 165) are excluded in the calculation of fieldwork time and efficiency. c Complete household mapping was only needed in compact segment surveys. The activity was conducted on a different day before survey, by field supervisors with the help of one or two interviewers. In large clusters, the mapping could take multiple days. d In EPI surveys, the process of locating the first household included: sketching a cluster map, roughly dividing the cluster into subdivisions and enumerating households in a selected subdivision. A team of one supervisor and three interviewers completed this process together, once arrived at the cluster on the morning of EPI survey day. e This category is only intended for activities that were implemented differently in the three sampling methods. Time spent on quality assurance, data inspection, training, piloting and statistical sampling preparation is excluded. Office work hours were not documented in logs, but approximates were provided here based on our experience. f Effective sample size = actual sample size/design effect (DEFF). View Large Next, we calculated the average implementation time per effective sample as a measure of survey efficiency and compared it across the three sampling methods. Effective sample size is a hypothetical concept which represents the sample size needed to achieve the same precision of a cluster survey if simple random sampling were performed instead. Since each coverage estimate was associated with a unique design effect, the effective sample size was calculated differently for each coverage estimate (Table 3). Generally, data collection per effective sample in the EPI surveys (54–60 min per effective sample) required less time than in the grid-based GIS surveys (84–128 min per effective sample) and considerably less than the compact segment surveys (118–181 min per effective sample). Discussion In this study of vaccine coverage survey methods, we conducted a series of equal-sized concurrent prospective surveys in Korangi town, Karachi, using three different household sampling methods: traditional EPI sampling, compact segment sampling, and grid-based GIS sampling. We found no differences in the vaccine coverage estimates across different sampling methods; however, due to stronger clustering effects and the use of sampling weights, the compact segment and the grid-based GIS surveys had higher design effects and therefore appeared to have lower statistical precision than the traditional EPI surveys. To achieve the same level of apparent precision, data collection activities in the compact segment surveys would require more than twice the time needed for the traditional EPI surveys in this setting. Whereas the study was planned before the 2018 revision of WHO EPI manual was available, we designed the study to provide evidence that can help the manual users, including governments, EPI programme managers and researchers, to understand how the traditional EPI household sampling approach may be modified to become more methodologically rigorous but still ‘rapid’ to implement; how coverage estimates may differ from alternative methods; and how much more effort is needed to implement alternative methods. Methodologically, the two alternative approaches studied are more rigorous than the traditional EPI sampling method. First, unlike the EPI surveys, the compact segment surveys use probability sampling, in which each eligible household has a known, non-zero probability of being selected. In grid-based GIS sampling, household selection probabilities are unknown, but can be approximated by the dwelling selection probabilities. The approximation is based on the assumption that the number of households in each dwelling is homogeneous within each quadrat, which is an acceptable assumption in Korangi town. Second, prespecifying the household selections on cluster maps and satellite imagery helps reduce the potential biases caused by interviewers’ unintentional and intentional household selection preference. Third, by avoiding census-based PPS cluster selection, the grid-based GIS sampling is not prone to potential biases caused by inaccurate or outdated census data. Our EPI and compact segment surveys relied on an official projection from the 1998 census. A new census was conducted in 2017 after we completed the study. Finally, the definition of quadrats is independent of the population distribution and enumeration area size. Thus, through oversampling households in less populated areas and enumeration areas of large geographical size, the grid-based GIS sampling method theoretically captures a larger variety of the target population than enumeration area-based methods (Figure 2). However, compared with a simple random household sample within each cluster, there are shared disadvantages among the three rapid methods. For example, they are all more prone to estimation variability and high design effects caused by the ‘pocketing’ of unvaccinated children, i.e. the tendency for households near to one another to share the same vaccination status. The study was not designed to assess if any one sampling method results in a more valid estimate of true vaccine coverage since no ‘gold standard’ value was available. Based on the four rounds of survey with a combined sample size to detect coverage differences of at least 8.5% across survey methods, we concluded that the coverage estimates were not systematically different. Though some estimates appeared to be 5% higher in the EPI and grid-based GIS surveys than in the compact segment surveys, we do not consider these differences to be important in practice, given that national and subnational vaccine coverage surveys are usually designed with ±5% to ±10% error margin. We found that the design effects and the implementation time of the compact segment surveys and the grid-based GIS surveys were higher than the EPI surveys in our study. Household survey design effects are usually associated with two primary factors: the clustering effect and the survey weights. The low design effect of the EPI surveys was likely caused by the questionable assumption of self-weighting, i.e. sampling weights were neglected when they were needed. When the compact segment survey data were analysed without sampling weights, the design effect of Penta3 coverage estimates (HBR or verbal history) reduced from 2.03 to 1.57, almost the same as the EPI surveys (1.57); and the grid-based GIS survey design effect reduced from 1.72 to 1.67. In this study, the extra efforts required by the alternative methods to achieve more rigorous estimates were quantified by the calculation of implementation time per effective sample (Table 3, Figure 3). In rural settings, we predict an even larger difference of implementation time between the EPI method and the alternatives, due to low dwelling density. Though not shown in the tables, two additional days of training were provided to the field teams to implement the compact segment mapping and sampling and to use satellite imagery in the sampling process; additional financial costs and efforts were also needed to establish a GIS work station and train the staff to mark dwellings on satellite imagery. We designed the grid-based GIS sampling method with the intention to minimize the hours spent in the field, which is an especially important feature for surveys in areas with security concerns. For this purpose, the latest satellite imagery was used to replace the mapping steps in other surveys; and, instead of preselecting dwellings across the quadrat, teams were only expected to visit households located close to the quadrat centre in a predefined sequence. However, the grid-based GIS surveys required more field time than the EPI surveys, due to walking distances and pathways between dwellings in the predefined sequence (Table 3). In quadrats with low dwelling density, interviewers were sometimes required to visit dwellings at the edge of quadrats to find enough eligible children, where the subsequent households were frequently more than 150 me away on the other end of the quadrat. During in-depth interviews and focus group discussions, fieldwork staff reported preferring other methods over grid-based GIS sampling because slums were often difficult to navigate and required much more walking. These qualitative results will be further discussed in a future publication. We recommend that future work be done to combine and evaluate the two alternative methodologies described in this paper. In particular, we believe that the grid-based GIS sampling method could be altered to reduce the number of hours required in the field while maintaining its advantages by introducing some compact segment concepts: first, reduce the quadrat size so that each quadrat only contains one or two expected eligible children, but allow each primary sampling unit (PSU, i.e. cluster) to contain multiple quadrats; second, employ simple random samping of N quadrats, with N equal to the desired number of clusters; third, visit all households in the selected quadrats; finally, if the desired number of enrolments is not reached by the end of quadrat, expand visits to nearby quadrats until it is, and consider this cluster of quadrats as a PSU. The number of quadrats in each PSU is dynamic. This design is an example of adaptive cluster sampling,24,25 where an initial probability sample of quadrates is selected and the addition of neighbouring quadrates is allowed given pre-specified conditions. Although our results showed that EPI surveys result in similar coverage estimates as more methodologically rigorous sampling approaches but require less implementation time, we do not conclude that the traditional EPI method is preferred. First, the precision of coverage estimates is likely overstated in EPI surveys due to the questionable assumption of self-weighting. Second, the sampling methods were only compared in one peri-urban setting, and the results may not be generalizable to rural settings or other countries. Third, the traditional EPI method is more prone to implementation-related biases, such as subjective household selection. In this research study, all surveys were carefully implemented following 20 days of intensive staff training, and were accompanied by an operational study on data quality inspection. This may have reduced the potential biases of EPI sampling in our study compared with real-world practices. Fourth, biases may have also been reduced due to our relative rigorous procedures for the selection of the first household in the EPI surveys. Larger biases might be observed if the original ‘spin-the-pen’ method had been used. Fifth, further improvements to the alternative methods may help to reduce their implementation barriers while maintaining their advantages. Besides the improvements already suggested in this paper for the grid-based GIS sampling method, the compact segment method may also be improved by using GIS techniques to reduce the burden of complete household mapping. Finally, as immunization programmes become better established, resources are increasingly available to produce more accurate and precise vaccine coverage estimates. Survey designers should not focus on survey costs and implementation ease alone. An appropriate sampling design should ensure sufficient estimation precision, maintain good methodological rigour and low risk of biases, and also be feasible and affordable in local settings. Funding This work was supported by the Pfizer Foundation, a charitable foundation established by Pfizer Inc. The funder had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. Acknowledgements We would like to express our gratitude to the EPI-Sindh, Korangi Town Health Office, Interactive Research and Development (IRD) and Indus Hospital for ensuring the successful field implementation of the research project. Conflict of interest: None declared. References 1 Cutts FT , Claquin P , Danovaro-Holliday MC , Rhoda DA. Monitoring vaccination coverage: defining the role of surveys . Vaccine 2016 ; 34 : 4103 – 09 . Google Scholar Crossref Search ADS PubMed 2 Henderson RH , Sundaresan T. Cluster sampling to assess immunization coverage: a review of experience with a simplified sampling method . Bull World Health Organ 1982 ; 60 : 253 – 60 . Google Scholar PubMed 3 Grais RF , Rose AM , Guthmann JP. Don't spin the pen: two alternative methods for second-stage sampling in urban cluster surveys . Emerg Themes Epidemiol 2007 ; 4 : 8. Google Scholar Crossref Search ADS PubMed 4 Bennett S , Woods T , Liyanage WM , Smith DL. A simplified general method for cluster-sample surveys of health in developing countries . World Health Stat Q 1991 ; 44 : 98 – 106 . Google Scholar PubMed 5 Myatt M , Feleke T , Sadler K , Collins S. A field trial of a survey method for estimating the coverage of selective feeding programmes . Bull World Health Organ 2005 ; 83 : 20 – 26 . Google Scholar PubMed 6 World Health Organization. Training for Mid-level Managers: The EPI Coverage Survey . Geneva : WHO Expanded Programme on Immunization , 1991 . 7 World Health Organization. World Health Organization Vaccination Coverage Cluster Surveys: Reference Manual . Geneva : World Health Organization , 2018 . 8 Danovaro-Holliday MC , Dansereau E , Rhoda DA , Brown DW , Cutts FT , Gacic-Dobo M. Collecting and using reliable vaccination coverage survey estimates: summary and recommendations from the “Meeting to share lessons learnt from the roll-out of the updated WHO Vaccination Coverage Cluster Survey Reference Manual and to set an operational research agenda around vaccination coverage surveys”, Geneva 18–21 April 2017 . Vaccine 2018 ; 36 : 5150 – 59 . Google Scholar Crossref Search ADS PubMed 9 Cutts FT , Izurieta HS , Rhoda DA. Measuring coverage in MNCH: design, implementation, and interpretation challenges associated with tracking vaccination coverage using household surveys . PLoS Med 2013 ; 10 : e1001404. Google Scholar Crossref Search ADS PubMed 10 Bennett S , Radalowicz A , Vella V , Tomkins A. A computer simulation of household sampling schemes for health surveys in developing countries . Int J Epidemiol 1994 ; 23 : 1282 – 91 . Google Scholar Crossref Search ADS PubMed 11 Brogan D , Flagg EW , Deming M , Waldman R. Increasing the accuracy of the Expanded Programme on Immunization's Cluster Survey design . Ann Epidemiol 1994 ; 4 : 302 – 11 . Google Scholar Crossref Search ADS PubMed 12 Turner AG , Magnani RJ , Shuaib M. A not quite as quick but much cleaner alternative to the Expanded Programme on Immunization (EPI) Cluster Survey design . Int J Epidemiol 1996 ; 25 : 198 – 203 . Google Scholar Crossref Search ADS PubMed 13 Milligan P , Njie A , Bennett S. Comparison of two cluster sampling methods for health surveys in developing countries . Int J Epidemiol 2004 ; 33 : 469 – 76 . Google Scholar Crossref Search ADS PubMed 14 Luman ET , Worku A , Berhane Y , Martin R , Cairns L. Comparison of two survey methodologies to assess vaccination coverage . Int J Epidemiol 2007 ; 36 : 633 – 41 . Google Scholar Crossref Search ADS PubMed 15 Lowther SA , Curriero FC , Shields T , Ahmed S , Monze M , Moss WJ. Feasibility of satellite image-based sampling for a health survey among urban townships of Lusaka, Zambia . Trop Med Int Health 2009 ; 14 : 70 – 78 . Google Scholar Crossref Search ADS PubMed 16 Galway L , Bell N , Sae AS et al. A two-stage cluster sampling method using gridded population data, a GIS, and Google Earth(TM) imagery in a population-based mortality survey in Iraq . Int J Health Geogr 2012 ; 11 : 12. Google Scholar Crossref Search ADS PubMed 17 World Health Organization. WHO_IVB_04.23. Immunization Coverage Cluster Survey Reference Manual. Geneva: WHO, 2005 . 18 Karachi Metropolitan Corporation. Official Web Portal of Karachi Metropolitan Corporation. 2017 . http://www.kmc.gos.pk (22 June 2017, date last accessed). 19 National Institute of Population Studies (NIPS) [Pakistan] and ICF International. Pakistan Demographic and Health Survey 2012–13. Islamabad, and Calverton, MD: NIPS and ICF Int, 2013 . 20 World Health Organization, Biostat Global Consulting. VCQI User's Guide, Version 2.52017. Geneva: WHO, 2017 . 21 Dean N , Pagano M. Evaluating confidence interval methods for binomial proportions in clustered surveys . J Surv Stat Methodol 2015 ; 3 : 484 – 503 . Google Scholar Crossref Search ADS 22 Kott PS , Carr DA. Developing an estimation strategy for a pesticide data program . J Off Stat 1997 ; 13 : 367 . 23 Wilson EB. Probable inference, the law of succession, and statistical inference . J Am Stat Assoc 1927 ; 22 : 209 – 12 . Google Scholar Crossref Search ADS 24 Thompson SK. Adaptive cluster sampling . J Am Stat Assoc 1990 ; 85 : 1050 – 59 . Google Scholar Crossref Search ADS 25 Thompson SK. Adaptive cluster sampling: designs with primary and secondary units . Biometrics 1991 ; 47 : 1103 – 15 . Google Scholar Crossref Search ADS © The Author(s) 2018; all rights reserved. Published by Oxford University Press on behalf of the International Epidemiological Association This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)
TI - Comparison of three rapid household survey sampling methods for vaccination coverage assessment in a peri-urban setting in Pakistan
JF - International Journal of Epidemiology
DO - 10.1093/ije/dyy263
DA - 2019-04-01
UR - https://www.deepdyve.com/lp/oxford-university-press/comparison-of-three-rapid-household-survey-sampling-methods-for-rUpEfSAaCV
SP - 583
VL - 48
IS - 2
DP - DeepDyve
ER -