Health Policy and Planning

Chowdhury, Reajul; McKague, Kevin; Krause, Heather

doi: 10.1093/heapol/czaa162pmid: 33881139

Abstract This paper investigates the effect of a non-financial incentive—a competitive annual award—on community health workers’ (CHWs) performance, an issue in the public health literature that has not been explored to its potential. Combining data on a competitive social ‘Best CHW’ award with the monthly performance of 4050 CHWs across Uganda, we examined if introducing social recognition awards improved the performance of CHWs. In contrast to predominant explanations about the effect of awards on motivation, our first multilevel mixed-effect models found that an award within a branch (consisting of ∼30 CHWs) was negatively associated with the performance of the local peers of the winning CHW. Models focused on non-winning branch offices revealed two additional findings. First, a branch showed underperformance if a CHW from any of the three neighbouring branches won an award in the previous year, with average monthly performance scores dropping by 27 percentage points. Second, this negative association was seen only in the top 50th percentile of CHWs. The bottom 50th percentile of CHWs exhibited increased performance by 13 percentage points. These counter-intuitive results suggest that the negative response from high performers might be explained by their frustration of not winning the award or by emotions such as envy and jealousy generated by negative social comparisons. Our results suggest that more fine-grained examination of data pertaining to motivators for CHWs in low-income countries is needed. Motivational incentives like awards may need to be customized for higher- and lower-performing CHWs. Community health workers, motivation, social recognition, status competition KEY MESSAGES Combining data on a competitive social reward intervention and monthly performance of 4050 community health workers (CHW) from Uganda, this study documents a negative association between an awarded health worker and the performance of neighbouring colleagues. The discretionary nature of the award, high degree of freedom of the supervisors in selecting nominees, rarity and low frequency of the award, and the social comparison costs can be potential sources of this negative association. Our results also showed that this association between social award and health workers’ performance varied across CHW quality tiers. Non-winning high performer CHWs exhibited underperformance when someone from a neighbouring branch wins the award, whereas the low performers CHWs reacted by improving their performance. Our results are in line with literature claiming upward-comparison induced by social awards at workplace can cause the high performers react with negative emotion like envy, and jealousy. Introduction Half of the world’s population lacks access to essential health services due to financial and human resource constraints, especially in low-income countries (World Health Organization, 2017). Finding a cost-effective strategy to reach marginalized populations with basic health services has been the priority of many low- and middle-income countries, as well as the World Health Organization for decades (Perry et al., 2014). Use of community health workers (CHWs) has emerged as a cost-effective approach to extend the reach of health systems to millions of the world’s most vulnerable populations (Swider, 2002; Zachariah et al., 2009; Neupane et al., 2018). CHWs are volunteers or paid workers that are often used in communities beyond the reach of health facilities. CHWs are ∼70% women and are generally chosen from the communities in which they live to minimize language or cultural barriers (McKague and Harrison, 2019). They receive basic training which often includes pregnancy care, the assessment and treatment of malaria and diarrhoea and the promotion of healthy behaviours such as hygiene, immunizations, nutrition and family planning (Aitken, 2014). CHWs often go door-to-door providing primary healthcare services to the population and some also sell non-prescription medicines (e.g. contraceptives, zinc tablets, soap and period products) (Reichenbach and Shimul, 2011). The CHW approach and its effectiveness in reaching marginalized communities with basic health services have been rigorously studied in the public health literature. Evidence shows that CHWs can be successful in low-income countries in reducing under-five child mortality by up to 25% (Perry et al., 2014; Neupane et al., 2018) and maternal mortality by up to 37% (Kane et al., 2010). CHWs can also increase breastfeeding practice among new mothers (Hermann et al., 2009). This low-cost approach to health service delivery is now a vital part of health systems in many low-income countries (Perry et al., 2014). However, some studies question the sustainability of the CHW approach, pointing to two important management challenges: inconsistent delivery of services (Rowe et al., 2005; Nkonki et al., 2011; Singh et al., 2015) and a high CHW drop-out rate (Rowe et al., 2005; Brunie et al., 2014; Singh et al., 2015; Vareilles et al., 2015). Since CHWs are not directly supervised on a daily basis, their level of service delivery can be inconsistent. The high drop-out rate of CHWs is believed to be the consequence of low motivation due to insufficient financial incentives (Nkonki et al., 2011; Brunie et al., 2014; Kok et al., 2015), as they are not paid well or not paid at all (Perry et al., 2017). Although it is often assumed that low levels of compensation contribute to high drop-out rates among CHWs, a field experiment from Uganda found that offering financial incentives may not provide the desired or expected health outcomes in communities served by CHWs (Deserranno, 2019). In this study, the offer of greater financial incentives attracted less socially motivated people to apply to CHW jobs and resulted in high drop-out rates. Altruistic motivation and the desire to elevate their social status in their communities have also been identified as important motivating factors for CHWs (Mlotshwa et al., 2015; Kane et al., 2020). Several studies have documented evidence that non-financial incentives such as moral support, appreciation and recognition from communities and organizations improve the performance of CHWs (Maes and Kalofonos, 2013; Druetz et al., 2015; Vareilles et al., 2017). However, quantitative empirical evidence on how non-financial social incentives affect the motivation and performance of CHWs is limited in the existing literature (Brunie et al., 2014; Kok et al., 2015, 2017; Vareilles et al., 2015; Rachlis et al., 2016; de Vries and Pool, 2017). Non-financial incentives such as social recognition and awards have been researched in other fields, including labour economics and organizational behaviour, where there is evidence that giving social rewards to workers has a positive effect on both the recognized workers and their co-workers (Fehr and Gächter, 2000; Charness and Rabin, 2002; Beersma et al., 2003; Rege and Telle, 2003; Bewley, 2007; Segal and Sobel, 2007). Beersma et al. (2003) showed evidence from an experimental study that a competitive non-financial reward structure (publicly appreciating and rewarding the best performer among the workers) enhanced status competition among workers and improved the performance of workers in terms of speed (Beersma et al., 2003). A social reward such as a ‘Best CHW Award’ involves providing recognition to individuals for their good performance and celebrating their success in public. Social rewards are distinguished from other forms of rewards by the feature of publicity (Gallus and Frey, 2017). Public recognition of an award elevates the status of the recipients and has been shown to motivate the performance of co-workers and colleagues (Fehr and Gächter, 2000; Beersma et al., 2010; Ager et al., 2016). Given the positive outcomes of non-financial social rewards in other fields, and building on models proposed by Besley and Ghatak (2008), our study set out to consider whether these outcomes are transferrable to the unique situation of CHWs in Uganda working for Building Resource Across Communities (BRAC), a large non-governmental organization. Intervention and data We used data from 4050 CHWs in BRAC Uganda to investigate whether introducing social rewards in the form of ‘Best CHW Awards’ enhanced status competition and improved performance of CHWs. In Uganda, CHWs are managed through 134 branch offices across the country (Figure 1) with ∼30 CHWs in each branch. The branch offices are where CHWs are recruited, trained, supervised and given supplies. BRAC recruits only female volunteer who have at least a primary level of education and have no children under-5 years of age at home. Each CHW is expected to serve ∼100 households. Figure 1 Open in new tabDownload slide BRAC’s CHW branch offices in Uganda. Figure 1 Open in new tabDownload slide BRAC’s CHW branch offices in Uganda. Intervention In 2015, BRAC Uganda launched a competitive reward programme for its CHWs in the form of a ‘Best CHW Award’ that gave social recognition awards to the best-performing CHWs. At the end of each year, branch offices supplied the country headquarters of BRAC with a list of nominations of their best-performing CHWs. Nominations were based on the qualitative criteria of whether the CHW was hardworking, whether the CHW was well supported in the community and whether the CHW had been regularly attending monthly refresher trainings. The Monitoring and Evaluation (M&E) Department at BRAC headquarters then followed-up each nomination to verify the performance information. The country headquarters finalized the list of winners based on this follow-up. Award recipients received a certificate and gift items worth approximately US$100. The names of the winners and their corresponding branch offices were then shared with all CHWs and health programme staff in the following month’s refresher training meetings. Starting in 2015, the award was given annually for 3 years. In 2015, the award was given to 46 CHWs; in 2016, the award was given to 43 CHWs; and in 2017, the award was given to 46 CHWs. In total, only 3% of the 4050 CHWs received an award, and none of the winners were awarded more than once in the 3-year period. Over the three intervention years, 88 of 134 branches (66%) had CHWs winning an award at least once. Among these 88 branch offices, 48 branches had a winner in only one year, 34 had a winner in two of the years and 6 branches had winners in each of the 3 years (Supplementary Table S1). Data Our data consisted of information on the monthly activities and performance of CHWs for 21 months from October 2016 to June 2018. Table 1 presents basic descriptive statistics of the performance indicators. Table 1 Descriptive statistics of major performance indicators CHW performance indicators . Obs . Mean . SD . Min . Max . Household visits  Number of months the CHW was active (out of 21) 85 050 13 3.81 0 21  Required follow-ups 85 050 3 5.70 0 128  On-time follow-ups made 85 050 2 4.39 0 105  Number of families registered 85 050 5 11.62 0 210  Family surveys 85 050 22 36.78 0 458 Assessment  Number of pregnancies registered 85 050 2 2 0 103  Total health prenatal care visits 85 050 0.56 1.29 0 55  All first prenatal care visit 85 050 0.61 1.37 0 55  Assessed any patient 85 050 11 14 0 257  Number of U1 children assessed 85 050 2 3 0 80  Number of U5 children assessed 85 050 9 12 0 184 Treatment services  Number of U1 children treated 85 050 2 3 0 76  Treatment U1: malaria 85 050 0.94 1.51 0 34  Treatment U1: diarrhoea 85 050 0.79 1.29 0 28  Treatment U1: pneumonia 85 050 0.42 0.87 0 26  Number of U5 children treated 85 050 9 11 0 306  Treatment U5: malaria 85 050 4 5 0 155  Treatment U5: diarrhoea 85 050 3 4 0 151  Treatment U5: pneumonia 85 050 2 3 0 100  Malarial all ages 85 050 5 6 0 151 CHW performance indicators . Obs . Mean . SD . Min . Max . Household visits  Number of months the CHW was active (out of 21) 85 050 13 3.81 0 21  Required follow-ups 85 050 3 5.70 0 128  On-time follow-ups made 85 050 2 4.39 0 105  Number of families registered 85 050 5 11.62 0 210  Family surveys 85 050 22 36.78 0 458 Assessment  Number of pregnancies registered 85 050 2 2 0 103  Total health prenatal care visits 85 050 0.56 1.29 0 55  All first prenatal care visit 85 050 0.61 1.37 0 55  Assessed any patient 85 050 11 14 0 257  Number of U1 children assessed 85 050 2 3 0 80  Number of U5 children assessed 85 050 9 12 0 184 Treatment services  Number of U1 children treated 85 050 2 3 0 76  Treatment U1: malaria 85 050 0.94 1.51 0 34  Treatment U1: diarrhoea 85 050 0.79 1.29 0 28  Treatment U1: pneumonia 85 050 0.42 0.87 0 26  Number of U5 children treated 85 050 9 11 0 306  Treatment U5: malaria 85 050 4 5 0 155  Treatment U5: diarrhoea 85 050 3 4 0 151  Treatment U5: pneumonia 85 050 2 3 0 100  Malarial all ages 85 050 5 6 0 151 Open in new tab Table 1 Descriptive statistics of major performance indicators CHW performance indicators . Obs . Mean . SD . Min . Max . Household visits  Number of months the CHW was active (out of 21) 85 050 13 3.81 0 21  Required follow-ups 85 050 3 5.70 0 128  On-time follow-ups made 85 050 2 4.39 0 105  Number of families registered 85 050 5 11.62 0 210  Family surveys 85 050 22 36.78 0 458 Assessment  Number of pregnancies registered 85 050 2 2 0 103  Total health prenatal care visits 85 050 0.56 1.29 0 55  All first prenatal care visit 85 050 0.61 1.37 0 55  Assessed any patient 85 050 11 14 0 257  Number of U1 children assessed 85 050 2 3 0 80  Number of U5 children assessed 85 050 9 12 0 184 Treatment services  Number of U1 children treated 85 050 2 3 0 76  Treatment U1: malaria 85 050 0.94 1.51 0 34  Treatment U1: diarrhoea 85 050 0.79 1.29 0 28  Treatment U1: pneumonia 85 050 0.42 0.87 0 26  Number of U5 children treated 85 050 9 11 0 306  Treatment U5: malaria 85 050 4 5 0 155  Treatment U5: diarrhoea 85 050 3 4 0 151  Treatment U5: pneumonia 85 050 2 3 0 100  Malarial all ages 85 050 5 6 0 151 CHW performance indicators . Obs . Mean . SD . Min . Max . Household visits  Number of months the CHW was active (out of 21) 85 050 13 3.81 0 21  Required follow-ups 85 050 3 5.70 0 128  On-time follow-ups made 85 050 2 4.39 0 105  Number of families registered 85 050 5 11.62 0 210  Family surveys 85 050 22 36.78 0 458 Assessment  Number of pregnancies registered 85 050 2 2 0 103  Total health prenatal care visits 85 050 0.56 1.29 0 55  All first prenatal care visit 85 050 0.61 1.37 0 55  Assessed any patient 85 050 11 14 0 257  Number of U1 children assessed 85 050 2 3 0 80  Number of U5 children assessed 85 050 9 12 0 184 Treatment services  Number of U1 children treated 85 050 2 3 0 76  Treatment U1: malaria 85 050 0.94 1.51 0 34  Treatment U1: diarrhoea 85 050 0.79 1.29 0 28  Treatment U1: pneumonia 85 050 0.42 0.87 0 26  Number of U5 children treated 85 050 9 11 0 306  Treatment U5: malaria 85 050 4 5 0 155  Treatment U5: diarrhoea 85 050 3 4 0 151  Treatment U5: pneumonia 85 050 2 3 0 100  Malarial all ages 85 050 5 6 0 151 Open in new tab Most of the performance indicators reported in Table 1 have a wide variation with high standard deviations. While there is a chance that CHWs over-reported their performance in some cases, the unusual outperformance in some months can also be explained by the recruitment of new CHWs in those months. In their first few months, newly recruited CHWs usually survey all households in their catchment areas and identify all cases of pregnancy, infant children and other patients. Though it is more likely that the unusual outperformance in some months is due to the high efforts that CHWs have to assert at the beginning of their job, we cannot claim this with certainty, as we do not have records of when a CHW drops out and get replaced by a new CHW. The 0 minimum values in the indicators suggest that there were CHWs who either dropped out or were inactive (did not work) in a particular month. Since being a CHW is not a regular salaried job and their activities are not directly monitored in the field, it is not uncommon for CHWs to become inactive in some months. Branch office supervisors are meant to terminate CHWs who remain inactive for 2 months, but they prefer to avoid such termination because it is often difficult to find new eligible candidates from their communities. Table 2 gives a timeline of the intervention and data availability. Though the award was launched in December 2015, we could obtain the performance data only for months starting from October 2016. In October 2016, BRAC integrated mobile phone technology in its health programme, and CHWs were provided with smartphones and were required to report their activities through a mobile application. The monthly performance data for each CHW was then generated by aggregating these day-to-day activity reports. Prior to October 2016, reports were generated manually at the end of month when CHWs came to branch offices to receive refresher training. For this study, we did not have the resources to digitize pre-2016 handwritten monthly data for the 4050. Considering these challenges in our data, we designed our empirical strategy with multilevel mixed-effect models that take the hierarchical structure of the data into consideration and control for unobserved heterogeneity at different levels (see Methods and empirical models section for detail). Table 2 Timeline of intervention and data availability 2015 . . 2016 . 2017 . 2018 . November to December . January to September . October . November to December . January to October . November to December . January to June . 46 awards provided Mobile reporting launched 43 awards provided 46 awards provided 2015 . . 2016 . 2017 . 2018 . November to December . January to September . October . November to December . January to October . November to December . January to June . 46 awards provided Mobile reporting launched 43 awards provided 46 awards provided Note: Monthly performance data were not available in 2015 or from January to September in 2016 (shown in grey). Open in new tab Table 2 Timeline of intervention and data availability 2015 . . 2016 . 2017 . 2018 . November to December . January to September . October . November to December . January to October . November to December . January to June . 46 awards provided Mobile reporting launched 43 awards provided 46 awards provided 2015 . . 2016 . 2017 . 2018 . November to December . January to September . October . November to December . January to October . November to December . January to June . 46 awards provided Mobile reporting launched 43 awards provided 46 awards provided Note: Monthly performance data were not available in 2015 or from January to September in 2016 (shown in grey). Open in new tab Despite the above-mentioned shortcomings in our data, there are several reasons that make BRAC’s CHW programme a good setting to examine the relationship between social incentives and CHW’s performance. First, the catchment area of each CHW is well defined and does not overlap with that of other CHWs, which controls for any task complementarities among them at the field level. Next, the remuneration package is identical for all CHWs regardless of their experience and educational qualifications, controlling for any externalities arising from remuneration. As the CHW job is voluntary in nature and day-to-day CHW activities are not monitored by branch managers, the externalities arising from supervision quality is expected to be limited. Outcome variable: performance index Of the 21 performance indicators, selecting one or a few as the primary outcomes of interest would ignore the multi-dimensionality of CHW performance. Measuring the effect of the awards on some particular performance indicators would make sense if the nominees were selected based on clearly defined measurable indicators (known as confirmatory rewards); however, the selection of the award recipients was made following a subjective process. Considering these challenges, we decided to generate a performance index value for each CHW and use the index as the primary outcome of interest. The idea comes from the latent variable approach, which assumes that different dimensions of CHW performance cannot be directly observed but can be represented by an index value that partially explains the variation in the observed performance indicators (Zeller et al., 2006; Williams et al., 2010). A latent variable approach is used widely in development economics to measure household welfare (e.g. poverty indices) (see Zeller et al., 2006; Morel and Chowdhury, 2015). The two most widely used models of latent variable approach are the principal component analysis and factor analysis (FA) (Zeller et al., 2006; Krishnakumar and Nagar, 2008; Williams et al., 2010). In this paper, we used FA to construct the performance index value. To improve the explanatory power of the performance index, some indicators have been filtered out from the FA. First, all but one indicator representing a particular aspect have been dropped because having many variables of a particular aspect would bias the index. For example, the variables U-1 children treated for malaria, U-1 children treated for diarrhoea and U-1 children treated for pneumonia were dropped from the analysis because these are closely correlated with the variable number of U-1 children treated. Second, following the standard procedure in the literature, the selected variables have been log transformed before being used in the FA. Therefore, any variable presenting the ratio of two other variables have been excluded. Third, dichotomous response variables such as whether organized any community event have not been included in the FA model, as the model requires continuous data. The final FA model includes a total of 12 variables covering different aspects of CHW performance. We used maximum likelihood estimation to extract the number of latent factors from the data. Supplementary Table S2 in the Supplementary data shows the indicators selected in the final model and their respective factor loadings. Factor loadings show the degree and sign of the correlation between the factors and indicators. The coefficients or loadings in Factor 1 appear with expected signs in relation to CHW performance and are treated as the relative performance index. Factor 2 does not seem to consistently capture variance related to performance, as some variable loadings have unexpected signs and all but two of them have insignificant values. Using the loadings of Factor 1, we estimated the factor scores, or performance index, for each observation of monthly CHW performance. Methods and empirical models Given the non-ignorable clustering and lack of independence across our observations, we recognized the need for more complex empirical models that would take the hierarchical structure of the data into consideration and control for unobserved heterogeneity at different cluster levels. Multilevel mixed-effects models—also known as hierarchical linear models and random-effects models—offered solutions to these problems (Snijders and Bosker, 2011; Gelman, 2006; Bell and Jones, 2015). We used a multilevel mixed-effect model to estimate the influence of social recognition on CHW performance. In the absence of data on CHWs and branch-level characteristics, multilevel mixed-effect method allowed us to explicitly model externalities arising from unobserved variables at the branch and CHW levels. By explicitly modelling for such externalities, the multilevel mixed-effect method separated the effect of unobserved externalities from the estimated coefficients of interest (Steele, 2008; Snijders and Bosker, 2011; Mathieu et al., 2012; Aguinis et al., 2013). Multilevel modelling also allowed us to investigate the nature of between-cluster variability (Gelman, 2006; Gelman and Hill, 2006; Bell and Jones, 2015), or—whether monthly performance observations varied across branch offices and CHWs and whether the variation between branch offices differed by quality of CHW. We began by examining how performance of a CHW changed if someone from her branch won the award in the previous year. We fit a three-level hierarchical model where the monthly performance scores (level 1) are nested within CHWs (level 2) and CHWs are nested within branch offices (level 3). In our fitted model, we allowed the intercept to vary at both branch and CHW levels and let the coefficient of our variable of interest vary across branch level only. This was the best-performing model found from an exercise where several models were fitted letting the intercept and slope vary at either or both of the CHW and branch levels. Our preferred estimated model took the following form: $${Y_{tij}} = {\gamma _{000}} + {\text{}}{\beta _1}Branch{\,\,\text{}}Siz{e_j} + {\,\,\text{}}{\beta _2}\,\,Awarded{\,\,\text{}}Last{\,\,\text{}}Yea{r_j} + {v_j}{\,\,\text{}}Awarded{\,\,\text{}}Last{\,\,\text{}}Yea{r_j} + {v_j} + {u_{ij}} + {\,\,\text{}}{e_{tij{\,\,\text{}},}}$$(1) where: |${Y_{t,i,j}} = {\text{Performance index score of a CHW}}\,i\,{\text{from branch}}\,j\,{\text{in month}}\,t;$| |${\gamma _{000}} = {\text{Grand mean of performance scores across all months}},{\mkern 1mu} {\mkern 1mu} {\text{CHWs}},{\text{and branches}};$| |$Branch\,Siz{e_j} = {\text{Number of CHWs serving in the same branch}}{\mkern 1mu} j;$| |$Awarded\,Last\,Yea{r_j} = I{\text{f any CHW from the same branch received an award in the last year}};$| |${v_j} = {\text{Residual coefficient for performance score for branch}}\,j;$| |${u_{ij}} = {\text{Residual coefficient for performance score for CHW}}\,i\,{\text{in branch}}\,j;$| |${e_{tij}} = {\text{Residual coefficient for performance score for month t of CHW}}\,i\,{\text{in branch}}\,j$| We took the performance index score estimated through FA as the outcome variable. In estimating this model, we excluded the observations of the award-winning CHWs. We also made an assumption that giving a social reward to a CHW affects the performance of her colleagues in the immediate next year only. Using the same multilevel modelling approach, we examined whether the effect of social recognition varied by CHW performance quality. As a measure of the performance quality, we took the percentile rank of CHWs based on their relative position in the distribution of the previous year’s average performance index scores. While constructing the quality measures based on the previous year’s performance scores instead of the current year’s performance scores implies losing observations and reducing power, this would make the possible endogeneity problem less severe given that the relative performance is also an outcome in our model. The estimated model was: $$\begin{gathered}{Y_{tij}} = {\gamma _{000}} + {\beta _1}Branch\,Siz{e_j} + {\beta _2}Awarded\,Last\,Yea{r_j} + {\beta _3}\,Percentil{e_{i,j}} + {\beta _4}\,Percentil{e_{ij}} \\ \times Awarded\,Last\,Yea{r_j} + {v_j}\,Percentil{e_{ij}} \\ \times Awarded\,Last\,Yea{r_j} + {v_j} + {u_{ij}} + {e_{tij}}\end{gathered}$$(2) In Model 2, the percentileij is a dummy indicating the percentile bracket (top 50% i.e. 50–100th percentile, bottom 50% and bottom 20%) to which a CHW belongs based on the distribution of the average performance indicator values in the previous year. The coefficient of the interaction between the nth percentile rank of a CHW and the presence of an award-winning colleague (β4) is the coefficient of interest. The β4 measures how the performance of a CHW from the nth percentile changes if a colleague received the award in the previous year. Finally, in the later part of our analysis, we used a more restricted sample, focusing on only the non-winner branches and examining how CHW performance from those branches changed if someone from their neighbouring three branches won the award. Restricting our focus on non-winner branches allowed us to control for externalities arising from the quality of supervision and training that might make the winner branches systematically different from the non-winner branches. Results Effect on immediate colleagues in next year Table 3 presents the results estimated from the model specified in Equation (1). The estimate shows a non-significant negative association between the social award and CHW performance: having an immediate colleague awarded in the previous year is associated with a reduction of 4.9 percentage points on the performance index. The estimate also shows that branch size, which indicates the number of CHWs in the same branch, had a positive but statistically insignificant association with the performance scores. The likelihood ratio (LR) test comparing this model to one without clustering was highly significant supporting the choice of the fitted model. An important takeaway from the results is that the variance of the random effect (slope) of the Awarded Last Year variable alone accounts for a 39% variance in performance score.1 This further confirms our model’s assumption that the effect of having an award-winning colleague on performance of other CHWs varies significantly across branch offices. Table 3 Effects on CHW peers from the same branch office . Random intercept and slope at branch level and random intercept at CHW level . (1) . Treatment: award-winning colleague −0.049 [−0.225 0.126] (0.089) Branch size 0.001 [−0.004 0.006] (0.003) Constant −0.089 [−0.258 0.079] (0.086) ICC  Branch 0.174 [0.141 0.214]  CHWs within branch 0.259 [0.228 0.294] Additional information:  N 85 050  LR test: compared with pooled OLS (chi) 14844*** . Random intercept and slope at branch level and random intercept at CHW level . (1) . Treatment: award-winning colleague −0.049 [−0.225 0.126] (0.089) Branch size 0.001 [−0.004 0.006] (0.003) Constant −0.089 [−0.258 0.079] (0.086) ICC  Branch 0.174 [0.141 0.214]  CHWs within branch 0.259 [0.228 0.294] Additional information:  N 85 050  LR test: compared with pooled OLS (chi) 14844*** Note: Performance index score is the dependent variable. 95% confidence intervals are shown in brackets and standard errors are shown in parentheses. *** denotes statistical significance of coefficients at the 1% confidence level. Open in new tab Table 3 Effects on CHW peers from the same branch office . Random intercept and slope at branch level and random intercept at CHW level . (1) . Treatment: award-winning colleague −0.049 [−0.225 0.126] (0.089) Branch size 0.001 [−0.004 0.006] (0.003) Constant −0.089 [−0.258 0.079] (0.086) ICC  Branch 0.174 [0.141 0.214]  CHWs within branch 0.259 [0.228 0.294] Additional information:  N 85 050  LR test: compared with pooled OLS (chi) 14844*** . Random intercept and slope at branch level and random intercept at CHW level . (1) . Treatment: award-winning colleague −0.049 [−0.225 0.126] (0.089) Branch size 0.001 [−0.004 0.006] (0.003) Constant −0.089 [−0.258 0.079] (0.086) ICC  Branch 0.174 [0.141 0.214]  CHWs within branch 0.259 [0.228 0.294] Additional information:  N 85 050  LR test: compared with pooled OLS (chi) 14844*** Note: Performance index score is the dependent variable. 95% confidence intervals are shown in brackets and standard errors are shown in parentheses. *** denotes statistical significance of coefficients at the 1% confidence level. Open in new tab As mentioned earlier, we obtained this preferred model of Equation (1) from an exercise of fitting several two-level and three-level hierarchical models with intercept and slope varied at either or both of the CHW and branch levels. The results from these estimated models are presented in Supplementary Table S3 in the Supplementary data. To complement our multilevel modelling estimates, we also fitted a fixed-effect model and performed the Hausman test to see if the assumption of non-correlation between the error terms and the covariates hold or not. The results of the fixed-effect estimations are presented in Supplementary Table S4 in the Supplementary data. Results of the Hausman test shows non-correlation between the covariates and the error terms supporting the random-effect assumption of our empirical model. Effects of award by CHW quality In Table 4, we present the results estimated following the empirical model specified in Equation (2). The purpose was to investigate if the effect of the social recognition award varied across CHWs by their quality. In estimating the model, the CHW’s nth percentile variable was generated by taking the different percentile cut-offs of the average performance index score in the previous year. The variable is a dummy that takes the value of 1 if a CHW belonged to the nth percentile rank of the average performance indicator in the previous year. The percentile ranks are indicated in the column headings. For example, in Column (1) the CHW’s nth percentile variable takes a value of 1 if a CHW’s average performance in last year belonged to the top 50th percentile, otherwise it takes 0. Table 4 Effects on the peers of different quality from the same branches . Random intercept and slope at branch level and random intercept at CHW level . Top 50 percentile . Last 50 percentile . Last 20 percentile . (1) . (2) . (3) . Treatment: Award-winning colleague 0.013 [−0.014 0.039] 0.102 [0.074 0.130] 0.035 [0.013 0.056] (0.013) (0.014)*** (0.011)*** Branch size 0.001 [−0.001 0.004] 0.001 [−0.003 0.003] −0.001 [−0.003 0.002] (0.001) (0.001) (0.001) CHW’s nth percentile 0.391 [0.367 0.415] −0.369 [−0.395 −0.345] −0.569 [−0.599 −0.539] (0.012)*** (0.013)*** (0.015)*** Treatment × CHW’s nth percentile 0.052 [−0.011 0.115] −0.057 [−0.179 0.066] 0.094 [−0.051 0.239] (0.032) (0.063) (0.074) Constant −0.048 [−0.134 0.039] 0.365 [0.255 0.475] 0.339 [0.252 0.427] (0.044)*** (0.056)*** (0.045)*** ICC  Branch level 0.054 0.087 0.057 [0.042 0.069] [0.069 0.111] [0.044 0.074]  CHW within branch 0.148 0.188 0.145 [0.135 0.162] [0.169 0.209] [0.131 0.160] Additional information  N 72 900 72 900 72 900  LR test: compared with pooled OLS 6576.31*** 8073.24*** 5527.30*** . Random intercept and slope at branch level and random intercept at CHW level . Top 50 percentile . Last 50 percentile . Last 20 percentile . (1) . (2) . (3) . Treatment: Award-winning colleague 0.013 [−0.014 0.039] 0.102 [0.074 0.130] 0.035 [0.013 0.056] (0.013) (0.014)*** (0.011)*** Branch size 0.001 [−0.001 0.004] 0.001 [−0.003 0.003] −0.001 [−0.003 0.002] (0.001) (0.001) (0.001) CHW’s nth percentile 0.391 [0.367 0.415] −0.369 [−0.395 −0.345] −0.569 [−0.599 −0.539] (0.012)*** (0.013)*** (0.015)*** Treatment × CHW’s nth percentile 0.052 [−0.011 0.115] −0.057 [−0.179 0.066] 0.094 [−0.051 0.239] (0.032) (0.063) (0.074) Constant −0.048 [−0.134 0.039] 0.365 [0.255 0.475] 0.339 [0.252 0.427] (0.044)*** (0.056)*** (0.045)*** ICC  Branch level 0.054 0.087 0.057 [0.042 0.069] [0.069 0.111] [0.044 0.074]  CHW within branch 0.148 0.188 0.145 [0.135 0.162] [0.169 0.209] [0.131 0.160] Additional information  N 72 900 72 900 72 900  LR test: compared with pooled OLS 6576.31*** 8073.24*** 5527.30*** Note: Performance index score is the dependent variable. 95% confidence intervals are shown in brackets and standard errors are shown in parentheses. *** denotes statistical significance of coefficients at the 1% confidence level. Open in new tab Table 4 Effects on the peers of different quality from the same branches . Random intercept and slope at branch level and random intercept at CHW level . Top 50 percentile . Last 50 percentile . Last 20 percentile . (1) . (2) . (3) . Treatment: Award-winning colleague 0.013 [−0.014 0.039] 0.102 [0.074 0.130] 0.035 [0.013 0.056] (0.013) (0.014)*** (0.011)*** Branch size 0.001 [−0.001 0.004] 0.001 [−0.003 0.003] −0.001 [−0.003 0.002] (0.001) (0.001) (0.001) CHW’s nth percentile 0.391 [0.367 0.415] −0.369 [−0.395 −0.345] −0.569 [−0.599 −0.539] (0.012)*** (0.013)*** (0.015)*** Treatment × CHW’s nth percentile 0.052 [−0.011 0.115] −0.057 [−0.179 0.066] 0.094 [−0.051 0.239] (0.032) (0.063) (0.074) Constant −0.048 [−0.134 0.039] 0.365 [0.255 0.475] 0.339 [0.252 0.427] (0.044)*** (0.056)*** (0.045)*** ICC  Branch level 0.054 0.087 0.057 [0.042 0.069] [0.069 0.111] [0.044 0.074]  CHW within branch 0.148 0.188 0.145 [0.135 0.162] [0.169 0.209] [0.131 0.160] Additional information  N 72 900 72 900 72 900  LR test: compared with pooled OLS 6576.31*** 8073.24*** 5527.30*** . Random intercept and slope at branch level and random intercept at CHW level . Top 50 percentile . Last 50 percentile . Last 20 percentile . (1) . (2) . (3) . Treatment: Award-winning colleague 0.013 [−0.014 0.039] 0.102 [0.074 0.130] 0.035 [0.013 0.056] (0.013) (0.014)*** (0.011)*** Branch size 0.001 [−0.001 0.004] 0.001 [−0.003 0.003] −0.001 [−0.003 0.002] (0.001) (0.001) (0.001) CHW’s nth percentile 0.391 [0.367 0.415] −0.369 [−0.395 −0.345] −0.569 [−0.599 −0.539] (0.012)*** (0.013)*** (0.015)*** Treatment × CHW’s nth percentile 0.052 [−0.011 0.115] −0.057 [−0.179 0.066] 0.094 [−0.051 0.239] (0.032) (0.063) (0.074) Constant −0.048 [−0.134 0.039] 0.365 [0.255 0.475] 0.339 [0.252 0.427] (0.044)*** (0.056)*** (0.045)*** ICC  Branch level 0.054 0.087 0.057 [0.042 0.069] [0.069 0.111] [0.044 0.074]  CHW within branch 0.148 0.188 0.145 [0.135 0.162] [0.169 0.209] [0.131 0.160] Additional information  N 72 900 72 900 72 900  LR test: compared with pooled OLS 6576.31*** 8073.24*** 5527.30*** Note: Performance index score is the dependent variable. 95% confidence intervals are shown in brackets and standard errors are shown in parentheses. *** denotes statistical significance of coefficients at the 1% confidence level. Open in new tab Table 4 shows that the interaction term between the Awarded Last Year variable and the nth percentile rank (top 50th, bottom 50th and bottom 20th) of the performance indicator was found to be trivial in all of the estimated models, suggesting there were no important differences between the subgroups. However, this relationship is quite heterogenous across the CHW and branch levels, as suggested by the random-effect parameters, variances of the intercepts and coefficient slope, and the corresponding likelihood ratio test statistics of the models. Effects on CHW peers in the neighbouring branches In Tables 3 and 4, we presented findings from models focusing on immediate peers and examined how their performances changed in the presence of an award-winning CHW in the same branch. These models did not control for any correlated shocks that CHWs might have faced within a branch. Correlated shocks might have come from having the same supervisor at the branch level, receiving training from the same trainer, or serving in similar types of communities. This means that performance index scores of CHWs from award-winning branch offices might have been systematically different from that of non-winning branches. Furthermore, the inclusion of observations from winning branches in the sample while measuring the spill-over effect of the social recognition award may have diluted this effect. To control for this threat, we re-estimated the models excluding the observations from the branches having at least one CHW who received the award in the previous year. We redefined our treatment variable of Awarded Last Yearj as an indicator variable that takes a value of 1 if any CHW within the nearest three neighbouring branches received the award in the previous year. For example, for any of the months in year 2017, this dummy variable took the value of 1 if and only if any CHW within the nearest three branches received the award in December 2016. As with the previous model, we assumed that the social recognition award affected the performance of CHWs only in the preceding year. The results are presented in Table 5. Estimates revealed a significant negative association between the variable Awarded Last Year and the monthly performance scores of CHWs. Presence of an award-winning CHW among the nearest three neighbouring branches was associated with a 27 percentage points decrease in the average performance score of a non-winner branch, which is significant at the 1% level. The likelihood ratio test statistics (χ2 = 4258.47 and P < 0000) comparing this model with the ordinary least squares (OLS) model showed that the current model substantially better fit the data. Table 5 Effects on the peers from the neighbouring branches . Random intercept and slope at branch level and random intercept at CHW level . (1) . Treatment: a winner within the nearest three branches −0.270 [−0.451 −0.089] (0.092)*** Branch size −0.004 [−0.012 0.003] (0.004) Constant 0.177 [−0.059 0.413] (0.121) ICC  Branch level 0.137 [0.092 0.199]  CHW within branch 0.216 [0.171 0.269] Additional information  N 26 922  LR test: compared with pooled OLS 4258.47*** . Random intercept and slope at branch level and random intercept at CHW level . (1) . Treatment: a winner within the nearest three branches −0.270 [−0.451 −0.089] (0.092)*** Branch size −0.004 [−0.012 0.003] (0.004) Constant 0.177 [−0.059 0.413] (0.121) ICC  Branch level 0.137 [0.092 0.199]  CHW within branch 0.216 [0.171 0.269] Additional information  N 26 922  LR test: compared with pooled OLS 4258.47*** Note: Performance index score is the dependent variable. 95% confidence intervals are shown in brackets and standard errors are shown in parentheses. *** denotes statistical significance of coefficients at the 1% confidence level. Open in new tab Table 5 Effects on the peers from the neighbouring branches . Random intercept and slope at branch level and random intercept at CHW level . (1) . Treatment: a winner within the nearest three branches −0.270 [−0.451 −0.089] (0.092)*** Branch size −0.004 [−0.012 0.003] (0.004) Constant 0.177 [−0.059 0.413] (0.121) ICC  Branch level 0.137 [0.092 0.199]  CHW within branch 0.216 [0.171 0.269] Additional information  N 26 922  LR test: compared with pooled OLS 4258.47*** . Random intercept and slope at branch level and random intercept at CHW level . (1) . Treatment: a winner within the nearest three branches −0.270 [−0.451 −0.089] (0.092)*** Branch size −0.004 [−0.012 0.003] (0.004) Constant 0.177 [−0.059 0.413] (0.121) ICC  Branch level 0.137 [0.092 0.199]  CHW within branch 0.216 [0.171 0.269] Additional information  N 26 922  LR test: compared with pooled OLS 4258.47*** Note: Performance index score is the dependent variable. 95% confidence intervals are shown in brackets and standard errors are shown in parentheses. *** denotes statistical significance of coefficients at the 1% confidence level. Open in new tab Table 6 follows models similar to those in Table 4. The results show significant heterogeneity across the three groups of CHWs. The top 50th percentile group showed underperformance by 20 percentage points (P < 0.01) while the bottom half and bottom 20th percentile groups showed outperformance by 13 percentage points (P < 0.05) and 17 percentage points (P < 0.05), respectively. The estimates implied that CHWs from the top 50th percentile group underperformed by 20 percentage points when a CHW from any of the nearest neighbouring three branches received the award. On the other hand, CHWs from the bottom half showed an outperformance in response to the intervention. For the bottom 20th percentile group, the magnitude of outperformance was even larger. Table 6 Effects on the peers of different quality from the neighbouring branches . Random intercept and slope at branch level and random intercept at CHW level . Top 50 percentile . Last 50 percentile . Last 20 percentile . (1) . (2) . (3) . Treatment: a winner within the nearest three branches 0.274 [0.231 0.317] 0.008 [−0.033 0.049] 0.090 [0.059 0.122] (0.022) *** (0.021)*** (0.016)*** Branch size 0.002 [−0.002 0.006] 0.004 [−0.001 0.009] 0.000 [−0.004 0.004] (0.002) (0.003) (0.002) CHW’s nth percentile 0.466 [0.415 0.517] −0.382 [−0.434 −0.331] −0.612 [−0.666 −0.558] (0.026)*** (0.026)*** (0.028)*** Treatment × CHW’s nth percentile −0.199 [−0.291 −0.107] 0.131 [−0.011 0.273] 0.165 [0.007 0.323] (0.047)*** (0.073)** (0.081)** Cons −0.227 [−0.363 −0.091] 0.219 [0.049 0.391] 0.239 [0.113 0.367] (0.069)*** (0.087)*** (0.065)*** ICC  Branch level 0.054 [0.034 0.082] 0.082 [0.053 0.124] 0.048 [0.031 0.074]  CHW within branch 0.142 [0.119 0.167] 0.172 [0.142 0.208] 0.120 [0.099 0.144] Additional information  N 23 076 23 076 23 076  LR test: compared with pooled OLS 1934.38*** 2225.05*** 945.94*** . Random intercept and slope at branch level and random intercept at CHW level . Top 50 percentile . Last 50 percentile . Last 20 percentile . (1) . (2) . (3) . Treatment: a winner within the nearest three branches 0.274 [0.231 0.317] 0.008 [−0.033 0.049] 0.090 [0.059 0.122] (0.022) *** (0.021)*** (0.016)*** Branch size 0.002 [−0.002 0.006] 0.004 [−0.001 0.009] 0.000 [−0.004 0.004] (0.002) (0.003) (0.002) CHW’s nth percentile 0.466 [0.415 0.517] −0.382 [−0.434 −0.331] −0.612 [−0.666 −0.558] (0.026)*** (0.026)*** (0.028)*** Treatment × CHW’s nth percentile −0.199 [−0.291 −0.107] 0.131 [−0.011 0.273] 0.165 [0.007 0.323] (0.047)*** (0.073)** (0.081)** Cons −0.227 [−0.363 −0.091] 0.219 [0.049 0.391] 0.239 [0.113 0.367] (0.069)*** (0.087)*** (0.065)*** ICC  Branch level 0.054 [0.034 0.082] 0.082 [0.053 0.124] 0.048 [0.031 0.074]  CHW within branch 0.142 [0.119 0.167] 0.172 [0.142 0.208] 0.120 [0.099 0.144] Additional information  N 23 076 23 076 23 076  LR test: compared with pooled OLS 1934.38*** 2225.05*** 945.94*** Note: Performance index score is the dependent variable. 95% confidence intervals are shown in brackets and standard errors are shown in parentheses. ** and *** denote statistical significance of coefficients at the 5% and 1%, respectively, confidence level. Open in new tab Table 6 Effects on the peers of different quality from the neighbouring branches . Random intercept and slope at branch level and random intercept at CHW level . Top 50 percentile . Last 50 percentile . Last 20 percentile . (1) . (2) . (3) . Treatment: a winner within the nearest three branches 0.274 [0.231 0.317] 0.008 [−0.033 0.049] 0.090 [0.059 0.122] (0.022) *** (0.021)*** (0.016)*** Branch size 0.002 [−0.002 0.006] 0.004 [−0.001 0.009] 0.000 [−0.004 0.004] (0.002) (0.003) (0.002) CHW’s nth percentile 0.466 [0.415 0.517] −0.382 [−0.434 −0.331] −0.612 [−0.666 −0.558] (0.026)*** (0.026)*** (0.028)*** Treatment × CHW’s nth percentile −0.199 [−0.291 −0.107] 0.131 [−0.011 0.273] 0.165 [0.007 0.323] (0.047)*** (0.073)** (0.081)** Cons −0.227 [−0.363 −0.091] 0.219 [0.049 0.391] 0.239 [0.113 0.367] (0.069)*** (0.087)*** (0.065)*** ICC  Branch level 0.054 [0.034 0.082] 0.082 [0.053 0.124] 0.048 [0.031 0.074]  CHW within branch 0.142 [0.119 0.167] 0.172 [0.142 0.208] 0.120 [0.099 0.144] Additional information  N 23 076 23 076 23 076  LR test: compared with pooled OLS 1934.38*** 2225.05*** 945.94*** . Random intercept and slope at branch level and random intercept at CHW level . Top 50 percentile . Last 50 percentile . Last 20 percentile . (1) . (2) . (3) . Treatment: a winner within the nearest three branches 0.274 [0.231 0.317] 0.008 [−0.033 0.049] 0.090 [0.059 0.122] (0.022) *** (0.021)*** (0.016)*** Branch size 0.002 [−0.002 0.006] 0.004 [−0.001 0.009] 0.000 [−0.004 0.004] (0.002) (0.003) (0.002) CHW’s nth percentile 0.466 [0.415 0.517] −0.382 [−0.434 −0.331] −0.612 [−0.666 −0.558] (0.026)*** (0.026)*** (0.028)*** Treatment × CHW’s nth percentile −0.199 [−0.291 −0.107] 0.131 [−0.011 0.273] 0.165 [0.007 0.323] (0.047)*** (0.073)** (0.081)** Cons −0.227 [−0.363 −0.091] 0.219 [0.049 0.391] 0.239 [0.113 0.367] (0.069)*** (0.087)*** (0.065)*** ICC  Branch level 0.054 [0.034 0.082] 0.082 [0.053 0.124] 0.048 [0.031 0.074]  CHW within branch 0.142 [0.119 0.167] 0.172 [0.142 0.208] 0.120 [0.099 0.144] Additional information  N 23 076 23 076 23 076  LR test: compared with pooled OLS 1934.38*** 2225.05*** 945.94*** Note: Performance index score is the dependent variable. 95% confidence intervals are shown in brackets and standard errors are shown in parentheses. ** and *** denote statistical significance of coefficients at the 5% and 1%, respectively, confidence level. Open in new tab Discussion In this paper, we examined the relationship between social rewards and CHW performance using data from 4050 CHWs organized within 134 branch offices of BRAC Uganda, a non-governmental organization. In 2015, BRAC introduced an annual competitive award for the best-performing CHWs. Between 2015 and 2018, a total of 135 CHWs from 88 branch offices won the award. We designed an empirical strategy with multilevel mixed-effect models that allowed us to control for unobserved heterogeneity at the branch and CHW levels. Our analysis began by looking at how CHWs reacted when a colleague from the same branch received the award in the previous year. The results showed a negative but statistically non-significant association between the award and CHW performance. However, the estimations showed that CHW-level and branch-level factors accounted for a substantial amount of variation in performance scores. The variation in performances across branch offices can be explained by differences among branch offices in terms of geographical location, quality of field supervisors and other characteristics, such as population size, infrastructure and the presence of health facilities in the community. The variation in performances across CHWs can be attributed to differences in individual factors, such as age, education, years of experience, marital status and number of children. We also found that the spill-over effect of the award significantly varied among branches but not among CHWs. Our estimates from a restricted sample controlling for correlated shocks at branch level found mixed effects. We found a significant negative association between CHW performance and the awards: having an award-winning CHW among the nearest three neighbouring branches was associated with a 27 percentage points decrease in the average performance score of a non-winner branch. One possible source of this negative association might be the design of the award itself. The nominations for the award were based on qualitative dimensions which gave the branch supervisors a high degree of freedom to decide upon whom the awards are bestowed. Gallus and Frey (2017) defined such awards as discretionary awards. While discretionary awards allow the supervisors to signal their intent and desired quality, such awards are also sensitive to issues of lack of objectivity. Discretionary awards can be perceived as a signal of favouritism and can deteriorate the reputation of both the supervisors and the award winners if the supervisors are not seen to invest enough time and effort in the selection of the candidates and the winners (Gallus and Frey, 2017). The discretionary nature of the awards might lead CHWs to believe that those selected by the branch supervisors were selected because of personal connections with the supervisors. Therefore, the awards might have been considered unfair, inducing a negative spill-over on CHW performance. Similarly, a review of literature on factors affecting CHW performance found that recognition by the organization improves CHW performance when the recognition is believed to be based on fairness and equity (Vareilles et al., 2017). Another source of this negative effect might come from the non-recipients perceiving the award as a signal of them not being meritorious—a phenomenon referred to as ‘social comparison cost’ in the literature (Harlow and Cantor, 1995; Exline and Lobel, 1997; Exline et al., 2004; Clark et al., 2010). Giving an award to a selected group of employees involves the risk of offending the non-recipients, particularly in small and homogenous group of employees where interpersonal comparisons are dominant (Frey, 2007; Gallus and Frey, 2017). Non-recipients often respond to such a workplace award with feelings of dejection or inferiority (Salovey and Rodin, 1984; Exline et al., 2004). This negative emotional response by the non-recipients may result in reduced efforts, increased jealousy and even sabotage (Charness et al., 2014; Gallus and Frey, 2017). The rarity of the award might also have induced the overall negative performance (only 3% of CHWs received the award). The low frequency of the award might have made the CHWs feel they had a low chance of winning it, which might have demotivated them and reduced their performance (Carnahan et al., 2010). A logical extrapolation of this point implies that high-performing workers have a higher chance of winning the award and are more likely to be positively affected by the award compared with their low-performing colleagues. This hypothesis is also suggested by several studies arguing that high performers respond positively to social awards by identifying themselves with the award winners (Ybema and Buunk, 1995; Collins, 1996; Hoyt, 2013) or seeing the winners as a source of inspiration (Ybema and Buunk, 1995; Exline and Lobel, 1997). However, our findings suggest the opposite. Our multilevel modelling estimates on the CHWs from the non-winning branches found that the presence of an award-winning CHW in the neighbouring three branches was correlated with a significant drop in performance scores of the top 50th percentile of CHWs. On the other hand, the bottom 50th percentile CHWs exhibited outperformance. Such negative response from the high performers can be explained by their frustration of not winning the award despite their higher chances of winning, or emotions such as envy, and jealousy as suggested in several studies (e.g. Salovey and Rodin, 1984; Pelham and Wachsmuth, 1995; Carnahan et al., 2010; Charness et al., 2014; Gallus and Frey, 2017). Besides the factors related to award design, and individual differences between CHWs, the effectiveness of an award also depends on organizational culture. Literatures in organization studies and labour psychology suggest that compensation and reward systems need to be congruent with management systems and the culture of the organization to have a positive effect on workers’ performance (Morgenstern, 1995; Lundby et al., 1999). For example, a competitive reward system might backfire in organizations that promote a culture of participation and teamwork within the workplace, since such reward systems contradict the espoused values of the organizations (Cameron, 1985). Competitive reward systems would be more congruent with organizational cultures where competitiveness and goal achievement are valued most. BRAC, like most other non-profit organizations, might have promoted a culture of cohesiveness and teamwork with the organization. Introducing a competitive award system, therefore, might have resulted in rejection and low performance from its high-performing CHWs. Overall, our results contrast with some of the existing literature on upward social comparison (e.g. Ybema and Buunk, 1995; Collins, 1996; Exline and Lobel, 1997) that suggests that high-performing workers are more likely to react positively when other high performers are awarded and that low-performing workers are more likely to react by reducing their performance because of a perception that the chance of getting the award is low. Our results are more in line with Salovey and Rodin (1984) where empirical evidence found that individuals experience social comparison envy or jealousy when they compare themselves with similar and successful rivals. This also follows Ager et al. (2016) where evidence was found that in World War II, low-skilled pilots in the German air force showed significant outperformance when one of their fellow pilots was mentioned on the national radio for his outstanding performance. Conclusion Our study focused on CHWs in Uganda. Although the findings may help shed light on important issues of motivating CHWs in other low-income countries, each country will have its own unique history and policies related to CHWs. Findings from Uganda should be interpreted in light of any country-level differences that may vary between low-income countries. Non-random allocation of the awards to branch offices and CHWs is a major shortcoming of our study, which restricts our ability to claim any causal relation between social award and CHW performance. However, our study made use of a unique and large sample of 4050 CHWs with 21 months of observational data. This study, within its limitations, offers two important contributions to the existing literature. First, we contribute to the scarce literature on the effects of non-financial incentives to keep CHWs motivated in low-income settings. Second, we contribute to the literature on the use of awards and the potential for upward social comparisons in the workplace. Our results suggest that best-performer awards can be used as an effective social incentive in boosting up the performance of low-performing CHWs in low-income countries. We also propose that the overall effectiveness of awards can be improved by investing more efforts in, and increasing the transparency of, the nominee selection process. Supplementary data Supplementary data are available at Health Policy and Planning online. Conflict of interest statement. Reajul Chowdhury worked for BRAC South Sudan from 2012 to 2016, before the start of the current research. Kevin McKague and Heather Krause declare that they have no conflict of interest.. Ethical approval. We received IRB approval from two institutions: The Uganda National Council for Science and Technology (number ARC 186) and the Research Ethics Board of Cape Breton University, Canada (number 1718067). Endnotes 1 |$\frac{{\sigma _{v1}^2}}{{\sigma _v^2 + {\rm{}}\sigma _{v1}^2 + \sigma _u^2 + {\rm{}}\sigma _e^2}} = \frac{{0.686}}{{0.184 + 0.686 + 0.091 + 0.782}} = 0.39$| Funding This work was carried out with the aid of a grant from the Innovating for Maternal and Child Health in Africa initiative—a partnership of Global Affairs Canada (GAC), the Canadian Institutes of Health Research (CIHR) and Canada’s International Development Research Centre (IDRC), grant number 108033. Any opinions expressed here are those of the authors and not those of any of the donor organizations. Acknowledgements We are grateful to Munshi Sulaiman, Regional Research Lead in Africa at BRAC International; Patrick Olobo Okello, Research Fellow at BRAC Uganda; and Dr Jenipher Twebaze Musoke, Research Coordinator at BRAC International, for their assistance. References Ager P , Bursztyn L, Voth HJ. 2016 . Killer Incentives: Status Competition and Pilot Performance during World War II. Working Paper w22992 . Cambridge, MA : National Bureau of Economic Research . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Aguinis H , Gottfredson RK, Culpepper SA. 2013 . Best-practice recommendations for estimating cross-level interaction effects using multilevel modeling . Journal of Management 39 : 1490 – 528 . Google Scholar OpenURL Placeholder Text WorldCat Aitken I. 2014 . Training Community Health Workers for Large-Scale Community-Based Health Care Programs in Developing and Strengthening Community Health Worker Programs at Scale: A Reference Guide and Case Studies for Program Managers and Policymakers . Washington, DC : USAID/Maternal and Child Health Integrated Project . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Beersma B , Hollenbeck JR, Humphrey SE et al. . 2003 . Cooperation competition and team performance: toward a contingency approach . Academy of Management Journal 46 : 572 – 90 . Google Scholar OpenURL Placeholder Text WorldCat Bell A , Jones K. 2015 . Explaining fixed effects: random effects modeling of time-series cross-sectional and panel data . Political Science Research and Methods 3 : 133 – 53 . Google Scholar OpenURL Placeholder Text WorldCat Besley T , Ghatak M. 2008 . Status incentives . American Economic Review 98 : 206 – 11 . Google Scholar OpenURL Placeholder Text WorldCat Bewley TF. 2007 . Fairness reciprocity and wage rigidity. In: Vartiainen Diamond P (eds). Behavioral Economics and Its Applications . Princeton, NJ : Princeton University Press , 157 – 88 . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Brunie A , Wamala-Mucheri P, Otterness C et al. . 2014 . Keeping community health workers in Uganda motivated: key challenges facilitators and preferred program inputs . Global Health: Science and Practice 2 : 103 – 16 . Google Scholar OpenURL Placeholder Text WorldCat Cameron KS. 1985 . Cultural Congruence Strength and Type: Relationships to Effectiveness. ASHE Annual Meeting Paper . Washington, DC : National Institute of Education . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Carnahan , S. , Agarwal , R. and Campbell , B.A., 2012 . Heterogeneity in turnover: The effect of relative compensation dispersion of firms on the mobility and entrepreneurship of extreme performers. Strategic Management Journal, 33(12): 1411 – 1430 . Charness G , Rabin M. 2002 . Understanding social preferences with simple tests . The Quarterly Journal of Economics 117 : 817 – 69 . Google Scholar OpenURL Placeholder Text WorldCat Charness G , Masclet D, Villeval MC. 2014 . The dark side of competition for status . Management Science 60 : 38 – 55 . Google Scholar OpenURL Placeholder Text WorldCat Clark AE , Masclet D, Villeval MC. 2010 . Effort and comparison income: experimental and survey evidence . ILR Review 63 : 407 – 26 . Google Scholar OpenURL Placeholder Text WorldCat Collins RL. 1996 . For better or worse: the impact of upward social comparison on self-evaluations . Psychological Bulletin 119 : 51 – 69 . Google Scholar OpenURL Placeholder Text WorldCat de Vries DH , Pool R. 2017 . The influence of community health resources on effectiveness and sustainability of community and lay health worker programs in lower-income countries: a systematic review . PLoS One 12 : e0170217 . Google Scholar OpenURL Placeholder Text WorldCat Deserranno E. 2019 . Financial incentives as signals: experimental evidence from the recruitment of village promoters in Uganda . American Economic Journal: Applied Economics 11 : 277 – 317 . Google Scholar OpenURL Placeholder Text WorldCat Druetz T , Kadio K, Haddad S et al. . 2015 . Do community health workers perceive mechanisms associated with the success of community case management of malaria? A qualitative study from Burkina Faso . Social Science & Medicine 124 : 232 – 40 . Google Scholar OpenURL Placeholder Text WorldCat Exline JJ , Lobel M. 1997 . Views of the self and affiliation choices: a social comparison perspective . Basic and Applied Social Psychology 19 : 243 – 59 . Google Scholar OpenURL Placeholder Text WorldCat Exline JJ , Single PB, Lobel M et al. . 2004 . Glowing praise and the envious gaze: social dilemmas surrounding the public recognition of achievement . Basic and Applied Social Psychology 26 : 119 – 30 . Google Scholar OpenURL Placeholder Text WorldCat Fehr E , Gächter S. 2000 . Fairness and retaliation: the economics of reciprocity . Journal of Economic Perspectives 14 : 159 – 81 . Google Scholar OpenURL Placeholder Text WorldCat Frey BS. 2007 . Awards as compensation . European Management Review 4 : 6 – 14 . Google Scholar OpenURL Placeholder Text WorldCat Gallus J , Frey BS. 2017 . Awards as strategic signals . Journal of Management Inquiry 26 : 76 – 85 . Google Scholar OpenURL Placeholder Text WorldCat Gelman A. 2006 . Multilevel (hierarchical) modeling: what it can and cannot do . Technometrics 48 : 432 – 5 . Google Scholar OpenURL Placeholder Text WorldCat Gelman A , Hill J. 2006 . Data Analysis Using Regression and Multilevel/Hierarchical Models . Cambridge : Cambridge University Press . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Harlow RE , Cantor N. 1995 . Overcoming a lack of self-assurance in an achievement domain. In: Kernis M (ed). Efficacy, Agency, and Self-Esteem . Boston, MA : Springer , 171 – 95 . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Hermann K , Van Damme W, Pariyo GW et al. . 2009 . Community health workers for ART in sub-Saharan Africa: learning from experience—capitalizing on new opportunities . Human Resources for Health 7 : 31 . Google Scholar OpenURL Placeholder Text WorldCat Hoyt CL. 2013 . Inspirational or self-deflating: the role of self-efficacy in elite role model effectiveness . Social Psychological and Personality Science 4 : 290 – 8 . Google Scholar OpenURL Placeholder Text WorldCat Kane S , Radkar A, Gadgil M et al. . 2020 . Community health workers as influential health system actors and not “just another pair of hands” . International Journal of Health Policy and Management . doi:10.34172/ijhpm.2020.58, accessed 27 September 2020. Google Scholar OpenURL Placeholder Text WorldCat Kane SS , Gerretsen B, Scherpbier R et al. . 2010 . A realist synthesis of randomised control trials involving use of community health workers for delivering child health interventions in low and middle income countries . BMC Health Services Research 10 : 286 . Google Scholar OpenURL Placeholder Text WorldCat Kok MC , Dieleman M, Taegtmeyer M et al. . 2015 . Which intervention design factors influence performance of community health workers in low-and middle-income countries? A systematic review . Health Policy and Planning 30 : 1207 – 27 . Google Scholar OpenURL Placeholder Text WorldCat Kok MC , Ormel H, Broerse JE et al. . 2017 . Optimising the benefits of community health workers’ unique position between communities and the health sector: a comparative analysis of factors shaping relationships in four countries . Global Public Health 12 : 1404 – 32 . Google Scholar OpenURL Placeholder Text WorldCat Krishnakumar J , Nagar AL. 2008 . On exact statistical properties of multidimensional indices based on principal components factor analysis MIMIC and structural equation models . Social Indicators Research 86 : 481 – 96 . Google Scholar OpenURL Placeholder Text WorldCat Lundby KM , DeMatteo JS, Rush MC. 1999 . Organizational culture and the total quality organization. In M. J. Stahl (ed). Perspective in Total Quality. Cambridge, MA : Blackwell Publishers , 135 – 152 . Maes K , Kalofonos I. 2013 . Becoming and remaining community health workers: perspectives from Ethiopia and Mozambique . Social Science & Medicine 87 : 52 – 9 . Google Scholar OpenURL Placeholder Text WorldCat Mathieu JE , Aguinis H, Culpepper SA et al. . 2012 . Understanding and estimating the power to detect cross-level interaction effects in multilevel modelling . Journal of Applied Psychology 97 : 951 – 66 . Google Scholar OpenURL Placeholder Text WorldCat McKague , K. and Harrison , S., 2019. Gender and health social enterprises in Africa: a research agenda . International journal for equity in health, 18(1), p. 95 . OpenURL Placeholder Text WorldCat Mlotshwa L , Harris B, Schneider H. et al. . 2015. Exploring the perceptions and experiences of community health workers using role identity theory . Global Health Action , 8 ( 1 ): 28045. OpenURL Placeholder Text WorldCat Morel R , Chowdhury R. 2015 . Reaching the ultra‐poor: adapting targeting strategy in the context of South Sudan . Journal of International Development 27 : 987 – 1011 . Google Scholar OpenURL Placeholder Text WorldCat Morgenstern ML. 1995 . The board's perspective: compensation and the new employment relationship . Compensation & Benefits Review 27 : 37 – 44 . Google Scholar OpenURL Placeholder Text WorldCat Neupane D , McLachlan CS, Mishra SR et al. . 2018 . Effectiveness of a lifestyle intervention led by female community health volunteers versus usual care in blood pressure reduction (COBIN): an open-label cluster-randomised trial . The Lancet Global Health 6 : e66 – 73 . Google Scholar OpenURL Placeholder Text WorldCat Nkonki L , Cliff J, Sanders D. 2011 . Lay health worker attrition: important but often ignored . Bulletin of the World Health Organization 89 : 919 – 23 . Google Scholar OpenURL Placeholder Text WorldCat Pelham BW , Wachsmuth JO. 1995 . The waxing and waning of the social self: assimilation and contrast in social comparison . Journal of Personality and Social Psychology 69 : 825 – 38 . Google Scholar OpenURL Placeholder Text WorldCat Perry H , Zulliger R, Scott K et al. . 2017 . Case Studies of Large-Scale Community Health Worker Programs: Examples from Afghanistan, Bangladesh, Brazil, Ethiopia, India, Indonesia, Iran, Nepal, Niger, Pakistan, Rwanda, Zambia, and Zimbabwe . Washington, DC : USAID . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Perry HB , Zulliger R, Rogers MM. 2014 . Community health workers in low- middle- and high-income countries: an overview of their history recent evolution and current effectiveness . Annual Review of Public Health 35 : 399 – 421 . Google Scholar OpenURL Placeholder Text WorldCat Rachlis B , Naanyu V, Wachira J et al. . 2016 . Community perceptions of community health workers (CHWs) and their roles in management for HIV tuberculosis and hypertension in Western Kenya . PLoS One 11 : e0149412 . Google Scholar OpenURL Placeholder Text WorldCat Rege M , Telle K. 2003 . Indirect Social Sanctions from Monetarily Unaffected Strangers in a Public Good Game. Discussion Paper 359. Kongsvinger: Statistics Norway. Reichenbach L , Shimul SN. 2011 . Sustaining health: the role of BRAC's community health volunteers in Bangladesh, Afghanistan and Uganda . Research Reports Economic Studies 28 : 224 – 333 . Google Scholar OpenURL Placeholder Text WorldCat Rowe AK , De Savigny D, Lanata CF et al. . 2005 . How can we achieve and maintain high-quality performance of health workers in low-resource settings? The Lancet 366 : 1026 – 35 . Google Scholar OpenURL Placeholder Text WorldCat Salovey P , Rodin J. 1984 . Some antecedents and consequences of social-comparison jealousy . Journal of Personality and Social Psychology 47 : 780 – 92 . Google Scholar OpenURL Placeholder Text WorldCat Segal U , Sobel J. 2007 . Tit for tat: foundations of preferences for reciprocity in strategic settings . Journal of Economic Theory 136 : 197 – 216 . Google Scholar OpenURL Placeholder Text WorldCat Singh D , Negin J, Otim M et al. . 2015 . The effect of payment and incentives on motivation and focus of community health workers: five case studies from low-and middle-income countries . Human Resources for Health 13 : 58 . Google Scholar OpenURL Placeholder Text WorldCat Snijders TA , Bosker RJ. 2011 . Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling . London : Sage . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Steele F. 2008 . Module 5: Introduction to Multilevel Modelling Concepts. LEMMA VLE. University of Bristol, Centre for Multilevel Modelling. https://www.cmm.bris.ac.uk/lemma/, accessed 27 September 2020. Swider SM. 2002 . Outcome effectiveness of community health workers: an integrative literature review . Public Health Nursing 19 : 11 – 20 . Google Scholar OpenURL Placeholder Text WorldCat Vareilles G , Pommier J, Kane S et al. . 2015 . Understanding the motivation and performance of community health volunteers involved in the delivery of health programmes in Kampala Uganda: a realist evaluation protocol . BMJ Open 5 : e006752 . Google Scholar OpenURL Placeholder Text WorldCat Vareilles G , Pommier J, Marchal B et al. . 2017 . Understanding the performance of community health volunteers involved in the delivery of health programmes in underserved areas: a realist synthesis . Implementation Science 12 : 22 . Google Scholar OpenURL Placeholder Text WorldCat Williams B , Onsman A, Brown T. 2010 . Exploratory factor analysis: a five-step guide for novices . Australasian Journal of Paramedicine 8 : http://ro.ecu.edu.au/jephc/vol8/iss3/1, accessed 27 September 2020. Google Scholar OpenURL Placeholder Text WorldCat World Health Organization. 2017 . World Bank and WHO: Half the world lacks access to essential health services, 100 million still pushed into extreme poverty because of health expenses. Press Release. 13 December. Ybema JF , Buunk BP. 1995 . Affective responses to social comparison: a study among disabled individuals . British Journal of Social Psychology 34 : 279 – 92 . Google Scholar OpenURL Placeholder Text WorldCat Zachariah R , Ford N, Philips M et al. . 2009 . Task shifting in HIV/AIDS: opportunities challenges and proposed actions for sub-Saharan Africa . Transactions of the Royal Society of Tropical Medicine and Hygiene 103 : 549 – 58 . Google Scholar OpenURL Placeholder Text WorldCat Zeller M , Sharma M, Henry C et al. . 2006 . An operational method for assessing the poverty outreach performance of development policies and projects: results of case studies in Africa . World Development 34 : 446 – 64 . Google Scholar OpenURL Placeholder Text WorldCat © The Author(s) 2020. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. © The Author(s) 2020. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine.

Kielmann, Karina; Dickson-Hall, Lindy; Jassat, Waasila; Le Roux, Sacha; Moshabela, Mosa; Cox, Helen; Grant, Alison D; Loveday, Marian; Hill, Jeremy; Nicol, Mark P; Mlisana, Koleka; Black, John

doi: 10.1093/heapol/czaa147pmid: 33582787

Abstract In 2011, the South African National TB Programme launched a policy of decentralized management of drug-resistant tuberculosis (DR-TB) in order to expand the capacity of facilities to treat patients with DR-TB, minimize delays to access care and improve patient outcomes. This policy directive was implemented to varying degrees within a rapidly evolving diagnostic and treatment landscape for DR-TB, placing new demands on already-stressed health systems. The variable readiness of district-level systems to implement the policy prompted questions not only about differences in health systems resources but also front-line actors’ capacity to implement change in resource-constrained facilities. Using a grounded theory approach, we analysed data from in-depth interviews and small group discussions conducted between 2016 and 2018 with managers (n = 9), co-ordinators (n = 15), doctors (n = 7) and nurses (n = 18) providing DR-TB care. Data were collected over two phases in district-level decentralized sites of three South African provinces. While health systems readiness assessments conventionally map the availability of ‘hardware’, i.e. resources and skills to deliver an intervention, a notable absence of systems ‘hardware’ meant that systems ‘software’, i.e. health care workers (HCWs) agency, behaviours and interactions provided the basis of locally relevant strategies for decentralized DR-TB care. ‘Software readiness’ was manifest in four areas of DR-TB care: re-organization of service delivery, redressal of resource shortages, creation of treatment adherence support systems and extension of care parameters for vulnerable patients. These strategies demonstrate adaptive capacity and everyday resilience among HCW to withstand the demands of policy change and innovation in stressed systems. Our work suggests that a useful extension of health systems ‘readiness’ assessments would include definition and evaluation of HCW ‘software’ and adaptive capacities in the face of systems hardware gaps. Policy implementation, decentralization, drug-resistant tuberculosis, health systems, readiness KEY MESSAGES Health system readiness assessments highlight critical resource gaps but fail to capture local contextual factors and values and the dynamic responses of health systems actors in driving policy implementation. Different levels of readiness to decentralize drug-resistant tuberculosis care observed in South African districts can be partly explained by health systems actors’ capacity to adapt to ongoing challenges and new demands placed on a stressed health system. Health systems’ actors’ capacities to adapt, self-organize and devise locally relevant strategies to implement policy directives reflect dimensions of resilience to systems stressors as well as readiness for organizational change. Conventional readiness assessment tools could be usefully extended to include questions about how health workers and managers respond to both the ‘everyday’ crises and the sporadic policy changes that can disrupt service delivery. This policy is in place, but we don’t know how to implement it… so let’s work with the district, because they had it on paper, it was there… but we actually had to modify the [National Department of Health] policy (Senior clinician, KwaZulu-Natal). Introduction In 2006, medical journals worldwide reported the outbreak of extensively drug-resistant tuberculosis (XDR-TB) in a hospital in Tugela Ferry, in the province of KwaZulu-Natal (KZN), South Africa. Tugela Ferry was a critical ‘tipping point’ for drug-resistant (DR)-TB in the country and drew global attention to the threat of DR-TB (Saidi et al., 2017). However, despite recognition that the conventional model of institutionalized treatment was inadequate (Padayatchi and Friedland, 2008), programmatic response to DR-TB was variable and fairly sluggish in some provinces (WHO, 2016). The numbers of DR-TB patients rose from ∼8000 between 2007 and 2010 to >16 000 in 2017 (WHO, 2018). Five years after the outbreak, the South African National TB Programme launched a policy of ‘decentralized and deinstitutionalized’ management of DR-TB (Department of Health and Republic of South Africa, 2011a,b) to expand facilities’ capacity to manage DR-TB. Concurrently, the diagnostic and treatment landscape for DR-TB was rapidly evolving1 and placing new demands on an already-stressed health system. Significant gaps between the number of people found to have DR-TB and those starting on second-line treatment (Cox et al., 2017a; Evans et al., 2018) prompted questions about the health system’s capacity to provide timely and appropriate treatment at decentralized levels of the system. In South Africa, large regional disparities in disease burden, human resources, financing and investment, administration and management capacity are mirrored in considerable differences in service readiness and availability (Dookie and Singh, 2012; Fusheini and Eyles, 2016). By 2015, substantially different ‘models’ of decentralized care for DR-TB were emerging, reflecting not only different interpretations of the policy, but variability in district health systems contexts, capacity and readiness to implement decentralized care (Cox et al., 2015; Department of Health, Republic of South Africa, 2019). Within the literature on policy implementation, ‘readiness’ refers to systems capability to initiate and sustain organizational change in response to initiatives intended to improve systems performance (Manu et al., 2018; Zurovac et al., 2018). Assessments of health systems readiness traditionally involve an evaluation of the minimum ‘hardware’ requirements to ensure successful delivery of health services. Monitoring tools, e.g. the WHO SARA (Service Availability Readiness Assessment) track the functional availability of health system’s ‘building blocks’ components, i.e. ‘…trained staff, guidelines, equipment, diagnostic capacity, and medicines and commodities’ (WHO, 2015). In settings where resources are lacking or inequitably distributed, assessments like the SARA can highlight critical gaps. Yet, they fail to capture contextual factors influencing the supply and demand of health services, the role of values as ‘steering mechanisms’ (Van Olmen et al., 2012), and the dynamic responses of health systems actors in driving or obstructing change (Blaauw et al., 2006). In this paper, we examine early responses to the mandate to decentralize DR-TB care in three South African provinces to illustrate the dynamic relationship between human agency, health systems readiness and policy implementation. Our aim is to highlight the neglected role of actors’ behaviours and interactions—often referred to as ‘software’ (Sheikh et al., 2011)—in assessments of health systems ‘readiness’. Against the ongoing challenges of providing DR-TB care in resource-constrained facilities, e.g. poorly maintained infrastructure, inadequate drug supplies, overworked staff and insufficient training on DR-TB management, we draw attention to the critical role of adaptive responses in policy implementation. Conceptual framework Calls to more explicitly link policy and systems research (Gilson, 2012; Ghaffar et al., 2016) suggest the need to consider how systems actors respond and adapt to changes that are introduced as a result of new policy initiatives. Policy implementation requires more than a template of standard operating procedures; it is a ‘…challenging process, working through the whole health system and ultimately taking effect or being blocked at the frontlines of service delivery and community engagement’ (Gilson, 2016). To examine the role of human agency in the adoption of health systems interventions, some implementation researchers suggest greater inclusion of concepts used in organizational theory and management (Birken et al., 2017). Here, ‘readiness’ refers to willingness and capacity to implement a particular innovation (Weiner, 2009; Scaccia et al., 2015); the research on organizational readiness explicitly considers interactions between ‘emergent expressions of human agency’ and context as critical to enactment of change within the system (May, 2013). The concept of ‘tinkering’ further helps to elucidate flexibility and adaptation in local-level responses to policy initiatives. ‘Tinkering’ has been used to describe how actors ‘adjust the protocol to unforeseen events’ (Timmermans and Berg, 1997) through an opportunistic rearrangement of existing elements that opens space for new ways of doing things. Policy translation is both creative and pragmatic, and characterized by ‘fluid multi-actor processes of interpretation, mutation and assemblage. […]’ (Stone, 2017, p. 67). Husain (2017) refers to emergent ‘tinkering’ in decentralized health policymaking in China: in the absence of national standardization and expert support, space became available for local discretion in pragmatic problem-solving, and local context-specific approaches in policy implementation. Health systems actors’ tinkering can also strengthen the system’s capacity to manage everyday crises faced in resource-constrained settings. An ‘agency centred’ focus of recent thinking on resilience in development work (Jeans et al., 2017) argues that we should ‘….move away from simply looking at what a person, household, or system has and recognise and enhance what it does’ (sic). Though the concept of resilience commonly refers to systems’ capacity to cope and ‘bounce back’ after significant shocks such as war, natural disasters or humanitarian crises, the term has also been applied to the micro-level adjustments that actors make to address ongoing challenges in constructive ways. For example, Gilson et al. (2017) focus on ‘…internally generated chronic stresses, some of which are even infused into the routine organisational life of health systems’ that both generate and demand expressions of ‘everyday resilience’ among district-level managers. Here, we examine the role of health workers and managers’ adaptive responses to move the agenda on decentralized DR-TB care forward in pragmatic ways, against a backdrop of structural resource constraints, and policy tensions (Moshabela et al., 2020). This involved attention to the small but meaningful changes in normative practice made to adapt to the ‘moving target’ of DR-TB care innovations in the district health system. Materials and methods This paper draws on data from a 4-year project that aimed to gain an understanding of the policy context, patient care pathways and models of decentralization of DR-TB care in three provinces of South Africa: Western Cape (WC); KZN and Eastern Cape (EC). The project entailed three phases of qualitative research conducted between 2016 and 2018 (see Figure 1): a key informant interview (KII) study (Phase 1); facility process-mapping and interviews in sites providing decentralized DR-TB care (Phases 2A and 2B, respectively) and an in-depth study of specific emergent models of decentralized care in the three provinces (Phase 3). We draw mainly on the interviews conducted in Phases 2B and 3, the vast majority of which took place in WC and KZN. We refer to a few of the KII (Phase 1) to elucidate the context within which the policy was launched, more fully described elsewhere (Moshabela etal., 2020). Figure 1 Open in new tabDownload slide Data collection methods, participants and areas of inquiry. WC, Western Cape; KZN, KwaZulu-Natal; EC, Eastern Cape. Source: Adapted from Department of Health, Republic of South Africa (2019), p. 17. Figure 1 Open in new tabDownload slide Data collection methods, participants and areas of inquiry. WC, Western Cape; KZN, KwaZulu-Natal; EC, Eastern Cape. Source: Adapted from Department of Health, Republic of South Africa (2019), p. 17. Ethical considerations All research procedures for the project were approved by the Human Research Ethics committee at the University of Cape Town and by the Observational/Interventions Research Ethics Committee of the London School of Hygiene & Tropical Medicine. Permission to conduct the site visits and interviews was granted by the Department of Health research committees in the EC, WC and KZN Provinces. Written informed consent was obtained from all participants. Data collection During Phase 1, KIIs were conducted by co-authors MM and WJ with national and provincial stakeholders (see Figure 1 for details), exploring their understanding of the evidence, timeline and initial strategies for launching the national policy on decentralized DR-TB care. During Phase 2A, site visits were conducted by co-authors LDH, SLR, and a third researcher (LM) in 21 facilities within 13 district-level decentralized sites of care in WC, KZN and EC. Researchers used a structured tool, administered to the facility manager or lead TB nurse, to conduct facility process-mapping while cross-checking pathways of care for patients with rifampicin-resistant TB identified previously using National Health Laboratory Services (NHLS) data, and identify emerging ‘models’ of decentralized care (Hill et al., 2020). In Phase 2B, WJ conducted in-depth interviews in WC and KZN to understand the policy implementation processes, and factors enabling and hindering decentralized DR-TB care across districts. Informants were selected purposively and included those more closely associated with implementation of the decentralization policy. Permissions for interviews conducted in Phases 1 and 2 were obtained from the respective provinces, district and facility management. While most interviews were conducted face to face, some were done telephonically. After obtaining written informed consent, interviews were voice-recorded; all but one were conducted in English and transcribed by SLR, checked and edited by MM and WJ. One interview was conducted in Afrikaans, translated and transcribed by LDR. In Phase 3, researchers returned to one district in each province that represented a distinct ‘model’ of decentralized care, as identified in Phase 2A. They conducted small group discussions and informal interviews with staff, mainly at three district ‘hub’ hospitals, to probe contextual features and mechanisms influencing ‘optimal’ delivery of care in the different sites. These conversations were recorded with consent of the participants; one of the researchers (SLR) made detailed notes based on the recordings. Data analysis Adopting a grounded theory approach to analysis (Glaser and Strauss, 1967), researchers (KK, LDH, MM, SLR and WJ) read transcripts and fieldnotes several times to discuss emerging concepts. For this paper, we identified three broad themes: systems readiness, health care worker (HCW) adaptive capacities and local innovations in responding to the mandate to decentralize DR-TB care. Constant comparison of text segments identified as relevant to themes helped to generate more nuanced sub-themes ( Appendix 1) that formed the basis of a coding system. The data were manually coded by three researchers (LDH; SLR; KK) in Microsoft Word; relevant quotes were extracted to facilitate comparison and identification of patterns. All individual and health facility identifiers were anonymized, with individuals identified only through their function within the system. Results The move to decentralize DR-TB care in South Africa: gaps in ‘readiness’ Policy guidelines to manage DR-TB in South Africa were drafted in 2000, and successively revised to emphasize standardized treatment regimens as well as monitoring and surveillance requirements for a uniform approach to organizing DR-TB services. The revised policy framework for decentralized DR-TB care (Department of Health Republic of South Africa, 2019) foresaw the transfer of responsibility for treating DR-TB patients from regional specialist centres to district-level facilities closer to patients’ homes. The policy distinguished between a hospitalized model for patients who were clinically unwell, had second-line resistance, were sputum smear positive or had comorbid conditions; and an ambulatory model for patients who were otherwise well and could be treated in their community. When introduced in 2011, the policy was not accompanied by dedicated or ring-fenced funding, except for limited support for building and infrastructure through the Global Fund. Provincial and district health officials noted that the policy was ‘an unfunded mandate’; some resorted to funding additional staff and equipment through their general health budgets as well as through the national grant ear-marked for HIV (Jassat, 2020). Essential elements to assess readiness of a site to provide decentralized DR-TB care included access to laboratory services for diagnosis of DR-TB, uninterrupted supplies of TB drugs and audiology services (Department of Health and Republic of South Africa, 2019). Beyond the ‘hardware’ necessary to deliver services, prerequisites included ‘software’ components such as functionality of multidisciplinary teams, integration of DR-TB care within PHC services, good communication across levels of the system, and effective advocacy and social mobilisation in the community ( Appendix 2). These components, indicative of effective management of organizational change processes, could not be assumed to be uniformly present across district health systems in the country. Key informants consistently pointed to gaps in readiness to implement the policy. These included a lack of sufficient evidence, insufficient resources and time and the absence of concrete plans (Moshabela et al., 2020). Underpinning these considerations was the wider perception of tuberculosis as a stagnant field, slow to change its ways. One senior NGO representative noted that the prospect of decentralising TB was ‘going against tradition’; those who managed TB were ‘very set in their ways…a real old boys’ club’. This key informant further elaborated: ‘There was no expert opinion…it was just that we have this back log, people are dying, what strategy can we employ?’ adding that ‘…as much as a researcher and a scientist you want the evidence before you can implement new things, we didn’t have that luxury’. In EC, one clinical manager referred to the decentralization policy framework as a ‘zero plan’ recalling that ‘we just did what we thought’. Practical implementation of a decentralized DR-TB service required that patients who had previously been admitted for specialised treatment in a highly monitored environment would now be initiated on to treatment and monitored at district-level units, which traditionally did not handle such complex conditions. The centralized, provincial Centre of Excellence was to remain responsible for initiation and treatment monitoring for XDR-TB patients and other complications. Following treatment initiation, patients would be referred to their nearest primary health care facility for daily observed treatment (DOT), injections, and monitoring of side effects and adherence (Vanleeuw et al., 2020). However, limited funding, inadequate infrastructure, differences in systems capacity (Cox et al., 2017b) and the absence of operational guidelines for implementation left the policy open to interpretation, leading to adaptive responses to compensate and ‘buffer’ the additional clinical load and scope of work placed on systems. In the following sections, we highlight health systems actors’ ability to cope, adapt and devise locally relevant strategies to deliver decentralized care against the backdrop of limited systems ‘readiness’. This was manifest in four key areas: first, re-organization of service delivery; second, redressal of resource shortages; third, treatment adherence support systems; and fourth, extension of care parameters for vulnerable patients. Re-organization of service delivery Getting patients on to treatment as soon as possible is deemed ‘…one of the pillars of success in TB management’ (HAST co-ordinator, female, white, rural WC). To improve access and timely initiation of DR-TB treatment at the district level, services were re-configured: Out of the sites that we had identified as our outreach points, we changed the order of things to accommodate seeing the [DR-TB] patients there (…) it sort of became like a motto that they need to ensure that the space is on a certain day that you don’t have a clinic for instance, like an under-5 for instance, or you don’t have an ANC on the day (Clinician, female, coloured, rural clinic, WC). In some rural settings, clinics only had a medical officer in attendance once a week, so arrangements were made to accommodate growing numbers of DR-TB patients. For example, in WC, staff from a rural clinic drove out to satellite clinics to reach farm workers as there were no nearby sites initiating treatment. In KZN, clustering arrangements were made to accommodate patients with poor access to clinics: Patients can’t even get to any clinic (…) So they know that they are seeing this cluster of clinics, they know the doctor will be there the first day of the month so on that day the patient will be reviewed by the doctor. The patient will have collection of sputum, collection of bloods, [they will] do everything on that day, because it’s the only time they get access to the health services (Clinical manager, male, coloured, urban TB hospital, KZN). In WC, senior clinicians’ close, cohesive relationship with hospitals in their districts allowed them to manipulate resources to optimize patient care, e.g. moving patients between different facilities to accommodate newer, sicker patients in exchange for stable, recuperating patients. Rural clinics without frequent access to a clinician devised a system of ‘virtual consultations’ through remote faxing of prescriptions. In rural areas of WC, this involved sending results through to the Infectious Diseases hospital, with the attending physician faxing back the prescription. In other instances, nurses went out of their way to ensure timely access to treatment, e.g. taking sputum samples to the lab personally in order to obtain a diagnostic result quickly. Redressal of resource shortages In developing strategies to deliver decentralized care for DR-TB patients, clinic staff had to absorb existing or anticipated resource gaps. Staff, infrastructure, equipment and drugs were temporarily transferred within and across sites to ensure that patients could access services without interruption. Staff In areas with limited trained staff or staffing shortages, interventions to deliver the minimum of care were achieved through task shifting and sharing. One rural WC clinic lacking a clinician trained in DR-TB was visited monthly by a doctor from a neighbouring facility who wrote out 6-month prescriptions to be facilitated by the pharmacy and the attending TB nurse. Outreach services were organized to extend DR-TB expertise to rural and underserved areas. In KZN, the manager (male, coloured) of a specialized TB hospital commented: ‘I think it is a good compromise… shifting the staff to the area of need since we are still getting the patients coming in as outpatients…it is even nicer to know that the doctor is there and the nurses are also there’. Recognition of systems weaknesses and gaps in specialized care further led to referrals to enable better care pathways for patients who required specific services, such as surgery, mental health resources or rehabilitation due to more complicated forms of DR-TB. Infrastructure and equipment Infrastructural changes were made to accommodate patients, e.g. an old canteen area for staff was converted into a four-bed DR-TB ward in rural EC. In another rural hospital in EC, staff created their own NGO to provide transport funding for patients unable to access a hospital ambulance. When essential equipment recommended to monitor the effects of DR-TB drugs was not functional or readily available, adaptive solutions were found: There was a time that our ECG [electrocardiogram] was faulty, so we just went to a nearby clinic …I would try to get patients to come on one specific day if they needed an ECG and then one of SPN’s [Senior Professional Nurse] would go and collect the machine [in her car] from the other clinic and then I would do all of them (DR-TB nurse, female, coloured, urban clinic, WC). I don't have my own machine. Currently, I'm using one of trauma’s machines in my room (DR-TB nurse, female, coloured, urban Community Health Centre (CHC), WC). In some settings, addressing the infrastructural and equipment gaps included leveraging resources through other programmes: The equipment and the services might not always be on site but we have been able to access those … with the whole Ideal Clinic2and all of those other things coming up, it has also been an opportunity to motivate for additional equipment (Operational manager, female, coloured, urban clinic, WC). Drugs Successful TB management relies on a steady supply of drugs. Clinic staff anticipated drug stock-outs by ordering larger quantities of drugs in advance or by balancing stock levels across facilities: We know that for TB, HIV and certain chronic medical conditions you should always try and make sure that you always order at least 3 months’ worth of stock, every month (…) because from time to time there are drug shortages in the country (Pharmacist, female, black, rural hospital, EC). From my side, I will find out if it [drug stockout] is only here or if it is a problem from our pharmacy’s side or it if it is out of stock in general from the depot … then I will speak to the pharmacist in charge and she will contact other clinics to ask what their stock levels are. If we are really short, we will do our utmost to go out to other clinics to get some stock to cover (DR-TB nurse, female, coloured, urban clinic, WC). Creation of treatment adherence support systems The strict monitoring demands of decentralized delivery of complex drugs in ambulatory settings obliged HCWs to adopt flexible practices related to drug dispensing, keeping patients on treatment, documentation practices and sharing of expertise. Drug dispensing Clinicians described adaptive prescribing practices to support patients on treatment, e.g. tapping into the existing packaging and delivery services for the patient’s monthly anti-retroviral (ARV) drug provision or giving patients supplies of treatment ‘under the table’ when patients were unable to come back on a daily basis. Patient costs incurred through multiple visits were also a reason to offer a more flexible schedule of treatment: Most of our MDR paeds [paediatric cases] will have to pay up to two hundred rand to get to the hospital… you know four hundred rand [27 US$] for a return trip for the mother and child or children where there are two or three of them on treatment is tricky. So, I often give them two months of treatment at a time (Hospital clinician, female, white, rural hospital, EC). In order to establish and maintain treatment routines, HCW adopted common-sense modifications of existing prescribing practices to facilitate dispensing of medication. These included pre-packing, bulk preparation and colour-coding medicines: We prepack the injections and then I literally write thirty scripts effectively and they have to sign on each one every day (Provincial hospital doctor, male, white, EC). I was using the medicine containers, the small ones in which they decant ointment in… I have a small booklet where I will put the sticker of who is coming tomorrow then I will say ok this one [this patient], I can give [medicines] weekly (DR-TB nurse, female, coloured, urban clinic, WC). You find, like in one household there’s three kids. So, what we usually do with those is, in terms of dispensing, we colour-code the medication (Pharmacist, female, black, rural district hospital, EC). Keeping patients on treatment To reduce the frequency of visits, HCW assessed adjusted treatment schedules based on individual patients’ situations. For example, nurses reported giving treatment on a weekly basis to patients who were working, no longer infectious, or deemed stable in terms of their treatment adherence. Strategies were devised for patients unable to attend a daily clinic, sometimes enlisting the help of other patients to collect treatment or an NGO to provide DOT at home: Even those who have gone back to work we organise a system for them. I don't know but with this patient now currently there are always two that are staying near to each other and then this one is going to work and the other one will come fetch his treatment and give it to him (Operational Manager, female, coloured, urban CHC, WC). The patients prefer to be at home and we also prefer them to be at home. So, we would involve our NGOs (…) and they would DOT [directly observe treatment] them at home. The patient would still have to come once a month to see myself and the sisters for the investigations and the clinical examinations (Family physician, female, white, rural Community Day Centre (CDC), WC). Clinic staff described numerous ways of motivating patients to stay on treatment. Some organized adherence workshops to ‘boost’ patient morale. The operational manager of an urban CHC in WC spoke of maintaining ‘open communication’ with patients, e.g. through WhatsApp. One DR-TB nurse, also in an urban WC clinic celebrated treatment ‘successes’ by organizing parties and treatment completion ‘certificates’ that had been designed by the facility co-ordinator. When patients needed to be in in-patient care longer than recommended, e.g. because of their specific difficulties to stay on treatment, HCW found ways to extend the prescribed length of stay. At times you get patients that request the admission for a little bit longer than actually clinically needed and very often they say the temptations out in the community are just too bad and they know that they’re going to have difficulty managing it [adherence] (Doctor, female, white, rural hospital, WC). Referring to patients who had to return to work soon after they were no longer considered infectious and their government TB disability grant had run out, one DR-TB nurse (female, coloured) in an urban WC clinic commented: ‘Occasionally, you have to sort of override the protocol a bit if you want to keep the patient in care’, later adding that ‘their bosses don’t always understand’ the long-term debilitating effects of the illness and of being on treatment. Documentation ‘Tinkering’ was also evident in initiatives to facilitate record-keeping and support the complex monitoring and documentation needs of decentralized DR-TB treatment. Monitoring forms that were seen to be cumbersome were re-designed to make them more user-friendly. In part, these modifications were aimed at reducing HCW unfamiliarity with new protocols and processes that were introduced as a result of the mandate to decentralize care: At the beginning it is not that easy you know for somebody that sees them [the patients]. Often you know what to do but even the medicines and the names were completely new. So, I developed like a worksheet that will tell you or guide to do sputum monthly, it will guide you to how often you need the ALT, how often you need the blood tests, how often audiology, how often X-rays so if a doctor went according to the worksheet, you couldn't miss something (Clinician, female, coloured, rural district CDC, WC). There are certain things that I have designed to make it easier for them [HCW] to work, like I redesigned the monitoring tool of the drug-resistant TB. So, we have a shorter one and a longer conventional one that I have designed (Sub-district co-ordinator, female, coloured, urban clinic, WC). Sharing expertise Some clinicians developed paper-based registers and templates to identify problems for discussion on a monthly basis, enabling timely and ongoing in-service training. Existing gaps in expertise were also addressed through support and mentorship networks among health professionals, often using WhatsApp as a platform. These served to discuss difficult cases, disseminate information and access experts. In a rural district hospital in EC, the senior medical officer managed DR-TB patients in collaboration with an off-site specialist consultant. Extending parameters of care for vulnerable patients Beyond meeting basic requirements, some staff actively sought to address the social needs of patients who were impoverished or had difficult life circumstances. Nurses in rural areas of WC reported recording lower patient weights or adjusting scales in order to help patients to get into nutritional support programmes. One nurse in WC described building relationships with ward councillors and community members to organize food parcels for vulnerable patients. There was a thing where they said that there were food trolleys … I jumped forward because I know what the needs of the family with five MDRs are. I would write to the ward councillor [to say] ‘there are so many people in that house and all of them are not getting the grant with the exception of the grandmother and she is also sick’. I give them a hand (…) Food packages that we make up on our own are accepted at certain NGOs to help them (Nurse, female, black, rural clinic, WC). Clinic staff pro-actively intervened to mobilize community assistance for patients living in sub-standard housing. There was this family, a sister was living with her two brothers, the mother passed away and then this boy contracted TB when he was 16 years of age and he was diagnosed with HIV (…) they were living in a one room shack then I had to intervene while we were still waiting for a bed. I had to ask the community to get involved because there was no material to find if whoever can then make a shack for him (Clinic manager, female, black, rural clinic, WC). I will go and have a look and see where you [referring to a patient] live. I link up with the ward councillor. I link up with housing and I will go and look at what the problem is there and then I will talk to them. [I will say :] ‘There are so many people in the house…can’t you add a bungalow to it or can you give me a hand?’ (Nurse, female, black, rural hospital, WC). Finally, initiatives to cater to children’s needs were observed, e.g. in a rural hospital in EC, where ‘individual plans’ were made in order to accommodate mothers and children together. In this hospital, staff also supported out-of-school children during their treatment: We’ve helped them [kids] with school and stuff as well. Our OTs [occupational therapists] will go and give them extra lessons or our social worker will help get them into the local school for two months which helps a lot because for children that is the most important thing (Senior Medical Officer, female, white, rural hospital, EC). Discussion Recognition of the importance of strengthening health systems capacity and readiness to deliver priority interventions has increased over the past 15 years, particularly in low- and middle-income countries (LMIC). For the most part, tools to assess ‘readiness’ for uptake of specific policies or scale-up of existing interventions in LMIC involve mapping essential resources, knowledge and skills needed to implement a new intervention or initiative at facility-level (WHO 2015). While useful in ranking facilities according to their capacity, in principle, to provide basic health services at minimum standards (Jigjidsuren et al., 2019; Ssensamba et al., 2019), these tools lack consideration of the software dimensions of ‘readiness’ related to individual and collective agency to implement change. Consequently, a few researchers have begun to articulate their own fit-for-purpose ‘systems readiness frameworks’. Reporting on a novel framework for assessing readiness to implement a domestic violence intervention at primary care level in Palestine, Colombini et al. (2020) conclude that even if all the necessary ‘hardware’ elements are in place, ‘…the materialization of collective readiness is dependent on the software elements also being ready’. Conversely, Akinyemi et al. (2019) discuss how, in the absence of adequate ‘hardware’ for the scale-up of community-based distribution of injectable contraceptives in Northern Nigeria, health workers enable policy implementation through their adaptive responses: ‘…they often modify the process in order to adapt to the realities on the ground’. Our study of health systems actors’ emerging responses to the policy of decentralizing DR-TB care in South Africa suggests there are useful bridges to be made across the currently distinct bodies of literature on health systems and organizational readiness. We concur with May et al. (2016) that understanding organizational aspects of implementation requires attention to how ‘they are shaped by the behaviours and actions of participants as they negotiate the normative and relational environment in which they are set’. We observed numerous instances of bottom-up ‘tinkering’ that challenge a linear interpretation of policy implementation, reflecting actors’ resilience in managing everyday ‘micro-level crises’ (Barasa et al., 2017, 2018) but also their capacity for managing change. Observed practices contributed to strengthening different capacities of resilient systems, described in the development literature as absorptive, adaptive and transformative. For the most part, HCWs and managers strived to maintain functional services in the face of policy change. Under absorptive capacity, we noted actions that sought to restore balance in observed disparities in resource allocation and capacity across sub-components of the system. Adaptive capacity was evident in HCWs’ refinement of existing tools and practices and their extension of tasks to accommodate patients’ unique and challenging circumstances. Less frequently, HCW and managers’ actions demonstrated transformative capacity in their attempts to organize additional or novel ways of facilitating patient access, care and follow-up. ‘Tinkering’ may thus serve different purposes in this setting: the absence of operational guidelines for policy implementation may open the space for ‘tinkering’ that is undertaken to meet minimum requirements for a functional delivery system. In other instances, however, health systems actors’ ‘tinkering’ extends beyond the status quo, demonstrating readiness to implement change towards improving quality of care (Mussie et al., 2020). Relevant to this distinction are the kinds of organizational cultures that support adaptive practices in the clinic environment (Weiner, 2009); HCWs’ capacity to provide individualized care may have less to do with available resources or the policy architecture, than with its ‘soft periphery’ (Langley and Denis, 2011) that allows for discretionary decision-making space and power within specific contexts. Our focus is on decentralized DR-TB care in South Africa, yet health systems actors’ ‘tinkering’ occurs in most settings where new service delivery initiatives are introduced. Studying the ‘micro-politics’ of implementing interventions to improve health care delivery (Langley and Denis, 2011) is a relatively recent turn in high-income countries, but still rare in studies on health systems in LMIC. Most literature on ‘organisational readiness’ stems from high-income settings that do not share the resource constraints and challenges of many LMIC health systems; accordingly, assumptions regarding both individual and collective agency may not apply. Our study suggests that assessing front-line health workers’ capacity to cope, adapt and innovate within particular organizational contexts may enhance existing tools to assess ‘systems readiness’ for implementing policy initiatives. Currently, standardized assessments use binary checklists to establish the presence or absence of components needed to deliver a service. Relatively simple adjustments to both tool and method of application would enable assessment of organizational and individual capacity to withstand negative shocks (resilience) and be prepared for change (readiness). A limited set of open-ended questions or vignette scenarios might be added to assess when and why resource gaps occur, how they compromise service delivery, and what HCWs and managers do to address these situations. In addition to understanding how health systems actors ‘get by’ and cope with what they have, it is important to document instances where they go ‘over and beyond’ what is required to provide patient-centred care. While these instances may partly reflect the social fabric of health facilities, they conversely may also signal the potential for stress and exploitation in chronically under-resourced settings. Study strengths and limitations Our study draws on a large data set collected over 2 years by researchers with extensive familiarity with the changing landscape of TB and DR-TB care and its delivery. Triangulation of methods, discussions among research team members following each phase of data collection, and iterative consultations with staff at research sites increased trustworthiness of the data obtained. We focused on three out of nine provinces in South Africa and did not collect data from private health care facilities; furthermore, data on which this paper is based stems mainly from our interviews in WC and KZN. However, earlier research and the insights gained from Phase 1 (Cox et al. 2017a; Moshabela et al 2020) provide evidence that the practices observed across urban and rural facilities in these two provinces were fairly uniform and representative of early responses to the policy across the country. We note, however, that the majority of our cited examples stem from WC. This is likely due to the fact that decentralization of DR-TB care was already considerably advanced in WC by the time the national strategy was released in 2011 as compared with KZN, for example (Vanleeuw et al., 2020). WC also has a history of implementing their own policies, as can be seen in the early introduction of decentralized delivery of ARV therapy for HIV (WHO, 2003). Although we argue that ‘tinkering’ may provide clues as to why some systems are more ‘ready’ to implement policy than others despite resource gaps, the study this paper draws on did not explicitly set out to compare early vs delayed implementation of policy in the districts studied. Furthermore, while our study focused on positive practices supporting decentralized DR-TB care, we are aware that these may be difficult to sustain, creating an unacceptable burden for some HCW who have to ‘make do’ with inadequate resources or support. HCW ‘tinkering’ may also have detrimental consequences for patient care (Mwamba et al., 2018). For example, dispensing medicines to ambulatory DR-TB patients for lengthier time periods may mean HCWs miss the opportunity to monitor side effects, compromising the quality of care provided. Conclusion In a quickly moving landscape of policy, funding and technological developments in DR-TB care in South Africa, HCWs and managers responded to the policy initiative to decentralize DR-TB care through small acts of ‘tinkering’ as well as more deliberate strategies to deliver sustained services. Our focus on ‘tinkering’ illustrates some of ‘the things that people do to make something happen’ (May, 2013) in the implementation of complex interventions. A bottom-up examination of these practices can shed light on the conditions that generate variability in interpretation and ‘successful’ implementation of policy directives, but also raise moral questions about placing accountability for policy implementation on HCW operating in sub-optimal conditions. Our observations support the need to develop actor-oriented frameworks of health systems ‘readiness.’ Currently, piloting of a ‘harmonised approach’ to health facility assessments that intends to overcome the ‘piecemeal’ focus on specific service areas is underway (WHO, 2019), a promising, but limited move in our view. Advancing the field of health systems ‘readiness’ assessment will require more radical revision to include real-time capture of human capacities not only to mitigate systems constraints, but to drive systems change. For TB services in South Africa and elsewhere, acute gaps between rhetoric and reality of people-centred care (Odone et al., 2018; Furin et al., 2020) suggest that close attention to the conditions that promote adaptive capacity as well as the emergence of ‘change agents’ is critical. Conflict of interest statement. None declared. Funding The work presented in this paper was supported by the Joint Health Systems Research Initiative, jointly supported by the Department for International Development (DFID), the Economic and Social Research Council (ESRC), the Medical Research Council (MRC) and the Wellcome Trust [grant number MR/N015924/1]. This UK funded award is part of the EDCTP2 programme supported by the European Union. Ethical approval for the project was obtained through the University of Cape Town Human Research Ethics Committee (HREC REF 350/2016). HC is supported by a Wellcome Trust Fellowship. Endnotes 1 A rapid automated diagnostic test called Xpert MTB/RIF that could detect resistance to rifampicin, a key drug used to treat tuberculosis, was introduced between 2011 and 2013, resulting in an increase in the number of individuals identified with rifampicin-resistant TB. In 2014, bedaquiline—a drug with considerably less toxic side-effects than standard regimen—was registered for ‘compassionate use’ in South Africa; thousands of individuals received the drug through the expanded access programme until it was more widely released in 2018 (Ndjeka et al., 2015). 2 The Ideal Clinic programme, launched in South Africa in July 2013, intended to systematically improve the quality of care provided in Primary Health Care facilities. Acknowledgements The authors wish to thank and acknowledge Dr Norbert Ndjeka (SA NDOH) and the health facility staff interviewed in the provinces of the WC, EC, KZN, for their time, inputs and assistance. We gratefully acknowledge Leila Mitrani's contribution to the facility-based data collection. Ethical approval. Ethical clearance was obtained through the Human Research Ethics committee at the University of Cape Town and by the Observational/Interventions Research Ethics Committee of the London School of Hygiene & Tropical Medicine. References Akinyemi O , Harris B, Kawonga M. 2019 . Health system readiness for innovation scale-up: the experience of community-based distribution of injectable contraceptives in Nigeria . BMC Health Services Research 19 : 938. 10.1186/s12913-019-4786-6 Google Scholar OpenURL Placeholder Text WorldCat Crossref Barasa E , Cloete K, Gilson L. 2017 . From bouncing back, to nurturing emergence: reframing the concept of resilience in health systems strengthening . Health Policy and Planning 32 : iii91 – 4 . Google Scholar OpenURL Placeholder Text WorldCat Barasa E , Mbau R, Gilson L. 2018 . What is resilience and how can it be nurtured? A systematic review of empirical literature on organizational resilience . International Journal of Health Policy and Management 7 : 491 – 503 . p Google Scholar OpenURL Placeholder Text WorldCat Birken SA , Bunger AC, Powell BJ et al. . 2017 . Organizational theory for dissemination and implementation research . Implementation Science 12 : 62 . Google Scholar OpenURL Placeholder Text WorldCat Blaauw D , Ambegaokar M, Penn-Kekana L et al. . 2006 . Neither Angels nor Robots: The ‘Dynamic Responses’ of Health Care Workers and the Unintended Effects of Health System Functioning. London : Health System Development Programme . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Colombini M , Alkaiyat A, Shaheen A et al. . 2020 . Exploring health system readiness for adopting interventions to address intimate partner violence: a case study from the occupied Palestinian Territory . Health Policy and Planning 35 : 245 – 56 . Google Scholar OpenURL Placeholder Text WorldCat Cox H , Daniels J, Muller O et al. . 2015 . Impact of decentralized care and the Xpert MTB/RIF test on rifampicin-resistant tuberculosis treatment initiation in Khayelitsha, South Africa . Open Forum Infectious Diseases 2 : ofv014 . Google Scholar OpenURL Placeholder Text WorldCat Cox H , Dickson-Hall L, Jassat W et al. . 2017 . Drug-resistant tuberculosis in South Africa: history, progress and opportunities for achieving universal access to diagnosis and effective treatment . South African Health Review 2017 : 157 – 67 . Google Scholar OpenURL Placeholder Text WorldCat Cox H , Dickson-Hall L, Ndjeka N et al. . 2017a . Delays and loss to follow-up before treatment of drug-resistant tuberculosis following implementation of Xpert MTB/RIF in South Africa: a retrospective cohort study . PLoS Medicine 14 : e1002238 . Google Scholar OpenURL Placeholder Text WorldCat Department of Health, Republic of South Africa . 2011a . Drug-Resistant Tuberculosis, TB & HIV Directorate. Multi-Drug Resistant Tuberculosis: A Policy Framework on Decentralised and Deinstitutionalised Management for South Africa . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Department of Health, Republic of South Africa . 2011b . Management of Drug-Resistant Tuberculosis: Policy Guidelines. Department of Health, Republic of South Africa . 2019 . Multi-Drug Resistant Tuberculosis. A Policy Framework on Decentralised and Deinstitutionalised Management for South Africa Dookie S , Singh S. 2012 . Primary health services at district level in South Africa: a critique of the primary health care approach . BMC Family Practice 13 : 67. 10.1186/1471-2296-13-67 Google Scholar OpenURL Placeholder Text WorldCat Crossref Evans D , Sineke T, Schnippel K et al. . 2018 . Impact of Xpert MTB/RIF and decentralized care on linkage to care and drug-resistant tuberculosis treatment outcomes in Johannesburg, South Africa . BMC Health Services Research 18 : 973. doi:10.1186/s12913-018-3762-x Google Scholar OpenURL Placeholder Text WorldCat Furin J , Loveday M, Hlangu S et al. . 2020 . “A very humiliating illness”: a qualitative study of patient-centered Care for Rifampicin-Resistant Tuberculosis in South Africa . BMC Public Health 20 : 76 . Google Scholar OpenURL Placeholder Text WorldCat Fusheini A , Eyles J. 2016 . Achieving universal health coverage in South Africa through a district health system approach: conflicting ideologies of health care provision . BMC Health Services Research 16 : 558 . Google Scholar OpenURL Placeholder Text WorldCat Ghaffar A , Gilson L, Tomson G, Viergever R, Røttingen JA. 2016 . Where is the policy in health policy and systems research agenda? Bulletin of the World Health Organization 94 : 306 – 8 . Google Scholar OpenURL Placeholder Text WorldCat Gilson L (ed). 2012 . Health Policy and Systems Research: A Methodology Reader. Geneva : Alliance for Health Policy and Systems Research, World Health Organisation . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Gilson L. 2016 . Everyday politics and the leadership of health policy implementation . Health Systems and Reform 2 : 187 – 93 . Google Scholar OpenURL Placeholder Text WorldCat Gilson L , Barasa E, Nxumalo N et al. . 2017 . Everyday resilience in district health systems: emerging insights from the front lines in Kenya and South Africa . BMJ Global Health 2 : e000224 . Google Scholar OpenURL Placeholder Text WorldCat Glaser B , Strauss A. 1967 . The Discovery of Grounded Theory: Strategies for Qualitative Research . Chicago, IL : Aldine . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Hill J, Dickson-Hall L, Grant AD et al. 2020. Drug-resistant tuberculosis patient care journeys in South Africa: a pilot study using routine laboratory data. The International Journal Tuberculosis and Lung Disease 24(1): 83 – 91 . Husain L. 2017 . Policy experimentation and innovation as a response to complexity in China’s management of health reforms . Globalization and Health 13 : 54 . Google Scholar OpenURL Placeholder Text WorldCat Jassat W. 2020. Implementing the decentralised drug-resistant TB policy in South Africa: The politics of resource allocation. Manuscript in preparation. Jeans H , Castillo GE, Thomas S. 2017 . Absorb, Adapt, Transform: Resilience Capacities . Oxford : Oxfam Great Britain . https://oxfamilibrary.openrepository.com/bitstream/handle/10546/620178/gd-resilience-capacities-absorb-adapt-transform-250117-en.pdf? sequence=4&isAllowed=y, accessed 31 October 2020. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Jigjidsuren A , Byambaa T, Altangerel E et al. . 2019 . Free and universal access to primary healthcare in Mongolia: the service availability and readiness assessment . BMC Health Services Research 19 : 129 . Google Scholar OpenURL Placeholder Text WorldCat Langley A , Denis J. 2011 . Beyond evidence: the micropolitics of improvement . BMJ Quality & Safety 20 : i43 – 6 . Google Scholar OpenURL Placeholder Text WorldCat Manu A , Arifeen S, Williams J et al. . 2018 . Assessment of facility readiness for implementing the WHO/UNICEF standards for improving quality of maternal and newborn care in health facilities—experiences from UNICEF’s implementation in three countries of South Asia and sub-Saharan Africa . BMC Health Services Research 18 : 531 . Google Scholar OpenURL Placeholder Text WorldCat May C. 2013 . Towards a general theory of implementation . Implementation Science 8 : 18 . Google Scholar OpenURL Placeholder Text WorldCat May CR , Johnson M, Finch T. 2016 . Implementation, context and complexity . Implementation Science 11 : 141 . Google Scholar OpenURL Placeholder Text WorldCat Moshabela, M, Cox, H., Jassat et al. 2020. “We were caught sleeping”: Tensions between ‘control-based’ and ‘access-based’ approaches in the formulation of a national policy to decentralise management of drug-resistant tuberculosis in South Africa. Manuscript under review, BMJ Global Health. Mussie KM , Gradmann C, Manyazewal T. 2020 . Bridging the gap between policy and practice: a qualitative analysis of providers’ field experiences tinkering with directly observed therapy in patients with drug-resistant tuberculosis in Addis Ababa, Ethiopia . BMJ Open 10 : e035272 . Google Scholar OpenURL Placeholder Text WorldCat Mwamba C , Sharma A, Mukamba N et al. . 2018 . ‘They care rudely!’: resourcing and relational health system factors that influence retention in care for people living with HIV in Zambia . BMJ Global Health 3 : e001007 . Google Scholar OpenURL Placeholder Text WorldCat Ndjeka N , Conradie F, Schnippel K et al. . 2015 . Treatment of drug-resistant tuberculosis with bedaquiline in a high HIV prevalence setting: an interim cohort analysis . The International Journal of Tuberculosis and Lung Disease 19 : 979 – 85 . Google Scholar OpenURL Placeholder Text WorldCat Odone A , Roberts B, Dara M et al. . 2018 . People and patient-centered care for tuberculosis: models of care for tuberculosis . The International Journal of Tuberculosis and Lung Disease 22 : 133 – 8 . Google Scholar OpenURL Placeholder Text WorldCat Padayatchi N , Friedland G. 2008 . Decentralised management of drug-resistant tuberculosis (MDR- and XDR-TB) in South Africa: an alternative model of care . The International Journal of Tuberculosis and Lung Disease 12 : 978 – 80 . Google Scholar OpenURL Placeholder Text WorldCat Saidi T , Salie F, Douglas T. 2017 . Towards understanding the drivers of policy change: a case study of infection control policies for multi-drug resistant tuberculosis in South Africa . Health Research Policy and Systems 15 : 41 . Google Scholar OpenURL Placeholder Text WorldCat Scaccia JP , Cook BS, Lamont A et al. . 2015 . A practical implementation science heuristic for organizational readiness: R = MC2 . Journal of Community Psychology 43 : 484 – 501 . Google Scholar OpenURL Placeholder Text WorldCat Sheikh K , Gilson L, Agyepong IA et al. . 2011 . Building the field of health policy and systems research: framing the questions . PLoS Medicine 8 : e1001073 . Google Scholar OpenURL Placeholder Text WorldCat Ssensamba JT , Mukuru M, Nakafeero M et al. . 2019 . Health systems readiness to provide geriatric friendly care services in Uganda: a cross-sectional study . BMC Geriatrics 19 : 256 . Google Scholar OpenURL Placeholder Text WorldCat Stone D. 2017 . Understanding the transfer of policy failure: bricolage, experimentalism and translation . Policy & Politics 45 : 55 – 70 . Google Scholar OpenURL Placeholder Text WorldCat Timmermans S , Berg M. 1997 . Standardization in action: achieving local universality through medical protocols . Social Studies of Science 27 : 273 – 305 . Google Scholar OpenURL Placeholder Text WorldCat Van Olmen J , Criel B, Bhojani U et al. . 2012 . The health systems dynamics framework . Health, Culture and Society 2 : 1 – 12 . Google Scholar OpenURL Placeholder Text WorldCat Vanleeuw L , Atkins S, Zembe-Mkabile W et al. . 2020 . Provider perspectives of the introduction and implementation of care for drug-resistant tuberculosis patients in district-level facilities in South Africa: a qualitative study . BMJ Open 10 : e032591 . Google Scholar OpenURL Placeholder Text WorldCat Weiner B. 2009 . A theory of organizational readiness for change . Implementation Science 4 :67. 10.1186/1748-5908-4-67 Google Scholar OpenURL Placeholder Text WorldCat Crossref World Health Organization (WHO) . 2003 . Antiretroviral Therapy in Primary Health Care: Experience of the Khayelitsha Programme in South Africa . Geneva : WHO . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC World Health Organization (WHO) . 2015 . Service Availability and Readiness Assessment (SARA): An Annual Monitoring System for Service Delivery . Geneva : WHO . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC World Health Organization (WHO) . 2016 . Towards Universal Health Coverage: Report of the Evaluation of South Africa Drug Resistant TB Programme and Its Implementation of the Policy Framework on Decentralised and Deinstitutionalised Management of Multidrug Resistant TB . Pretoria : WHO . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC World Health Organization (WHO) . 2018 . Global Tuberculosis Report 2018 [Internet]. World Health Organization. 231 p. http://apps.who.int/iris/handle/10665/274453, accessed 4 December 2018. World Health Organization (WHO) . 2019 . Standardised Health Facility Survey Modules: A Harmonised Approach to Health Facility Assessments (HFA) . Geneva : WHO . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Zurovac D , Machini B, Kiptui R et al. . 2018 . Monitoring health systems readiness and inpatient malaria case-management at Kenyan county hospitals . Malaria Journal 17 : 213 . Google Scholar OpenURL Placeholder Text WorldCat Appendix 1 List of themes Key theme . Sub-themes . Systems readiness gaps DR-TB policy vs practice discursive gaps Lack of operational guidelines Lack of funding Adaptive capacity Re-organizing mode of service delivery Addressing resource shortages and gaps (drugs, equipment, staff) Local innovations Novel communication and knowledge transfer mechanisms Monitoring and motivating patients on treatment Extending job/care parameters for specific populations Key theme . Sub-themes . Systems readiness gaps DR-TB policy vs practice discursive gaps Lack of operational guidelines Lack of funding Adaptive capacity Re-organizing mode of service delivery Addressing resource shortages and gaps (drugs, equipment, staff) Local innovations Novel communication and knowledge transfer mechanisms Monitoring and motivating patients on treatment Extending job/care parameters for specific populations Open in new tab Key theme . Sub-themes . Systems readiness gaps DR-TB policy vs practice discursive gaps Lack of operational guidelines Lack of funding Adaptive capacity Re-organizing mode of service delivery Addressing resource shortages and gaps (drugs, equipment, staff) Local innovations Novel communication and knowledge transfer mechanisms Monitoring and motivating patients on treatment Extending job/care parameters for specific populations Key theme . Sub-themes . Systems readiness gaps DR-TB policy vs practice discursive gaps Lack of operational guidelines Lack of funding Adaptive capacity Re-organizing mode of service delivery Addressing resource shortages and gaps (drugs, equipment, staff) Local innovations Novel communication and knowledge transfer mechanisms Monitoring and motivating patients on treatment Extending job/care parameters for specific populations Open in new tab Appendix 2 Open in new tabDownload slide Open in new tabDownload slide Source: Department of Health, Republic of South Africa (2019, p. 17). © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine.

Muhoza, Pierre; Koffi, Alain K; Anglewicz, Philip; Gichangi, Peter; Guiella, Georges; OlaOlorun, Funmilola; Omoluabi, Elizabeth; Sodani, P R; Thiongo, Mary; Akilimali, Pierre; Tsui, Amy; Radloff, Scott

doi: 10.1093/heapol/czaa197pmid: 33454786

Abstract Approximately 214 million women of reproductive age lack adequate access to contraception for their family planning needs, yet patterns of contraceptive availability have seldom been examined. With growing demand for contraceptives in some areas, low contraceptive method availability and stockouts are thought to be major drivers of unmet need among women of reproductive age, though evidence for this is limited. In this research, we examined trends in stockouts, method availability and consumption of specific contraceptive methods in urban areas of four sub-Saharan African countries (Burkina Faso, Democratic Republic of Congo, Kenya and Nigeria) and India. We used representative survey data from the Performance Monitoring for Action Agile Project that were collected in quarterly intervals at service delivery points (SDP) stratified by sector (public vs private), with all countries having five to six quarters of surveys between 2017 and 2019. Among SDPs that offer family planning, we calculated the percentage offering at least one type of modern contraceptive method (MCM) for each country and quarter, and by sector. We examined trends in the percentage of SDPs with stockouts and which currently offer condoms, emergency contraception, oral pills, injectables, intrauterine devices and implants. We also examined trends of client visits for specific methods and the resulting estimated protection from pregnancy by quarter and country. Across all countries, the vast majority of SDPs had at least one type of MCM in-stock during the study period. We find that the frequency of stockouts varies by method and sector and is much more dynamic than previously thought. While the availability and distribution of long-acting reversible contraceptives (LARCs) were limited compared to other methods across countries, LARCs nonetheless consistently accounted for a larger portion of couple years of protection. We discuss findings that show the importance of engaging the private sector towards achieving global and national family planning goals. Family planning, contraception, reproductive health, survey methods, surveillance, public/private, health services KEY MESSAGES PMA Agile is a useful platform for routinely tracking the family planning supply environment in urban/suburban areas of low- and middle-income countries Changes in contraceptive supply are common and vary by method and sector (public vs private) Pharmacies and drug shops could be further leveraged in addressing unmet need in contraceptives Introduction Providing access to contraception is critical for achieving several global health outcomes, such as reducing maternal mortality, maximizing the health benefits of birth spacing, and promoting the economic empowerment of women (Stover and Ross, 2010; Ahmedet al., 2012; Do and Kurimoto, 2012). Considering the efforts of family planning (FP) programs worldwide to address the unmet need for contraception, progress has been slow in many low-resource countries (Darroch et al., 2011). Millions of women continue to lack access to contraception due to ‘supply-side’ barriers such as poor accessibility to health facilities, low levels of contraceptive method availability and contraceptive stockouts (Hubacher et al., 2008; Chandra-Mouliet al., 2014). In 2017, more than 214 million women of reproductive age worldwide lacked adequate access to contraception (Guttmacher Institute, 2017). Stockouts likely play a role in this lack of access, though the relationship between contraceptive availability and use remains poorly understood. Existing evidence suggests that contraceptive stockouts and method availability range widely across countries and across methods (Ali et al., 2018; Babazadeh et al., 2018; Zimmerman et al., 2019). Different countries may experience different contexts of varying supply chain challenges that ultimately impact method availability and stockouts differentially. The policy environment on family planning also likely contributes to the variation in contraceptive stockouts and method availability across health delivery sectors. For instance, a 2018 USAID report suggested that among the 36 countries providing information on supply chain challenges, 15% cited formal and informal policy barriers that hinder the ability of the private sector to provide contraceptive methods (USAID, 2018). Stockouts and low method availability restrict choice in contraception, forcing individuals to choose methods that may not suit their preferences and needs. For example, a woman’s contraceptive preferences may vary depending on spousal preferences, cultural acceptability of contraceptive methods, socioeconomic influences, her motivation for spacing or limiting births and her preference for hormonal or non-hormonal methods (Brown and Eisenberg, 1995; Wang and Mallick, 2019). In limiting contraceptive method choice, stockouts and low method availability promote conditions that ultimately discourage the use of modern methods and likely lead to increased contraceptive discontinuation (Ross and Hardee, 2013; Grindlay et al., 2016). Therefore, ensuring an adequate range of methods at various levels of the health care system is crucial to guarantee that individuals and couples can select their contraceptive method of choice, thereby allowing them to achieve their fertility goals. Despite the importance of measuring stockouts, existing research is limited. Data on the family planning supply side are not systematically available: the Service Provision Assessment (SPA) of the Demographic and Health Surveys (DHS) occurs at irregular intervals and only in a limited set of countries where DHS operates (DHS Program, 2017). Overall, data on stockouts by contraceptive method are scarce, and only a few countries are able to monitor stockouts routinely at the facility level (FP, 2015, 2020). Furthermore, since there have not been many studies that have examined contraceptive stockouts in the context of client volume, it remains unclear whether stockouts are primarily a response to a break-down in the supply chain or increase in demand, or both. It is plausible that stockouts may be broadly problematic for the most popularly used methods, especially short-acting methods that require frequent revisits to maintain protection against unintended pregnancy. Due to these data limitations, little is known about patterns in stockouts, or the FP supply-side picture overall. Data from the PMA Agile surveys provide an opportunity to address existing knowledge gaps on contraceptive stockouts. A unique feature of the PMA Agile platform is a design that allows the monitoring of progress at the subnational level of each country through the collection of facility-level data in quarterly intervals each year. They thus enable a continuous assessment of contraceptive commodity and service provision in urban facilities, both private and public. In this research, we describe trends in stockouts, method availability and consumption of specific contraceptive methods using locally representative data from facilities in urban areas of Burkina Faso, the Democratic Republic of Congo (DRC), Kenya, Nigeria and India. We expect that this information may help monitor progress towards FP2020 goals (Brown et al., 2014) and inform cross-country strategies to anticipate, reduce and prevent stockouts. Methods Study design, sampling and data collection Data for this study come from the Performance Monitoring for Action (PMA) Agile Project. PMA Agile is a continuous data monitoring and evaluation system that collects data on family planning service delivery and consumption through quarterly public and private health facility surveys and semi-annual client exit interviews (CEIs) in urban areas. A phone follow-up survey is conducted with consenting female clients four months after their interviews (www.pmadata.org/technical-areas/pma-agile). It operates in selected urban areas of six countries, Burkina Faso, DRC, India, Kenya, Niger and Nigeria. PMA Agile has multiple sites in urban areas of four countries: Lagos, Kano and Ogun in Nigeria; Uasin Gishu, Migori and Kericho in Kenya; Indore, Firozabad and Puri in India; Ouagadougou and Koudougou in Burkina Faso. There is one PMA Agile site in each of the two remaining countries: Kinshasa in the DRC and Niamey in Niger. We note that the sites in Kenya are a mix of urban and suburban sites. Data from Niger are excluded from this study since they were not available at the time of this analysis. The population estimates for each of the aforementioned sites are included in the PMA Agile protocol paper (Tsui et al., 2020). The surveys in this study were implemented between November 2017 and December 2019 across countries. Within each country survey, data were collected by locally recruited and trained female resident enumerators (REs). Data collection was implemented using questionnaires powered by smartphone and open source software (Open Data Kit—ODK) technologies. Our analysis included service delivery points (SDPs) data from official listings provided by the country Ministry of Health (MoH) and other government agencies within each PMA Agile study site. The sampling scheme allowed for a 10% non-response rate, and thus a maximum sample of 220 SDPs was randomly selected from each site. The surveys used a two-stage cluster sampling design where SDPs were first stratified by public and private and then further sampled based on probability proportional to size to select facility types with at least 20 SDPs. In smaller areas such as Koudougou where the number of SDPs was relatively limited, the full census of SDPs was used. More information about PMA Agile can be found at the project’s website: www.pmadata.org/technical-areas/pma-agile, and in the PMA Agile protocol paper (Tsui et al., 2020). Measurement The main measures of interest were the provision and demand of specific modern contraceptive methods (MCM) among SDPs offering family planning services. We define contraceptive stockouts as when one or more contraceptive methods are temporarily unavailable at a health facility that routinely provides that method (Grindlay et al., 2016). Method availability, in turn, measures the percentage of health facilities offering a given contraceptive method over a period of time. Information on the country-specific brands of contraceptive methods considered is available on the project’s website and dashboard. The availability of family planning services/products at a given SDP was determined by the response to the survey question, ‘Do you usually offer FP services/products?’. If the SDP provided FP services/products, the respondent was then asked about the availability of specific contraceptive methods. If the SDP reported offering a given contraceptive method, the RE then asked if the method was in-stock on the day of the survey and if there had been a stockout of the method at any point within the 3 months preceding the interview. A method was considered in-stock only if the RE could visually confirm its availability on the day of the survey. To assess the demand of contraceptive methods, the RE requested to see the facility logbook and recorded the total number of family planning visits (new and continuing) in the last completed month for each method. A secondary measure examined in this study was a couple years of protection (CYP). The CYP is a commonly used family planning metric that quantifies the level of protection offered by contraceptives against unintended pregnancies over a period of time (Stover and Ross, 2010). It is obtained by multiplying the quantity of each method distributed to clients by a conversion factor resulting in an estimate of the duration of contraceptive protection provided per unit of that method (USAID, 2019). To obtain the quantity of methods distributed to clients, we combined the total number of contraceptive method units sold with the total number of visits for each method. This was necessary since certain types of SDPs such as pharmacies, drugstores and chemists track product distribution as sales whereas other facility types use clinical visits as measures of client volume. To standardize these different measures, we assumed that clients received six condoms, four sachets of oral pills or one unit of the other methods for each relevant visit to an eligible facility. We defined the following contraceptive methods as modern according to the WHO (WHO, 2018): oral pills, intrauterine devices (IUDs), injectables, male and female condoms, implants and emergency contraception. Though PMA Agile collects SDP data on other methods such as contraceptive beads, foam/jelly and sterilization, these are excluded from this study since the first two methods (beads, foam/jelly) are uncommon in all PMA Agile settings, and sterilization is a medical procedure for which stock does not directly apply as a measure. We note that PMA Agile did not collect information on implants in India as the method was not offered at the time of this study and is currently undergoing consideration for introduction into the national family planning program (Joshi et al., 2019). Data analysis We limited the analysis to SDPs offering family planning and stratified our analyses by sector (i.e. public vs private). The proportions of surveyed SDPs offering family planning for each country are shown in Supplementary Table S1. For each country, we first tabulated SDP types by quarter for those with stock of at least one type of modern contraceptive method. Though the standard in the field is to calculate the percentage of SDPs offering at least three to five methods in order to evaluate the level of client choice in methods, we found these thresholds to be too restrictive for analyses focused on assessing method stockouts. Next, we calculated the quarterly percentages of SDPs that typically offer specific contraceptive methods, those that had the indexed methods in stock, out-of-stock or had experienced a stockout within the 3-month period preceding the survey. We then estimated the total monthly volume of clients visiting SDPs for specific contraceptive methods. Finally, we estimated the percent contribution of individual contraceptive methods to overall CYP units produced. Based on the sampling procedures, we constructed facility weights using SDP selection probabilities. The weighting process took into account the quarterly changes in non-response rates and any SDP classification change by public/private or facility type that happened over time. We report weighted results to account for the stratified two-stage cluster sampling design and the variances of the estimates are adjusted accordingly using Taylor series linearization. We used Stata version 14.2 (StataCorp LLC, College Station, TX) with the SVY command to conduct design‐based analyses that accounted for stratification, clustering and probability of selection of the SDPs (Heeringa et al., 2017). Results In Table 1, we present the number of SDPs by country and quarter, separately for public and private SDPs. We also show the percentage of each offering at least one family planning method in-stock, again by country, quarter and public/private. Table 1 Percentage of SDPs (by type, country and quarter) with at least one type of modern contraceptive method in stock at time of survey Country name . Quarters . Public SDPs . Private SDPs . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . N . % . N . % . N . % . N . % . N . % . . . N . % . N . % . N . % . N . % . N . % . . . Burkina Faso  Hospital 2 100.0 2 73.9 2 100.0 2 72.7 1 100.0 2 0.0 0 0.0 2 0.0 2 0.0 2 0.0  Medical Center/ Health Clinic/ Health Center 27 100.0 26 98.0 27 100.0 30 93.3 24 95.3 49 69.5 44 68.7 51 63.9 43 72.5 45 72.8  Maternity clinic 13 56.7 11 49.9 13 52.6 11 48.2 11 42.7 4 50.0 4 55.9 3 66.7 3 66.7 4 50.0  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 54 100.0 60 90.6 55 100.0 52 96.9 60 100.0  All SDP types 42 86.9 39 82.7 41 84.5 43 81.4 36 78.9 109 83.0 108 80.6 111 81.2 100 83.8 111 85.7 DRC  Hospital 13 81.8 19 93.8 16 92.9 17 100.0 17 78.6 1 100.0 1 100.0 1 100.0 1 100.0 1 100.0  Health Center 64 78.9 54 93.5 60 90.5 59 96.0 60 94.1 11 90.0 11 100.0 11 100.0 12 81.8 12 100.0  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 58 92.9 62 93.2 54 98.5 54 93.6 53 95.2  All SDP types 77 86.9 73 93.5 76 91.0 76 96.9 77 90.7 70 92.6 74 94.2 66 98.8 67 91.4 66 95.9 India  Hospital 5 88.9 4 100.0 4 100.0 4 100.0 4 100.0 3 100.0 57 24.4 61 34.1 64 17.5 59 10.6 62 13.4 55 10.9  Health Clinic/Health Center 17 87.0 16 93.1 13 96.6 13 96.8 13 96.8 11 100.0 21 31.9 22 30.1 21 32.1 23 20.1 23 20.1 19 2.0  Pharmacy/Drugstore 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 217 94.0 224 96.8 216 96.1 217 78.0 216 78.8 168 68.6  Dispensary 4 100.0 3 100.0 4 100.0 3 100.0 3 100.0 3 100.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0  All SDP types 25 89.3 23 95.4 21 97.8 20 97.9 20 97.9 17 100.0 295 76.1 307 79.6 301 74.9 299 60.1 301 61.0 242 50.2 Kenya  Hospital 14 100.0 14 95.5 14 90.9 14 100.0 14 90.9 14 90.3 15 84.0 16 90.2 15 95.9 16 100.0 14 100.0 15 100.0  Health Clinic/Health Center 34 95.8 33 100.0 33 98.1 33 96.1 33 97.9 35 92.1 63 94.7 66 97.0 64 96.4 65 95.8 67 93.0 66 94.8  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 166 100.0 181 99.5 195 100.0 202 99.5 188 99.5 198 99.5  Dispensary 184 97.7 175 99.3 172 99.5 173 97.5 171 98.6 174 97.9 22 91.9 25 84.7 21 100.0 23 100.0 22 94.6 22 100.0  All SDP types 232 97.6 222 99.1 219 98.7 220 97.5 218 98.0 223 96.5 266 97.2 288 97.1 295 99.0 306 98.8 293 97.6 301 98.6 Nigeria  Hospital 5 94.9 6 100.0 6 95.0 6 95.1 5 95.0 5 100.0 133 70.5 164 78.8 169 69.9 168 59.5 151 56.9 145 63.9  Health Post/Health Center 75 89.3 83 92.8 83 91.7 83 91.6 77 94.3 74 91.2 30 26.7 43 56.3 34 84.1 34 88.5 30 85.6 30 93.4  Maternity clinic 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 23 63.0 29 63.9 31 78.7 30 53.2 28 63.7 27 53.5  Pharmacy/Chemist 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 138 96.5 168 99.7 180 99.5 180 97.1 164 99.0 158 99.0  All SDP types 80 89.7 89 93.3 89 91.9 89 91.8 82 94.3 79 91.8 324 76.9 404 84.0 414 84.6 412 77.9 373 78.3 360 81.0 Country name . Quarters . Public SDPs . Private SDPs . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . N . % . N . % . N . % . N . % . N . % . . . N . % . N . % . N . % . N . % . N . % . . . Burkina Faso  Hospital 2 100.0 2 73.9 2 100.0 2 72.7 1 100.0 2 0.0 0 0.0 2 0.0 2 0.0 2 0.0  Medical Center/ Health Clinic/ Health Center 27 100.0 26 98.0 27 100.0 30 93.3 24 95.3 49 69.5 44 68.7 51 63.9 43 72.5 45 72.8  Maternity clinic 13 56.7 11 49.9 13 52.6 11 48.2 11 42.7 4 50.0 4 55.9 3 66.7 3 66.7 4 50.0  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 54 100.0 60 90.6 55 100.0 52 96.9 60 100.0  All SDP types 42 86.9 39 82.7 41 84.5 43 81.4 36 78.9 109 83.0 108 80.6 111 81.2 100 83.8 111 85.7 DRC  Hospital 13 81.8 19 93.8 16 92.9 17 100.0 17 78.6 1 100.0 1 100.0 1 100.0 1 100.0 1 100.0  Health Center 64 78.9 54 93.5 60 90.5 59 96.0 60 94.1 11 90.0 11 100.0 11 100.0 12 81.8 12 100.0  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 58 92.9 62 93.2 54 98.5 54 93.6 53 95.2  All SDP types 77 86.9 73 93.5 76 91.0 76 96.9 77 90.7 70 92.6 74 94.2 66 98.8 67 91.4 66 95.9 India  Hospital 5 88.9 4 100.0 4 100.0 4 100.0 4 100.0 3 100.0 57 24.4 61 34.1 64 17.5 59 10.6 62 13.4 55 10.9  Health Clinic/Health Center 17 87.0 16 93.1 13 96.6 13 96.8 13 96.8 11 100.0 21 31.9 22 30.1 21 32.1 23 20.1 23 20.1 19 2.0  Pharmacy/Drugstore 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 217 94.0 224 96.8 216 96.1 217 78.0 216 78.8 168 68.6  Dispensary 4 100.0 3 100.0 4 100.0 3 100.0 3 100.0 3 100.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0  All SDP types 25 89.3 23 95.4 21 97.8 20 97.9 20 97.9 17 100.0 295 76.1 307 79.6 301 74.9 299 60.1 301 61.0 242 50.2 Kenya  Hospital 14 100.0 14 95.5 14 90.9 14 100.0 14 90.9 14 90.3 15 84.0 16 90.2 15 95.9 16 100.0 14 100.0 15 100.0  Health Clinic/Health Center 34 95.8 33 100.0 33 98.1 33 96.1 33 97.9 35 92.1 63 94.7 66 97.0 64 96.4 65 95.8 67 93.0 66 94.8  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 166 100.0 181 99.5 195 100.0 202 99.5 188 99.5 198 99.5  Dispensary 184 97.7 175 99.3 172 99.5 173 97.5 171 98.6 174 97.9 22 91.9 25 84.7 21 100.0 23 100.0 22 94.6 22 100.0  All SDP types 232 97.6 222 99.1 219 98.7 220 97.5 218 98.0 223 96.5 266 97.2 288 97.1 295 99.0 306 98.8 293 97.6 301 98.6 Nigeria  Hospital 5 94.9 6 100.0 6 95.0 6 95.1 5 95.0 5 100.0 133 70.5 164 78.8 169 69.9 168 59.5 151 56.9 145 63.9  Health Post/Health Center 75 89.3 83 92.8 83 91.7 83 91.6 77 94.3 74 91.2 30 26.7 43 56.3 34 84.1 34 88.5 30 85.6 30 93.4  Maternity clinic 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 23 63.0 29 63.9 31 78.7 30 53.2 28 63.7 27 53.5  Pharmacy/Chemist 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 138 96.5 168 99.7 180 99.5 180 97.1 164 99.0 158 99.0  All SDP types 80 89.7 89 93.3 89 91.9 89 91.8 82 94.3 79 91.8 324 76.9 404 84.0 414 84.6 412 77.9 373 78.3 360 81.0 N represents the total weighted number of SDPs offering family planning Open in new tab Table 1 Percentage of SDPs (by type, country and quarter) with at least one type of modern contraceptive method in stock at time of survey Country name . Quarters . Public SDPs . Private SDPs . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . N . % . N . % . N . % . N . % . N . % . . . N . % . N . % . N . % . N . % . N . % . . . Burkina Faso  Hospital 2 100.0 2 73.9 2 100.0 2 72.7 1 100.0 2 0.0 0 0.0 2 0.0 2 0.0 2 0.0  Medical Center/ Health Clinic/ Health Center 27 100.0 26 98.0 27 100.0 30 93.3 24 95.3 49 69.5 44 68.7 51 63.9 43 72.5 45 72.8  Maternity clinic 13 56.7 11 49.9 13 52.6 11 48.2 11 42.7 4 50.0 4 55.9 3 66.7 3 66.7 4 50.0  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 54 100.0 60 90.6 55 100.0 52 96.9 60 100.0  All SDP types 42 86.9 39 82.7 41 84.5 43 81.4 36 78.9 109 83.0 108 80.6 111 81.2 100 83.8 111 85.7 DRC  Hospital 13 81.8 19 93.8 16 92.9 17 100.0 17 78.6 1 100.0 1 100.0 1 100.0 1 100.0 1 100.0  Health Center 64 78.9 54 93.5 60 90.5 59 96.0 60 94.1 11 90.0 11 100.0 11 100.0 12 81.8 12 100.0  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 58 92.9 62 93.2 54 98.5 54 93.6 53 95.2  All SDP types 77 86.9 73 93.5 76 91.0 76 96.9 77 90.7 70 92.6 74 94.2 66 98.8 67 91.4 66 95.9 India  Hospital 5 88.9 4 100.0 4 100.0 4 100.0 4 100.0 3 100.0 57 24.4 61 34.1 64 17.5 59 10.6 62 13.4 55 10.9  Health Clinic/Health Center 17 87.0 16 93.1 13 96.6 13 96.8 13 96.8 11 100.0 21 31.9 22 30.1 21 32.1 23 20.1 23 20.1 19 2.0  Pharmacy/Drugstore 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 217 94.0 224 96.8 216 96.1 217 78.0 216 78.8 168 68.6  Dispensary 4 100.0 3 100.0 4 100.0 3 100.0 3 100.0 3 100.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0  All SDP types 25 89.3 23 95.4 21 97.8 20 97.9 20 97.9 17 100.0 295 76.1 307 79.6 301 74.9 299 60.1 301 61.0 242 50.2 Kenya  Hospital 14 100.0 14 95.5 14 90.9 14 100.0 14 90.9 14 90.3 15 84.0 16 90.2 15 95.9 16 100.0 14 100.0 15 100.0  Health Clinic/Health Center 34 95.8 33 100.0 33 98.1 33 96.1 33 97.9 35 92.1 63 94.7 66 97.0 64 96.4 65 95.8 67 93.0 66 94.8  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 166 100.0 181 99.5 195 100.0 202 99.5 188 99.5 198 99.5  Dispensary 184 97.7 175 99.3 172 99.5 173 97.5 171 98.6 174 97.9 22 91.9 25 84.7 21 100.0 23 100.0 22 94.6 22 100.0  All SDP types 232 97.6 222 99.1 219 98.7 220 97.5 218 98.0 223 96.5 266 97.2 288 97.1 295 99.0 306 98.8 293 97.6 301 98.6 Nigeria  Hospital 5 94.9 6 100.0 6 95.0 6 95.1 5 95.0 5 100.0 133 70.5 164 78.8 169 69.9 168 59.5 151 56.9 145 63.9  Health Post/Health Center 75 89.3 83 92.8 83 91.7 83 91.6 77 94.3 74 91.2 30 26.7 43 56.3 34 84.1 34 88.5 30 85.6 30 93.4  Maternity clinic 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 23 63.0 29 63.9 31 78.7 30 53.2 28 63.7 27 53.5  Pharmacy/Chemist 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 138 96.5 168 99.7 180 99.5 180 97.1 164 99.0 158 99.0  All SDP types 80 89.7 89 93.3 89 91.9 89 91.8 82 94.3 79 91.8 324 76.9 404 84.0 414 84.6 412 77.9 373 78.3 360 81.0 Country name . Quarters . Public SDPs . Private SDPs . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . N . % . N . % . N . % . N . % . N . % . . . N . % . N . % . N . % . N . % . N . % . . . Burkina Faso  Hospital 2 100.0 2 73.9 2 100.0 2 72.7 1 100.0 2 0.0 0 0.0 2 0.0 2 0.0 2 0.0  Medical Center/ Health Clinic/ Health Center 27 100.0 26 98.0 27 100.0 30 93.3 24 95.3 49 69.5 44 68.7 51 63.9 43 72.5 45 72.8  Maternity clinic 13 56.7 11 49.9 13 52.6 11 48.2 11 42.7 4 50.0 4 55.9 3 66.7 3 66.7 4 50.0  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 54 100.0 60 90.6 55 100.0 52 96.9 60 100.0  All SDP types 42 86.9 39 82.7 41 84.5 43 81.4 36 78.9 109 83.0 108 80.6 111 81.2 100 83.8 111 85.7 DRC  Hospital 13 81.8 19 93.8 16 92.9 17 100.0 17 78.6 1 100.0 1 100.0 1 100.0 1 100.0 1 100.0  Health Center 64 78.9 54 93.5 60 90.5 59 96.0 60 94.1 11 90.0 11 100.0 11 100.0 12 81.8 12 100.0  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 58 92.9 62 93.2 54 98.5 54 93.6 53 95.2  All SDP types 77 86.9 73 93.5 76 91.0 76 96.9 77 90.7 70 92.6 74 94.2 66 98.8 67 91.4 66 95.9 India  Hospital 5 88.9 4 100.0 4 100.0 4 100.0 4 100.0 3 100.0 57 24.4 61 34.1 64 17.5 59 10.6 62 13.4 55 10.9  Health Clinic/Health Center 17 87.0 16 93.1 13 96.6 13 96.8 13 96.8 11 100.0 21 31.9 22 30.1 21 32.1 23 20.1 23 20.1 19 2.0  Pharmacy/Drugstore 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 217 94.0 224 96.8 216 96.1 217 78.0 216 78.8 168 68.6  Dispensary 4 100.0 3 100.0 4 100.0 3 100.0 3 100.0 3 100.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0  All SDP types 25 89.3 23 95.4 21 97.8 20 97.9 20 97.9 17 100.0 295 76.1 307 79.6 301 74.9 299 60.1 301 61.0 242 50.2 Kenya  Hospital 14 100.0 14 95.5 14 90.9 14 100.0 14 90.9 14 90.3 15 84.0 16 90.2 15 95.9 16 100.0 14 100.0 15 100.0  Health Clinic/Health Center 34 95.8 33 100.0 33 98.1 33 96.1 33 97.9 35 92.1 63 94.7 66 97.0 64 96.4 65 95.8 67 93.0 66 94.8  Pharmacy 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 166 100.0 181 99.5 195 100.0 202 99.5 188 99.5 198 99.5  Dispensary 184 97.7 175 99.3 172 99.5 173 97.5 171 98.6 174 97.9 22 91.9 25 84.7 21 100.0 23 100.0 22 94.6 22 100.0  All SDP types 232 97.6 222 99.1 219 98.7 220 97.5 218 98.0 223 96.5 266 97.2 288 97.1 295 99.0 306 98.8 293 97.6 301 98.6 Nigeria  Hospital 5 94.9 6 100.0 6 95.0 6 95.1 5 95.0 5 100.0 133 70.5 164 78.8 169 69.9 168 59.5 151 56.9 145 63.9  Health Post/Health Center 75 89.3 83 92.8 83 91.7 83 91.6 77 94.3 74 91.2 30 26.7 43 56.3 34 84.1 34 88.5 30 85.6 30 93.4  Maternity clinic 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 23 63.0 29 63.9 31 78.7 30 53.2 28 63.7 27 53.5  Pharmacy/Chemist 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 138 96.5 168 99.7 180 99.5 180 97.1 164 99.0 158 99.0  All SDP types 80 89.7 89 93.3 89 91.9 89 91.8 82 94.3 79 91.8 324 76.9 404 84.0 414 84.6 412 77.9 373 78.3 360 81.0 N represents the total weighted number of SDPs offering family planning Open in new tab Overall, some patterns in the health system structure are evident in Table 1. The number of private facilities is larger than public ones in all countries except DRC. Pharmacies are generally private in all countries. In all countries, public facilities are mostly made up of health posts, health centres and medical centres. The overall number of facilities by type is generally stable across quarters for each country. Of these facilities, the vast majority offers at least one modern method. For example, 86.9% of all public SDPs and 83.0% of all private SDPs in quarter 1 in Burkina Faso offer at least one modern method. In some settings, however, there is variation across SDP facility types in the percentage offering one modern method: relatively few private hospitals or health clinics/centres in India offer at least one method, but the majority of private pharmacies and drugstores offer at least one method. Similarly, most private pharmacies offer at least one method in Burkina Faso but smaller percentages of medical centres/health centres/health clinics do in all quarters. Stockouts and consumption of contraceptive methods Next, in Figures 1–6, we present trends in stockouts for the different types of MCMs (condoms, EC, oral pills, injectables, IUDs and implants), for the public, private and all facilities. In the same figures, we also show the corresponding trends in monthly client volume (in dotted lines). We present this separately for each of the five PMA Agile countries. Figure 1 Open in new tabDownload slide Contraceptive method availability and client volume by quarter and country. NB: Client volume is measured as the total number of visits by method in the month preceding the interview. Error bars represent standard errors Figure 2 Open in new tabDownload slide EC pill. Figure 3 Open in new tabDownload slide Pill. Figure 4 Open in new tabDownload slide Injectables. Figure 5 Open in new tabDownload slide IUD. Figure 6 Open in new tabDownload slide Implant. Condoms In most countries, condoms were more commonly distributed through the public SDPs instead of the private sector, as shown by higher percentages of public SDPs offering condoms and the higher numbers of monthly client visits purchasing condoms from public SDPs. The average percentage of public SDPs not offering condoms was 11.2% compared to 24.8% of the private SDPs. Among the public SDPs, the average client volume for condoms was 6 842 monthly visits, ranging from 178 visits in Burkina Faso to 22 775 in Kenya. Among the private SDPs, the average client volume was 2018 monthly visits ranging from 32 visits in India to 3077 in Kenya. Regardless of SDP managing authority, an average of 7 858 monthly visits for condoms were observed across all countries ranging from 296 in Burkina Faso to 25 852 in Kenya. The average SDP stockout rate for condoms was 4.2% across all countries and it ranged from 2.5% in Burkina Faso and India to 6.8% in the DRC. Among the surveyed public SDPs, the average condom stockout rate was 5.3% ranging from 2.5% in Nigeria to 7.3% in Kenya. The average condom stockout rate among the private SDPs was 4.1% ranging from 2.2% in India to 6.6% in the DRC. Furthermore, an average of 12.5% of all the SDPs across all countries had experienced a condom stockout within the 3 months preceding the survey. This percentage varied from 2.8% in India to 25.6% in the DRC. Emergency contraception We found that the private sector was generally the main distributor of emergency contraceptive pills among the Sub-Saharan African (SSA) countries. In fact, the distribution was conducted almost exclusively through the private sector in Burkina Faso and Nigeria. Among the private facilities with FP, the proportion that did not offer emergency contraception over the study period fluctuated around 28.4% for the DRC, 22.6% for Kenya with that number exceeding 50% for both Burkina Faso and Nigeria. Compared to the other African countries in the study, Burkina Faso had the lowest volume of clients (9 monthly visits on average) purchasing EC with the number progressively decreasing during the study period. In contrast, there were 327, 241 and 221 monthly client visits on average, in the DRC, Nigeria and Kenya, respectively. Across all the African countries, the trends for monthly client visits remained largely highly variable. Contrary to African countries, the provision of EC in Indian sites was largely via the public sector. The proportion of government-owned SDPs that had EC in stock on the day of the survey increased from 30.6% to 87.4%. The percentage of SDPs experiencing a stockout within the 3 months preceding the survey decreased from 3.1% to 1.1%. This positive trend was further strengthened by the almost 60% reduction in SDPs experiencing a stock out in EC between Q2 and Q6. The steady gains and stability observed in method provision within the public sector were not accompanied by similar increases in client visits for EC. Client visits for EC through the private sector in Indian sites were negligible. Oral pills Similar to EC, the distribution of oral pills in India was higher in the public sector compared to the private sector. Over the study period, the proportion of public SDPs that did not offer the method in India decreased from 14.8% to 2.3%. This trend was reversed within the private sector with the percentage of SDPs not offering pills increasing from 39.0% to 57.5%. The average number of monthly visits for pills was 718 among the public SDPs compared to only 13 among the private SDPs. Similarly, in SSA countries, the number of clients obtaining pills was generally higher among public SDPs compared to private SDPs. The ratio of the average monthly client volume among public SDPs to that among private SDPs ranged from 2.2 in Burkina Faso to 4.4 in Nigeria. Compared to other African geographies, Burkina Faso generally experienced lower stockouts of oral pills with an average stockout rate of 3.8% over the study period. In contrast, Kenya experienced an overall average stockout rate of 17% in oral pills with that number approximating 25% among public SDPs. The overall average stockout rate in the DRC was 19.7% (21.2% among public SDPs; 17.8% private SDPs). In Nigeria, the stockout rate for oral pills varied from quarter to quarter at an overall average of 10.6% across surveyed SDPs (10.1% among public SDPs; 10.8% private SDPs). With the exception of Burkina Faso, where the percentage of SDPs not offering oral pills increased slightly from 20.7% to 22.2% over the study period, the availability of oral pills generally improved among SSA countries. In the DRC, the average percentage of SDPs that did not offer oral pills was 29%, decreasing from 40% to 21%; a 47.5% decrease (49.5% decrease among public SDPs compared to 45.7% decrease among private SDPs). In Kenya, the percentage of SDPs not offering oral contraceptives decreased by 25% over the study period to an average of approximately only 10%. Only 7% of the private SDPs reported that they do not offer the method compared to 15% of the public SDPs. An average of only 14% of the private Nigerian SDPs compared to 38% of the public SDPs did not offer the method, the overall average being 34%. Injectables Across all geographies, public SDPs generally offered injectables at higher rates than private SDPs. While only an average of 15.8% of the public SDPs did not offer injectables during the study period, this figure was 56.5% among the private SDPs. The combined average was 45.6%. With the exception of Burkina Faso, net improvements were generally observed in the average number of monthly client visits. The most remarkable improvements were noted among public SDPs in India where the number of client visits for injectables increased by more than 3.5-fold over the first five quarters with an associated 93% decrease in the percentage of SDPs not offering the method. Despite these gains, however, the overall availability of injectables remains low in India due to the low rates of injectable availability among private SDPs. On average, 91% of the private SDPs in India did not offer injectables compared to 31.4% among public SDPs and the combined average of 87.2%. Across all SDPs, the average stockout rate in injectable provision was 13.0% and it varied considerably across geographies ranging from 1.6% in India to 21.6% in the DRC over the study period. The average stockout rate for the public SDPs across all geographies was 13.6% compared to the average stockout rate of 11.9% among private SDPs. Roughly, 5.1% of all SDPs reported a stockout in injectables within the previous 3 months. There was geographic variation in this rate ranging from 0.4% in India to 11.9% in Kenya. Of all public SDPs, 7.4% experienced a stockout during the 3-month period preceding the interview compared to 4.0% of the private SDPs. Intrauterine devices Compared to the short-acting methods described above, IUDs were generally offered at lower levels across the different geographies. Across all geographies, the percentage of SDPs not offering IUDs was 69.2% and it ranged from 58.8% in Nigeria to 88.4% in India. It was also notable that, across the board, the public sector offered IUDs at considerably higher levels compared to the private sector and was furthermore likely to have the method in-stock over a 3-month period. The average percentage of public SDPs not offering IUDs was 36.0% compared to 81.2% of the private SDPs. Nonetheless, with the exception of Nigeria, IUD stockouts were more common among public SDPs. Of the public SDPs that offered IUDs, the average stockout rate was 7.3% compared to 3.7% among the private SDPs. Across all countries, the combined average stockout rate was 5.4%. On average, 1.1% of the IUD-providing SDPs reported a stockout during the 3-month period preceding the interview. This figure was 2.5% among public SDPs compared to 0.7% among the private SDPs. Trends in the client visits for the purchase of IUDs were centred at an average of approximately 372 monthly visits. It varied across countries ranging from an average of 39 monthly visits in the DRC to 776 monthly visits in Kenya. Implants The average percentage of SDPs not offering implants over the study period was 52.0%, ranging from 38.1% in Kenya to 61.0% in Burkina Faso. In all cases, the percentage of public SDPs offering implants was greater than private SDPs. Overall, only 17.1% of the public SDPs did not offer implants compared to 72.4% of the private SDPs. This contrast was starkest in Kenya where a quarterly average of only 3.7% of the public SDPs did not offer implants compared to 64.3% in the private sector (combined average being 38.1%). Among the African countries, the overall average stockout rate was 11.5% ranging from 5.2% in Burkina Faso to 14% in both Kenya and Nigeria. The DRC had an average implant stockout rate of 13.1%. The stockout rates were higher among the public SDPs and varied more across quarters relative to the private SDPs. Among the public SDPs, the average stockout rate was 18.9% compared to 6.6% observed among the private SDPs. Furthermore, whereas the percentage of public SDPs that experienced a stockout in implants 3 months prior to the interview was 6.3%, this number was only 1.3% among the private SDPs. The average number of monthly client visits was 1 913 monthly visits and ranged from 559 visits in Burkina Faso to 4 815 in Kenya. Couple-years of protection of contraceptives CYP data for the different countries are shown in Table 2. Of all the MCMs considered in this study, implants on average accounted for the most CYPs among the public SDPs. With the exception of India where implants are currently not offered, implants accounted for 58.9% of the CYPs provided by the public SDPs across all countries ranging from 55.2% in Burkina Faso to 61.5% in the DRC. This trend did not hold among the private SDPs. Table 2 Quarterly changes in couple years of protection (CYP) by country and method . PUBLIC . PRIVATE . Country namea . Quarters . Quarters . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Burkina Faso N = 34 N = 32 N = 34 N = 29 N = 15 N = 82 N = 73 N = 75 N = 73 N = 79  Condoms 48.4 0.8 4.1 0.1 3.0 0.1 4.3 0.3 2.1 0.2 80.8 9.8 83.9 9.9 70.6 9.4 4.3 0.3 74.9 7.9  Pill 240.3 4.2 200.6 5.4 200.7 4.8 122.8 7.4 64.2 5.2 229.8 27.9 293.8 34.7 221.7 29.7 122.8 7.4 209.0 22.1  Injectable 652.1 11.4 501.1 13.6 516.4 12.4 274.0 16.5 202.7 16.6 131.4 16.0 114.9 13.6 117.0 15.6 274.0 16.5 131.6 13.9  EC pill 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 93.5 11.4 104.4 12.3 75.1 10.0 0.0 0.0 151.7 16  IUD 1545.1 26.9 962.3 26.1 1184.5 28.4 330.3 19.8 293.9 24.0 141.2 17.2 36.3 4.3 122.4 16.4 330.3 19.8 230.0 24.3  Implant 3257.8 56.7 2016.8 54.7 2268.8 54.4 934.3 56.1 661.5 54.0 145.5 17.7 214.5 25.3 141.0 18.9 934.3 56.1 149.0 15.7  Total 5743.7 100 3684.8 100 4173.3 100 1665.6 100 1224.4 100 822.2 100 847.9 100 747.8 100 1666 100 946.2 100 DRC N = 33 N = 58 N = 54 N = 66 N = 70 N = 57 N = 70 N = 61 N = 67 N = 65  Condoms 296.8 20.5 392.9 18.9 379.6 13.9 339.0 17.0 378.7 17.4 127.5 12.3 103.6 6.3 100.6 6.9 158.3 3.1 138.9 4.1  Pill 26.0 1.8 140.1 6.8 107.6 3.9 187.0 9.4 226.5 10.4 528.6 51.1 981.7 59.5 529.2 36.3 1199.0 23.5 773.7 22.7  Injectable 83.4 5.7 140.2 6.8 167.5 6.1 191.1 9.6 197.2 9.1 34.6 3.4 71.2 4.3 136.0 9.3 84.5 1.7 68.2 2.0  EC pill 4.1 0.3 9.6 0.5 76.9 2.8 4.3 0.2 11.2 0.5 92.6 9.0 83.0 5.0 84.0 5.8 92.1 1.8 168.9 5.0  IUD 130.6 9.0 206.1 9.9 224.5 8.2 44.2 2.2 38.6 1.8 28.5 2.8 41.9 2.5 116.4 8.0 3280.7 64.3 994.1 29.2  Implant 910.3 62.7 1185.8 57.2 1783.5 65.1 1225.0 61.5 1321.8 60.8 221.8 21.5 369.3 22.4 492.5 33.8 284.3 5.6 1265.8 37.1  Total 1451.2 100 2074.6 100 2739.6 100 1990.5 100 2174.0 100 1033.5 100 1650.7 100 1458.5 100 5098.8 100 3409.5 100 India N = 24 N = 22 N = 20 N = 19 N = 20 N = 17 N = 205 N = 246 N = 190 N = 213 N = 227 N = 193  Condoms 164.0 12.0 238.5 20.8 41.0 7.3 73.8 4.8 69.6 5.6 62.6 4.2 231.1 14.3 377.7 27.0 336.6 26.3 288.7 8.6 305.5 26.0 134.7 13.7  Pill 303.4 22.1 162.4 14.2 89.3 15.9 217.6 14.2 197.9 16.0 246.4 16.4 319.9 19.8 297.8 21.3 240.7 18.8 117.5 3.5 235.0 20.0 173.0 17.6  Injectable 41.3 3.0 76.2 6.7 90.0 16.0 136.1 8.9 150.7 12.1 30.0 2.0 49.3 3.1 40.0 2.9 30.5 2.4 508.3 15.1 37.9 3.2 40.6 4.1  EC pill 3.3 0.2 7.8 0.7 2.3 0.4 4.0 0.3 6.1 0.5 4.0 0.3 66.9 4.1 79.5 5.7 70.0 5.5 441.5 13.2 101.1 8.6 91.9 9.3  IUD 859.7 62.7 659.6 57.6 339.0 60.4 1102.6 71.9 816.0 65.8 1156.9 77.1 946.2 58.6 602.1 43.1 603.5 47.1 2000.5 59.6 496.3 42.2 544.2 55.3  Total 1371.8 100 1144.6 100 561.6 100 1534.0 100 1240.3 100 1499.9 100 1613.4 100 1397.0 100 1281.4 100 3356.4 100 1175.8 100 984.4 100 Kenya N = 232 N = 223 N = 220 N = 220 N = 218 N = 224 N = 265 N = 283 N = 293 N = 300 N = 292 N = 300  Condoms 1647.8 15.0 1393.7 9.8 1229.6 7.9 1447.7 10.3 1601.4 11.9 887.4 6.4 584.5 5.3 388.7 3.4 587.4 6.0 730.0 7.9 606.2 7.9 579.1 7.2  Pill 291.9 2.7 203.5 1.4 306.4 2.0 318.3 2.3 363.9 2.7 363.6 2.6 978.6 8.9 668.8 5.9 813.3 8.3 852.8 9.3 599.3 7.8 827.9 10.3  Injectable 1260.9 11.5 1558.3 10.9 1465.0 9.5 1724.9 12.3 1753.7 13.1 1861.2 13.4 1565.6 14.2 1465.4 12.9 1708.7 17.5 1674.6 18.2 1533.4 19.9 1502.6 18.7  EC pill 0.4 0.0 1.1 0.0 1.8 0.0 1.6 0.0 0.5 0.0 0.6 0.0 231.3 2.1 703.4 6.2 947.1 9.7 826.2 9.0 687.2 8.9 635.9 7.9  IUD 1534.1 14.0 1961.4 13.8 2922.8 18.9 1769.2 12.6 2121.1 15.8 2035.5 14.6 2193.7 19.9 3259.6 28.7 1575.0 16.1 1013.8 11.0 1041.0 13.5 1177.6 14.6  Implant 6221.0 56.8 9136.3 64.1 9553.3 61.7 8755.0 62.5 7591.0 56.5 8768.0 63.0 5472.0 49.6 4860.3 42.8 4159.8 42.5 4104.0 44.6 3230.5 42.0 3326.8 41.3  Total 10956.1 100 14254.2 100 15479 100 14016 100 13432 100 13916 100 11025.7 100 11346.2 100 9791.3 100 9201.5 100 7697.6 100 8049.9 100 Nigeria N = 79 N = 87 N = 88 N = 88 N = 82 N = 79 N = 270 N = 325 N = 342 N = 338 N = 302 N = 285  Condoms 271.0 4.7 232.1 4.0 174.9 3.4 248.6 4.1 209.1 4.0 243.0 4.4 235.2 4.7 230.1 4.1 170.1 3.7 187.3 5.0 205.8 5.3 281.4 7.8  Pill 152.7 2.7 198.5 3.4 222.6 4.3 181.8 3.0 194.6 3.7 238.7 4.3 328.5 6.6 301.4 5.4 302.8 6.5 281.1 7.5 349.1 8.9 299.8 8.4  Injectable 499.7 8.7 606.1 10.5 549.8 10.7 677.6 11.3 691.6 13.1 724.7 13.1 287.7 5.8 391.0 7.0 409.8 8.9 360.2 9.6 522.4 13.4 303.6 8.5  EC pill 0.0 0.0 0.1 0.0 1.0 0.0 0.4 0.0 0.0 0.0 0.8 0.0 153.9 3.1 206.9 3.7 163.0 3.5 248.1 6.6 511.0 13.1 319.2 8.9  IUD 1103.1 19.2 1491.8 25.8 1013.8 19.7 1779.3 29.6 1299.5 24.6 994.1 17.9 2499.2 50.3 2984.5 53.1 2261.4 48.9 1587.0 42.3 845.5 21.7 753.9 21.0  Implant 3707.8 64.7 3246.5 56.2 3184.8 61.9 3115.5 51.9 2883.3 54.6 3337.3 60.3 1460.3 29.4 1504.3 26.8 1320.8 28.5 1085.3 28.9 1470.5 37.7 1627.0 45.4  Total 5734.3 100 5775.0 100 5146.8 100 6003.1 100 5278.1 100 5538.5 100 4964.7 100 5618.2 100 4627.8 100 3748.8 100 3904.3 100 3584.9 100 . PUBLIC . PRIVATE . Country namea . Quarters . Quarters . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Burkina Faso N = 34 N = 32 N = 34 N = 29 N = 15 N = 82 N = 73 N = 75 N = 73 N = 79  Condoms 48.4 0.8 4.1 0.1 3.0 0.1 4.3 0.3 2.1 0.2 80.8 9.8 83.9 9.9 70.6 9.4 4.3 0.3 74.9 7.9  Pill 240.3 4.2 200.6 5.4 200.7 4.8 122.8 7.4 64.2 5.2 229.8 27.9 293.8 34.7 221.7 29.7 122.8 7.4 209.0 22.1  Injectable 652.1 11.4 501.1 13.6 516.4 12.4 274.0 16.5 202.7 16.6 131.4 16.0 114.9 13.6 117.0 15.6 274.0 16.5 131.6 13.9  EC pill 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 93.5 11.4 104.4 12.3 75.1 10.0 0.0 0.0 151.7 16  IUD 1545.1 26.9 962.3 26.1 1184.5 28.4 330.3 19.8 293.9 24.0 141.2 17.2 36.3 4.3 122.4 16.4 330.3 19.8 230.0 24.3  Implant 3257.8 56.7 2016.8 54.7 2268.8 54.4 934.3 56.1 661.5 54.0 145.5 17.7 214.5 25.3 141.0 18.9 934.3 56.1 149.0 15.7  Total 5743.7 100 3684.8 100 4173.3 100 1665.6 100 1224.4 100 822.2 100 847.9 100 747.8 100 1666 100 946.2 100 DRC N = 33 N = 58 N = 54 N = 66 N = 70 N = 57 N = 70 N = 61 N = 67 N = 65  Condoms 296.8 20.5 392.9 18.9 379.6 13.9 339.0 17.0 378.7 17.4 127.5 12.3 103.6 6.3 100.6 6.9 158.3 3.1 138.9 4.1  Pill 26.0 1.8 140.1 6.8 107.6 3.9 187.0 9.4 226.5 10.4 528.6 51.1 981.7 59.5 529.2 36.3 1199.0 23.5 773.7 22.7  Injectable 83.4 5.7 140.2 6.8 167.5 6.1 191.1 9.6 197.2 9.1 34.6 3.4 71.2 4.3 136.0 9.3 84.5 1.7 68.2 2.0  EC pill 4.1 0.3 9.6 0.5 76.9 2.8 4.3 0.2 11.2 0.5 92.6 9.0 83.0 5.0 84.0 5.8 92.1 1.8 168.9 5.0  IUD 130.6 9.0 206.1 9.9 224.5 8.2 44.2 2.2 38.6 1.8 28.5 2.8 41.9 2.5 116.4 8.0 3280.7 64.3 994.1 29.2  Implant 910.3 62.7 1185.8 57.2 1783.5 65.1 1225.0 61.5 1321.8 60.8 221.8 21.5 369.3 22.4 492.5 33.8 284.3 5.6 1265.8 37.1  Total 1451.2 100 2074.6 100 2739.6 100 1990.5 100 2174.0 100 1033.5 100 1650.7 100 1458.5 100 5098.8 100 3409.5 100 India N = 24 N = 22 N = 20 N = 19 N = 20 N = 17 N = 205 N = 246 N = 190 N = 213 N = 227 N = 193  Condoms 164.0 12.0 238.5 20.8 41.0 7.3 73.8 4.8 69.6 5.6 62.6 4.2 231.1 14.3 377.7 27.0 336.6 26.3 288.7 8.6 305.5 26.0 134.7 13.7  Pill 303.4 22.1 162.4 14.2 89.3 15.9 217.6 14.2 197.9 16.0 246.4 16.4 319.9 19.8 297.8 21.3 240.7 18.8 117.5 3.5 235.0 20.0 173.0 17.6  Injectable 41.3 3.0 76.2 6.7 90.0 16.0 136.1 8.9 150.7 12.1 30.0 2.0 49.3 3.1 40.0 2.9 30.5 2.4 508.3 15.1 37.9 3.2 40.6 4.1  EC pill 3.3 0.2 7.8 0.7 2.3 0.4 4.0 0.3 6.1 0.5 4.0 0.3 66.9 4.1 79.5 5.7 70.0 5.5 441.5 13.2 101.1 8.6 91.9 9.3  IUD 859.7 62.7 659.6 57.6 339.0 60.4 1102.6 71.9 816.0 65.8 1156.9 77.1 946.2 58.6 602.1 43.1 603.5 47.1 2000.5 59.6 496.3 42.2 544.2 55.3  Total 1371.8 100 1144.6 100 561.6 100 1534.0 100 1240.3 100 1499.9 100 1613.4 100 1397.0 100 1281.4 100 3356.4 100 1175.8 100 984.4 100 Kenya N = 232 N = 223 N = 220 N = 220 N = 218 N = 224 N = 265 N = 283 N = 293 N = 300 N = 292 N = 300  Condoms 1647.8 15.0 1393.7 9.8 1229.6 7.9 1447.7 10.3 1601.4 11.9 887.4 6.4 584.5 5.3 388.7 3.4 587.4 6.0 730.0 7.9 606.2 7.9 579.1 7.2  Pill 291.9 2.7 203.5 1.4 306.4 2.0 318.3 2.3 363.9 2.7 363.6 2.6 978.6 8.9 668.8 5.9 813.3 8.3 852.8 9.3 599.3 7.8 827.9 10.3  Injectable 1260.9 11.5 1558.3 10.9 1465.0 9.5 1724.9 12.3 1753.7 13.1 1861.2 13.4 1565.6 14.2 1465.4 12.9 1708.7 17.5 1674.6 18.2 1533.4 19.9 1502.6 18.7  EC pill 0.4 0.0 1.1 0.0 1.8 0.0 1.6 0.0 0.5 0.0 0.6 0.0 231.3 2.1 703.4 6.2 947.1 9.7 826.2 9.0 687.2 8.9 635.9 7.9  IUD 1534.1 14.0 1961.4 13.8 2922.8 18.9 1769.2 12.6 2121.1 15.8 2035.5 14.6 2193.7 19.9 3259.6 28.7 1575.0 16.1 1013.8 11.0 1041.0 13.5 1177.6 14.6  Implant 6221.0 56.8 9136.3 64.1 9553.3 61.7 8755.0 62.5 7591.0 56.5 8768.0 63.0 5472.0 49.6 4860.3 42.8 4159.8 42.5 4104.0 44.6 3230.5 42.0 3326.8 41.3  Total 10956.1 100 14254.2 100 15479 100 14016 100 13432 100 13916 100 11025.7 100 11346.2 100 9791.3 100 9201.5 100 7697.6 100 8049.9 100 Nigeria N = 79 N = 87 N = 88 N = 88 N = 82 N = 79 N = 270 N = 325 N = 342 N = 338 N = 302 N = 285  Condoms 271.0 4.7 232.1 4.0 174.9 3.4 248.6 4.1 209.1 4.0 243.0 4.4 235.2 4.7 230.1 4.1 170.1 3.7 187.3 5.0 205.8 5.3 281.4 7.8  Pill 152.7 2.7 198.5 3.4 222.6 4.3 181.8 3.0 194.6 3.7 238.7 4.3 328.5 6.6 301.4 5.4 302.8 6.5 281.1 7.5 349.1 8.9 299.8 8.4  Injectable 499.7 8.7 606.1 10.5 549.8 10.7 677.6 11.3 691.6 13.1 724.7 13.1 287.7 5.8 391.0 7.0 409.8 8.9 360.2 9.6 522.4 13.4 303.6 8.5  EC pill 0.0 0.0 0.1 0.0 1.0 0.0 0.4 0.0 0.0 0.0 0.8 0.0 153.9 3.1 206.9 3.7 163.0 3.5 248.1 6.6 511.0 13.1 319.2 8.9  IUD 1103.1 19.2 1491.8 25.8 1013.8 19.7 1779.3 29.6 1299.5 24.6 994.1 17.9 2499.2 50.3 2984.5 53.1 2261.4 48.9 1587.0 42.3 845.5 21.7 753.9 21.0  Implant 3707.8 64.7 3246.5 56.2 3184.8 61.9 3115.5 51.9 2883.3 54.6 3337.3 60.3 1460.3 29.4 1504.3 26.8 1320.8 28.5 1085.3 28.9 1470.5 37.7 1627.0 45.4  Total 5734.3 100 5775.0 100 5146.8 100 6003.1 100 5278.1 100 5538.5 100 4964.7 100 5618.2 100 4627.8 100 3748.8 100 3904.3 100 3584.9 100 N represents the total weighted number of SDPs contributing to CYP units Couple years of protection (CYP) is the estimated protection provided by contraceptive methods during a 1-year period, based upon the volume of all contraceptives sold or distributed free of charge to clients during that period (USAID, 2019). the bold values differentiate country totals from other values for the individual methods Open in new tab Table 2 Quarterly changes in couple years of protection (CYP) by country and method . PUBLIC . PRIVATE . Country namea . Quarters . Quarters . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Burkina Faso N = 34 N = 32 N = 34 N = 29 N = 15 N = 82 N = 73 N = 75 N = 73 N = 79  Condoms 48.4 0.8 4.1 0.1 3.0 0.1 4.3 0.3 2.1 0.2 80.8 9.8 83.9 9.9 70.6 9.4 4.3 0.3 74.9 7.9  Pill 240.3 4.2 200.6 5.4 200.7 4.8 122.8 7.4 64.2 5.2 229.8 27.9 293.8 34.7 221.7 29.7 122.8 7.4 209.0 22.1  Injectable 652.1 11.4 501.1 13.6 516.4 12.4 274.0 16.5 202.7 16.6 131.4 16.0 114.9 13.6 117.0 15.6 274.0 16.5 131.6 13.9  EC pill 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 93.5 11.4 104.4 12.3 75.1 10.0 0.0 0.0 151.7 16  IUD 1545.1 26.9 962.3 26.1 1184.5 28.4 330.3 19.8 293.9 24.0 141.2 17.2 36.3 4.3 122.4 16.4 330.3 19.8 230.0 24.3  Implant 3257.8 56.7 2016.8 54.7 2268.8 54.4 934.3 56.1 661.5 54.0 145.5 17.7 214.5 25.3 141.0 18.9 934.3 56.1 149.0 15.7  Total 5743.7 100 3684.8 100 4173.3 100 1665.6 100 1224.4 100 822.2 100 847.9 100 747.8 100 1666 100 946.2 100 DRC N = 33 N = 58 N = 54 N = 66 N = 70 N = 57 N = 70 N = 61 N = 67 N = 65  Condoms 296.8 20.5 392.9 18.9 379.6 13.9 339.0 17.0 378.7 17.4 127.5 12.3 103.6 6.3 100.6 6.9 158.3 3.1 138.9 4.1  Pill 26.0 1.8 140.1 6.8 107.6 3.9 187.0 9.4 226.5 10.4 528.6 51.1 981.7 59.5 529.2 36.3 1199.0 23.5 773.7 22.7  Injectable 83.4 5.7 140.2 6.8 167.5 6.1 191.1 9.6 197.2 9.1 34.6 3.4 71.2 4.3 136.0 9.3 84.5 1.7 68.2 2.0  EC pill 4.1 0.3 9.6 0.5 76.9 2.8 4.3 0.2 11.2 0.5 92.6 9.0 83.0 5.0 84.0 5.8 92.1 1.8 168.9 5.0  IUD 130.6 9.0 206.1 9.9 224.5 8.2 44.2 2.2 38.6 1.8 28.5 2.8 41.9 2.5 116.4 8.0 3280.7 64.3 994.1 29.2  Implant 910.3 62.7 1185.8 57.2 1783.5 65.1 1225.0 61.5 1321.8 60.8 221.8 21.5 369.3 22.4 492.5 33.8 284.3 5.6 1265.8 37.1  Total 1451.2 100 2074.6 100 2739.6 100 1990.5 100 2174.0 100 1033.5 100 1650.7 100 1458.5 100 5098.8 100 3409.5 100 India N = 24 N = 22 N = 20 N = 19 N = 20 N = 17 N = 205 N = 246 N = 190 N = 213 N = 227 N = 193  Condoms 164.0 12.0 238.5 20.8 41.0 7.3 73.8 4.8 69.6 5.6 62.6 4.2 231.1 14.3 377.7 27.0 336.6 26.3 288.7 8.6 305.5 26.0 134.7 13.7  Pill 303.4 22.1 162.4 14.2 89.3 15.9 217.6 14.2 197.9 16.0 246.4 16.4 319.9 19.8 297.8 21.3 240.7 18.8 117.5 3.5 235.0 20.0 173.0 17.6  Injectable 41.3 3.0 76.2 6.7 90.0 16.0 136.1 8.9 150.7 12.1 30.0 2.0 49.3 3.1 40.0 2.9 30.5 2.4 508.3 15.1 37.9 3.2 40.6 4.1  EC pill 3.3 0.2 7.8 0.7 2.3 0.4 4.0 0.3 6.1 0.5 4.0 0.3 66.9 4.1 79.5 5.7 70.0 5.5 441.5 13.2 101.1 8.6 91.9 9.3  IUD 859.7 62.7 659.6 57.6 339.0 60.4 1102.6 71.9 816.0 65.8 1156.9 77.1 946.2 58.6 602.1 43.1 603.5 47.1 2000.5 59.6 496.3 42.2 544.2 55.3  Total 1371.8 100 1144.6 100 561.6 100 1534.0 100 1240.3 100 1499.9 100 1613.4 100 1397.0 100 1281.4 100 3356.4 100 1175.8 100 984.4 100 Kenya N = 232 N = 223 N = 220 N = 220 N = 218 N = 224 N = 265 N = 283 N = 293 N = 300 N = 292 N = 300  Condoms 1647.8 15.0 1393.7 9.8 1229.6 7.9 1447.7 10.3 1601.4 11.9 887.4 6.4 584.5 5.3 388.7 3.4 587.4 6.0 730.0 7.9 606.2 7.9 579.1 7.2  Pill 291.9 2.7 203.5 1.4 306.4 2.0 318.3 2.3 363.9 2.7 363.6 2.6 978.6 8.9 668.8 5.9 813.3 8.3 852.8 9.3 599.3 7.8 827.9 10.3  Injectable 1260.9 11.5 1558.3 10.9 1465.0 9.5 1724.9 12.3 1753.7 13.1 1861.2 13.4 1565.6 14.2 1465.4 12.9 1708.7 17.5 1674.6 18.2 1533.4 19.9 1502.6 18.7  EC pill 0.4 0.0 1.1 0.0 1.8 0.0 1.6 0.0 0.5 0.0 0.6 0.0 231.3 2.1 703.4 6.2 947.1 9.7 826.2 9.0 687.2 8.9 635.9 7.9  IUD 1534.1 14.0 1961.4 13.8 2922.8 18.9 1769.2 12.6 2121.1 15.8 2035.5 14.6 2193.7 19.9 3259.6 28.7 1575.0 16.1 1013.8 11.0 1041.0 13.5 1177.6 14.6  Implant 6221.0 56.8 9136.3 64.1 9553.3 61.7 8755.0 62.5 7591.0 56.5 8768.0 63.0 5472.0 49.6 4860.3 42.8 4159.8 42.5 4104.0 44.6 3230.5 42.0 3326.8 41.3  Total 10956.1 100 14254.2 100 15479 100 14016 100 13432 100 13916 100 11025.7 100 11346.2 100 9791.3 100 9201.5 100 7697.6 100 8049.9 100 Nigeria N = 79 N = 87 N = 88 N = 88 N = 82 N = 79 N = 270 N = 325 N = 342 N = 338 N = 302 N = 285  Condoms 271.0 4.7 232.1 4.0 174.9 3.4 248.6 4.1 209.1 4.0 243.0 4.4 235.2 4.7 230.1 4.1 170.1 3.7 187.3 5.0 205.8 5.3 281.4 7.8  Pill 152.7 2.7 198.5 3.4 222.6 4.3 181.8 3.0 194.6 3.7 238.7 4.3 328.5 6.6 301.4 5.4 302.8 6.5 281.1 7.5 349.1 8.9 299.8 8.4  Injectable 499.7 8.7 606.1 10.5 549.8 10.7 677.6 11.3 691.6 13.1 724.7 13.1 287.7 5.8 391.0 7.0 409.8 8.9 360.2 9.6 522.4 13.4 303.6 8.5  EC pill 0.0 0.0 0.1 0.0 1.0 0.0 0.4 0.0 0.0 0.0 0.8 0.0 153.9 3.1 206.9 3.7 163.0 3.5 248.1 6.6 511.0 13.1 319.2 8.9  IUD 1103.1 19.2 1491.8 25.8 1013.8 19.7 1779.3 29.6 1299.5 24.6 994.1 17.9 2499.2 50.3 2984.5 53.1 2261.4 48.9 1587.0 42.3 845.5 21.7 753.9 21.0  Implant 3707.8 64.7 3246.5 56.2 3184.8 61.9 3115.5 51.9 2883.3 54.6 3337.3 60.3 1460.3 29.4 1504.3 26.8 1320.8 28.5 1085.3 28.9 1470.5 37.7 1627.0 45.4  Total 5734.3 100 5775.0 100 5146.8 100 6003.1 100 5278.1 100 5538.5 100 4964.7 100 5618.2 100 4627.8 100 3748.8 100 3904.3 100 3584.9 100 . PUBLIC . PRIVATE . Country namea . Quarters . Quarters . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Q1 . Q2 . Q3 . Q4 . Q5 . Q6 . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Total CYP . % . Burkina Faso N = 34 N = 32 N = 34 N = 29 N = 15 N = 82 N = 73 N = 75 N = 73 N = 79  Condoms 48.4 0.8 4.1 0.1 3.0 0.1 4.3 0.3 2.1 0.2 80.8 9.8 83.9 9.9 70.6 9.4 4.3 0.3 74.9 7.9  Pill 240.3 4.2 200.6 5.4 200.7 4.8 122.8 7.4 64.2 5.2 229.8 27.9 293.8 34.7 221.7 29.7 122.8 7.4 209.0 22.1  Injectable 652.1 11.4 501.1 13.6 516.4 12.4 274.0 16.5 202.7 16.6 131.4 16.0 114.9 13.6 117.0 15.6 274.0 16.5 131.6 13.9  EC pill 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 93.5 11.4 104.4 12.3 75.1 10.0 0.0 0.0 151.7 16  IUD 1545.1 26.9 962.3 26.1 1184.5 28.4 330.3 19.8 293.9 24.0 141.2 17.2 36.3 4.3 122.4 16.4 330.3 19.8 230.0 24.3  Implant 3257.8 56.7 2016.8 54.7 2268.8 54.4 934.3 56.1 661.5 54.0 145.5 17.7 214.5 25.3 141.0 18.9 934.3 56.1 149.0 15.7  Total 5743.7 100 3684.8 100 4173.3 100 1665.6 100 1224.4 100 822.2 100 847.9 100 747.8 100 1666 100 946.2 100 DRC N = 33 N = 58 N = 54 N = 66 N = 70 N = 57 N = 70 N = 61 N = 67 N = 65  Condoms 296.8 20.5 392.9 18.9 379.6 13.9 339.0 17.0 378.7 17.4 127.5 12.3 103.6 6.3 100.6 6.9 158.3 3.1 138.9 4.1  Pill 26.0 1.8 140.1 6.8 107.6 3.9 187.0 9.4 226.5 10.4 528.6 51.1 981.7 59.5 529.2 36.3 1199.0 23.5 773.7 22.7  Injectable 83.4 5.7 140.2 6.8 167.5 6.1 191.1 9.6 197.2 9.1 34.6 3.4 71.2 4.3 136.0 9.3 84.5 1.7 68.2 2.0  EC pill 4.1 0.3 9.6 0.5 76.9 2.8 4.3 0.2 11.2 0.5 92.6 9.0 83.0 5.0 84.0 5.8 92.1 1.8 168.9 5.0  IUD 130.6 9.0 206.1 9.9 224.5 8.2 44.2 2.2 38.6 1.8 28.5 2.8 41.9 2.5 116.4 8.0 3280.7 64.3 994.1 29.2  Implant 910.3 62.7 1185.8 57.2 1783.5 65.1 1225.0 61.5 1321.8 60.8 221.8 21.5 369.3 22.4 492.5 33.8 284.3 5.6 1265.8 37.1  Total 1451.2 100 2074.6 100 2739.6 100 1990.5 100 2174.0 100 1033.5 100 1650.7 100 1458.5 100 5098.8 100 3409.5 100 India N = 24 N = 22 N = 20 N = 19 N = 20 N = 17 N = 205 N = 246 N = 190 N = 213 N = 227 N = 193  Condoms 164.0 12.0 238.5 20.8 41.0 7.3 73.8 4.8 69.6 5.6 62.6 4.2 231.1 14.3 377.7 27.0 336.6 26.3 288.7 8.6 305.5 26.0 134.7 13.7  Pill 303.4 22.1 162.4 14.2 89.3 15.9 217.6 14.2 197.9 16.0 246.4 16.4 319.9 19.8 297.8 21.3 240.7 18.8 117.5 3.5 235.0 20.0 173.0 17.6  Injectable 41.3 3.0 76.2 6.7 90.0 16.0 136.1 8.9 150.7 12.1 30.0 2.0 49.3 3.1 40.0 2.9 30.5 2.4 508.3 15.1 37.9 3.2 40.6 4.1  EC pill 3.3 0.2 7.8 0.7 2.3 0.4 4.0 0.3 6.1 0.5 4.0 0.3 66.9 4.1 79.5 5.7 70.0 5.5 441.5 13.2 101.1 8.6 91.9 9.3  IUD 859.7 62.7 659.6 57.6 339.0 60.4 1102.6 71.9 816.0 65.8 1156.9 77.1 946.2 58.6 602.1 43.1 603.5 47.1 2000.5 59.6 496.3 42.2 544.2 55.3  Total 1371.8 100 1144.6 100 561.6 100 1534.0 100 1240.3 100 1499.9 100 1613.4 100 1397.0 100 1281.4 100 3356.4 100 1175.8 100 984.4 100 Kenya N = 232 N = 223 N = 220 N = 220 N = 218 N = 224 N = 265 N = 283 N = 293 N = 300 N = 292 N = 300  Condoms 1647.8 15.0 1393.7 9.8 1229.6 7.9 1447.7 10.3 1601.4 11.9 887.4 6.4 584.5 5.3 388.7 3.4 587.4 6.0 730.0 7.9 606.2 7.9 579.1 7.2  Pill 291.9 2.7 203.5 1.4 306.4 2.0 318.3 2.3 363.9 2.7 363.6 2.6 978.6 8.9 668.8 5.9 813.3 8.3 852.8 9.3 599.3 7.8 827.9 10.3  Injectable 1260.9 11.5 1558.3 10.9 1465.0 9.5 1724.9 12.3 1753.7 13.1 1861.2 13.4 1565.6 14.2 1465.4 12.9 1708.7 17.5 1674.6 18.2 1533.4 19.9 1502.6 18.7  EC pill 0.4 0.0 1.1 0.0 1.8 0.0 1.6 0.0 0.5 0.0 0.6 0.0 231.3 2.1 703.4 6.2 947.1 9.7 826.2 9.0 687.2 8.9 635.9 7.9  IUD 1534.1 14.0 1961.4 13.8 2922.8 18.9 1769.2 12.6 2121.1 15.8 2035.5 14.6 2193.7 19.9 3259.6 28.7 1575.0 16.1 1013.8 11.0 1041.0 13.5 1177.6 14.6  Implant 6221.0 56.8 9136.3 64.1 9553.3 61.7 8755.0 62.5 7591.0 56.5 8768.0 63.0 5472.0 49.6 4860.3 42.8 4159.8 42.5 4104.0 44.6 3230.5 42.0 3326.8 41.3  Total 10956.1 100 14254.2 100 15479 100 14016 100 13432 100 13916 100 11025.7 100 11346.2 100 9791.3 100 9201.5 100 7697.6 100 8049.9 100 Nigeria N = 79 N = 87 N = 88 N = 88 N = 82 N = 79 N = 270 N = 325 N = 342 N = 338 N = 302 N = 285  Condoms 271.0 4.7 232.1 4.0 174.9 3.4 248.6 4.1 209.1 4.0 243.0 4.4 235.2 4.7 230.1 4.1 170.1 3.7 187.3 5.0 205.8 5.3 281.4 7.8  Pill 152.7 2.7 198.5 3.4 222.6 4.3 181.8 3.0 194.6 3.7 238.7 4.3 328.5 6.6 301.4 5.4 302.8 6.5 281.1 7.5 349.1 8.9 299.8 8.4  Injectable 499.7 8.7 606.1 10.5 549.8 10.7 677.6 11.3 691.6 13.1 724.7 13.1 287.7 5.8 391.0 7.0 409.8 8.9 360.2 9.6 522.4 13.4 303.6 8.5  EC pill 0.0 0.0 0.1 0.0 1.0 0.0 0.4 0.0 0.0 0.0 0.8 0.0 153.9 3.1 206.9 3.7 163.0 3.5 248.1 6.6 511.0 13.1 319.2 8.9  IUD 1103.1 19.2 1491.8 25.8 1013.8 19.7 1779.3 29.6 1299.5 24.6 994.1 17.9 2499.2 50.3 2984.5 53.1 2261.4 48.9 1587.0 42.3 845.5 21.7 753.9 21.0  Implant 3707.8 64.7 3246.5 56.2 3184.8 61.9 3115.5 51.9 2883.3 54.6 3337.3 60.3 1460.3 29.4 1504.3 26.8 1320.8 28.5 1085.3 28.9 1470.5 37.7 1627.0 45.4  Total 5734.3 100 5775.0 100 5146.8 100 6003.1 100 5278.1 100 5538.5 100 4964.7 100 5618.2 100 4627.8 100 3748.8 100 3904.3 100 3584.9 100 N represents the total weighted number of SDPs contributing to CYP units Couple years of protection (CYP) is the estimated protection provided by contraceptive methods during a 1-year period, based upon the volume of all contraceptives sold or distributed free of charge to clients during that period (USAID, 2019). the bold values differentiate country totals from other values for the individual methods Open in new tab Consistent with the lower availability rates and client volumes observed among public SDPs offering emergency contraception, we found that this method did not provide many CYPs relative to other MCMs offered by the public SDPs. In Burkina Faso, an average of 4 146 CYPs was provided each quarter, of which 3 298 were provided by public SDPs with the remainder being provided by the private SDPs. The combination of IUDs and implants accounted for 80.2% of the CYPs provided by public SDPs and 33.8% of those provided by private SDPs. In the DRC, public SDPs provided an average of 2 086 CYPs each quarter compared to 2 530 among the private SDPs. Oral pills contributed the highest amount of CYPs in the private sector. They accounted for 38.6% of the total CYPs in the private sector. IUDs and implants together accounted for 45.5% of the CYPs provided by the private SDPs. In India, public SDPs provided an average of 1 225 CYPs whereas the private SDPs contributed 1 635 CYPs. Among the MCMs tracked in this study, IUDs provided the highest amount of CYPs against unintended pregnancies. This was consistent over time among both public and private SDPs. Relative to other MCMs, injectables and emergency contraceptive pills did not seem to provide high levels of protection in India. We found that the public SDPs in Kenya provided 13 676 CYPs against unintended pregnancies on average compared to the 9 519 observed in the private sector. Implants provided the most protection across all SDPs regardless of public–private ownership. Similar to the public SDPs in Kenya, those in Nigeria provided a higher amount of protection (5 579 CYPs) compared to their private counterparts (4 408 CYPS). Together, implants and IUDs generally provided the most couple-years of protection across both public and private SDPs. Discussion Modern contraceptive use is a product of both supply of and demand for contraception. While the latter has been examined extensively, the former has been studied less often, largely due to data limitations. In this research, we use locally representative data to measure trends in the family planning supply environment in urban or suburban areas of Burkina Faso, DRC, India, Kenya and Nigeria. Specifically, we describe trends in stockouts, method availability and consumption of specific contraceptive methods. Although each setting has its own unique FP supply profile, some of our results are consistent across settings. We find that public sector facilities are more likely to have a comprehensive mix of short- and long-acting contraceptive methods, compared to private sector facilities, similar to results from other studies (Hutchinson et al., 2011; Kakoko et al., 2012). There are, however, a few exceptions to this pattern, such as EC in SSA countries and oral pills in Kenyan sites. Indeed, previous work by others has shown that private sector SDPs are the major outlets for EC in a range of SSA countries (Maharaj and Rogan, 2008; Mané et al., 2015; Hernandez et al., 2018) and for oral pills in Kenyan sites (Corroon et al., 2016). Furthermore, it is worthwhile to point out that, for some SSA countries such as Kenya, the distribution of EC in public facilities is usually provided only to victims of sexual assault or rape whereas it is readily available over the counter without a doctor prescription in private pharmacies (Thompson et al., 2018). The exact difference between public and private facilities in contraceptive availability differs by country and method. For example, the levels of condom availability are similar between public and private facilities in Kenya, but the difference is larger for IUD and implant availability. Analysis of MCM stocks by facility type confirmed previous findings by others that while primary care facilities are important outlets for contraceptives in the public sector, pharmacies and drug shops dominate the private sector. Pharmacies and drug shops have an important role to play in achieving FP2020 objectives particularly in SSA countries where they serve as crucial outlets for addressing the unmet need in short-acting contraceptives for vulnerable and hard-to-reach populations (Corroon et al., 2016; Riley et al., 2018). Recent evidence points to the popularity of drug shops, in particular, as preferred SDPs for young women and unmarried women (High-Impact Practices in Family Planning, 2013; Corroon et al., 2016; Weinberger and Callahan, 2017), though their potential likely remains under-leveraged. In general, we noted that the patterns of stock supply and client volume corroborated findings from previous studies examining client-level data in contraceptive use. For example, the high client volume and method availability observed across the sites in Kenya relative to other SSA countries is consistent with a previous study showing that Kenya had the highest modern contraceptive rate among the countries in the study (Ahmed et al., 2019). Consistent with other studies, our findings showed limited availability for long-acting reversible contraceptives (LARCs), including IUDs and implants, with great variability between countries (Grindlay et al., 2016; Thanel et al., 2018). The findings further showed that LARCs were generally more readily available and distributed through the public sector as compared to the private sector. LARCs are highly effective forms of reversible birth control as evidenced by the large proportion of CYPs they provided in our study despite the consistently high levels of stockouts and lower numbers of client visits that they were subject to. Compared to short-acting methods, they have higher upfront costs but are more cost-effective over time, generally well-tolerated by women and thus less likely to be discontinued (World Health Organization, 2012; Diedrich et al., 2015; Thanel et al., 2018; Römer and Linsberger, 2009). Nonetheless, the requirements for service readiness for LARC delivery are decidedly more complex than those for short-acting methods. For instance, service readiness for implant and IUD insertion typically requires the availability of the contraceptive commodity, appropriately trained and credentialed providers and the necessary equipment for insertions. It is thus likely that the low levels of LARC availability observed across countries can be attributed to the limited capacity of country health systems to sustain adequate levels of service readiness for LARCs. With the aim of addressing the shortcomings related to service readiness, some countries, including Nigeria, have made commitments to increase FP program funding while pursuing task-shifting policies geared towards expanding service delivery for LARCs (Scoggins and Bremner, 2020). Nonetheless, our findings suggest that further action is needed from governments, FP program managers and international donors to leverage underutilized private sector primary care providers and continue striving towards improved access to LARCs. Based on client volume data for LARCs, we observed that implants were consistently distributed at higher levels than IUDs in SSA. This is consistent with previous studies that have shown higher levels of implant use relative to IUDs in several African countries (Duvall et al., 2014; Tsui et al., 2017). While implants possess relative advantages and attributes that users like about them as previously discussed by others (Ortayli, 2002; Jacobstein and Stanley, 2013), it is worthwhile to also note that the introduction of implants in SSA benefitted from considerable international assistance to expand access to the method and manage supply chains (Duvall et al., 2014; Tsui et al., 2017). Total CYPs in India appeared to be lower than those in SSA countries. This was likely due to the lack of contribution of implants to total CYPs as the method is currently not offered in India. Though implants are offered in the DRC, the total CYPs provided were still low compared to other African geographies likely due to the low client volume for IUDs. These findings not only highlight the importance of ensuring the availability of LARC choice for maximum protection against unintended pregnancies but also the value of examinations of country-specific method mix when attempting cross-country comparisons. As others have previously reported, trends in method mix may vary considerably across and within geographies thereby raising programmatic issues that need to be addressed in order to address client needs (Bertrand et al., 2020). Our goal in this research is primarily descriptive; we do not seek to explain why stockouts occur or why certain methods are consumed more than others, as doing so would require the measurement of environmental and supply factors that were out of the scope of PMA Agile. For example, we understand from correspondence with PMA Agile personnel that declines in stock of MCMs and in client visits for MCMs in Burkina Faso were due to a public sector strike that took place between June and November of 2019. Additionally, the Indian MoH launched the ‘Antara Programme’ in 2017 with the aim of providing injectables free of charge in public facilities (Ministry of Health and Family Welfare, 2017b). Though this potentially explains the remarkable increase in injectable availability and consumption in the public sector in India, we do not capture information on such programmatic changes, potential disruptions to service delivery or other similar environmental impacts on stockouts. Although PMA Agile did not collect information on the reason for stockouts, the PMA Core project (PMA, 2020) recently did so in four geographies that overlap with the countries chosen here: Kenya, Burkina Faso, DRC and Nigeria. Across all countries and methods, the most common reason for stockouts is that the facility claims to have ordered the method but did not receive the requested shipment. The delays in making contraceptives available in India are likely due to weak monitoring of the contraceptive method supply and consumption at all levels of the health system (Ministry of Health and Family Welfare, 2017a). In response, the Indian government recently rolled out the Family Planning Logistics Management Information System (FPLMIS) (Ministry of Health and Family Welfare, 2017a). The FPLMIS is expected to empower state-level program managers, facility-level stock managers as well as community health workers manage stocks of contraceptive commodities more effectively. Although a detailed examination of the relationship between contraceptive procurement, supply chain management and stockouts was beyond the scope of this research, our findings nonetheless have the potential to inform ongoing debates about the merits of different contraceptive distribution models vis-à-vis stockouts. For example, whereas Burkina Faso and DRC use a ‘pull system’ in which procurement is decentralized and multiple low-level actors place contraceptive orders from central hubs based on their forecasted needs (PATH, 2017; Babazadeh et al., 2018), Nigeria uses a variation of the push system in which the central hubs are responsible for resupplying SDPs with contraceptives (USAID | DELIVER PROJECT, 2014) with Kenya using a combination of the two systems (Eliya Msiyaphazi Zulu et al., 2012). Yet, it was not entirely clear from our findings whether contraceptive stockouts vary based on the distribution model employed, considering the differences in contraceptive consumption across countries. Given our observations that stockouts vary based on geographic setting, sector and contraceptive method, we recommend that country stakeholders adopt context-specific and problem-based approaches to address procurement challenges for individual methods. Considering the rapid fluctuations observed in both contraceptive consumption and stockout rates, the approaches will need to be flexible and cost-effective in order to sustainably address unmet need. On the basis of the PMA Core observation that failure to receive requested contraceptive orders was the most common cause of stockouts among SDPs in African geographies, it is probable that countries may benefit from the Informed Push Model (IPM), which achieved dramatic reductions in stockouts in Senegal largely by transferring the responsibility of order placement and delivery from SDPs to an external professional logistician (Daff et al., 2014) The model’s ability to couple information collection and product distribution by dedicated professionals enhances supply chain performance ultimately strengthening the health system’s ability to respond to the healthcare needs of the population. Furthermore, IPM avoids the requirement for facilities to pay for contraceptive supplies upfront, a situation which may exacerbate supply chain issues (Hasselback et al., 2017). As others have indicated, upfront payments may drive some facilities to use their working capital thus delaying the replenishment of funds until after clients have purchased contraceptive commodities (Hasselback et al., 2017). The funding delays may in turn result in cash flow problems for some facilities or encourage the diversion of the remaining capital towards more profitable non-contraceptive commodities. Given the success of IPM in addressing these issues, we therefore recommend country-specific analyses comparing costs associated with current distribution channels for individual methods to the costs associated with alternative distribution models including IPM. There is limited data on MCM stockouts in low- and middle-income settings, and most available data offer annual averages that do not provide the opportunity to examine short-term trends in stockouts. The quarterly PMA Agile data therefore fill an important gap in the literature, by tracking within-year changes in stockouts, CYPs and client volume across five different settings. We also provide a contrast between public and private facilities which greatly increases the representativeness of our findings. This research also has some limitations. The PMA Agile platform offers an online dashboard displaying trends obtained from the analysis of repeated cross-sectional surveys (PMA Agile, 2020). There may be instances where our results differ slightly from the dashboard results since our analytical dataset contained SDPs that were followed longitudinally across at least two quarters. We also note that in some instances, such as in the public sectors for Burkina Faso and India, the sample size of facilities is small. Furthermore, our analyses combine different types of SDPs which are in turn probably subject to different variation in terms of method availability and client volume. In future studies, we plan to conduct facility-level analyses that address these concerns. Though REs were able to access facility logbooks and record product sales and client visit data, we are unable to comment on the extent to which the data were accurate or complete. With the exception of SDPs in Burkina Faso and public SDPs in DRC, the proportion of facilities contributing data was generally high across countries as evidenced by the number of SDPs contributing to CYP units. Finally, the survey data were subject to sampling and non-sampling errors. The majority of research on FP and contraceptive use has focused on demand-side measures, such as fertility preferences, individual-level predictors of modern contraceptive use and characteristics associated with unmet need. In this research, we demonstrate the value of supply-side measures by showing trends in contraceptive supply and stockout over a short period of time. We expect that this information may help monitor progress towards addressing an unmet need and inform cross-country strategies to anticipate, reduce and prevent stockouts. Acknowledgements The PMA Agile team wishes to acknowledge the helpful cooperation of The Challenge Initiative project and DKT International and the support of the many staff on the in-country Agile teams and local health departments. Funding This work was supported by the Bill and Melinda Gates Foundation [grant numbers OPP1163884, OPP1079004]. PMA Agile is a survey research project implemented by the Bill and Melinda Gates Institute for Population and Reproductive Health at the Johns Hopkins Bloomberg School of Public Health, funded by the Bill and Melinda Gates Foundation. The findings and conclusions contained within are those of the authors and do not necessarily reflect positions or policies of the Bill and Melinda Gates Foundation. Conflict of interest statement. None declared. Ethical approval. PMA Agile data collection protocols were reviewed and approved by the Johns Hopkins Bloomberg School of Public Health Institutional Review Board and the in-country counterpart review board: Kenyatta National Hospital-University of Nairobi Ethics Research Committee; National Health Research Ethics Committee of Nigeria; MOH-Burkina Comité d’Ethique pour la Recherche en Santé; University of Kinshasa School of Public Health Institutional Review Board; Indian Institute for Health Management Research Ethical Review Board. All participants provided verbal consent in accordance with country-specific approved consent procedures. References Ahmed S , Li Q, Liu L et al. . 2012 . Maternal deaths averted by contraceptive use: an analysis of 172 countries . The Lancet 380 : 111 – 25 . Google Scholar OpenURL Placeholder Text WorldCat Ahmed S , Choi Y, Rimon JG et al. . 2019 . Trends in contraceptive prevalence rates in sub-Saharan Africa since the 2012 London Summit on Family Planning: results from repeated cross-sectional surveys . The Lancet Global Health 7 : e904 – e911 . Google Scholar OpenURL Placeholder Text WorldCat Ali M, Farron M, Dilip TR et al. . 2018 . Assessment of family planning service availability and readiness in 10 African countries . Global Health: Science and Practice 6 : 473 – 83 . Google Scholar OpenURL Placeholder Text WorldCat Babazadeh S, Lea S, Kayembe P et al. . 2018 . Assessing the contraceptive supply environment in Kinshasa . DRC: trend Data from PMA2020, Health Policy and Planning 33 : 155 – 62 . Google Scholar OpenURL Placeholder Text WorldCat Bertrand JT , Ross J, Sullivan TM et al. . 2020 . Contraceptive method mix: updates and implications . Global Health: Science and Practice 8 : 666 – 79 . Google Scholar OpenURL Placeholder Text WorldCat Brown SS , Eisenberg L. 1995 . Socioeconomic and Cultural Influences on Contraceptive Use the Best Intentions: Unintended Pregnancy and the Well-Being of Children and Families . Washington, DC: National Academies Press (US ). Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Brown W , Druce N, Bunting J et al. . 2014 . Developing the “120 by 20” goal for the Global FP2020 Initiative . Studies in Family Planning 45 : 73 – 84 . Google Scholar OpenURL Placeholder Text WorldCat Chandra-Mouli V , McCarraher DR, Phillips SJ et al. . 2014 . Contraception for adolescents in low and middle income countries: needs, barriers, and access . Reproductive Health 11 : 1 . Google Scholar OpenURL Placeholder Text WorldCat Corroon M , Kebede E, Spektor G et al. . 2016 . Key role of drug shops and pharmacies for family planning in urban Nigeria and Kenya . Global Health: Science and Practice 4 : 594 – 609 . Google Scholar OpenURL Placeholder Text WorldCat Daff BM, Seck C, Belkhayat H et al. . 2014 . Informed push distribution of contraceptives in Senegal reduces stockouts and improves quality of family planning services . Global Health: Science and Practice 2 : 245 – 52 . Google Scholar OpenURL Placeholder Text WorldCat Darroch JE , Sedgh G, Ball H. 2011 . Contraceptive Technologies: Responding to Women’s Needs , Vol. 201. New York : Guttmacher Institute . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC DHS Program . 2017 . The Service Provision Assessment (SPA). https://dhsprogram.com/What-We-Do/Survey-Types/SPA.cfm, accessed 7 July 2020. Diedrich JT, Zhao Q, Madden T et al. . 2015 . Three-year continuation of reversible contraception . Am J Obstet Gynecol 213 : 662e661 – 8 . Google Scholar OpenURL Placeholder Text WorldCat Do M , Kurimoto N. 2012 . Women's empowerment and choice of contraceptive methods in selected African countries . International Perspectives on Sexual and Reproductive Health 38: 23 – 33 . Google Scholar OpenURL Placeholder Text WorldCat Duvall S, Thurston S, Weinberger M et al. . 2014 . Scaling up delivery of contraceptive implants in sub-Saharan Africa: operational experiences of Marie Stopes International . Global Health: Science and Practice 2 : 72 – 92 . Google Scholar OpenURL Placeholder Text WorldCat Eliya Msiyaphazi Zulu D, Musila NR, Murunga V et al. . 2012 . Assessment of Drivers of Progress in Increasing Contraceptive use in sub-Saharan Africa: Case Studies from Eastern and Southern Africa. African Institute for Development Policy (AFIDEP). Lilongwe, Malawi. https://media.africaportal.org/documents/Assessment_of_Drivers.pdf, accessed 19 November 2020. FP 2020 . 2015 . FP2020. Momentum at the Midpoint 2015-16. http://2015-2016progress.familyplanning2020.org/page/measurement/contraceptive-stock-outs-and-availability-indicators-10-11, accessed 18 June 2020. Grindlay K, Turyakira E, Kyamwanga IT et al. . 2016 . The experience and impact of contraceptive stockouts among women, providers and policymakers in two districts of Uganda . International Perspectives on Sexual and Reproductive Health 42 : 141 – 50 . Google Scholar OpenURL Placeholder Text WorldCat Guttmacher Institute . 2017 . Adding It up: Investing in Contraception and Maternal and Newborn Health, 2017 . New York : Guttmacher Institute . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Hasselback L, Dicko M, Viadro C et al. . 2017 . Understanding and Addressing Contraceptive Stockouts to Increase Family Planning Access and Uptake in Senegal , Vol. 17 , 373 . Heeringa SG , West BT, Berglund PA. 2017 . Applied Survey Data Analysis . Boca Raton, Florida: Chapman and Hall/CRC . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Hernandez JH , Mbadu MF, Garcia M et al. . 2018 . The provision of emergency contraception in Kinshasa's private sector pharmacies: experiences of mystery clients . Contraception 97 : 57 – 61 . Google Scholar OpenURL Placeholder Text WorldCat High-Impact Practices in Family Planning . 2013 . Drug Shops and Pharmacies: sources for family planning commodities and information. Brief. USAID Washington DC, USA. http://www.fphighimpactpractices.org/briefs/drug-shops-and-pharmacies/, accessed 19 November 2020. Hubacher D , Mavranezouli I, McGinn E. 2008 . Unintended pregnancy in sub-Saharan Africa: magnitude of the problem and potential role of contraceptive implants to alleviate it . Contraception 78 : 73 – 8 . Google Scholar OpenURL Placeholder Text WorldCat Hutchinson PL , Do M, Agha S. 2011 . Measuring client satisfaction and the quality of family planning services: a comparative analysis of public and private health facilities in Tanzania, Kenya and Ghana . BMC Health Services Research 11 : 203 . Google Scholar OpenURL Placeholder Text WorldCat Jacobstein R , Stanley H. 2013 . Contraceptive implants: providing better choice to meet growing family planning demand . Global Health: Science and Practice 1 : 11 – 7 . Google Scholar OpenURL Placeholder Text WorldCat Joshi B , Sachina O, Begum S. 2019 . Health Technology Assessment of Long Acting Reversible Contraceptives in India . Mumbai, India : National Institute for Research in Reproductive Health. Indian Council of Medical Research . https://dhr.gov.in/sites/default/files/HTA_%20LARC%20in%20India.pdf, accessed 19 November 2020. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Kakoko DC, Ketting E, Kamazima SR et al. . 2012 . Provision of family planning services in Tanzania: a comparative analysis of public and private facilities . African Journal of Reproduction Health 16 : 140 – 8 . Google Scholar OpenURL Placeholder Text WorldCat Maharaj P , Rogan M. 2008 . Emergency contraception in South Africa: a literature review . The European Journal of Contraception & Reproductive Health Care 13 : 351 – 61 . Google Scholar OpenURL Placeholder Text WorldCat Mané B , Brady M, Ramarao S et al. . 2015 . Emergency contraception in Senegal: challenges and opportunities . The European Journal of Contraception & Reproductive Health Care 20 : 64 – 73 . Google Scholar OpenURL Placeholder Text WorldCat Ministry of Health and Family Welfare . 2017a . Family Planning Logistics Management Information System (FP-LMIS): Training Manual and User Guide . Family Planning Division : Ministry of Health and Family Welfare: Government of India. http://www.rajswasthya.nic.in/FPLIMIS/FP-LMIS%20Training%20Manual.pdf, accessed 23 December 2020. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Ministry of Health and Family Welfare . 2017b . Health Ministry Launches Two New Contraceptives: Press Information Bureau, Government of India. https://pib.gov.in/newsite/PrintRelease.aspx?relid=170537, accessed 19 November 2020. Ortayli N. 2002 . Users’ perspectives on implantable contraceptives for women . Contraception 65 : 107 – 11 . Google Scholar OpenURL Placeholder Text WorldCat PATH . 2017 . How to Introduce and Scale up Sayana Press (DMPA-SC in Uniject): Practical Guidance from PATH Based on Lessons Learned during Pilot Introduction . Seattle, WA, USA : PATH . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC PMA . 2020 . PMA: Results from Phase 1 Baseline Survey. https://www.pmadata.org, accessed 19 November 2020. PMA Agile . 2020 . www.pmadata.org/technical-areas/pma-agile, accessed 28 June 2020. Riley C , Garfinkel D, Thanel K, on behalf of the FPwatch Group et al. . 2018 . Getting to FP2020: harnessing the private sector to increase modern contraceptive access and choice in Ethiopia, Nigeria, and DRC . PLoS One 13 : e0192522 . Google Scholar OpenURL Placeholder Text WorldCat Römer T , Linsberger D. 2009 . User satisfaction with a Levonorgestrel-releasing intrauterine system (LNG-IUS): data from an international survey . The European Journal of Contraception & Reproductive Health Care 14 : 391 – 8 . Google Scholar OpenURL Placeholder Text WorldCat Ross J , Hardee K. 2013 . Access to contraceptive methods and prevalence of use . Journal of Biosocial Science 45 : 761 – 78 . Google Scholar OpenURL Placeholder Text WorldCat Scoggins S , Bremner J. 2020 . FP2020: The Way Ahead, 2016–2017: Family Planning . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Stover J , Ross J. 2010 . How increased contraceptive use has reduced maternal mortality . Maternal and Child Health Journal 14 : 687 – 95 . Google Scholar OpenURL Placeholder Text WorldCat Thanel K , Garfinkel D, Riley C, on behalf of the FPwatch Group et al. . 2018 . Leveraging long acting reversible contraceptives to achieve FP2020 commitments in sub-Saharan Africa: the potential of implants . PLoS One 13 : e0195228 . Google Scholar OpenURL Placeholder Text WorldCat Thompson J , Undie C-C, Amin A et al. . 2018 . Harmonizing national abortion and pregnancy prevention laws and policies for sexual violence survivors with the Maputo Protocol: proceedings of a 2016 regional technical meeting in sub-Saharan Africa . BMC Proceedings 12 : 5 . Google Scholar OpenURL Placeholder Text WorldCat Tsui A , Anglewicz P, Akinlose T et al. . 2020 . Performance monitoring and accountability: the Agile Project’s protocol, record and experience . Gates Open Research 4 : 30 . Google Scholar OpenURL Placeholder Text WorldCat Tsui AO , Brown W, Li Q. 2017 . Contraceptive practice in sub-Saharan Africa, Population and development review . 43(Suppl 1 ): 166 . USAID . 2018 . USAID Global Health Supply Chain Program: 2017 Contraceptive Security Indicators Report . Washington DC : Chemonics International Inc . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC USAID . 2019 . Couple of Years of Protection (CYP). https://www.usaid.gov/global-health/health-areas/family-planning/couple-years-protection-cyp, accessed 18 June, 2020. USAID | DELIVER PROJECT, Task Order 4 and Task Order 7 . 2014 . Nigeria: Direct Delivery and Information Capture Activities: March 2012–August 2014 . Abuja, Nigeria : USAID . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Wang W , Mallick L. 2019 . Understanding the relationship between family planning method choices and modern contraceptive use: an analysis of geographically linked population and health facilities data in Haiti . BMJ Global Health 4(Suppl 5 ): e000765 . Google Scholar OpenURL Placeholder Text WorldCat Weinberger M , Callahan S. 2017 . The Private Sector: Key to Achieving Family Planning 2020 Goals. Brief. Bethesda, MD : Sustaining Health Outcomes through Private Sector Project, Abt Associates . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC WHO 2018 . Family planning/Contraception. https://www.who.int/news-room/fact-sheets/detail/family-planning-contraception, accessed 15 February 2019. World Health Organization . 2012 . From Evidence to Policy: Expanding Access to Family Planning. Strategies to Increase Use of Long-acting and Permanent Contraception . Geneva : WHO . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Zimmerman LA , Bell SO, Li Q, et al. . PMA2020 Principal Investigators Group . 2019 . Individual, community and service environment factors associated with modern contraceptive use in five Sub-Saharan African countries: a multilevel, multinomial analysis using geographically linked data from PMA2020 . PLoS One 14 : e0218157 . Google Scholar OpenURL Placeholder Text WorldCat © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine.

Odoch, Walter Denis; Dambisya, Yoswa; Peacocke, Elizabeth; Sandberg, Kristin Ingstad; Hembre, Berit Sofie Hustad

doi: 10.1093/heapol/czaa189pmid: 33569583

Abstract The WHO Model List of Essential Medicines (MLEM) has since 1977 helped prioritize and ensure availability of medicines especially in low- and middle-income countries. The MLEM consists mainly of generic medicines, though recent trends point towards listing expensive on-patent medicines and increasing global support for medicines against non-communicable diseases. However, the implications of such changes for national essential medicines list (NEML) updates for access to essential medicines has received relatively little attention. This study examined how government agencies and other actors in Kenya, Uganda and Tanzania participate in and influence the NEML update process and subsequent availability of prioritized medicines; and the alignment of these processes to WHO guidance. A mixed study design was used, with qualitative documentary review, key informant interviews and thematic data analysis. Results show that NEML updating processes were similar amongst the three countries and aligned to WHO guidelines, albeit conducted irregularly, with tendency to reprioritization during procurement stages, and were not always accompanied by revision of clinical guidelines. Variations were noted in the inclusion of medicines against cancer and hepatitis C, and the utilization of health technology assessment (HTA). For medicines against diseases with high global engagement, such as HIV/AIDS and TB, national stakeholders had more limited inputs in prioritization and funding. Furthermore, national actors were not influenced by the pharmaceutical industry during the NEML update process, nor were any conflicting agendas identified between health, trade and industrial policies. Hence, the study suggests that more attention should be paid to the combination of HTAs and NEMLs, particularly as countries work towards universal health coverage, in addition to heightened awareness of how global disease-specific initiatives may confound national implementation of the NEML. The study concludes with a call to strengthen country-level policy and procedural coherence around the process of prioritizing and ensuring availability of essential medicines. Model list of essential medicines, national essential medicines list, health policy, actors, Uganda, Kenya, Tanzania, access to medicines KEY MESSAGES Evidence around governance issues on access to medicines remains scarce, relative to what is available on breakthroughs in biomedical technologies. The WHO Model List of Essential Medicines (MLEM) and national essential medicine list development, update and implementation is one such policy and governance issue. The process for the essential medicines list development and implementation takes place within a national policy space in which national key stakeholders deliberate and choose their policy options to determine availability and access to medicines at point of care. The national policy-level analysis of essential medicines lists development suggest that prioritization and availability are two sides of the same coin and benefit from being analysed as such and essential medicines lists represents an important interface between demand, and supply-side factors, however, this interface varies according to medicines that come into play with new and more expensive medicines tampering the balance between demand and supply. The differences in content of Essential Medicines Lists amongst countries are a constellation of actors and elements within the process. Introduction Evidence around governance issues regarding access to medicines remains scarce, relative to what is known about breakthroughs in biomedical technologies (Simao et al., 2018), and yet medicines account for a large proportion of national health expenditure (WHO, 2011). The concept of essential medicines advanced by the World Health Organization (WHO) in 1977 has been adopted as a global practice (Laing et al., 2003). Essential medicines are defined as those meeting priority health care needs of the population (WHO, 2002). However, considerations of essential medicines also include evidence of efficacy and safety, and both comparative costs and cost-effectiveness (Magrini et al., 2015). WHO revises the model list of essential medicines (MLEM) biennially, providing additional guidance for developing and implementing NEML (Magrini et al., 2015). The usefulness of the list has been tagged to its adaptation by member states as they develop national lists (Laing et al., 2003). The governance aspects of updating national essential medicines lists (NEML) is subject to limited systematic comparison, and especially in low- and middle-income countries (LMICs) (Mori et al., 2014). A common assumption that needs to be critiqued, is that once a select list of medicines is prioritized, e.g. through the NEML, there is improved access, higher quality of care, and rational and safe use of such medicines (WHO, 2002; Mugiraneza, 2009). Even though essential medicines policies are found to be associated with quality use of medicines, there is a stated need for better data about the co-implementation of medicine-related policies as well as the interaction between public and private sectors (Holloway et al., 2020). The national policy space and discourse where key domestic stakeholders deliberate and choose their policy options, often in collaboration with international partners, is a key determinant related to the availability and access to medicines at point of care (Mugiraneza, 2009). Therefore, more attention needs to be paid to how governance and financing at both global and national levels affect the availability of essential medicines (Bernstein and Cashore, 2012; Shaffer and Ginsburg, 2017). Greater engagement of health policy and systems research actors on issues of access to medicines in LMICs is necessary to fill current gaps in published literature (Adam et al., 2011). The WHO MLEM mainly includes off-patent medicines, which are generally affordable for most countries. However, in 2001, the WHO Executive Board (EB) indicated that absolute treatment cost should not disqualify a proposed addition to the model list, as long as a particular medicine meets the criteria for benefit and public health relevance. The issue of affordability, the EB argued, should be considered as a consequence to be managed after the list is developed (Magrini et al., 2015). Subsequently, in the past 15 years, the inclusion of these on-patent medicines on the MLEM has fuelled discussions within the global health community and among countries about what should be regarded as an ‘essential’ or ‘cost-effective’ product (Laing et al., 2003). Following the 2015 revision of MLEM, many new or relatively new and expensive medicines to treat cancer, hepatitis C and multidrug-resistant tuberculosis were added to the list (Magrini et al., 2015). The extent to which these aforementioned medicines become part of the access norm, through inclusion in revised editions of NEMLs, remains to be established (Ferrario et al., 2018). In addition to the increasing number of expensive on-patent medicines to the MLEM and specific guidance by WHO on the update process, the last two decades have witnessed changes in pharmaceutical markets with increasing importance of emerging country manufacturers, together with a growing density of global health governance arrangements designed to assist LMIC in adopting and delivering at point of care medicines against key infectious diseases, particularly HIV/AIDS and tuberculosis (WHO 2001a; Laing et al., 2003; Bigdeli et al., 2013; Hill et al., 2018). As a result, national authorities have to contend with increasing number of actors with interest in the development and implementation of national policies relevant to medicines access. Furthermore, country commitments to the sustainable development goals, which sought to reposition access to quality essential medicines as part of efforts towards universal health coverage (UHC), adds to this complexity (Bigdeli et al., 2015). A rising global concern for increased prevalence of non-communicable diseases and hence the case for equitable access to these medicines points towards the possibility of a more inclusive global agenda. Evidently, however, understanding these dynamics and their implications for the national policy processes requires a broader conceptual view (Bigdeli et al., 2013). There is limited evidence on how countries manage these governance situations, including coordination of multiple stakeholders, funding sources and procurement organizations, and various supply chains within a single national policy environment to support and facilitate access to essential medicines (Kraiselburd and Yadav, 2013; Simao et al., 2018). Therefore, in this study we sought to examine how government agencies and other actors, including non-state actors and international partners in Kenya, Uganda and Tanzania participate in and influence the process of updating their NEML and making prioritized medicines available. In addition, we examined the extent to which the processes in the three countries were aligned to the WHO guidance, paying particular attention to new addition to the 2015 MLEM with focus on tuberculosis, hepatitis C and cancer medicines. This group of medicines formed the bulked of the new addition to the 2015 MLEM. In balancing a wide scope with small-scale empirical data, the study is suited as an initial building block for exploring policy-options and further paths for inquiry. The comparative design, examining the policy-processes in three settings, further strengthens the methodological approach we adopted. Methodology A mixed method case study approach was used, relying mainly on qualitative data, collected using semi-structured interviews and a document review guide. Documents for the review were searched from four electronic databases (Allied and Complementary Medicine, Embase, MEDLINE and PsycINFO). Google scholar and websites of Ministries of Health, National Medicine Regulatory Authorities (NMRAs) and national medicines procurement agencies of Kenya, Uganda and Tanzania were also searched to supplement what was obtained from the above databases. The choice of the study sites was informed largely due to the timeframe and the available resources. However, the data generated, and findings is adequate, and likely the same picture holds for countries of similar contextual environment; the LMIC. Searching databases involved the use of standard Boolean operators. Search on the databases, Google scholar and websites of NMRAs, Ministries of Health yielded 512 documents. Given time restraints the title and abstract review for the initial screening was completed by one reviewer (EP). The final documents used in the analysis were 42 after limiting to broad relevance to study objectives; and availability of full-length articles. The 42 documents also included the three countries’ NEMLs, national pharmaceutical (medicines/drug) policies, national health strategic plans, national health policies; and WHO MLEM 2013 and 2015. The document review was conducted to provide context and empirical literature on discourses in NEML, as well as facilitate identification of the new medicines added to the WHO 2015 MLEM and of those, which ones the countries added to their national lists. To complement information from the documentary review, key informant interviews were conducted between May and July 2018. The selected participants were senior officials at national level and had partaken in or had knowledge of the most recent NEML update process. Study participants were from Ministries of Health (MOH), NMRAs, Medical Procurement agencies, WHO country offices, Country offices of The Global Fund and non-governmental organizations. A 45 min in-person interview on average was conducted with each of the 20 purposely selected participants (Uganda 8, Kenya 7 and Tanzania 5). None of the invited study participants declined the invitation for interviews. We did not specifically set out to pilot test the interview guide, the authors (WDO, KIS and YD) discussed and revised the question guide. We also used the first two interviews to highlight additional probes to employ during successive interviews given that the questions were already broad and open ended. Interviews were digitally recorded and subsequently transcribed verbatim. The interviews complemented the documentary review by eliciting perspectives from key stakeholders on main drivers and barriers as countries consider adapting WHO revisions to their NEML (Supplementary data S1: Interview guide). The information from interviews was extracted by two authors; WDO and KIS. Where there were disagreements, it was addressed through discussion and consensus. In addition, eight (n = 8) of the study participants took part in a 2-day validation workshop with members of the research team in Mombasa, Kenya, in March 2019, during which preliminary findings were presented and a focus group discussion on select issues requiring clarification were addressed. A policy analysis approach adapted from Walt and Gilson’s Policy Triangle Framework guided the analysis. This framework (Walt and Gilson, 1994) posits that policy research needs to consider not just the content of policies (the NEML in this instance), but also the actors, context and process in order to be able to explain outcomes and assess implementation. The analysis was conducted based on preconceived thematic categories of the Policy Triangle Framework. This includes the four categories content (inclusions of the selected medicines on the NEML), actors, the process of updating the list and making medicines available, and finally the context. On the process, we developed a schema of the WHO process in order to examine alignment between the national process and WHO recommendations (Figure 1). Figure 1 Open in new tabDownload slide WHO guidance on the process for updating NEML. Guidance on the process for updating NEML [based on interview data and literature (Fulone et al. 2016; Albert et al. 2007; Perumal-Pillay and Suleman 2017)]. Ethical clearance was obtained from the three countries’ Institutional Review bodies: Kenya’s Kenyatta National Hospital and University of Nairobi Ethics and Research Committee, Tanzania’s National Institute for Medical Research, and Uganda’s Higher Degrees, Research and Ethics Committee at Makerere University School of Public Health. Informed consent of all research participants was obtained, and their confidentiality and privacy observed by anonymizing participants. Only synthesized responses are presented in the results section. Results Our survey of selected medicines and interviews with country policy practitioners suggest that the national prioritization of essential medicines is a task that extends beyond the update of the NEML, and is followed by decision-making junctures that are then applied to further down-select the medicines to be provided by the public health system. We found that these decisions play out differently according to medicine groups and contributes to complexity from the health systems perspective. The results are presented thematically according to the elements of the policy triangle framework and summarized in Figure 1 and Tables 1 and 2. Table 1 Thematic summary of findings regarding NEML updates in Kenya, Tanzania and Uganda based on the policy triangle framework Country . Content . Actors Those involved in the updating and related implementation . Process The interactions between countries new list(s) and the prioritized medicines being available . Context Affecting update process and ensuring the availability or use of the medicines . Kenya Essential Medicines List (KEML) First published 1981; Most recent: 2016 (5th edition) The total number of medicines increased by 46 (to a total of 674) Medicines grouped as core or specialist Hepatitis C: did not include 2015 WHO EML additions, but added Pegylated interferon alfa-2a and Ribvarin from the 2013 WHO EML (see Table 2). List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministries of Health National medicines and therapeutics committees lead process Global stakeholders—DANIDA through Health Sector Programme Support, USAID, Global Fund, UNICEF Followed the WHO recommended process Around 20 meetings of the technical committees No challenge reaching consensus The latest update not done concurrently with the revisions of clinical guidelines, as is the norm Cost-effectiveness findings and challenges with the necessary data means exclusion of its use in decision-making Fragmentation of medicines funding Challenges with dissemination mean the list is used less frequently for procurement and clinical practice Hepatitis C: unknown prevalence, thought to be low Tuberculosis: well-documented data on prevalence These common contextual factors were relevant in all three countries Health sector reforms, especially devolution of health services Global push for universal health coverage Limited knowledge and awareness about WHO MLEL and NML concept amongst health workers Global funding prioritizations and evolving health technologies influences the process and the content of the WHO EML, especially the inclusion of new medicines for the treatment of HIV and TB that are largely funded through global health financing initiatives Lack of country consultation in the WHO-EML In the three countries, there is a link between updated national medicines lists and procurement, namely that medicines are classified into sub-groups to facilitate reprioritization during procurement. Tanzania National Essential Medicines List for Tanzania (NEMLIT) First published 1971a; Most recent: 2017 (5th edition) Reduced the total number of medicines to 400 Antibiotics classified as access, watch or reserve List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministry of Health National medicines and therapeutics committees lead process Global stakeholders—WHO, iDSI, PRICELESS; PATH, Swiss Agency for Development and Cooperation and the Global Fund. Followed the WHO recommended process Over 10 meetings of the technical committees No challenge reaching consensus The latest list was updated concurrently with revision of standard treatment (clinical) guidelines HTA integrated in the process, which includes the use of cost-effectiveness findings Fragmentation of medicines funding The use of the list for procurement and clinical practice is widespread Cancer: Lacking reliable data on prevalence; Hepatitis C: prevalence unknown, though to be high Tuberculosis: well-documented data on prevalence Uganda Essential Medicines and Health Supplies List for Uganda (EMHSLU) First published 1991; Most recent: 2016 (6th edition) The total number of medicines increased by 206 (to a total of 687) Medicines grouped into Vital, Essential and Necessary List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministry of Health National medicines and therapeutics committees as transitioned into the Appropriate Medicine Use Advisory Group, which meets regularly Global stakeholders—WHO, Management Sciences for Health, Clinton Health Access Initiative, UNFPA, USAID Followed the WHO recommended process Over 10 meetings of the technical committees No challenge reaching consensus The latest list was updated concurrently with revision of standard treatment (clinical) guidelines Cost-effectiveness findings and challenges with the necessary data means exclusion of its use in decision-making The use of the list for procurement and in clinical practice is widespread Cancer: Lacking reliable data on prevalence; Hepatitis C: Unknown prevalence, thought to be low; Tuberculosis: Well documented data on prevalence Country . Content . Actors Those involved in the updating and related implementation . Process The interactions between countries new list(s) and the prioritized medicines being available . Context Affecting update process and ensuring the availability or use of the medicines . Kenya Essential Medicines List (KEML) First published 1981; Most recent: 2016 (5th edition) The total number of medicines increased by 46 (to a total of 674) Medicines grouped as core or specialist Hepatitis C: did not include 2015 WHO EML additions, but added Pegylated interferon alfa-2a and Ribvarin from the 2013 WHO EML (see Table 2). List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministries of Health National medicines and therapeutics committees lead process Global stakeholders—DANIDA through Health Sector Programme Support, USAID, Global Fund, UNICEF Followed the WHO recommended process Around 20 meetings of the technical committees No challenge reaching consensus The latest update not done concurrently with the revisions of clinical guidelines, as is the norm Cost-effectiveness findings and challenges with the necessary data means exclusion of its use in decision-making Fragmentation of medicines funding Challenges with dissemination mean the list is used less frequently for procurement and clinical practice Hepatitis C: unknown prevalence, thought to be low Tuberculosis: well-documented data on prevalence These common contextual factors were relevant in all three countries Health sector reforms, especially devolution of health services Global push for universal health coverage Limited knowledge and awareness about WHO MLEL and NML concept amongst health workers Global funding prioritizations and evolving health technologies influences the process and the content of the WHO EML, especially the inclusion of new medicines for the treatment of HIV and TB that are largely funded through global health financing initiatives Lack of country consultation in the WHO-EML In the three countries, there is a link between updated national medicines lists and procurement, namely that medicines are classified into sub-groups to facilitate reprioritization during procurement. Tanzania National Essential Medicines List for Tanzania (NEMLIT) First published 1971a; Most recent: 2017 (5th edition) Reduced the total number of medicines to 400 Antibiotics classified as access, watch or reserve List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministry of Health National medicines and therapeutics committees lead process Global stakeholders—WHO, iDSI, PRICELESS; PATH, Swiss Agency for Development and Cooperation and the Global Fund. Followed the WHO recommended process Over 10 meetings of the technical committees No challenge reaching consensus The latest list was updated concurrently with revision of standard treatment (clinical) guidelines HTA integrated in the process, which includes the use of cost-effectiveness findings Fragmentation of medicines funding The use of the list for procurement and clinical practice is widespread Cancer: Lacking reliable data on prevalence; Hepatitis C: prevalence unknown, though to be high Tuberculosis: well-documented data on prevalence Uganda Essential Medicines and Health Supplies List for Uganda (EMHSLU) First published 1991; Most recent: 2016 (6th edition) The total number of medicines increased by 206 (to a total of 687) Medicines grouped into Vital, Essential and Necessary List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministry of Health National medicines and therapeutics committees as transitioned into the Appropriate Medicine Use Advisory Group, which meets regularly Global stakeholders—WHO, Management Sciences for Health, Clinton Health Access Initiative, UNFPA, USAID Followed the WHO recommended process Over 10 meetings of the technical committees No challenge reaching consensus The latest list was updated concurrently with revision of standard treatment (clinical) guidelines Cost-effectiveness findings and challenges with the necessary data means exclusion of its use in decision-making The use of the list for procurement and in clinical practice is widespread Cancer: Lacking reliable data on prevalence; Hepatitis C: Unknown prevalence, thought to be low; Tuberculosis: Well documented data on prevalence a The 1977, 1981 and 1986 lists were only drug lists. The NEMLIT &STGs edition started in 1991, then 1997, 2001, 2007, 2013 and 2017. Open in new tab Table 1 Thematic summary of findings regarding NEML updates in Kenya, Tanzania and Uganda based on the policy triangle framework Country . Content . Actors Those involved in the updating and related implementation . Process The interactions between countries new list(s) and the prioritized medicines being available . Context Affecting update process and ensuring the availability or use of the medicines . Kenya Essential Medicines List (KEML) First published 1981; Most recent: 2016 (5th edition) The total number of medicines increased by 46 (to a total of 674) Medicines grouped as core or specialist Hepatitis C: did not include 2015 WHO EML additions, but added Pegylated interferon alfa-2a and Ribvarin from the 2013 WHO EML (see Table 2). List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministries of Health National medicines and therapeutics committees lead process Global stakeholders—DANIDA through Health Sector Programme Support, USAID, Global Fund, UNICEF Followed the WHO recommended process Around 20 meetings of the technical committees No challenge reaching consensus The latest update not done concurrently with the revisions of clinical guidelines, as is the norm Cost-effectiveness findings and challenges with the necessary data means exclusion of its use in decision-making Fragmentation of medicines funding Challenges with dissemination mean the list is used less frequently for procurement and clinical practice Hepatitis C: unknown prevalence, thought to be low Tuberculosis: well-documented data on prevalence These common contextual factors were relevant in all three countries Health sector reforms, especially devolution of health services Global push for universal health coverage Limited knowledge and awareness about WHO MLEL and NML concept amongst health workers Global funding prioritizations and evolving health technologies influences the process and the content of the WHO EML, especially the inclusion of new medicines for the treatment of HIV and TB that are largely funded through global health financing initiatives Lack of country consultation in the WHO-EML In the three countries, there is a link between updated national medicines lists and procurement, namely that medicines are classified into sub-groups to facilitate reprioritization during procurement. Tanzania National Essential Medicines List for Tanzania (NEMLIT) First published 1971a; Most recent: 2017 (5th edition) Reduced the total number of medicines to 400 Antibiotics classified as access, watch or reserve List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministry of Health National medicines and therapeutics committees lead process Global stakeholders—WHO, iDSI, PRICELESS; PATH, Swiss Agency for Development and Cooperation and the Global Fund. Followed the WHO recommended process Over 10 meetings of the technical committees No challenge reaching consensus The latest list was updated concurrently with revision of standard treatment (clinical) guidelines HTA integrated in the process, which includes the use of cost-effectiveness findings Fragmentation of medicines funding The use of the list for procurement and clinical practice is widespread Cancer: Lacking reliable data on prevalence; Hepatitis C: prevalence unknown, though to be high Tuberculosis: well-documented data on prevalence Uganda Essential Medicines and Health Supplies List for Uganda (EMHSLU) First published 1991; Most recent: 2016 (6th edition) The total number of medicines increased by 206 (to a total of 687) Medicines grouped into Vital, Essential and Necessary List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministry of Health National medicines and therapeutics committees as transitioned into the Appropriate Medicine Use Advisory Group, which meets regularly Global stakeholders—WHO, Management Sciences for Health, Clinton Health Access Initiative, UNFPA, USAID Followed the WHO recommended process Over 10 meetings of the technical committees No challenge reaching consensus The latest list was updated concurrently with revision of standard treatment (clinical) guidelines Cost-effectiveness findings and challenges with the necessary data means exclusion of its use in decision-making The use of the list for procurement and in clinical practice is widespread Cancer: Lacking reliable data on prevalence; Hepatitis C: Unknown prevalence, thought to be low; Tuberculosis: Well documented data on prevalence Country . Content . Actors Those involved in the updating and related implementation . Process The interactions between countries new list(s) and the prioritized medicines being available . Context Affecting update process and ensuring the availability or use of the medicines . Kenya Essential Medicines List (KEML) First published 1981; Most recent: 2016 (5th edition) The total number of medicines increased by 46 (to a total of 674) Medicines grouped as core or specialist Hepatitis C: did not include 2015 WHO EML additions, but added Pegylated interferon alfa-2a and Ribvarin from the 2013 WHO EML (see Table 2). List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministries of Health National medicines and therapeutics committees lead process Global stakeholders—DANIDA through Health Sector Programme Support, USAID, Global Fund, UNICEF Followed the WHO recommended process Around 20 meetings of the technical committees No challenge reaching consensus The latest update not done concurrently with the revisions of clinical guidelines, as is the norm Cost-effectiveness findings and challenges with the necessary data means exclusion of its use in decision-making Fragmentation of medicines funding Challenges with dissemination mean the list is used less frequently for procurement and clinical practice Hepatitis C: unknown prevalence, thought to be low Tuberculosis: well-documented data on prevalence These common contextual factors were relevant in all three countries Health sector reforms, especially devolution of health services Global push for universal health coverage Limited knowledge and awareness about WHO MLEL and NML concept amongst health workers Global funding prioritizations and evolving health technologies influences the process and the content of the WHO EML, especially the inclusion of new medicines for the treatment of HIV and TB that are largely funded through global health financing initiatives Lack of country consultation in the WHO-EML In the three countries, there is a link between updated national medicines lists and procurement, namely that medicines are classified into sub-groups to facilitate reprioritization during procurement. Tanzania National Essential Medicines List for Tanzania (NEMLIT) First published 1971a; Most recent: 2017 (5th edition) Reduced the total number of medicines to 400 Antibiotics classified as access, watch or reserve List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministry of Health National medicines and therapeutics committees lead process Global stakeholders—WHO, iDSI, PRICELESS; PATH, Swiss Agency for Development and Cooperation and the Global Fund. Followed the WHO recommended process Over 10 meetings of the technical committees No challenge reaching consensus The latest list was updated concurrently with revision of standard treatment (clinical) guidelines HTA integrated in the process, which includes the use of cost-effectiveness findings Fragmentation of medicines funding The use of the list for procurement and clinical practice is widespread Cancer: Lacking reliable data on prevalence; Hepatitis C: prevalence unknown, though to be high Tuberculosis: well-documented data on prevalence Uganda Essential Medicines and Health Supplies List for Uganda (EMHSLU) First published 1991; Most recent: 2016 (6th edition) The total number of medicines increased by 206 (to a total of 687) Medicines grouped into Vital, Essential and Necessary List indicates the level of healthcare level in which a particular medicine is to be found State and non-state, national and international actors Clearly prescribed roles under purview of Ministry of Health National medicines and therapeutics committees as transitioned into the Appropriate Medicine Use Advisory Group, which meets regularly Global stakeholders—WHO, Management Sciences for Health, Clinton Health Access Initiative, UNFPA, USAID Followed the WHO recommended process Over 10 meetings of the technical committees No challenge reaching consensus The latest list was updated concurrently with revision of standard treatment (clinical) guidelines Cost-effectiveness findings and challenges with the necessary data means exclusion of its use in decision-making The use of the list for procurement and in clinical practice is widespread Cancer: Lacking reliable data on prevalence; Hepatitis C: Unknown prevalence, thought to be low; Tuberculosis: Well documented data on prevalence a The 1977, 1981 and 1986 lists were only drug lists. The NEMLIT &STGs edition started in 1991, then 1997, 2001, 2007, 2013 and 2017. Open in new tab Table 2 New tuberculosis, cancer and hepatitis C medicines on the 2015 WHO Model List vis à vis the addition to the latest National EMLs Kenya (2016), Uganda (2016) and Tanzania (2017) WHO Model List 2015 Update . Kenya (2016) . Uganda (2016) . Tanzania (2017) . Tuberculosis Bedaquiline Delamanid Linezolid Rifapentine Bedaquilinea Capreomycin Cycloserine Delamanida Levofloxacin Linezolida Moxifloxacin p-aminosalicylic acid Prothionamide Rifabutin Amikacin Bedaquilinea Capreomycin Clofazimine Cycloserine Ethionamide Kanamycin Levofloxacin Linezolida Moxifloxacin P-Aminosalicylic acid Prothionamide Amoxicillin + Clavulanic Acid Bedaquilinea Delamanida Linezolida Cancer All-trans retinoid acid Bendamustine Capecitabine Cisplatin Filgrastim Fludarabine Gemicitabine Imatinib Irinotecan Oxaliplatin Rituximab Trastuzumab Vinorelbine Alendronic acidb Anastrozolea Bicalutamidea Capecitabinea Diethylstiboestrolb Docetaxel Filgrastima Gemicitabinea Goserelinb Ifosfamide Imatiniba Irinotecana Melphalanb Mesna Oxaliplatin Paclitaxel Rituximaba Tamoxifen Thalidomide Trastuzumaba Vinorelbine Anastrozolea Bicalutamidea Capecitabinea Dactinomycin Filgrastim Fluorouracil Gemicitabinea Goserelin Irinotecan Mesna Oxaliplatina Paclitaxel Thalidomide Vinblastine Rituximab already on the previous national list and retained in 2016 version Bicalutamidea Cisplatina Gemcitabinea Imatiniba Irinotecana Oxaliplatina Rituximaba Trastuzumaba HepatitisC Sofosbuvir Simeprevir Daclatasvir Dasabuvir ledipasvir + sofosbuvir ombitasvir+ paritaprevir + ritonavir Pegylated interferon alfa-2a Ribavirin Sofosbuvira Ledpasvira Ribavirin WHO Model List 2015 Update . Kenya (2016) . Uganda (2016) . Tanzania (2017) . Tuberculosis Bedaquiline Delamanid Linezolid Rifapentine Bedaquilinea Capreomycin Cycloserine Delamanida Levofloxacin Linezolida Moxifloxacin p-aminosalicylic acid Prothionamide Rifabutin Amikacin Bedaquilinea Capreomycin Clofazimine Cycloserine Ethionamide Kanamycin Levofloxacin Linezolida Moxifloxacin P-Aminosalicylic acid Prothionamide Amoxicillin + Clavulanic Acid Bedaquilinea Delamanida Linezolida Cancer All-trans retinoid acid Bendamustine Capecitabine Cisplatin Filgrastim Fludarabine Gemicitabine Imatinib Irinotecan Oxaliplatin Rituximab Trastuzumab Vinorelbine Alendronic acidb Anastrozolea Bicalutamidea Capecitabinea Diethylstiboestrolb Docetaxel Filgrastima Gemicitabinea Goserelinb Ifosfamide Imatiniba Irinotecana Melphalanb Mesna Oxaliplatin Paclitaxel Rituximaba Tamoxifen Thalidomide Trastuzumaba Vinorelbine Anastrozolea Bicalutamidea Capecitabinea Dactinomycin Filgrastim Fluorouracil Gemicitabinea Goserelin Irinotecan Mesna Oxaliplatina Paclitaxel Thalidomide Vinblastine Rituximab already on the previous national list and retained in 2016 version Bicalutamidea Cisplatina Gemcitabinea Imatiniba Irinotecana Oxaliplatina Rituximaba Trastuzumaba HepatitisC Sofosbuvir Simeprevir Daclatasvir Dasabuvir ledipasvir + sofosbuvir ombitasvir+ paritaprevir + ritonavir Pegylated interferon alfa-2a Ribavirin Sofosbuvira Ledpasvira Ribavirin a Medicines that were included in the NEMLs from the 2015 update of the WHO Model List. b Medicines not on the 2015 or 2013 WHO MLEM but on the national list. Open in new tab Table 2 New tuberculosis, cancer and hepatitis C medicines on the 2015 WHO Model List vis à vis the addition to the latest National EMLs Kenya (2016), Uganda (2016) and Tanzania (2017) WHO Model List 2015 Update . Kenya (2016) . Uganda (2016) . Tanzania (2017) . Tuberculosis Bedaquiline Delamanid Linezolid Rifapentine Bedaquilinea Capreomycin Cycloserine Delamanida Levofloxacin Linezolida Moxifloxacin p-aminosalicylic acid Prothionamide Rifabutin Amikacin Bedaquilinea Capreomycin Clofazimine Cycloserine Ethionamide Kanamycin Levofloxacin Linezolida Moxifloxacin P-Aminosalicylic acid Prothionamide Amoxicillin + Clavulanic Acid Bedaquilinea Delamanida Linezolida Cancer All-trans retinoid acid Bendamustine Capecitabine Cisplatin Filgrastim Fludarabine Gemicitabine Imatinib Irinotecan Oxaliplatin Rituximab Trastuzumab Vinorelbine Alendronic acidb Anastrozolea Bicalutamidea Capecitabinea Diethylstiboestrolb Docetaxel Filgrastima Gemicitabinea Goserelinb Ifosfamide Imatiniba Irinotecana Melphalanb Mesna Oxaliplatin Paclitaxel Rituximaba Tamoxifen Thalidomide Trastuzumaba Vinorelbine Anastrozolea Bicalutamidea Capecitabinea Dactinomycin Filgrastim Fluorouracil Gemicitabinea Goserelin Irinotecan Mesna Oxaliplatina Paclitaxel Thalidomide Vinblastine Rituximab already on the previous national list and retained in 2016 version Bicalutamidea Cisplatina Gemcitabinea Imatiniba Irinotecana Oxaliplatina Rituximaba Trastuzumaba HepatitisC Sofosbuvir Simeprevir Daclatasvir Dasabuvir ledipasvir + sofosbuvir ombitasvir+ paritaprevir + ritonavir Pegylated interferon alfa-2a Ribavirin Sofosbuvira Ledpasvira Ribavirin WHO Model List 2015 Update . Kenya (2016) . Uganda (2016) . Tanzania (2017) . Tuberculosis Bedaquiline Delamanid Linezolid Rifapentine Bedaquilinea Capreomycin Cycloserine Delamanida Levofloxacin Linezolida Moxifloxacin p-aminosalicylic acid Prothionamide Rifabutin Amikacin Bedaquilinea Capreomycin Clofazimine Cycloserine Ethionamide Kanamycin Levofloxacin Linezolida Moxifloxacin P-Aminosalicylic acid Prothionamide Amoxicillin + Clavulanic Acid Bedaquilinea Delamanida Linezolida Cancer All-trans retinoid acid Bendamustine Capecitabine Cisplatin Filgrastim Fludarabine Gemicitabine Imatinib Irinotecan Oxaliplatin Rituximab Trastuzumab Vinorelbine Alendronic acidb Anastrozolea Bicalutamidea Capecitabinea Diethylstiboestrolb Docetaxel Filgrastima Gemicitabinea Goserelinb Ifosfamide Imatiniba Irinotecana Melphalanb Mesna Oxaliplatin Paclitaxel Rituximaba Tamoxifen Thalidomide Trastuzumaba Vinorelbine Anastrozolea Bicalutamidea Capecitabinea Dactinomycin Filgrastim Fluorouracil Gemicitabinea Goserelin Irinotecan Mesna Oxaliplatina Paclitaxel Thalidomide Vinblastine Rituximab already on the previous national list and retained in 2016 version Bicalutamidea Cisplatina Gemcitabinea Imatiniba Irinotecana Oxaliplatina Rituximaba Trastuzumaba HepatitisC Sofosbuvir Simeprevir Daclatasvir Dasabuvir ledipasvir + sofosbuvir ombitasvir+ paritaprevir + ritonavir Pegylated interferon alfa-2a Ribavirin Sofosbuvira Ledpasvira Ribavirin a Medicines that were included in the NEMLs from the 2015 update of the WHO Model List. b Medicines not on the 2015 or 2013 WHO MLEM but on the national list. Open in new tab The process The process of updating NEMLs can be investigated from the perspective of the stages and interactions during each update, or more broadly, how the practice in each country has developed over time. While this study considers the former, the initial development of a national list and the frequency of updates form a backdrop. Tanzania was the first country of the three to issue an NEML in 1977, updated in 1981 and 1986 (Munishi, 1995); subsequent updates in 1991, 1997, 2007, 2013 and 2017 (Munishi, 1995; Tanzania Ministry of Health, 2017). From 1991, Tanzania NEML (NEMLIT) reviews were conducted concurrently with revisions of the standard treatment guidelines (STG) as recommended by the WHO (WHO 2013). The first edition of the Essential Medicines and Health Supplies List for Uganda (EMHSLU) was published in 1991 with subsequent updates in 1996, 2001, 2007, 2012 and 2016 (Uganda Ministry of Health, 2016). Kenya, on the other hand, developed its first essential medicines list in 1981, with subsequent updates published in 1993, 2003, 2010 and 2016 (Kenya Ministry of Health, 2016). While there is a consistency in how the process is implemented, the intervals vary and have at times been as long as 10 years. For an NEML update, the WHO recommends the selection of committee members, and the procedure for the structure of interactions and decision-making (WHO 1977, 2001b). Figure 1 summarizes WHO guidance on the process of updating the NEML based on available literature and the interview data. As depicted in Figure 1, WHO guidance is quite prescriptive and structured for the selection and committee processes until the list is in the publication stage. However, it is not explicit on the implementation stage, as the process enters into the broader practice of making the prioritized medicines available. Examination of the 2016 (Kenya and Uganda) and 2017 (Tanzania) updates, described in the following section, considered factors that influence actors’ coordination, as well as links between the nature of the process and observed outcomes. The data show that all three NEML update processes were coordinated by national medicines and therapeutic committees (NMTC) anchored at a high level within the Ministries of Health, with input from WHO country offices. The committees had representation from a wide range of stakeholders, or consulted other stakeholders not represented on the committees through technical working groups (TWGs), consultative meetings and written submissions (WHO, 2001a, 2002). The number of meetings varied from 10 to 20. Uganda and Tanzania updated the clinical guidelines (STG) and NEML concurrently. In Kenya, for the update in question, the STG was not concurrently updated with NEML as recommended, reportedly due to funding constraints. None of the interviewees reported any coordination challenges in terms of involving the recommended stakeholders during the update process of the national lists. In all three countries the activities related to updating the NEML, including meetings and workshops were largely funded by development partners. The responsible units at Ministries of Health (MOH) budget for this process, however this usually sits amongst MOH’s unfunded priorities. The main criteria for inclusion and exclusion of medicines on the lists were efficacy, safety, quality, clinical experience, disease prevalence, cost and cost-effectiveness. It is only Tanzania that incorporated cost-effective criteria using the health technology assessment (HTA) during the update. Uganda and Kenya intended to incorporate cost-effectiveness assessment; however, such methods were not used because HTA had yet to be implemented as a policy tool, and difficulties accessing the relevant data to be used for cost-effectiveness calculations. Neither market authorization, nor status of medicine registration in the country was amongst criteria for inclusion or exclusion in all the three countries. Once NEMLs are confirmed, the next step is the implementation, initial step being sharing the list with the health system at large. In Uganda and Tanzania, interviewees reported that the NEML was disseminated widely and it is used highly in public and private health facilities. In Kenya, the list has not been disseminated as widely, largely due to limitation in funding. The focus group discussion suggested that the limited dissemination of the list manifests itself as the concept of the essential medicines list not being well understood amongst key stakeholders in both public and private sectors, especially prescribers. On a more general level, the overall demand for and use of the list depends on the general awareness of the essential medicines concept, availability of the list and timely updates. Following the NEML update and dissemination, interview data show that funding and procurement are additional decision-making points that prioritize which medicines become available. In all the three countries, the procurement of medicines and other health supplies in the public sector are based on the NEML; however, final procurement decisions are influenced by the available budget allocated in the year. To a varying extent among the three countries, global initiatives also support procurement of medicines especially the programmatic medicines for HIV, Malaria and TB. However, they require co-funding arrangements with governments as a mechanism of promoting sustainability, and the co-funding is usually drawn from the same source as for other essential medicines, influencing prioritization of government funds for essential medicines (Uganda Ministry of Health et al., 2012). Actors In all the three countries, as mentioned above, the process was led by the NMTC. The WHO recommends inclusion of a wide range of stakeholders to the committee (WHO, 2001a, 2002). Documentation and interviews show that Kenya had the least number of committee members 10, compared with 20 each in Uganda and Tanzania. In Kenya, all the members of the NMTC were from MOH (Kenya Ministry of Health, 2016), while in Uganda and Tanzania the committees had membership from other stakeholder groups (Uganda Ministry of Health, 2016; Tanzania Ministry of Health, 2017). In all countries, the committees established a number of subordinate TWGs with representatives from a number of key stakeholder groups from both public and private sectors, including health professional associations, public and private hospitals, local government representatives, health development partners and the WHO country office. Not all TWG members are also members of the NMTC. Departing from this uniformity is a variation in development partners involved in the process, as is shown in Table 1. The terms of reference for the NMTCs in the three countries extend beyond coordination of updating of the national medicines list, it includes policy advice on rational use of medicine and clinical guideline development among others. The Committees tended to be inactive between updates, to be reestablished when the List is to be updated. However, in Uganda the NMTC has now been reconstituted into the Appropriate Medicine Use Advisory Group with regular meetings and issuance of guidance as and when required. With regard to pharmaceutical quality control and procurement, the National Medicines Regulatory Authorities (Kenya Pharmacy and Poisons Board, Uganda National Drugs Authority, and Tanzania Food and Drugs Authority) and the National Medical procurement and supplies agencies [National Medical Stores in Uganda (NMS), Kenya Medicine Supplies Agency (KEMSA) and Medical Stores department (MSD)] are cornerstone actors in all the three countries. These two governmental agencies administer and ensure the actual availability of medicines in the public health system, and are the technical agencies that manage logistics, quality, efficacy and safety of essential medicines. As a result, they bridge funding decisions by Ministries of Finance, global health initiatives, and pharmaceutical manufacturers. For all three countries most of the medicines are imported from Indian pharmaceutical companies, and to a limited extent from China. However, our interview data confirm that these companies or their representatives play no active role in the update process of the EML or further prioritization. The same applies for western multinational pharmaceutical industries. In this way, most of the actors who need to coordinate are firmly positioned within the health (as opposed to trade) sector. The context The contextual factors affecting update process and ensuring the availability or use of the medicines in the list are similar in the three countries and are summarized in Table 1. Our interview data suggest two broad categories of contextual drivers: firstly, domestic health system changes, which includes the transition towards UHC and reforms such as devolution of responsibilities from central to district level; and secondly, global health aid funding. Limited information related to changes at the global level and shifting co-financing arrangements challenge governments as they plan medicine procurement budgets. In addition, global initiatives can introduce coordination difficulties if they call for individual medicine updates, initiate parallel procurement and supply chains, or merely conduct negotiations with pharmaceutical industry outside the purview of national stakeholders. Discussion at global level, e.g. between global health financing initiatives and pharmaceutical industries usually do not involve national stakeholders. This has been reportedly so for medicines for TB and HIV/AIDS. Content Overall, the latest updates compared with the previous edition indicate there was a net increase of 206 medicines to the 2016 Kenya list (total 687), and 46 to the 2016 Uganda list (total of 674). In Tanzania, the total number of medicines in the list reduced to 400 compared with >500 that were in the 2013 national list (Uganda Ministry of Health, 2012, 2016; Kenya Ministry of Health, 2016). The increase in Kenya was attributed to the push by the specialist groups, especially the oncologists, whereas the reduction in Tanzania has been attributed to use of HTA. In reviewing how countries aligned their update with the 2015 MLEM, and with a focus of medicines against tuberculosis, cancer and hepatitis C, the most distinct variations among the countries were those medicines for cancer and hepatitis C. Kenya exceeded the additions to the 2015 WHO MLEM regarding treatments for cancer (21 new anticancer medicines were added to the list) (Table 2) (WHO, 2015). One reason, from interview and focus group discussion is that specialist treatment is more developed in the Kenyan health system compared with the other two countries and that clinicians advocate for enhanced access more. Regarding hepatitis C, Tanzania added three of the new medicines to its essential list, including sofosbuvir. Kenya and Uganda did not add the new drugs included in the 2015 MLEM. The non-inclusion of hepatitis C medicines in Kenya and Uganda lists was reportedly due to limited information on hepatitis C prevalence. All three countries included new and expensive treatment for MDR TB, Bedaquiline. All countries also included linezolid for treating tuberculosis, while only Tanzania and Kenya included delamanid to their lists. The inclusion of expensive, second line anti-TB medicines is, to an extent, driven by global funding available for these medicines through the global financing initiatives, especially the Global Fund to Fight AIDS, TB and Malaria. In the three countries, the medicines are classified into sub-groups to facilitate reprioritization during procurement. In the KEML, medicines have been grouped as either core or specialist. In Uganda, a medicine is vital, essential or necessary (VEN) and in Tanzania, for antibiotics, it has been classified as access, watch or reserve. Medicines considered Vital must always be available (for instance against life-threatening conditions, or first line treatment), while the Essential medicines are against common illnesses, and Necessary medicines are those against minor diseases with limited impact on the population, or high average cost for marginal therapeutic benefit (WHO, 1998; Kenya Ministry of Health, 2016; Uganda Ministry of Health, 2016; Tanzania Ministry of Health, 2017). For all the three countries, the lists indicate the level of healthcare facility in which a particular medicine is to be made available (Tanzania Ministry of Health, 2013, 2017; Kenya Ministry of Health, 2016). Discussion The three countries’ NEML update processes were similar and showed close alignment to recommended WHO procedures. All countries applied similar criteria for considering changes to the list. However, cost-effectiveness estimations were the most challenging and only Tanzania applied HTA methods through the support of the International Decision Support Initiative (IDSI) (International Decision Support Initiative, 2018). The limited use of HTA in the update process in the region is in line with findings from other studies (Mori et al., 2014; Perumal-Pillay and Suleman, 2016, 2017). Update processes in the studied countries have been largely funded by health development partners, and perhaps the unpredictability that comes with donor funding explains the irregularity in the intervals between updates, ranging from 5 to 10 years. Another notable feature for the countries included in the study was that many of the medicines have remained on countries NEMLs for a long period of time (Uganda Ministry of Health, 2012; Tanzania Ministry of Health, 2013, 2017; Kenya Ministry of Health, 2016). This is perhaps due to their enduring qualities as safe, efficacious and affordable. However, changes in the local burden of disease and pharmaceutical innovation prompt the need to trade older medicines for new ones, or to consider entire new classes of medicines. Availability of medicines on the NEML in the public sector rely on government funding allocations to the Ministries of Health (Wirtz et al., 2017). Therefore, financing and affordability are a consideration during the NEML update; whether this is implicitly or explicitly stated. This is reflected in prioritization within the list in terms of medicines classification as either VEN, with available public funding prioritized to vital and essential medicines. As such, even though medicine characteristics and local need represents a potential, its actual availability and accessibility especially in the public sector is inextricably linked to funding. Further downstream, the link between the updated NEML and procurement planning in the public sector is also the extent to which the Ministry of Health makes the list known to facilities and professionals who procure and prescribe medicines. Once known, its use also depends on it being considered relevant, and not outdated thus requiring the regular updates. However, utilization of the list is likely to work best when the concept of the NEML is integrated as a continuous practice, for instance through professional training, rather than as an implementation or a launch event following each update. Findings by Holloway et al. (2020) indicate that undergraduate training of prescribers in STG is associated with quality use of medicines in comparison to only the biennial updates of the list amongst other NEML implementation enablers. In developing the NEML, a key objective is that those medicines considered essential will be made available (Laing et al., 2003). Therefore, in the update process, the committee ought to take into account issues of supply chain system including the feasibility of procurement and funding wholesomely. For all of the countries, market authorization was not considered a criterion. According to the WHO, market approval is a regulatory decision on which availability may be conditioned given that it can be a proxy for availability of medicines (WHO, 2002). However, there are regulatory frameworks in the three countries that allow unregistered medicines (non-market authorized) to be brought into the country under special conditions, such as in public health emergencies. The differences in content of the three countries by way of the essential medicines list updates resulting in a longer or shorter list is likely a constellation of actors and elements within the process. The findings point to a growing dilemma of increasing the number of accessible medicines, for instance against non-communicable diseases such as cancers, and raising the governments’ ability to finance medicines in the public sector. While the process in Kenya appears to have emphasized the former, the introduction of HTA in Tanzania incorporated the latter to a greater extent. However, the present study did not examine the HTA experience in Tanzania beyond publicly available documentation, it cannot detail explicitly how additional economic analyses contributed to the NEML update process outcome (International Decision Support Initiative, 2018). A nascent literature on HTA and priority setting in Africa nevertheless suggests that there is room for discussion as to how economic analysis is carried out, balancing commissioned research with local capacity building (Doherty et al., 2017). An initial assumption of this study was that coordination between the health and trade sectors at the national policy level represents an obstacle for access to medicines in relation to the NEML. The country perspectives in this study, however, suggest that coordination challenges within the health sector may be more problematic. The reason is the multiple actors within the health sector, especially at the global level with influence or interest at the national level. Apart from the procurement by national medical stores or other national wholesalers, the intersections between national healthcare delivery systems and the pharmaceutical sector occur globally, involving public−private partnerships to incentivize development and introduction of new medicines, conduction of market-shaping activities including bulk procurement, and direct price negotiations. Activities centre around select medicines and may change over time, and country policymakers are poorly represented or rarely directly involved with such initiatives. While these activities of global initiatives may produce substantive gains in terms of access to new and affordable medicines, it conflicts with national ownership, sustainability and the continuation of government funded medicines at the national level (Kusemererwa et al., 2016; Rockers et al., 2018). At the nation level, our finding suggests that there is a window of opportunity to bring in the actors from the trade sector as part of the essential medicine policy development and as a way of engage the sector towards increasing local pharmaceutical production and supplies. Furthermore, other contextual factors, especially the evolving domestic policies on health systems financing including the establishment of national health insurance schemes and global commitment on UHC influences the process for updating the NEMLs. The way global initiatives support access to medicines varies with disease conditions, yet the update of the NEML is taken as a cross-cutting process at the national level. Therefore, there is a need for better engagement by global health financing initiatives and the country-level stakeholders to ensure smooth introduction of new medicines for programmatic medicines in light of the NEML and associated implementation constraints. Information about the disease prevalence affects inclusion decisions related to medicines on the list, as demonstrated by the case of hepatitis C medicines in the three countries. Bigdeli et al. (2013) have drawn attention to systemic factors at the national level that influence access, however, their argument largely focuses on going beyond the pharmaceutical sector to other sub-sectors of the health system at all levels; from local to the international. In studying the national policy level—this study suggests that there is still a need to connect demand and supply-side elements within the pharmaceutical area, spanning health and trade sectors. A key limitation of this study is that the exploratory nature of the study combined with modest resources did not allow for a full-fledged systematic review of literature, nor inclusion of additional informants that would be needed for a more in-depth analysis of supply-side factors. Still, the study confirms that by enabling broader representation and perspectives at the national level, the complexity of access becomes apparent, with its priorities, trade-offs, network of global, regional and national stakeholders. This complex system approach is echoed by Ozawa et al. (2019). However, the question remains how this can be best managed. This study suggests that when limited to the specific NEML update process, the three countries are well aligned in following the WHO guidance. In governing the frequency and interlinkages with other policy processes, however, there is a potential for cross-country harmonization and sharing of best practices. Bigdeli et al. (2013) argue that frameworks for understanding or exploring issue of access to medicines should take into consideration health system dynamics and complexities. For example, when focussing on the health sector, Bigdeli et al. (2013) suggest four constraints on access to medicines, including governance of the pharmaceutical sector (e.g. registration and procurement), prices, overall health sector governance and interaction between private and public services. They also recognize that the international context involves both the market for pharmaceuticals as well as donors’ agenda and funding, but that constraints in this domain is less documented in published literature (Bigdeli et al., 2013). In suggesting that the EML represents an important interface between demand and supply-side factors, this study finds that this interface varies according to medicines that come into play, with new and more expensive medicines tampering with the equilibrium between the two sides. In taking a broader view of health sector governance, one may also see this in relation to the capacity of government to manage actors, networks and institutions, constituting a health sector resilience (Topp, 2020). The remaining question is the extent to which governments are able to steer the engagement of international partners, mobilize own funding and decide on the extent to which new medicines will be offered in the private or public sector. Conclusions The present study shows that NEML updates tend to follow a structured process aligned to the WHO guidelines and are influenced by prioritization, the procurement planning, and considerations of demand, funding and cost. The national policy-level analysis of the three countries suggests that the implementation of NEMLs should be subject to heightened attention and systematic analysis, as prioritization and availability are two sides of the same coin and benefit from being analysed as such. Furthermore, STG should inform the NEML, this will make the list relevant to the practitioners and improve its use. Countries in the region need to ensure that health system financing and pharmaceutical policies at the national level promote the implementation of the Essential Medicines concept through building capacities for substantive coherence to ensure complementarity between policy outcomes. Furthermore, national policymakers need to exert their influence over intersections that are currently outside their own policy domains through enhancing global health diplomacy competencies of government officials involved in global pharmaceutical negotiations. In addition, for future research, it is worth further exploring the consequences of adding on-patent drugs for a variety of diseases on the MLEM and thus NEMLs in terms of access and the function of national EMLs. Supplementary data Supplementary data are available at Health Policy and Planning online. Conflict of interest statement. None declared. Funding This work was partly supported by the Research Council of Norway through the Global Health and Vaccination Programme (GLOBVAC Project 234608). Ethical approval. Ethical clearance was obtained from the Kenya’s Joint Kenyatta National Hospital and University of Nairobi Ethics and Research Committee (P118/03/2018), Tanzania’s National Institute for Medical Research (NIMR/HQ/R.8a/Vol.IX/2726), and Uganda’s Higher Degrees, Research and Ethics Committee at Makerere University School of Public Health (Protocol 543). Acknowledgements We thank the research assistants; Dr Darlington Muwhezi-Uganda, Ms Ann B. Masese-Kenya and Mr Ernest Nyoni—Tanzania. We also acknowledge the contribution of Trygve Ottersen; the overall project director of the i4C project that supported this study. References Adam T , Ahmad S, Bigdeli M, Ghaffar A, Røttingen J-A. 2011 . Trends in health policy and systems research over the past decade: still too little capacity in low-income countries . PLoS One 6 : e27263 . Google Scholar Crossref Search ADS PubMed WorldCat Albert MA , Fretheim A, Maiga D. 2007 . Factors influencing the utilization of research findings by health policy-makers in a developing country: the selection of Mali's essential medicines . Health Research Policy and Systems 5 : 2 . Google Scholar Crossref Search ADS PubMed WorldCat Bernstein S , Cashore B. 2012 . Complex global governance and domestic policies: four pathways of influence . International Affairs 88 : 585 – 604 . Google Scholar Crossref Search ADS WorldCat Bigdeli M , Jacobs B, Tomson G et al. . 2013 . Access to medicines from a health system perspective . Health Policy and Planning 28 : 692 – 704 . Google Scholar Crossref Search ADS PubMed WorldCat Bigdeli M , Laing R, Tomson G, Babar ZU. 2015 . Medicines and universal health coverage: challenges and opportunities . Journal of Pharmaceutical Policy and Practice 8 : 8 . Google Scholar Crossref Search ADS PubMed WorldCat Doherty J , Wilkinson T, Edoka I, Hofman K. 2017 . Strengthening expertise for health technology assessment and priority-setting in Africa . Global Health Action 10 : 1370194 . Google Scholar Crossref Search ADS PubMed WorldCat Ferrario A , Stephens P, Ross-Degnan D, Wagner A. 2018 . Trends in sales volumes of cancer medicines in six Asian countries working toward universal health coverage . Journal of Global Oncology 4 : 220s . Google Scholar Crossref Search ADS WorldCat Fulone I , Barberato-Filho S, dos Santos MF et al. . 2016 . Essential psychiatric medicines: wrong selection, high consumption and social problems . BMC Public Health 16 : 52 . Google Scholar Crossref Search ADS PubMed WorldCat Hill AM , Barber MJ, Gotham D. 2018 . Estimated costs of production and potential prices for the WHO Essential Medicines List . BMJ Global Health 3 : e000571 . Google Scholar Crossref Search ADS PubMed WorldCat Holloway KA , Ivanovska V, Manikandan S et al. . 2020 . Identifying the most effective essential medicines policies for quality use of medicines: a replicability study using three World Health Organisation data sets . PLoS One 15 : e0228201 . Google Scholar Crossref Search ADS PubMed WorldCat International Decision Support Initiative . 2018 . New Treatment Guidelines Launched in Tanzania [Online]. https://www.idsihealth.org/blog/new-treatment-guidelines-launched-in-tanzania/, accessed 2 June 2018. Kenya Ministry of Health . 2016 . Kenya Essential Medicines List 2016 . Nairobi : Kenya Ministry of Health . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Kraiselburd S , Yadav P. 2013 . Supply chains and global health: an imperative for bringing operations management scholarship into action . Production and Operations Management 22 : 377 – 381 . Google Scholar Crossref Search ADS WorldCat Kusemererwa D , Alban A, Obua OT, Trap B. 2016 . An exploratory study on equity in funding allocation for essential medicines and health supplies in Uganda's public sector . BMC Health Services Research 16 : 453 . Google Scholar Crossref Search ADS PubMed WorldCat Laing R , Waning B, Gray A, Ford N, Hoen E. 2003 . 25 years of the WHO essential medicines lists: progress and challenges . The Lancet 361 : 1723 – 9 . Google Scholar Crossref Search ADS WorldCat Magrini N , Robertson J, Forte G et al. . 2015 . Tough decisions on essential medicines in 2015 . Bulletin of the World Health Organization 93 : 283 – 4 . Google Scholar Crossref Search ADS PubMed WorldCat Mori AT , Kaale EA, Ngalesoni F, Norheim OF, Robberstad B. 2014 . The role of evidence in the decision-making process of selecting essential medicines in developing countries: the case of Tanzania . PLoS One 9 : e84824 . Google Scholar Crossref Search ADS PubMed WorldCat Mugiraneza CJ. 2009 . Is the World Health Organization Model List of Essential Drugs Relevant to Member States? National Essential Drugs Lists of Selected African Countries in Comparative Perspective . MSc, D’Youville College . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Munishi GK. 1991. The development of the essential drugs program and implications for self-reliance in Tanzania. Journal of Clinical Epidemiology 45 : 7 – 14 . Ozawa S , Shankar R, Leopold C, Orubu S. 2019 . Access to medicines through health systems in low- and middle-income countries . Health Policy and Planning 34 : iii1 – 3 . Google Scholar Crossref Search ADS PubMed WorldCat Perumal-Pillay VA , Suleman F. 2016 . Quantitative evaluation of essential medicines lists: the South African case study . BMC Health Services Research 16 : 687 . Google Scholar Crossref Search ADS PubMed WorldCat Perumal-Pillay VA , Suleman F. 2017 . Selection of essential medicines for South Africa - an analysis of in-depth interviews with national essential medicines list committee members . BMC Health Services Research 17 : 17 . Google Scholar Crossref Search ADS PubMed WorldCat Rockers PC , Laing RO, Wirtz VJ. 2018 . Equity in access to non-communicable disease medicines: a cross-sectional study in Kenya . BMJ Global Health 3 : e000828 . Google Scholar Crossref Search ADS PubMed WorldCat Shaffer G , Ginsburg T. 2017 . The empirical turn in international legal scholarship . American Journal of International Law 106 : 1 – 46 . Google Scholar Crossref Search ADS WorldCat Simao M , Wirtz VJ, Al-Ansary LA et al. . 2018 . A global accountability mechanism for access to essential medicines . Lancet (London, England) 392 : 2418 – 2420 . Google Scholar Crossref Search ADS PubMed WorldCat Tanzania Ministry of Health . 2013 . Standard Treatment Guidelines and Essential Medicines List. Dar es Salaam: Tanzania Ministry of Health. Tanzania Ministry of Health . 2017 . Standard Treatment Guidelines and National Essential Medicines List Tanzania Mainland . Dar es Salaam : Tanzania Ministry of Health . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Topp SM. 2020 . Power and politics: the case for linking resilience to health system governance . BMJ Global Health 5 : e002891 . Google Scholar Crossref Search ADS PubMed WorldCat Uganda Ministry of Health . 2012 . Essential Medicines and Health Supplies List for Uganda 2012 . Kampala : Uganda Ministry of Health . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Uganda Ministry of Health . 2016 . Essential Medicines and Health Supplies List for Uganda . Kampala : Uganda Ministry of Health . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Uganda Ministry of Health, Abt Associations Inc, Makerere University School of Public Health . 2012 . Uganda Health Systems Assessment 2011 [Online]. Abt. Associations Inc. http://health.go.ug/docs/hsa.pdf, accessed 12 June 2018. Walt G , Gilson L. 1994 . Reforming the health sector in developing countries: the central role of policy analysis . Health Policy and Planning 9 : 353 – 70 . Google Scholar Crossref Search ADS PubMed WorldCat WHO . 1977 . The selection of essential drugs-report of a WHO Expert Committee. WHO Technical Report Series . Geneva : WHO . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC WHO . 1998 . Guide to Drug Financing Mechanisms [Online]. Geneva : WHO . https://apps.who.int/medicinedocs/en/d/Jh2928e/, accessed 12 May 2018. WHO . 2001a . EB109/8—Revised procedure for updating WHO’s Model List of Essential Drugs—Report by the Secretariat. WHO Medicines Strategy. Executive Board, 109th Session, 7 December 2001 (Arabic version) [Online]. Geneva: WHO. http://apps.who.int/medicinedocs/en/m/abstract/Js22165ar/, accessed 20 May 2018. WHO . 2001b . WHO Medicines Strategy Revised Procedure for Updating. WHO’s Model List of Essential Drugs-Report by the Secretariat. WHO . 2002 . The Selection of Essential Medicines - WHO Policy Perspectives on Medicines [Online]. Geneva: WHO. https://apps.who.int/iris/bitstream/handle/10665/67375/WHO_EDM_2002.2.pdf; jsessionid=5009FBA4850DE535CDB0B6BCA54B39CA? sequence=1, accessed 19 June 2018. WHO . 2011 . The World Medicines Situation 2011: MEDICINE EXPENDITURES. Geneva : WHO . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC WHO . 2013 . WHO Model List of Essential Medicines-18th List . Geneva : WHO . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC WHO . 2015 . WHO Model List of Essential Medicines. 19th List (April 2015) (Amended November 2015) . Geneva : WHO . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Wirtz VJ , Hogerzeil HV, Gray AL et al. . 2017 . Essential medicines for universal health coverage . The Lancet 389 : 403 – 476 . Google Scholar Crossref Search ADS WorldCat © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contact [email protected] © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine.

Gage, Anna; Bauhoff, Sebastian

doi: 10.1093/heapol/czaa191pmid: 33491082

Abstract Maternal and newborn care has been a primary focus of performance-based financing (PBF) projects, which have been piloted or implemented in 21 countries in sub-Saharan Africa since 2007. Several evaluations of PBF have demonstrated improvements to facility delivery or quality of care. However, no studies have measured the impact of PBF programmes directly on neonatal health outcomes in Africa, nor compared PBF programmes against another. We assess the impact of PBF on early neonatal health outcomes and associated health care utilization and quality in Burundi, Lesotho, Senegal, Zambia and Zimbabwe. We pooled Demographic and Health Surveys and Multiple Indicator Cluster Surveys and apply difference-in-differences analysis to estimate the effect of PBF projects supported by the World Bank on early neonatal mortality and low birthweight. We also assessed the effect of PBF on intermediate outputs that are frequently explicitly incentivized in PBF projects, including facility delivery and antenatal care utilization and quality, and caesarean section. Finally, we examined the impact among births to poor or high-risk women. We found no statistically significant impact of PBF on neonatal health outcomes, health care utilization or quality in a pooled sample. PBF was also not associated with better health outcomes in each country individually, though in some countries and among poor women PBF improved facility delivery, antenatal care utilization or antenatal care quality. There was no improvement on the health outcomes among poor or high-risk women in the five countries. PBF had no impact on early neonatal health outcomes in the five African countries studied and had limited and variable effects on the utilization and quality of neonatal health care. These findings suggest that there is a need for both a deeper assessment of PBF and for other strategies to make meaningful improvements to neonatal health outcomes. Health financing, health systems, health facilities, maternal and child health, quality of care, health care utilization KEY MESSAGES We estimated the impact of performance-based financing (PBF) in Burundi, Lesotho, Senegal, Zambia and Zimbabwe on early neonatal death and low birthweight using a difference-in-differences approach. PBF did not reduce early neonatal mortality or low birthweight in across the five countries or in any country individually. There was also no improvement on the health outcomes among poor or high-risk women. There is a need for a deeper assessment of the costs and benefits of PBF projects and development of other strategies to improve neonatal health outcomes. Introduction Despite decades of declining neonatal mortality rates, many countries in sub-Saharan Africa are still not on track to reach the Sustainable Development Goal of 12 neonatal deaths per 1000 live births by 2030. Maternal and newborn care services provided in health care facilities are viewed as critical to accelerate progress on neonatal health outcomes (Gülmezoglu et al., 2016; World Health Organization, 2019). Improving the quantity and quality of maternal and newborn care services has been a primary focus of performance-based financing (PBF) projects in Africa in the past two decades, partly driven by support from the Health Results Innovation Trust Fund (HRITF) administered by the World Bank (Kandpal, 2016; Gergen et al., 2017). While there are many models of PBF, these projects generally entail a set of financing reforms that explicitly incentivize pre-defined quantity and quality indicators (Renmans et al., 2017). Through incentives, PBF aims to motivate providers to improve their performance, help attract more capable health workers, or provide additional funding that can support improvements (Lemière et al., 2013). Commonly incentivized maternal and newborn service indicators include the volume of antenatal care visits and facility deliveries and quality measures, such as all deliveries being conducted by qualified personnel and presence of proper maternity equipment (Gergen et al., 2017). A robust literature documents the variable impacts of African PBF projects on the quantity and quality of health care services. An influential early evaluation found that Rwanda’s PBF raised the number of facility deliveries and the quality of antenatal care, among other intermediate outputs, but did not improve the number of antenatal care visits (Basinga et al., 2011). Recent reviews similarly found that incentivizing health facilities to provide deliveries can increase their number, but mixed evidence on quality of care and quantity of antenatal care with variation across projects and indicators within projects (Witter et al., 2012; Lemière et al., 2013; Kandpal, 2016). There is currently no empirical evidence on the direct impact of PBF on neonatal health outcomes in African countries. Several studies have modelled health impacts of PBF based on changes to utilization and quality (Zeng et al., 2018; Chinkhumba et al., 2020), but direct evidence is critical for several reasons. First, changes in intermediate outputs may not always translate to better health. For example, increasing facility delivery may not improve neonatal health outcomes in the absence of high-quality care (Lim et al., 2010; Fink et al., 2015), and improved adherence to evidence-based checklists during delivery can fail to generate better maternal or newborn health outcomes (Semrau et al., 2015). Second, the evaluations to date have demonstrated mixed results, with improvements on some indicators, generally including facility delivery, but not on others, including delivery quality (Eichler et al., 2013; Turcotte-Tremblay et al., 2016). It is unclear how these inconsistent improvements may come together to affect health outcomes, and the modelling studies rely on strong assumptions about quality-adjusted coverage measures (Zeng et al., 2018; Chinkhumba et al., 2020). Third, PBF projects incentivize a particular set of indicators and it remains largely unclear whether there are negative or positive spill-overs. For example, PBF may inadvertently divert resources and attention but could also encourage closely associated beneficial behaviours that are not incentivized (Lemière et al., 2013; Sherry, 2016; Sherry et al., 2017). Finally, PBF projects generally pursue multiple strategies, so that focussing on intermediate outputs may miss other pathways to improved health outcomes. Examining the direct impact on health outcomes captures all pathways and spill-overs that are otherwise difficult to model in the context of complex adaptive systems (Paina and Peters, 2012). As improving maternal and child health outcomes, including neonatal health outcomes, is a primary objective of many PBF projects, it is important to evaluate these impacts directly (Bonfrer et al., 2014; Friedman et al., 2016a,b). In this paper, we empirically evaluate the impact of five PBF projects in Africa on two important neonatal health outcomes, neonatal mortality and low birthweight, as well as on intermediate outputs through which PBF may improve health outcomes: antenatal care utilization and quality, facility delivery utilization and quality, and caesarean section rates. We conduct both pooled and country-specific analyses, and also assess the impact of PBF for two vulnerable groups: poor women and women with high-risk births. Our analysis offers three primary contributions. First, we provide direct evidence of the impact of African PBF projects on neonatal mortality, avoiding the challenges faced by modelling studies. Second, we compare the effectiveness of PBF projects in different countries against one another using the same methods and data. Most evaluations focus on just one project and because they use differing methodologies, they are not directly comparable (Oxman and Fretheim, 2009; Eichler et al., 2013). As each project is implemented differently, a direct comparison can help to identify features of the health system context or project that may be more or less effective. Finally, our analysis represents a systematic replication of previous evaluations using alternative data sources (Bonfrer et al., 2014). Materials and methods Data and study countries Our analysis focussed on PBF projects in five African countries: Burundi, Lesotho, Senegal, Zambia and Zimbabwe. Countries were included into the study if they were in sub-Saharan Africa, had implemented an PBF project supported by the World Bank’s HRITF and for which the intervention provinces or districts are known, and had a publicly available nationally representative survey on health care and utilization both prior to and after implementation of the PBF project. Although Burundi did not have a survey prior to its PBF implementation, we were able to include Burundi by using just the post-DHS survey for a longer span of births. The DHS collects data on neonatal mortality for all births of the women respondents regardless of when the birth occurred. Burundi is excluded from the pooled analysis as a robustness check. Countries that assigned PBF to specific facilities or sub-districts within districts were further excluded from the study, as in this study the population’s treatment status was assigned by their district residence rather than by facility catchment areas. We used the Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) to assess the impact of PBF. Because there were differing amounts of time between the surveys and the PBF implementation in each country, we limited the analysis to births that occurred within 3 years before implementation and 2 years after. We also excluded all births from mothers outside of the defined treatment and control districts. In Zambia and Zimbabwe, data on the household’s district were not available directly from the surveys. In these cases, we used the cluster geocodes to place households in districts. Although DHS geocodes are displaced to maintain privacy, the displacement is restricted so that clusters stay within the second administrative level, or the district, in these countries (Burgert et al., 2013). We assumed that a household was treated if it was located within a PBF implementation district, and therefore that all facilities within implementation districts were treated and that women would have gone to facilities within her district. Table 1 summarizes the data sources used for each country. Table 1 PBF characteristics and data sources . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . First implementation date December 2006 July 2016 April 2012 April 2012 March 2012 Second implementation date October 2008 October 2016 N/A N/A N/A Additional rollout Expanded to control regions in April 2010 N/A Expanded to control regions in May 2016 Expanded to 39 districts in October 2016 Expanded to 44 districts in 2015 Pre-implementation survey DHS 2010a DHS 2014 DHS 2011 DHS 2007 DHS 2010 − 11 Post-implementation survey DHS 2010 and DHS 2017 MICS 2018 Continuous DHS 2013 − 17 DHS 2014 and DHS 2018 DHS 2015 District selection notes Randomized treatment to districts. Additional unconditional financing arm in 10 districts. Government selected implementation districts from pair-matched districts. Major related concurrent interventions Incentives for district teams for good quality of supervision and support to PBF project Demand-side vouchers also provided for four ANC visits and skilled delivery . Introduced simultaneously with national elimination of user fees for targeted services. Payment adjustment on other dimensions Remoteness, poverty, staff and facility needs Remoteness Remoteness Remoteness Allocation of PBF payment  Health facility 70% 50% 25% 40% 75%  Staff incentives 30% 50% 75% 60% 25% . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . First implementation date December 2006 July 2016 April 2012 April 2012 March 2012 Second implementation date October 2008 October 2016 N/A N/A N/A Additional rollout Expanded to control regions in April 2010 N/A Expanded to control regions in May 2016 Expanded to 39 districts in October 2016 Expanded to 44 districts in 2015 Pre-implementation survey DHS 2010a DHS 2014 DHS 2011 DHS 2007 DHS 2010 − 11 Post-implementation survey DHS 2010 and DHS 2017 MICS 2018 Continuous DHS 2013 − 17 DHS 2014 and DHS 2018 DHS 2015 District selection notes Randomized treatment to districts. Additional unconditional financing arm in 10 districts. Government selected implementation districts from pair-matched districts. Major related concurrent interventions Incentives for district teams for good quality of supervision and support to PBF project Demand-side vouchers also provided for four ANC visits and skilled delivery . Introduced simultaneously with national elimination of user fees for targeted services. Payment adjustment on other dimensions Remoteness, poverty, staff and facility needs Remoteness Remoteness Remoteness Allocation of PBF payment  Health facility 70% 50% 25% 40% 75%  Staff incentives 30% 50% 75% 60% 25% a Given the absence of earlier data sources in Burundi, we used the birth recode file from 2010 for the pre-implementation survey through including births that occurred prior to implementation. Open in new tab Table 1 PBF characteristics and data sources . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . First implementation date December 2006 July 2016 April 2012 April 2012 March 2012 Second implementation date October 2008 October 2016 N/A N/A N/A Additional rollout Expanded to control regions in April 2010 N/A Expanded to control regions in May 2016 Expanded to 39 districts in October 2016 Expanded to 44 districts in 2015 Pre-implementation survey DHS 2010a DHS 2014 DHS 2011 DHS 2007 DHS 2010 − 11 Post-implementation survey DHS 2010 and DHS 2017 MICS 2018 Continuous DHS 2013 − 17 DHS 2014 and DHS 2018 DHS 2015 District selection notes Randomized treatment to districts. Additional unconditional financing arm in 10 districts. Government selected implementation districts from pair-matched districts. Major related concurrent interventions Incentives for district teams for good quality of supervision and support to PBF project Demand-side vouchers also provided for four ANC visits and skilled delivery . Introduced simultaneously with national elimination of user fees for targeted services. Payment adjustment on other dimensions Remoteness, poverty, staff and facility needs Remoteness Remoteness Remoteness Allocation of PBF payment  Health facility 70% 50% 25% 40% 75%  Staff incentives 30% 50% 75% 60% 25% . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . First implementation date December 2006 July 2016 April 2012 April 2012 March 2012 Second implementation date October 2008 October 2016 N/A N/A N/A Additional rollout Expanded to control regions in April 2010 N/A Expanded to control regions in May 2016 Expanded to 39 districts in October 2016 Expanded to 44 districts in 2015 Pre-implementation survey DHS 2010a DHS 2014 DHS 2011 DHS 2007 DHS 2010 − 11 Post-implementation survey DHS 2010 and DHS 2017 MICS 2018 Continuous DHS 2013 − 17 DHS 2014 and DHS 2018 DHS 2015 District selection notes Randomized treatment to districts. Additional unconditional financing arm in 10 districts. Government selected implementation districts from pair-matched districts. Major related concurrent interventions Incentives for district teams for good quality of supervision and support to PBF project Demand-side vouchers also provided for four ANC visits and skilled delivery . Introduced simultaneously with national elimination of user fees for targeted services. Payment adjustment on other dimensions Remoteness, poverty, staff and facility needs Remoteness Remoteness Remoteness Allocation of PBF payment  Health facility 70% 50% 25% 40% 75%  Staff incentives 30% 50% 75% 60% 25% a Given the absence of earlier data sources in Burundi, we used the birth recode file from 2010 for the pre-implementation survey through including births that occurred prior to implementation. Open in new tab PBF projects The PBF projects differed in their design and implementation across the study countries. In general, the projects were structured to provide healthcare facilities financial incentives conditional on reaching certain performance targets. Maternal and newborn care was a priority for all of the study countries, and targets included both quantity and quality of services. The volume of facility deliveries provided by a skilled birth attendant and antenatal care visits were rewarded in all study countries. Quality measures included structural quality items, such as water and soap available in delivery room (Lesotho), and process quality measures, such as correct use of the partograph (Senegal). The programs all had quantity-based formulas for determining the incentive, which were then inflated (or deflated, in Senegal) based on a quality score. None of the projects directly rewarded improvements on early neonatal death or low birthweight. Further details about the implementation and incentivized measures are provided in Supplementary Appendix 1. Four of the five study countries employed purposive selection to select the districts for PBF implementation. For example, in Zimbabwe districts were pair-matched on baseline characteristics such as geographic accessibility and average catchment population and then government officials selected between the two districts for implementation. Implementation was randomized only in Zambia, where districts were also matched prior to randomization. In addition, Zambia also had a third treatment arm which gave facilities unconditional financing equivalent to the amount of the PBF arm. We use the pure control districts without unconditional financing as the controls in the primary analysis but conduct a sensitivity analysis which compares the conditional and unconditional arms in Zambia. We selected control districts in Zambia and Zimbabwe to match those from the World Bank’s impact evaluations (Friedman et al., 2016a,b). Burundi and Senegal both implemented a phased rollout; consequently, we defined the control districts as those that later received PBF in those countries (Bonfrer et al., 2014; Falisse et al., 2015). The additional rollouts did not occur within the time period considered in this study. Finally given the small size of Lesotho, we defined the control districts as all the remaining districts that had not received PBF. We excluded Quthing and Leribe districts in Lesotho because they had piloted PBF 2 years prior to the larger implementation of PBF (The World Bank, 2017). Supplementary Appendix 1 lists of all the implementation and control districts for each country. In a sensitivity analysis, we use all non-implementation districts in all of the countries as controls, only excluding districts that had a prior pilot implementation. Burundi and Lesotho rolled out the PBF project in two stages within the study period. In the primary analysis, we consider only the first set of implementation districts and the control districts; in a sensitivity analysis we separately compare the second set of implementation districts against the control districts. Dependent variables We examined the effect of the PBF projects on two primary neonatal health outcomes: early neonatal death and low birthweight. Early neonatal death, which is associated with facility delivery and quality (Fink et al., 2015; Leslie et al., 2016), was defined as a death before or including 7 days of birth. Low birthweight, which is associated with ANC quantity and content (Coria-Soto et al., 1996), was defined as a birthweight below or including 2500 g. If the baby was not weighed at birth, we used multiple imputation with five imputations to impute missing values based on the mother’s report of the baby’s size at birth and risk factors including multiple births, primipara, urban location, maternal age and primary education, wealth quintile and district (Katz et al., 2013). Although there may be measurement error in the mother’s report of the baby’s size, this measure is strongly correlated with related health outcomes such as prematurity and intrauterine growth restrictions (McClure et al., 2011; Fink et al., 2015). As robustness checks, we also tested whether PBF impacted the likelihood of birthweight being recorded and the impact of PBF on the subset of observations where birthweight was recorded. We also examined several pathways through which PBF might affect these health outcomes, including increased utilization or improved quality of antenatal or intrapartum care or increased caesarean sections. We defined antenatal utilization as at least four antenatal care visits and intrapartum utilization as delivering in a health facility. Antenatal and delivery quality were both defined as binary variables, where high quality care recipients received all of the recommended quality items while low-quality care recipients received fewer items. Quality measures were alternatively defined as the percent of items received as a robustness check. Antenatal care quality items included the recommended number of Tetanus Toxoid shots, iron supplementation, a blood sample test and antenatal care from a qualified provider. Iron supplementation was not measured in the 2018 Lesotho MICS, so quality in Lesotho during both waves was measured using the other three items. Delivery quality items included breastfeeding within an hour of delivery, postnatal check before discharge and delivery with a trained provider. Finally, caesarean section was defined as the mother’s report of a caesarean delivery. The sample for each dependent variable varied based on data availability. Neonatal death data was available for all births, and we imputed birthweight for all births as described above. Antenatal care utilization and quality were only collected for the most recent birth; delivery quality was collected about the most recent birth if the woman had a facility delivery. Facility delivery and caesarean sections were collected about all births. Analysis We pooled data from all study countries and used a difference in differences specification to assess the impact of PBF on the study dependent variables. $$\begin{equation} {Y_{{\text{idt}}}} = {\beta _0} + {\beta _1}\left( {{\text{PB}}{{\text{F}}_d}{\text{*Pos}}{{\text{t}}_t}} \right) + {\beta _2}{\gamma _{idt}} + \mathop \sum \limits_{j = 1}^{60} {\beta _j}{\text{*Mont}}{{\text{h}}_t} + \mathop \sum \limits_{k = 1}^{75} {\beta _k}{\text{*Distric}}{{\text{t}}_d} + {\varepsilon _{idt}}, \end{equation}$$ where Y is a dependent variable for an individual i in district d and month t, PBF is an indicator for whether the district was treated, Post is an indicator for whether the birth was after the date of implementation, Y is a set of covariates, Month is a set of fixed effects of the month of birth in relation to the date of implementation where PBF was implemented in month 37, and District is a set of district fixed effects. We used multivariable linear probability models with standard errors clustered by district. We similarly tested for parallel pre-trends between implementation and control districts by interacting quarter fixed effects prior to and after the PBF implementation with the binary PBF indicator, excluding the quarter that PBF was implemented. This method can also be used to examine the effect of PBF over time. Because PBF was not randomized to districts in most countries, we both matched on a set of covariates and controlled for them in our model to obtain a better balance on important characteristics and improve the precision of our estimates (Chen et al., 2016). We used coarsened exact matching (CEM) to first match births on the set of covariates. CEM is a method that corrects for imbalances between composition of treatment and control districts by coarsening a set of covariates into bins, creating a stratum per bin and assigning observations to the strata, then dropping any births whose stratum does not contain at least one treated and one control unit (Blackwell et al., 2009; Chen et al., 2016). We included covariates that are known to be associated with neonatal health outcomes, including multiple birth, primipara, maternal age, year of birth, mother’s completion of primary education, urban vs rural location, and whether the household is in the poorest two wealth quintiles in the country. We included these covariates directly in the model in addition to using the CEM weights in order to further control potential residual confounding and improve precision (Blackwell et al., 2009). We conducted several additional analyses to understand whether the effect differed among sub-populations of interest. First, we conducted the differences in differences model separately in each study country in addition to the pooled analysis. We did not further adjust the standard errors for the small number of clusters in some countries; doing so would result in even more conservative results. Second, we ran the pooled model among the subset of households that were in the poorest two wealth quintiles in the country and among the subset of high-risk births. We defined high-risk births as those to primipara women, to women younger than 18 years or older than 34, or multiple births. Descriptive statistics are presented with the DHS and MICS sampling weights. Analyses were conducted in Stata 15. The original survey implementers obtained ethical approvals for data collection; the authors’ institute approved this secondary analysis as exempt from human subjects review. Results A total of 30 200 births from DHS or MICS across the five study countries met the inclusion criteria for the study. These included 12 790 births born after the introduction of PBF in their respective countries and 12 700 births that occurred in districts that implemented PBF projects. After CEM, 28 619 births were retained in the analysis, removing 1016 births from control districts and 565 births from PBF districts that were not matched. Table 2 displays the study outcomes and key covariates by treatment district prior to PBF implementation among the matched sample. Across the study countries, 658 (2.3%) births resulted in early neonatal death, ranging from 174 (1.5%) in Senegal to 99 (3.5%) in Lesotho. A total of 4579 (16%) births were low birthweight. Facility delivery and antenatal care utilization rates were low in most countries prior to the intervention; only Lesotho had over 70% facility delivery rate and only 55% of births had four antenatal care visits. Birthweight was recorded on a card for less than half of births at baseline; the PBF interventions did not have an impact on whether birthweight was recorded (Supplementary Appendix 4). Table 2 Dependent variables and covariates in control and implementation districts prior to implementation among analytic sample . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . Total . . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Districts 6 3 4 4 4 2 10 10 16 16 40 35 Pre-implementation births 3229 1557 1013 980 3576 2499 931 990 729 993 9478 7019 Post-implementation births 2285 1217 427 418 3300 2244 602 611 392 626 7006 5116 Pre-implementation dependent variables  Early neonatal death 2.3% 3.2% 3.7% 3.2% 1.3% 1.7% 1.5% 1.6% 2.4% 3.0% 2.2% 2.5%  Low birthweight 18% 20% 13% 16% 21% 17% 15% 14% 13% 15% 17% 16%  Facility delivery 49% 50% 84% 76% 57% 46% 54% 59% 74% 64% 61% 57%  Delivery quality 86% 86% 56% 58% 55% 53% 71% 72% 54% 58% 61% 61%  C-section 1% 3% 13% 8% 3% 1% 2% 3% 5% 4% 4% 3%  4+ ANC visits 31% 38% 75% 69% 43% 32% 60% 59% 69% 63% 57% 52%  ANC quality 8% 0% 61% 60% 55% 49% 45% 44% 30% 30% 48% 43% Pre-implementation covariates  Mother’s age at birth (mean) 26.7 27.0 25.2 25.5 26.6 26.2 26.3 26.2 25.9 25.5 26.3 26.2  Mother primary education 38% 45% 100% 100% 31% 18% 88% 90% 99% 99% 57% 62%  Primipara 20% 23% 44% 39% 20% 19% 20% 17% 27% 29% 24% 24%  Multiple birth 0% 1% 0% 1% 1% 2% 2% 1% 1% 2% 1% 1% Urban 2% 3% 46% 19% 24% 13% 16% 11% 27% 23% 17% 13%  Poorest wealth quintile 21% 22% 13% 37% 41% 59% 28% 38% 28% 38% 25% 37% Birthweight recorded 7% 8% 44% 46% 45% 30% 51% 61% 51% 61% 32% 39% . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . Total . . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Districts 6 3 4 4 4 2 10 10 16 16 40 35 Pre-implementation births 3229 1557 1013 980 3576 2499 931 990 729 993 9478 7019 Post-implementation births 2285 1217 427 418 3300 2244 602 611 392 626 7006 5116 Pre-implementation dependent variables  Early neonatal death 2.3% 3.2% 3.7% 3.2% 1.3% 1.7% 1.5% 1.6% 2.4% 3.0% 2.2% 2.5%  Low birthweight 18% 20% 13% 16% 21% 17% 15% 14% 13% 15% 17% 16%  Facility delivery 49% 50% 84% 76% 57% 46% 54% 59% 74% 64% 61% 57%  Delivery quality 86% 86% 56% 58% 55% 53% 71% 72% 54% 58% 61% 61%  C-section 1% 3% 13% 8% 3% 1% 2% 3% 5% 4% 4% 3%  4+ ANC visits 31% 38% 75% 69% 43% 32% 60% 59% 69% 63% 57% 52%  ANC quality 8% 0% 61% 60% 55% 49% 45% 44% 30% 30% 48% 43% Pre-implementation covariates  Mother’s age at birth (mean) 26.7 27.0 25.2 25.5 26.6 26.2 26.3 26.2 25.9 25.5 26.3 26.2  Mother primary education 38% 45% 100% 100% 31% 18% 88% 90% 99% 99% 57% 62%  Primipara 20% 23% 44% 39% 20% 19% 20% 17% 27% 29% 24% 24%  Multiple birth 0% 1% 0% 1% 1% 2% 2% 1% 1% 2% 1% 1% Urban 2% 3% 46% 19% 24% 13% 16% 11% 27% 23% 17% 13%  Poorest wealth quintile 21% 22% 13% 37% 41% 59% 28% 38% 28% 38% 25% 37% Birthweight recorded 7% 8% 44% 46% 45% 30% 51% 61% 51% 61% 32% 39% Open in new tab Table 2 Dependent variables and covariates in control and implementation districts prior to implementation among analytic sample . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . Total . . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Districts 6 3 4 4 4 2 10 10 16 16 40 35 Pre-implementation births 3229 1557 1013 980 3576 2499 931 990 729 993 9478 7019 Post-implementation births 2285 1217 427 418 3300 2244 602 611 392 626 7006 5116 Pre-implementation dependent variables  Early neonatal death 2.3% 3.2% 3.7% 3.2% 1.3% 1.7% 1.5% 1.6% 2.4% 3.0% 2.2% 2.5%  Low birthweight 18% 20% 13% 16% 21% 17% 15% 14% 13% 15% 17% 16%  Facility delivery 49% 50% 84% 76% 57% 46% 54% 59% 74% 64% 61% 57%  Delivery quality 86% 86% 56% 58% 55% 53% 71% 72% 54% 58% 61% 61%  C-section 1% 3% 13% 8% 3% 1% 2% 3% 5% 4% 4% 3%  4+ ANC visits 31% 38% 75% 69% 43% 32% 60% 59% 69% 63% 57% 52%  ANC quality 8% 0% 61% 60% 55% 49% 45% 44% 30% 30% 48% 43% Pre-implementation covariates  Mother’s age at birth (mean) 26.7 27.0 25.2 25.5 26.6 26.2 26.3 26.2 25.9 25.5 26.3 26.2  Mother primary education 38% 45% 100% 100% 31% 18% 88% 90% 99% 99% 57% 62%  Primipara 20% 23% 44% 39% 20% 19% 20% 17% 27% 29% 24% 24%  Multiple birth 0% 1% 0% 1% 1% 2% 2% 1% 1% 2% 1% 1% Urban 2% 3% 46% 19% 24% 13% 16% 11% 27% 23% 17% 13%  Poorest wealth quintile 21% 22% 13% 37% 41% 59% 28% 38% 28% 38% 25% 37% Birthweight recorded 7% 8% 44% 46% 45% 30% 51% 61% 51% 61% 32% 39% . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . Total . . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Control . PBF . Districts 6 3 4 4 4 2 10 10 16 16 40 35 Pre-implementation births 3229 1557 1013 980 3576 2499 931 990 729 993 9478 7019 Post-implementation births 2285 1217 427 418 3300 2244 602 611 392 626 7006 5116 Pre-implementation dependent variables  Early neonatal death 2.3% 3.2% 3.7% 3.2% 1.3% 1.7% 1.5% 1.6% 2.4% 3.0% 2.2% 2.5%  Low birthweight 18% 20% 13% 16% 21% 17% 15% 14% 13% 15% 17% 16%  Facility delivery 49% 50% 84% 76% 57% 46% 54% 59% 74% 64% 61% 57%  Delivery quality 86% 86% 56% 58% 55% 53% 71% 72% 54% 58% 61% 61%  C-section 1% 3% 13% 8% 3% 1% 2% 3% 5% 4% 4% 3%  4+ ANC visits 31% 38% 75% 69% 43% 32% 60% 59% 69% 63% 57% 52%  ANC quality 8% 0% 61% 60% 55% 49% 45% 44% 30% 30% 48% 43% Pre-implementation covariates  Mother’s age at birth (mean) 26.7 27.0 25.2 25.5 26.6 26.2 26.3 26.2 25.9 25.5 26.3 26.2  Mother primary education 38% 45% 100% 100% 31% 18% 88% 90% 99% 99% 57% 62%  Primipara 20% 23% 44% 39% 20% 19% 20% 17% 27% 29% 24% 24%  Multiple birth 0% 1% 0% 1% 1% 2% 2% 1% 1% 2% 1% 1% Urban 2% 3% 46% 19% 24% 13% 16% 11% 27% 23% 17% 13%  Poorest wealth quintile 21% 22% 13% 37% 41% 59% 28% 38% 28% 38% 25% 37% Birthweight recorded 7% 8% 44% 46% 45% 30% 51% 61% 51% 61% 32% 39% Open in new tab Treatment and control districts were not balanced on all covariates prior to PBF implementation even after matching. PBF was implemented more often in poorer districts, particularly in Lesotho, Senegal and Zimbabwe, and in rural districts. Despite these differing characteristics, the trends in most outcomes do not significantly differ between implementation and control districts prior to implementation (Supplementary Appendix 3). Table 3 presents the results from the difference in differences estimation pooling together births from all the study countries and stratified by country. We found no statistically or substantially significant effect of the PBF intervention on any of the health outcomes or intermediate outputs in the pooled analysis. The unadjusted trends for early neonatal death and low birthweight are shown in Figure 1, while the results for the intermediate outputs are shown in Supplementary Appendix 2. These results were robust to excluding Burundi, to using all non-implementation districts as controls, to using the alternative definitions of the quality measures, to only including observations where birthweight was recorded, and to using the second implementation date in Burundi and Lesotho (Supplementary Appendix 4). There also do not appear to be delayed effects of PBF within the 2-year period assessed (Supplementary Appendix 3). Figure 1 Open in new tabDownload slide Pooled unadjusted trends in early neonatal death and low birthweight before and after PBF implementation. Figure 1 Open in new tabDownload slide Pooled unadjusted trends in early neonatal death and low birthweight before and after PBF implementation. Table 3 Effects of PBF on primary and secondary outcomes pooled and in all study countries . Pooled . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . Early-neonatal death  Coef. 0.00 0.00 0.02 0.00 −0.01 0.00  95% CI (−0.01, 0.01) (−0.01, 0.01) (−0.01, 0.05) (−0.01, 0.01) (−0.04, 0.01) (−0.02, 0.03)  N 28 619 8288 2838 11 619 3134 2740 Low birthweight  Coef. 0.01 0.01 −0.05 0.03 0.00 −0.02  95% CI (−0.02, 0.03) (−0.12, 0.13) (−0.14, 0.03) (−0.01, 0.08) (−0.06, 0.06) (−0.1, 0.06)  N 28 619 8288 2838 11 619 3134 2740 Facility delivery  Coef. 0.03 0.08 0.03 0.03 0.03 −0.02  95% CI (−0.01, 0.07) (0.02, 0.14) (−0.04, 0.09) (−0.03, 0.08) (−0.06, 0.12) (−0.1, 0.06)  N 21 471 2140 1849 11 619 3123 2740 Delivery quality  Coef. −0.05 −0.05 −0.09 −0.05 −0.05 −0.03  95% CI (−0.14, 0.04) (−0.16, 0.06) (−0.24, 0.06) (−0.11, 0.01) (−0.16, 0.06) (−0.14, 0.09)  N 13 054 1219 1558 6275 2026 1976 C-section  Coef. 0.00 0.01 −0.01 0.00 0.00 −0.02  95% CI (−0.01, 0.01) (−0.03, 0.05) (−0.11, 0.09) (−0.01, 0.01) (−0.03, 0.04) (−0.05, 0.01)  N 21 424 2145 1849 11 564 3128 2738 ANC 4 visits  Coef. 0.04 −0.06 0.12 0.02 0.06 0.00  95% CI (−0.02, 0.10) (−0.18, 0.07) (0.01, 0.22) (−0.1, 0.15) (−0.02, 0.13) (−0.12, 0.13)  N 14 383 793 1840 7383 2157 2210 ANC quality  Coef. 0.02 0.09 −0.03 0.03 0.09 −0.03  95% CI (−0.04, 0.09) (−0.05, 0.24) (−0.14, 0.08) (−0.1, 0.16) (0.01, 0.17) (−0.16, 0.09)  N 14 510 796 1869 7445 2172 2228 . Pooled . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . Early-neonatal death  Coef. 0.00 0.00 0.02 0.00 −0.01 0.00  95% CI (−0.01, 0.01) (−0.01, 0.01) (−0.01, 0.05) (−0.01, 0.01) (−0.04, 0.01) (−0.02, 0.03)  N 28 619 8288 2838 11 619 3134 2740 Low birthweight  Coef. 0.01 0.01 −0.05 0.03 0.00 −0.02  95% CI (−0.02, 0.03) (−0.12, 0.13) (−0.14, 0.03) (−0.01, 0.08) (−0.06, 0.06) (−0.1, 0.06)  N 28 619 8288 2838 11 619 3134 2740 Facility delivery  Coef. 0.03 0.08 0.03 0.03 0.03 −0.02  95% CI (−0.01, 0.07) (0.02, 0.14) (−0.04, 0.09) (−0.03, 0.08) (−0.06, 0.12) (−0.1, 0.06)  N 21 471 2140 1849 11 619 3123 2740 Delivery quality  Coef. −0.05 −0.05 −0.09 −0.05 −0.05 −0.03  95% CI (−0.14, 0.04) (−0.16, 0.06) (−0.24, 0.06) (−0.11, 0.01) (−0.16, 0.06) (−0.14, 0.09)  N 13 054 1219 1558 6275 2026 1976 C-section  Coef. 0.00 0.01 −0.01 0.00 0.00 −0.02  95% CI (−0.01, 0.01) (−0.03, 0.05) (−0.11, 0.09) (−0.01, 0.01) (−0.03, 0.04) (−0.05, 0.01)  N 21 424 2145 1849 11 564 3128 2738 ANC 4 visits  Coef. 0.04 −0.06 0.12 0.02 0.06 0.00  95% CI (−0.02, 0.10) (−0.18, 0.07) (0.01, 0.22) (−0.1, 0.15) (−0.02, 0.13) (−0.12, 0.13)  N 14 383 793 1840 7383 2157 2210 ANC quality  Coef. 0.02 0.09 −0.03 0.03 0.09 −0.03  95% CI (−0.04, 0.09) (−0.05, 0.24) (−0.14, 0.08) (−0.1, 0.16) (0.01, 0.17) (−0.16, 0.09)  N 14 510 796 1869 7445 2172 2228 Bolded estimates signify confidence intervals that do not contain zero. Estimated coefficients for β1 from multivariable difference-in-difference regressions representing the percentage point change in the outcome, with standard errors clustered at the district level. Open in new tab Table 3 Effects of PBF on primary and secondary outcomes pooled and in all study countries . Pooled . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . Early-neonatal death  Coef. 0.00 0.00 0.02 0.00 −0.01 0.00  95% CI (−0.01, 0.01) (−0.01, 0.01) (−0.01, 0.05) (−0.01, 0.01) (−0.04, 0.01) (−0.02, 0.03)  N 28 619 8288 2838 11 619 3134 2740 Low birthweight  Coef. 0.01 0.01 −0.05 0.03 0.00 −0.02  95% CI (−0.02, 0.03) (−0.12, 0.13) (−0.14, 0.03) (−0.01, 0.08) (−0.06, 0.06) (−0.1, 0.06)  N 28 619 8288 2838 11 619 3134 2740 Facility delivery  Coef. 0.03 0.08 0.03 0.03 0.03 −0.02  95% CI (−0.01, 0.07) (0.02, 0.14) (−0.04, 0.09) (−0.03, 0.08) (−0.06, 0.12) (−0.1, 0.06)  N 21 471 2140 1849 11 619 3123 2740 Delivery quality  Coef. −0.05 −0.05 −0.09 −0.05 −0.05 −0.03  95% CI (−0.14, 0.04) (−0.16, 0.06) (−0.24, 0.06) (−0.11, 0.01) (−0.16, 0.06) (−0.14, 0.09)  N 13 054 1219 1558 6275 2026 1976 C-section  Coef. 0.00 0.01 −0.01 0.00 0.00 −0.02  95% CI (−0.01, 0.01) (−0.03, 0.05) (−0.11, 0.09) (−0.01, 0.01) (−0.03, 0.04) (−0.05, 0.01)  N 21 424 2145 1849 11 564 3128 2738 ANC 4 visits  Coef. 0.04 −0.06 0.12 0.02 0.06 0.00  95% CI (−0.02, 0.10) (−0.18, 0.07) (0.01, 0.22) (−0.1, 0.15) (−0.02, 0.13) (−0.12, 0.13)  N 14 383 793 1840 7383 2157 2210 ANC quality  Coef. 0.02 0.09 −0.03 0.03 0.09 −0.03  95% CI (−0.04, 0.09) (−0.05, 0.24) (−0.14, 0.08) (−0.1, 0.16) (0.01, 0.17) (−0.16, 0.09)  N 14 510 796 1869 7445 2172 2228 . Pooled . Burundi . Lesotho . Senegal . Zambia . Zimbabwe . Early-neonatal death  Coef. 0.00 0.00 0.02 0.00 −0.01 0.00  95% CI (−0.01, 0.01) (−0.01, 0.01) (−0.01, 0.05) (−0.01, 0.01) (−0.04, 0.01) (−0.02, 0.03)  N 28 619 8288 2838 11 619 3134 2740 Low birthweight  Coef. 0.01 0.01 −0.05 0.03 0.00 −0.02  95% CI (−0.02, 0.03) (−0.12, 0.13) (−0.14, 0.03) (−0.01, 0.08) (−0.06, 0.06) (−0.1, 0.06)  N 28 619 8288 2838 11 619 3134 2740 Facility delivery  Coef. 0.03 0.08 0.03 0.03 0.03 −0.02  95% CI (−0.01, 0.07) (0.02, 0.14) (−0.04, 0.09) (−0.03, 0.08) (−0.06, 0.12) (−0.1, 0.06)  N 21 471 2140 1849 11 619 3123 2740 Delivery quality  Coef. −0.05 −0.05 −0.09 −0.05 −0.05 −0.03  95% CI (−0.14, 0.04) (−0.16, 0.06) (−0.24, 0.06) (−0.11, 0.01) (−0.16, 0.06) (−0.14, 0.09)  N 13 054 1219 1558 6275 2026 1976 C-section  Coef. 0.00 0.01 −0.01 0.00 0.00 −0.02  95% CI (−0.01, 0.01) (−0.03, 0.05) (−0.11, 0.09) (−0.01, 0.01) (−0.03, 0.04) (−0.05, 0.01)  N 21 424 2145 1849 11 564 3128 2738 ANC 4 visits  Coef. 0.04 −0.06 0.12 0.02 0.06 0.00  95% CI (−0.02, 0.10) (−0.18, 0.07) (0.01, 0.22) (−0.1, 0.15) (−0.02, 0.13) (−0.12, 0.13)  N 14 383 793 1840 7383 2157 2210 ANC quality  Coef. 0.02 0.09 −0.03 0.03 0.09 −0.03  95% CI (−0.04, 0.09) (−0.05, 0.24) (−0.14, 0.08) (−0.1, 0.16) (0.01, 0.17) (−0.16, 0.09)  N 14 510 796 1869 7445 2172 2228 Bolded estimates signify confidence intervals that do not contain zero. Estimated coefficients for β1 from multivariable difference-in-difference regressions representing the percentage point change in the outcome, with standard errors clustered at the district level. Open in new tab Consistent with the pooled results, PBF did not have a significant effect on early neonatal death or low birthweight in any of the study countries. Zambia’s PBF may have resulted in a slight decline in early neonatal death, but the 95% confidence interval (CI) contained zero. However, several countries did see some effect on intermediate outputs. Facility delivery rose 8 percentage points in Burundi (95% CI: 0.02, 0.14), antenatal care visits rose by 12 percentage points in Lesotho (95% CI: 0.01, 0.22) and antenatal visit quality improved by 9 percentage points in Zambia (95% CI: 0.01, 0.17). There were no effects on delivery quality or caesarean sections in any country. In Zambia, there were no effects on the primary or secondary outcomes when comparing the PBF districts to the unconditional financing arm rather than the pure control arm (Supplementary Appendix 4). Table 4 presents the results when the pooled sample is restricted to the two sub-populations of interest. PBF increased antenatal care utilization by 8 percentage points (95% CI: 0.00, 0.17) among poor women. It did not have any effect on the health outcomes or any of the other intermediate outputs in either of the populations of interest. Table 4 Effects of PBF on primary and secondary outcomes among populations of interest . . Poor women . . High-risk births . Outcome . Coef. . 95% CI . N . Coef. . 95% CI . N . Early neonatal death 0.00 (−0.01, 0.02) 9680 0.00 (−0.01, 0.02) 10 887 Low birthweight 0.00 (−0.03, 0.03) 9680 0.01 (−0.04, 0.05) 10 887 Facility delivery 0.02 (−0.04, 0.09) 8051 0.03 (−0.02, 0.09) 8222 Delivery quality −0.05 (−0.16, 0.06) 3476 −0.07 (−0.18, 0.05) 5570 C-section −0.01 (−0.02, 0.01) 8034 −0.01 (−0.03, 0.01) 8205 ANC 4 visits 0.08 (0, 0.17) 5122 0.04 (−0.02, 0.1) 5771 ANC quality 0.06 (−0.03, 0.14) 5152 0.00 (−0.06, 0.07) 5824 . . Poor women . . High-risk births . Outcome . Coef. . 95% CI . N . Coef. . 95% CI . N . Early neonatal death 0.00 (−0.01, 0.02) 9680 0.00 (−0.01, 0.02) 10 887 Low birthweight 0.00 (−0.03, 0.03) 9680 0.01 (−0.04, 0.05) 10 887 Facility delivery 0.02 (−0.04, 0.09) 8051 0.03 (−0.02, 0.09) 8222 Delivery quality −0.05 (−0.16, 0.06) 3476 −0.07 (−0.18, 0.05) 5570 C-section −0.01 (−0.02, 0.01) 8034 −0.01 (−0.03, 0.01) 8205 ANC 4 visits 0.08 (0, 0.17) 5122 0.04 (−0.02, 0.1) 5771 ANC quality 0.06 (−0.03, 0.14) 5152 0.00 (−0.06, 0.07) 5824 Bolded estimates signify confidence intervals that do not contain zero. Estimated coefficients for β1from pooled multivariable difference-in-difference regression, with standard errors clustered at the district level. Open in new tab Table 4 Effects of PBF on primary and secondary outcomes among populations of interest . . Poor women . . High-risk births . Outcome . Coef. . 95% CI . N . Coef. . 95% CI . N . Early neonatal death 0.00 (−0.01, 0.02) 9680 0.00 (−0.01, 0.02) 10 887 Low birthweight 0.00 (−0.03, 0.03) 9680 0.01 (−0.04, 0.05) 10 887 Facility delivery 0.02 (−0.04, 0.09) 8051 0.03 (−0.02, 0.09) 8222 Delivery quality −0.05 (−0.16, 0.06) 3476 −0.07 (−0.18, 0.05) 5570 C-section −0.01 (−0.02, 0.01) 8034 −0.01 (−0.03, 0.01) 8205 ANC 4 visits 0.08 (0, 0.17) 5122 0.04 (−0.02, 0.1) 5771 ANC quality 0.06 (−0.03, 0.14) 5152 0.00 (−0.06, 0.07) 5824 . . Poor women . . High-risk births . Outcome . Coef. . 95% CI . N . Coef. . 95% CI . N . Early neonatal death 0.00 (−0.01, 0.02) 9680 0.00 (−0.01, 0.02) 10 887 Low birthweight 0.00 (−0.03, 0.03) 9680 0.01 (−0.04, 0.05) 10 887 Facility delivery 0.02 (−0.04, 0.09) 8051 0.03 (−0.02, 0.09) 8222 Delivery quality −0.05 (−0.16, 0.06) 3476 −0.07 (−0.18, 0.05) 5570 C-section −0.01 (−0.02, 0.01) 8034 −0.01 (−0.03, 0.01) 8205 ANC 4 visits 0.08 (0, 0.17) 5122 0.04 (−0.02, 0.1) 5771 ANC quality 0.06 (−0.03, 0.14) 5152 0.00 (−0.06, 0.07) 5824 Bolded estimates signify confidence intervals that do not contain zero. Estimated coefficients for β1from pooled multivariable difference-in-difference regression, with standard errors clustered at the district level. Open in new tab Discussion PBF is considered an innovative approach to tackle the challenges to improving neonatal health outcomes that persist in many African countries. This study used quasi-experimental methods and population representative secondary data to assess the effect of PBF projects on neonatal health outcomes, and the quantity and quality of care in five African countries. Despite the large sample sizes from pooling the data, we found no effect on any of the examined outputs or outcomes. Although there were several positive impacts on utilization and antenatal care quality among individual country projects and among poor women, no project had a statistically detectable impact on either neonatal mortality or low birthweight. Furthermore, the PBF projects did not have detectable impacts on the health outcomes for two vulnerable sub-groups, poor women and women with a high-risk birth. There may be several reasons for our null findings. First, the potential of PBF may be constrained by the ability of health facilities or providers to adjust their behaviour to improve performance. In practice, they may already be operating at capacity given their environmental, educational and structural constraints. For example, chronic staff shortages limited sustained improvement in Zimbabwe (Moyo et al., 2015). Poorly functioning health systems may instead require greater foundational change than adjustments to provider performance (Kruk et al., 2018). Second, PBF may have both positive and negative effects on different aspects of provider motivation (Shen et al., 2017; Lohmann et al., 2018), and its effects on non-incentivized services can be ambiguous (Sherry, 2016; Sherry et al., 2017). Although improving health outcomes is a stated primary goal of all PBF projects in this study, it is possible that the projects had positive impacts on important clinical and non-clinical areas that we did not consider. Third, the specific design and implementation of the projects could affect their impacts. For example, the incentives may be too low or not be tied to the most effective behaviours. This may be particularly relevant for quality of care: PBF predominantly incentivizes structural quality (Gergen et al., 2017), which may be only weakly correlated with care processes (Leslie et al., 2017). Despite the large-pooled sample size, the study may also still not be adequately powered to detect changes in early neonatal death. An ex-post power calculation (Supplementary Appendix 5) suggests that the minimum detectable effect is a 0.67 percentage point change in the probability of early neonatal death, with the available sample size, 80% power and a 5% significance level. Smaller changes may be policy relevant, however, the small coefficient size and lack of effect in any of the intermediate outputs suggests that an effect would still not be detectable even with a larger sample size. Some of our results differ from those of earlier impact evaluations of these PBF projects, which are summarized in Table 5. While no prior study had directly assessed the impacts on health outcomes, several studies found positive impacts on utilization or quality, particularly on rates of facility delivery (Bonfrer et al., 2014; Friedman et al., 2016a,b). We found a positive impact on facility delivery in Burundi, though smaller effect size than in earlier studies (Bonfrer et al., 2014), and no impact in Zambia or Zimbabwe. There may be a number of explanations for this divergence, including differences in the sampling strategy, timing of data and inclusion criteria; differences in the covariates used to control for baseline differences; and our use matching to reduce covariate imbalance. There are also differences in how quality is measured. Our quality measures rely on a relatively small number of process measures from self-reports, whereas the earlier studies tend to use more indicators and rely more heavily on structural measures. For example, the Burundi evaluation uses a composite facility-based measure constructed using 57 structural and process indicators (Bonfrer et al., 2014), while the large impact on delivery quality in Zambia is driven by the availability of equipment, medicines and supplies in the delivery room (Friedman et al., 2016a,b). Table 5 Summary of effects from previous impact evaluations . Burundid . Lesotho . Senegal . Zambiae . Zimbabwef . Early neonatal death Not assessed Not assessed Not assessed Not assessed Not assessed Low birthweight Not assessed Not assessed Not assessed Not assessed Not assessed Facility delivery 22 pp Not assessed Not assessed 13 pp 13 pp Delivery quality 17 ppa Not assessed Not assessed 57 pp No effect C-section Not assessed Not assessed Not assessed Not assessed 7 pp ANC visits No effect Not assessed Not assessed No effect No effect ANC quality 17 ppa Not assessed Not assessed Mixedb Mixedc . Burundid . Lesotho . Senegal . Zambiae . Zimbabwef . Early neonatal death Not assessed Not assessed Not assessed Not assessed Not assessed Low birthweight Not assessed Not assessed Not assessed Not assessed Not assessed Facility delivery 22 pp Not assessed Not assessed 13 pp 13 pp Delivery quality 17 ppa Not assessed Not assessed 57 pp No effect C-section Not assessed Not assessed Not assessed Not assessed 7 pp ANC visits No effect Not assessed Not assessed No effect No effect ANC quality 17 ppa Not assessed Not assessed Mixedb Mixedc Statistically significant effects reported; all reported effects were positive. a Facility quality measured overall, rather than by service. b Found improvements in iron supplementation and malaria drugs, decrease in urine sample taken, and no change in other 5 ANC quality measures assessed. c Found improvements in urine sample taken and tetanus injections, and no change in other 6 ANC quality measures assessed. d Bonfrer et al. (2014). e Friedman et al. (2016b). f Friedman et al. (2016a). Open in new tab Table 5 Summary of effects from previous impact evaluations . Burundid . Lesotho . Senegal . Zambiae . Zimbabwef . Early neonatal death Not assessed Not assessed Not assessed Not assessed Not assessed Low birthweight Not assessed Not assessed Not assessed Not assessed Not assessed Facility delivery 22 pp Not assessed Not assessed 13 pp 13 pp Delivery quality 17 ppa Not assessed Not assessed 57 pp No effect C-section Not assessed Not assessed Not assessed Not assessed 7 pp ANC visits No effect Not assessed Not assessed No effect No effect ANC quality 17 ppa Not assessed Not assessed Mixedb Mixedc . Burundid . Lesotho . Senegal . Zambiae . Zimbabwef . Early neonatal death Not assessed Not assessed Not assessed Not assessed Not assessed Low birthweight Not assessed Not assessed Not assessed Not assessed Not assessed Facility delivery 22 pp Not assessed Not assessed 13 pp 13 pp Delivery quality 17 ppa Not assessed Not assessed 57 pp No effect C-section Not assessed Not assessed Not assessed Not assessed 7 pp ANC visits No effect Not assessed Not assessed No effect No effect ANC quality 17 ppa Not assessed Not assessed Mixedb Mixedc Statistically significant effects reported; all reported effects were positive. a Facility quality measured overall, rather than by service. b Found improvements in iron supplementation and malaria drugs, decrease in urine sample taken, and no change in other 5 ANC quality measures assessed. c Found improvements in urine sample taken and tetanus injections, and no change in other 6 ANC quality measures assessed. d Bonfrer et al. (2014). e Friedman et al. (2016b). f Friedman et al. (2016a). Open in new tab This study has a number of limitations. First, women’s treatment status may have been misclassified based on her district of residence at the time of the interview. This may be the case if the woman moved districts between the birth and the survey, sought care outside of her district, or visited a private facility which did not receive the RBF intervention within an RBF district. While these cases should affect a small per cent of women and should not differentially affect women in intervention or comparison districts, a misclassified status would bias the results towards the null. Second, the quality measures available in the DHS and MICS data sets were limited. We selected indicators for process quality that may have a large impact on neonatal health outcomes but only partially capture routine delivery and antenatal care quality. Third, the mostly non-randomized implementation of the PBF projects could result in residual confounding that persists despite matching at baseline. Although we found pre-trends to be largely parallel, there could be unobserved time-variant factors that differentially impacted the districts during the study period. Fourth, we were unable to look at a longer time frame beyond 2 years because of PBF implementation in the control areas in some of the countries at that time. Although neonatal mortality can be responsive to changes in the health system (Magge et al., 2020), it may take longer than this period to see an effect particularly if there were delays in signing contracts or delivering payments (Rajkotia et al., 2017; Ridde et al., 2018). Finally, we were unable to look at treatment heterogeneity at levels lower than the country because of limited sample sizes. The mixed and variable effects we observed across countries indicate scopes for learning from comparative studies. Such comparisons and innovations in measurement (e.g. of quality) can also be used to adjust ongoing projects (Fritsche and Peabody, 2018). The large number of HRITF-supported PBF pilots provides an important opportunity for such further research. Overall, our results indicate that PBF—as implemented in the five projects we examined—may have limited impacts on neonatal health outcomes, as well as the associated utilization and quality pathways. While this does not preclude PBF from having other effects, positive or negative, this finding suggests caution with designing and deploying PBF with the goal of improving neonatal health outcomes at the population level. PBF may have other benefits, e.g. arising from increased autonomy and supervision (Renmans et al., 2017), but must also contend with other criticisms, such the lack of domestic ownership and the diversion of attention and resources away from broader health systems strategies (Paul et al., 2018; Ridde et al., 2018). Different strategies will likely be needed to make meaningful progress on improving neonatal health outcomes in sub-Saharan Africa. Supplementary data Supplementary data are available at Health Policy and Planning online. Funding Publication fees were supported by The Bill and Melinda Gates Foundation grant number OPP1161450. Acknowledgements The authors thank Jed Friedman, Margaret Kruk, Hannah Leslie, Mirabel Levine and Ronald Mutasa for their comments on this work. Conflict of interest statement. None declared. Ethical approval. The original survey implementers obtained ethical approvals for data collection; the authors’ institute approved this secondary analysis as exempt from human subjects review. References Basinga P , Gertler PJ, Binagwaho A et al. . 2011 . Effect on maternal and child health services in Rwanda of payment to primary health-care providers for performance: an impact evaluation . The Lancet 377 : 1421 – 8 . Google Scholar OpenURL Placeholder Text WorldCat Blackwell M , Iacus S, King G, Porro G. 2009 . CEM: coarsened exact matching in Stata . The Stata Journal: Promoting Communications on Statistics and Stata 9 : 524 – 46 . Google Scholar OpenURL Placeholder Text WorldCat Bonfrer I , Van de Poel E, Van Doorslaer E. 2014 . The effects of performance incentives on the utilization and quality of maternal and child care in Burundi . Social Science & Medicine 123 : 96 – 104 . Google Scholar OpenURL Placeholder Text WorldCat Burgert CR , Colston J, Roy T, Zachary B. 2013 . Geographic Displacement Procedure and Georeferenced Data Release Policy for the Demographic and Health Surveys. Calverton, MD: ICF International. Chen B , Halliday TJ, Fan VY. 2016 . The impact of internal displacement on child mortality in post-earthquake Haiti: a difference-in-differences analysis . International Journal for Equity in Health 15 : 114 . Google Scholar OpenURL Placeholder Text WorldCat Chinkhumba J , De Allegri M, Brenner S, Muula A, Robberstad B. 2020 . The cost-effectiveness of using results-based financing to reduce maternal and perinatal mortality in Malawi . BMJ Global Health 5 : e002260 . Google Scholar OpenURL Placeholder Text WorldCat Coria-Soto IL , Bobadilla JL, Notzon F. 1996 . The Effectiveness of Antenatal Care in Preventing Intrauterine Growth Retardation and Low Birth Weight Due to Preterm Delivery . Oxford : Oxford University Press . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Eichler R , Agarwal K, Askew I et al. . 2013 . Performance-based incentives to improve health status of mothers and newborns: what does the evidence show? Journal of Health, Population, and Nutrition 31 : S36 . Google Scholar OpenURL Placeholder Text WorldCat Falisse J-B , Ndayishimiye J, Kamenyero V, Bossuyt M. 2015 . Performance-based financing in the context of selective free health-care: an evaluation of its effects on the use of primary health-care services in Burundi using routine data . Health Policy and Planning 30 : 1251 – 60 . Google Scholar OpenURL Placeholder Text WorldCat Fink G , Ross R, Hill K. 2015 . Institutional deliveries weakly associated with improved neonatal survival in developing countries: evidence from 192 Demographic and Health Surveys . International Journal of Epidemiology 44 : 1879 – 88 . Google Scholar OpenURL Placeholder Text WorldCat Friedman J , Das A, Mutasa R. 2016a . Rewarding Provider Performance to Improve Quality and Coverage of Maternal and Child Health Outcomes: Zimbabwe Results-Based Financing Pilot Program. Washington, DC: The World Bank. Friedman J , Qamruddin J, Chansa C, Das AK. 2016b . Impact Evaluation of Zambia’s Health Results-Based Financing Pilot Project . Washington, DC : World Bank Group . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Fritsche G , Peabody J. 2018 . Methods to improve quality performance at scale in lower-and middle-income countries . Journal of Global Health 8 : 1 – 15 . Google Scholar OpenURL Placeholder Text WorldCat Gergen J , Josephson E, Coe M et al. . 2017 . Quality of care in performance-based financing: how it is incorporated in 32 programs across 28 countries . Global Health: Science and Practice 5 : 90 – 107 . Google Scholar OpenURL Placeholder Text WorldCat Gülmezoglu AM , Lawrie TA, Hezelgrave N et al. . 2016 . Interventions to Reduce Maternal and Newborn Morbidity and Mortality. In: Disease Control Priorities 3 : 115 – 36 . Kandpal E. 2016 . Completed Impact Evaluations and Emerging Lessons from the Health Results Innovation Trust Fund Learning Portfolio. Washington, DC: The World Bank. Katz J , Lee AC, Kozuki N et al. . 2013 . Mortality risk in preterm and small-for-gestational-age infants in low-income and middle-income countries: a pooled country analysis . The Lancet 382 : 417 – 25 . Google Scholar OpenURL Placeholder Text WorldCat Kruk M , Gage A, Arsenault C et al. . 2018 . High quality health systems—time for a revolution: report of the Lancet Global Health Commission on High Quality Health Systems in the SDG Era . The Lancet Global Health 6 : e1196 – 252 . Google Scholar OpenURL Placeholder Text WorldCat Lemière C , Torsvik G, Mæstad O, Herbst CH, Leonard KL. 2013 . Evaluating the Impact of Results-Based Financing on Health Worker Performance, HNP Discussion Paper. Washington, DC: The World Bank. Leslie HH , Fink G, Nsona H, Kruk ME. 2016 . Obstetric facility quality and newborn mortality in Malawi: a cross-sectional study . PLoS Medicine 13 : e1002151 . Google Scholar OpenURL Placeholder Text WorldCat Leslie HH , Sun Z, Kruk ME. 2017 . Association between infrastructure and observed quality of care in 4 healthcare services: a cross-sectional study of 4,300 facilities in 8 countries . PLoS Medicine 14 : e1002464 . Google Scholar OpenURL Placeholder Text WorldCat Lim SS , Dandona L, Hoisington JA et al. . 2010 . India’s Janani Suraksha Yojana, a conditional cash transfer programme to increase births in health facilities: an impact evaluation . The Lancet 375 : 2009 – 23 . Google Scholar OpenURL Placeholder Text WorldCat Lohmann J , Muula AS, Houlfort N, De Allegri M. 2018 . How does performance-based financing affect health workers’ intrinsic motivation? A self-determination theory-based mixed-methods study in Malawi . Social Science & Medicine 208 : 1 – 8 . Google Scholar OpenURL Placeholder Text WorldCat Magge H , Nahimana E, Mugunga JC et al. . 2020 . The all babies count initiative: impact of a health system improvement approach on neonatal care and outcomes in Rwanda . Global Health: Science and Practice 8 : 505 – 17 . Google Scholar OpenURL Placeholder Text WorldCat McClure CK , Bodnar LM, Ness R, Catov JM. 2011 . Accuracy of maternal recall of gestational weight gain 4 to 12 years after delivery . Obesity 19 : 1047 – 53 . Google Scholar OpenURL Placeholder Text WorldCat Moyo I , Gandidzanwa C, Tsikiria T et al. . 2015 . Learning from implementation. In: Process Monitoring and Evaluation II of Zimbabwe’s Results-Based Financing Project: The Case of Mutoko, Chiredzi, Nkayi and Kariba Districts . Washington, DC: The World Bank . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Oxman AD , Fretheim A. 2009 . Can paying for results help to achieve the Millennium Development Goals? Overview of the effectiveness of results‐based financing . Journal of Evidence-Based Medicine 2 : 70 – 83 . Google Scholar OpenURL Placeholder Text WorldCat Paina L , Peters DH. 2012 . Understanding pathways for scaling up health services through the lens of complex adaptive systems . Health Policy and Planning 27 : 365 – 73 . Google Scholar OpenURL Placeholder Text WorldCat Paul E , Albert L, Bisala BN et al. . 2018 . Performance-based financing in low-income and middle-income countries: isn’t it time for a rethink? BMJ Global Health 3 : e000664 . Google Scholar OpenURL Placeholder Text WorldCat Rajkotia Y , Zang O, Nguimkeu P et al. . 2017 . The effect of a performance-based financing program on HIV and maternal/child health services in Mozambique—an impact evaluation . Health Policy and Planning 32 : 1386 – 96 . Google Scholar OpenURL Placeholder Text WorldCat Renmans D , Holvoet N, Criel B, Meessen B. 2017 . Performance-based financing: the same is different . Health Policy and Planning 32 : 860 – 8 . Google Scholar OpenURL Placeholder Text WorldCat Ridde V , Yaogo M, Zongo S, Somé P-A, Turcotte-Tremblay A-M. 2018 . Twelve months of implementation of health care performance-based financing in Burkina Faso: a qualitative multiple case study . The International Journal of Health Planning and Management 33 : e153 – 67 . Google Scholar OpenURL Placeholder Text WorldCat Semrau M , Evans-Lacko S, Alem A et al. . 2015 . Strengthening mental health systems in low- and middle-income countries: the Emerald programme . BMC Medicine 13 : 79 . Google Scholar OpenURL Placeholder Text WorldCat Shen GC , Nguyen HTH, Das A et al. . 2017 . Incentives to change: effects of performance-based financing on health workers in Zambia . Human Resources for Health 15 : 20 . Google Scholar OpenURL Placeholder Text WorldCat Sherry TB. 2016 . A note on the comparative statics of pay‐for‐performance in health care . Health Economics 25 : 637 – 44 . Google Scholar OpenURL Placeholder Text WorldCat Sherry TB , Bauhoff S, Mohanan M. 2017 . Multitasking and heterogeneous treatment effects in pay-for-performance in health care: evidence from Rwanda . American Journal of Health Economics 3 : 192 – 226 . Google Scholar OpenURL Placeholder Text WorldCat Walldorf J, De Walque DB, Shapira G, Yamashita-Allen K . 2017 . Baseline Study for the Maternal and Newborn Health Performance Based Financing Project in Lesotho. Washington, DC: The World Bank. Turcotte-Tremblay A-M , Spagnolo J, De Allegri M, Ridde V. 2016 . Does performance-based financing increase value for money in low- and middle- income countries? A systematic review . Health Economics Review 6 : 18 . Google Scholar OpenURL Placeholder Text WorldCat Witter S , Fretheim A, Kessy FL, Lindahl AK. 2012 . Paying for performance to improve the delivery of health interventions in low and middle income countries . Cochrane Effective Practice and Organisation of Care Group (2): 1–16 , 10.1002/14651858.CD007899.pub2 . Google Scholar OpenURL Placeholder Text WorldCat Crossref World Health Organization . 2019 . Newborns: Reducing Mortality, Fact Sheets . Geneva, Switzerland: World Health Organization . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Zeng W , Shepard DS, Nguyen H et al. . 2018 . Cost–effectiveness of results-based financing, Zambia: a cluster randomized trial . Bulletin of the World Health Organization 96 : 760 – 71 . Google Scholar OpenURL Placeholder Text WorldCat © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine.

Aujla, Navneet; Chen, Yen-Fu; Samarakoon, Yasara; Wilson, Anna; Grolmusová, Natalia; Ayorinde, Abimbola; Hofer, Timothy P; Griffiths, Frances; Brown, Celia; Gill, Paramjit; Mallen, Christian; Sartori, Jo; Lilford, Richard J

doi: 10.1093/heapol/czaa152pmid: 33313845

Boyce, Matthew R; Katz, Rebecca

doi: 10.1093/heapol/czaa194pmid: 33491068

Abstract Recent years have witnessed cities establishing themselves as major players in addressing global issues, often taking collective action through international city networks and organizations. These networks are important, as they amplify the voices of municipal officials, who are often excluded from high-level decision-making, and can also provide a platform for officials from low- or middle-income nations to participate in higher-level political forums. The global response to the COVID-19 pandemic has included traditional public health stakeholders—including supranational organizations, international non-governmental organizations and national authorities—but has also featured mayors and city networks, in an unprecedented fashion. Existing networks without an explicit focus on health have shifted their focuses to prioritize pandemic response and several new networks have been created. These developments are significant, not only because they represent a shift in health governance and policy, but also because cities and urban networks more broadly have exhibited a nimbleness and pragmatism unmatched by higher levels of governance. These characteristics could prove beneficial for addressing the current pandemic, as well as future health issues and emergencies. Furthermore, given the relative lack of engagement with health security issues before the COVID-19 pandemic, the drastic health and economic impacts associated with it, and the demonstrable value added by strong city leadership, there are an open policy window and a compelling case for continued city engagement in health security. Global health, governance, health security, policy, sustainable development, urban health KEY MESSAGES Cities have demonstrated an aptitude and nimbleness in responding to the COVID-19 pandemic that has proved useful and addressed the shortcomings of higher levels of governance. Municipal authorities have recently established themselves as major actors in global issues—often acting collectively through inter-city networks and organizations—but historically, these networks have not engaged substantively on health-related issues. The pandemic has shifted or altered the focus of several existing networks and galvanized the creation of several new networks focused on preparing and responding to the public health emergency. There is an open-window for sustained local-level engagement with health security issues that could have implications for better preparing for and responding to future public health emergencies. The rapid spread of the SARS-CoV-2 virus has prompted a collective, global response to the resulting COVID-19 pandemic. This response has included traditional public health stakeholders—supranational organizations (e.g. the World Health Organization), international non-governmental organizations (e.g. Médecins Sans Frontières) and national authorities (e.g. ministries of health, national public health institutes)—but has also featured collective action by sub-national authorities, such as mayors and municipal authorities, in an unprecedented fashion. In recent years, cities have established themselves as major players in addressing transnational issues, such as global warming and climate change, health and pervasive socioeconomic inequalities (Acuto, 2013). While a handful of global cities maintain the clout to influence national and international agendas on their own, to a growing extent, they have sought to do so primarily by forming and operating through organized networks. While this phenomenon is not new—as the first formal international city organization was established over a century ago—our world has never seen such a proliferation of city networks to address global issues (Fernández de Losada and Abdullah, 2019). Indeed, by some estimates, over 200 formal city networks—defined as formalized organizations, primarily composed of cities, and characterized by established patterns of communication, policy‐making and exchange—exist globally, and thousands of para-diplomatic connections further define the relationships between cities (Acuto and Rayner, 2016). While some of these networks have domestic focuses, many more are truly global—amplifying the voices of municipal authorities in low- or middle-income nations that otherwise may be lost to the forces of western dominated politics and norms. However, the utility of these networks is not exclusively confined to advocacy and influencing political agendas, but also extends to more practical considerations, such as taking strategic joint-action and sharing lessons, models and best practices. These developments are significant, not only because they represent a shift in the health governance and policy, but also because cities and urban networks more broadly have exhibited a nimbleness unmatched by more bureaucratic forms of governance that could prove beneficial for addressing future health issues and emergencies. We assume that there are advantages to city networking—namely those previously mentioned—that render them a valuable contribution to pandemic response. In this commentary, we provide a brief but substantiated overview of how city networks have contributed to the COVID-19 pandemic, pose questions and assert that there is a compelling need for the continued engagement of city networks in health security. Pandemic pivots of existing urban networks A 2018 review of 99 city networks found that a majority of them had a specific or very narrow focus (Foster and Swiney, 2018). Of this majority, most networks sought to address environmental issues (e.g. climate change, green infrastructure and transportation, environmental sustainability) and health was identified as a neglected theme. Those networks that did prioritize health, such as the World Health Organization’s European Healthy Cities Network, focused primarily on developing urban spaces to promote good health, addressing socioeconomic inequalities and determinants of health, or vague goals related to achieving health and well-being [World Health Organization Regional Office for Europe (WHO EURO), 2020]. It was within this context that, in 2018, the Global Parliament of Mayors (GPM) took up the cause of pandemic preparedness and prioritized it as a key component of urban health planning and committed to developing an intra-city mechanism to share information and experiences during an emergency response (Global Parliament of Mayors, 2018). In doing so, they became one of the first, if not the first, mayoral networks to explicitly prioritize preparing for responding to infectious disease threats. The GPM is now in good company. Many other networks, including many that did not prioritize health, have pivoted their focuses to address the COVID-19 pandemic. For example, the C40 Network, a group of 96 cities dedicated to addressing climate change, has assembled a regularly updated knowledge hub with resources, articles, policy briefs and indices to help cities better understand and respond to the pandemic. They have also hosted virtual meetings of mayors to share experiences in responding to the pandemic and created a COVID-19 Recovery Task Force to promote a fair and sustainable recovery (C40 Cities, 2020). However, mayors have also used the connections in the network to bolster public health capacities. For example, the City of Seattle leveraged its relationship with the City of Seoul, which was established through the C40 Network, to secure additional diagnostic tests as a means of improving access to COVID testing within the city (Anderson and De Jong, 2020). Similar actions have been taken by both international networks—such as the EuroCities Network (EuroCities, 2020), the Global Resilient Cities Network (formerly the 100 Resilient Cities Program) (Global Resilient Cities Network, 2020), the Metropolis Network (Metropolis, 2020) and the World Organization of United Cities and Local Governments [The World Organization of United Cities and Local Governments (UCLG), 2020]—and domestic networks—such as the National League of Cities (National League of Cities, 2020) and the United States Conference of Mayors [United States Conference of Mayors (USCM), 2020]. New mayoral networks for pandemic response In addition to the shifting focus of existing mayoral networks, the COVID-19 pandemic has spurred the creation of new networks, focused exclusively on responding to the public health emergency or improving the response to future events. Bloomberg Philanthropies has partnered with the National League of Cities, the United States Conference of Mayors and several academic institutions to create a new network—the COVID-19 Local Response Initiative—to help cities in the USA address the pandemic. Members of this network participate in weekly conference calls and daily e-mail updates are sent to all cities involved with the latest actions that member cities are taking to fight the pandemic, mobilize the public and support the local economy (Bloomberg, 2020). Similarly, the Rockefeller Foundation has created an ad hoc city network called the Testing Solutions Group (TSG), i.e. focused on facilitating the exchange of lessons and best practices between city public health officials and linking officials with public health experts to scale up COVID-19 testing while safeguarding the health of communities (Rockefeller Foundation, 2020). In addition to the peer-to-peer network, the TSG convenes experts to develop knowledge products (i.e. strategy and policy recommendations) for local authorities, help city officials collect real-time data to promote evidence-based testing strategies and provide technical and financial assistance for enhancing testing in vulnerable populations. The WHO has also discussed creating a Global Cities Network for Health Security that will be a part of their larger Global Strategic Preparedness Network. This network would be comprised of mayors and other local authorities and would strive to document, share and learn from the experiences among cities in advancing health security. The WHO has also considered allowing these authorities to advocate at future political-level meetings—another request of the GPM in 2018 (Global Parliament of Mayors, 2018). While time will only tell if these larger efforts will come to fruition, they have released specific interim COVID-19 preparedness guidance for local authorities in cities and urban environments that, at a minimum, acknowledge the important role cities have played in the response to this global crisis [World Health Organization (WHO), 2020]. The future of mayoral networks in health security The successful response to public health emergencies requires strong, evidence-based leadership. Some have argued that cities currently are the most powerful they have been since city-states dominated during the Renaissance (Swiney and Foster, 2019); and indeed, mayors and other city officials have stepped up to fill the gaps left by some higher levels of governance to provide consistent and decisive public-sector leadership from the bottom-up (Anderson and De Jong, 2020). But the future of mayoral networks in health security is opaque and likely to be influenced by a variety of factors. For instance, the involvement of cities and city networks in health security will be influenced by larger tensions in health policy and governance—especially concerning powers and authorities relating to outbreak response. This has been seen in a variety of cities throughout the COVID-19 pandemic, such as Atlanta (Bogel-Burroughs and Robertson, 2020), Madrid (Dombey, 2020), Manchester and Marseilles (The Economist, 2020). The involvement of city networks in health security also begs two major questions: first, is this recent trend beneficial; and second, are efforts sustainable in the long-term? The city network ecosystem has already been characterized as saturated, competitive (as opposed to complementary) and at risk of being duplicative—all of which could limit efficiency and waste limited resources (Fernández de Losada and Abdullah, 2019). Regarding long-term sustainability, pandemic preparation is notorious for suffering from cycles of panic and neglect in which efforts are increased when a threat is imminent—when it is too late for them to have optimal impacts—and then quickly decreased when the threat subsides [Global Preparedness and Monitoring Board (GPMB), 2019]. Furthermore, this second question relates not only to financing for pandemic preparedness but also with regard to superficial or temporary involvement of cities in international health security agendas, which can lead to frustration and eventual disengagement. It seems unlikely that networks with other stated priorities and agendas will continue to prioritize health security once the pandemic subsides. However, given the relative dearth of local-level attention or engagement with health security issues prior to the pandemic, the drastic health and economic impacts associated with it, and the value demonstrated by strong city leadership and engagement, we feel there is a clear and compelling case for continued city engagement in health security. Ultimately, it seems unlikely that it will be possible to talk about health security or sustainable development issues without including cities and as a part of the discussion following the COVID-19 pandemic. Funding This work did not receive funding support. Conflict of interest statement. None declared. Ethical approval. This work did not include research with human subjects and thus did not require ethical clearance as outlined in the World Medical Association's Declaration of Helsinki. References Acuto M. 2013 . City leadership in global governance . Global Governance: A Review of Multilateralism and International Organizations 19 : 481 – 98 . Google Scholar Crossref Search ADS WorldCat Acuto M , Rayner S. 2016 . City networks: breaking gridlocks or forging (new) lock-ins? International Affairs 92 : 1147 – 66 . Google Scholar Crossref Search ADS WorldCat Anderson J , De Jong J. 2020 . A nationwide response from an unusual place: city halls . The Hill , 10 July 2020. Google Scholar OpenURL Placeholder Text WorldCat Bogel-Burroughs N , Robertson C. 2020 . While virus surges, Georgia Governor sues Atlanta Mayor to block mask rule . The New York Times , 17 July 2020. Google Scholar OpenURL Placeholder Text WorldCat Bloomberg M. 2020 . To beat the global pandemic, empower local leaders. Bloomberg Opinion, 05 April 2020. C40 Cities . 2020 . 45 mayors & city leaders from 30 countries share knowledge and advice on tackling COVID-19 crisis. C40 Cities, 27 March 2020. Dombey D. 2020 . Madrid regional chief hits out at Spanish government Covid measures. The Financial Times, 13 October 2020. EuroCities . 2020 . Live Updates—COVID-19. https://covidnews.eurocities.eu/, accessed 24 Jul 2020. Fernández de Losada A , Abdullah H. 2019 . Rethinking the Ecosystem of International City Networks: Challenges and Opportunities . Barcelona : CIDOB . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Foster S , Swiney CF. 2019 . City power and powerlessness on the global stage. Barcelona : CIDOB . Global Parliament of Mayors . 2018 . The Bristol Declaration . Bristol : The Global Parliament of Mayors . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Global Preparedness and Monitoring Board (GPMB) . 2019 . A World at Risk: Annual Report on Global Preparedness for Health Emergencies . Geneva : World Health Organization . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Global Resilient Cities Network . 2020 . COVID-19 Response. https://www.rockefellerfoundation.org/covid-19-response/, accessed 24 July 2020. Metropolis . 2020 . Cities for Global Health. https://www.citiesforglobalhealth.org/, accessed 24 July 2020. National League of Cities . 2020 . COVID-19. https://www.nlc.org/topics/health-wellness/covid-19, accessed 24 July 2020. Rockefeller Foundation . 2020 . COVID-19 Testing Group Solutions. https://www.rockefellerfoundation.org/covid-19-national-testing-solutions-group/, accessed 24 July 2020. Swiney CF , Foster S. 2019 . Cities are rising in influence and power on the global stage. CityLab, 15 April 2019. The Economist . 2020 . Across the world central governments face local covid-19 revolts. The Economist, 12 October 2020. The World Organization of United Cities and Local Governments (UCLG) . 2020 . United Cities and Local Governments. https://www.uclg.org/en, accessed 24 July 2020. United States Conference of Mayors (USCM) . 2020 . COVID-19: What Mayors Need to Know. https://www.usmayors.org/issues/covid-19/, accessed 24 July 2020. World Health Organization (WHO) . 2019 . Strengthening Preparedness for COVID-19 in Cities and Urban Settings . Geneva : World Health Organization . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC World Health Organization Regional Office for Europe (WHO EURO) . 2020 . WHO European Healthy Cities Network. https://www.euro.who.int/en/health-topics/environment-and-health/urban-health/who-european-healthy-cities-network, accessed 24 July 2020. © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine.

Hensen, B; Mackworth-Young, C R S; Simwinga, M; Abdelmagid, N; Banda, J; Mavodza, C; Doyle, A M; Bonell, C; Weiss, H A

doi: 10.1093/heapol/czaa158pmid: 33881138

Introduction The coronavirus disease 2019 (COVID-19) pandemic, caused by the SARS-CoV-2 virus, has had unprecedented impacts on health systems, public health, societies and individuals globally (The Lancet Public Health, 2020). In response to outbreaks, physical distancing measures, national lockdowns and travel restrictions to control the spread of COVID-19 have been implemented in many countries (Chu et al., 2020). In response to these measures, many public health researchers are choosing to switch from standard face-to-face data collection methods to remote data collection in support of continued research. Remote data collection is defined here as the collection of data via the phone, online or other virtual platforms, with study participants and researchers physically distanced. The aim of this commentary is to summarize methods, key challenges and opportunities of remote qualitative and quantitative data collection for public health research in low- and middle-income countries (LMIC). The framework we use to structure our discussion is the research process, starting from sampling and culminating in analysis. Within this, we draw out the steps in research most likely to be affected by the pandemic and attendant need to cease face-to-face interactions with research participants. We identify which steps are most affected and what are potential alternatives based on interviews and discussions, held between May and June 2020, with ∼30 researchers from the London School of Hygiene and Tropical Medicine and collaborating partners, representing a range of disciplines. Interviewees were selected or volunteered themselves, based on their experience and expertise in designing and conducting remote data collection. These consultations identified the following as the steps in research most likely to require attention: sampling and recruitment; informed consent; response rates; rapport with participants; privacy and safety; and analysis. Whilst the focus of this commentary is on LMIC, many of the lessons learnt are relevant to remote data collection in high-income countries. What remote data collection methods can I use? Remote qualitative methods include online or phone-based interviews and focus group discussions (FGDs), audio-diary methods (Mupambireyi and Bernays, 2019), photovoice (use of photography to capture lived experiences) (Copes et al., 2018), video documenting, documentary analysis of social media (e.g. Facebook and WhatsApp groups, YouTube comments or podcasts) and auto-ethnography (ethnographic study on self) (Ellis and Bochner, 2000; Lupton, 2020). Remote quantitative methods include mobile phone surveys implemented using: interactive voice response (IVR), short messaging service (SMS) or computer-assisted telephone interviews (CATI) and self-completed online questionnaires, shared via email or social media platforms. These methods are not new (Gibson et al., 2017) with telephone and postal surveys used in higher-income countries; yet their use has become essential during the COVID-19 era to support the collection of data directly from individuals and populations. Each remote data collection method has advantages and disadvantages, which affect their feasibility and acceptability in specific settings (Table 1). For example, when considering a mobile phone survey, although IVR and SMS surveys are cheaper than CATI, they require participants to have high levels of literacy; CATI allows for the inclusion of individuals regardless of literacy and provides opportunities for researchers to encourage participation and study participants to clarify questions (Gibson et al., 2017). With widespread ownership of mobile phones in LMIC, but lower access to smartphones and the Internet, mobile phone methods are more commonly used than online methods and are a key focus of this commentary. Few of the experts interviewed had implemented or were planning online methods due, in part, to their limited reach in certain LMIC. Exceptions include online surveys planned with specific target groups, e.g. members of an established association of professionals and university students. Table 1 Available qualitative and quantitative remote data collection methods, their strengths, limitations and strategies to improve the data quality of different methods Method . Description . Strengths . Limitations . Strategies to improve data quality and navigate challenges . Qualitative methods  Phone interviews In-depth and semi-structured interviews can be conducted by phone. Interviews can also be via WhatsApp calls or online platforms (e.g. Skype or Zoom) Can facilitate the collection of high-quality data on personal experiences Real-time interviews allow for the interviewer to probe, check understanding and follow the direction of conversation Challenge to develop rapport and trust with the participant Inability to see visual cues reduces understanding and appropriate prompting Technological challenges with network, airtime, batteries Cost of phone technologies Disturbance by other noises and activities if participants cannot find a private space Responsibility of privacy placed on participant Training is key to developing rapport with participants over the phone, including role-play practice of interviews, especially how to set the tone at the beginning of the interview with informal conversation Phone interviews should be shorter than face-to-face interviews, to account for participant fatigue Perseverance is required to repeatedly call participants at different times of day and days of the week and call back if the interview is disturbed or cut off  Phone or online FGDs Online platforms or group phone calls can be used to facilitate group discussions remotely. These real-time discussions can be through writing, speaking or with video. Facilitates interaction to understand socially normative perceptions Can provide some peer support Moderation can be challenging, and requires a skilled facilitator Data security of online platforms need to be considered, including end-to-end encryption as some platforms (e.g. Zoom) are less secure If not participating anonymously, ensuring confidentiality is challenging If desired, participants can join anonymously, through providing a pseudonym and not using a video Group phone calls may be most appropriate in lower-income settings, where access to online is lower, but incurs additional airtime costs  Self-collection of data (including diaries, photovoice, video documenting and auto-ethnography) Participants record elements of their lived experiences themselves. Diaries or journals can be handwritten, voice memos or through online platforms or applications. Photovoice or video documenting involves participants taking photos or videos about their everyday practices and interactions that they can share with researchers. Auto-ethnography situates the researcher as the participant, documenting their own lives, experiences and perceptions. Enables participants to generate data at a time and a place that is convenient for them Facilitates tailoring data collection to participants’ personal experiences Attrition to continue data collection can be a problem, especially for longer term studies Data recorded may deviate from desired areas of enquiry or research questions Challenges in transferring self-collected data to researchers Participants using and keeping photo and video technology may not be appropriate in lower-income settings Asking participants what method of self-collection of data suits them can better tailor the method to the participant Providing relevant prompts and guiding questions can help direct participants’ documentation Keeping in touch with participants maintains participation Self-collection of data can be used as prompts for further discussion in combination with interviews and other methods  Documentary analysis Analysis of naturally occurring online data, e.g. analysing YouTube comments, or Facebook group discussions Data are already existing and available in the public domain Data are not produced for the purpose of research, and not directed by the researcher Lack of depth, or ability to probe, compared to interviews or FGDs Capturing and analysing emojis is a challenge, but important for understanding meaning Can capture and transfer text, videos and images to analytical software (e.g. Nvivo) Analytics (e.g. through YouTube) can provide data on demographics of those interacting (watching and commenting) Quantitative methods  SMS survey Individual questions sent via SMS to phone number Participants respond with a number corresponding to a response list Relative to other phone surveys, may be more representative of women (Lau et al., 2019) Appropriate for short surveys, quick to implement More inclusive for individuals who are hard of hearing May be appropriate for sensitive questions Relative to CATI, cheaper and may provide more language options Allows individuals to respond at times most convenient to them Least expensive relative to CATI and IVR (Lau et al., 2019) Low response rates compared to CATI Potential for question-level breakoff, which can increase with each subsequent question Under-represents participants with lower literacy levels Questions need to be highly specific, as there is no opportunity for participants to clarify questions and limits in character numbers Navigating keyboard can be a challenge (Leo et al., 2015), particularly for older populations (who are under-represented in SMS surveys) (Broich, 2015) Participants may have concerns regarding data charges for submitting responses Send reminders to increase response rates Combine with other mobile phone survey method to increase reach Provide an incentive to minimize concerns regarding data charges  IVR survey Automated phone survey, with individuals asked to key in or state their response to the questions Some evidence to suggest this is more inclusive of individuals with lower literacy levels (Gibson et al., 2017; Lau et al., 2019) More representative of rural populations Higher response rates than SMS survey, although not necessarily more representative overall (Lau et al., 2019) Relative to CATI, provides more language options (Lau et al., 2019) More expensive than SMS surveys, but cheaper than CATI Lacks personal touch of CATI surveys, no opportunity for rapport development Not inclusive for individuals who are hard of hearing No opportunity for individuals to clarify questions Individuals may not answer unknown phone number Send survey at different day/time combinations Combine with SMS reminder to increase response rates Combine with other mobile phone survey method to increase reach  CATI Participants are phoned by an individual and asked to complete a survey over the phone Higher response rates relative to IVR and SMS surveys (Lau et al., 2019) Higher survey completion rates, as interviewer able to build rapport and explain the purpose of the study Increased reach of populations with lower literacy Potential for less measurement error, as there is an opportunity for clarification of questions Can complete longer, more complex/sensitive surveys Costlier than IVR and SMS surveys Requires more quality control and training Potential for interviewer bias introduced (Gibson et al., 2017) Make phone calls on different days/times of day Schedule a time to interview the study participant Send SMS prior to the phone call to introduce the study  Online surveys A link to a self-completed questionnaire sent to potential study participants via email, WhatsApp or other social media or networking platform Cheaper and easier to execute than mobile phone survey methods More efficient than mobile phone surveys, reaching a high sample size in a short timeframe (particularly within a specific target population, e.g. university students or members of a professional association) Can be shared via multiple platforms (email, social media) Open-ended, more qualitative questions, can be embedded within the survey, which can provide a rich data source Under-represents individuals without an email address, smartphone or data on their phone (for survey completion) (Roy et al., 2020) If survey completion relies on snowball ‘sharing’ of the survey, the findings may be further subject to bias (Ball, 2019). Limited reach of individuals with lower literacy No opportunity for study participants to clarify questions Provide an incentive for survey completion and for inviting eligible individuals to complete the survey Conduct extensive questionnaire testing to minimize any ambiguity in eligibility criteria and questions Use with specific target group(s) rather than general population (Ball, 2019). Method . Description . Strengths . Limitations . Strategies to improve data quality and navigate challenges . Qualitative methods  Phone interviews In-depth and semi-structured interviews can be conducted by phone. Interviews can also be via WhatsApp calls or online platforms (e.g. Skype or Zoom) Can facilitate the collection of high-quality data on personal experiences Real-time interviews allow for the interviewer to probe, check understanding and follow the direction of conversation Challenge to develop rapport and trust with the participant Inability to see visual cues reduces understanding and appropriate prompting Technological challenges with network, airtime, batteries Cost of phone technologies Disturbance by other noises and activities if participants cannot find a private space Responsibility of privacy placed on participant Training is key to developing rapport with participants over the phone, including role-play practice of interviews, especially how to set the tone at the beginning of the interview with informal conversation Phone interviews should be shorter than face-to-face interviews, to account for participant fatigue Perseverance is required to repeatedly call participants at different times of day and days of the week and call back if the interview is disturbed or cut off  Phone or online FGDs Online platforms or group phone calls can be used to facilitate group discussions remotely. These real-time discussions can be through writing, speaking or with video. Facilitates interaction to understand socially normative perceptions Can provide some peer support Moderation can be challenging, and requires a skilled facilitator Data security of online platforms need to be considered, including end-to-end encryption as some platforms (e.g. Zoom) are less secure If not participating anonymously, ensuring confidentiality is challenging If desired, participants can join anonymously, through providing a pseudonym and not using a video Group phone calls may be most appropriate in lower-income settings, where access to online is lower, but incurs additional airtime costs  Self-collection of data (including diaries, photovoice, video documenting and auto-ethnography) Participants record elements of their lived experiences themselves. Diaries or journals can be handwritten, voice memos or through online platforms or applications. Photovoice or video documenting involves participants taking photos or videos about their everyday practices and interactions that they can share with researchers. Auto-ethnography situates the researcher as the participant, documenting their own lives, experiences and perceptions. Enables participants to generate data at a time and a place that is convenient for them Facilitates tailoring data collection to participants’ personal experiences Attrition to continue data collection can be a problem, especially for longer term studies Data recorded may deviate from desired areas of enquiry or research questions Challenges in transferring self-collected data to researchers Participants using and keeping photo and video technology may not be appropriate in lower-income settings Asking participants what method of self-collection of data suits them can better tailor the method to the participant Providing relevant prompts and guiding questions can help direct participants’ documentation Keeping in touch with participants maintains participation Self-collection of data can be used as prompts for further discussion in combination with interviews and other methods  Documentary analysis Analysis of naturally occurring online data, e.g. analysing YouTube comments, or Facebook group discussions Data are already existing and available in the public domain Data are not produced for the purpose of research, and not directed by the researcher Lack of depth, or ability to probe, compared to interviews or FGDs Capturing and analysing emojis is a challenge, but important for understanding meaning Can capture and transfer text, videos and images to analytical software (e.g. Nvivo) Analytics (e.g. through YouTube) can provide data on demographics of those interacting (watching and commenting) Quantitative methods  SMS survey Individual questions sent via SMS to phone number Participants respond with a number corresponding to a response list Relative to other phone surveys, may be more representative of women (Lau et al., 2019) Appropriate for short surveys, quick to implement More inclusive for individuals who are hard of hearing May be appropriate for sensitive questions Relative to CATI, cheaper and may provide more language options Allows individuals to respond at times most convenient to them Least expensive relative to CATI and IVR (Lau et al., 2019) Low response rates compared to CATI Potential for question-level breakoff, which can increase with each subsequent question Under-represents participants with lower literacy levels Questions need to be highly specific, as there is no opportunity for participants to clarify questions and limits in character numbers Navigating keyboard can be a challenge (Leo et al., 2015), particularly for older populations (who are under-represented in SMS surveys) (Broich, 2015) Participants may have concerns regarding data charges for submitting responses Send reminders to increase response rates Combine with other mobile phone survey method to increase reach Provide an incentive to minimize concerns regarding data charges  IVR survey Automated phone survey, with individuals asked to key in or state their response to the questions Some evidence to suggest this is more inclusive of individuals with lower literacy levels (Gibson et al., 2017; Lau et al., 2019) More representative of rural populations Higher response rates than SMS survey, although not necessarily more representative overall (Lau et al., 2019) Relative to CATI, provides more language options (Lau et al., 2019) More expensive than SMS surveys, but cheaper than CATI Lacks personal touch of CATI surveys, no opportunity for rapport development Not inclusive for individuals who are hard of hearing No opportunity for individuals to clarify questions Individuals may not answer unknown phone number Send survey at different day/time combinations Combine with SMS reminder to increase response rates Combine with other mobile phone survey method to increase reach  CATI Participants are phoned by an individual and asked to complete a survey over the phone Higher response rates relative to IVR and SMS surveys (Lau et al., 2019) Higher survey completion rates, as interviewer able to build rapport and explain the purpose of the study Increased reach of populations with lower literacy Potential for less measurement error, as there is an opportunity for clarification of questions Can complete longer, more complex/sensitive surveys Costlier than IVR and SMS surveys Requires more quality control and training Potential for interviewer bias introduced (Gibson et al., 2017) Make phone calls on different days/times of day Schedule a time to interview the study participant Send SMS prior to the phone call to introduce the study  Online surveys A link to a self-completed questionnaire sent to potential study participants via email, WhatsApp or other social media or networking platform Cheaper and easier to execute than mobile phone survey methods More efficient than mobile phone surveys, reaching a high sample size in a short timeframe (particularly within a specific target population, e.g. university students or members of a professional association) Can be shared via multiple platforms (email, social media) Open-ended, more qualitative questions, can be embedded within the survey, which can provide a rich data source Under-represents individuals without an email address, smartphone or data on their phone (for survey completion) (Roy et al., 2020) If survey completion relies on snowball ‘sharing’ of the survey, the findings may be further subject to bias (Ball, 2019). Limited reach of individuals with lower literacy No opportunity for study participants to clarify questions Provide an incentive for survey completion and for inviting eligible individuals to complete the survey Conduct extensive questionnaire testing to minimize any ambiguity in eligibility criteria and questions Use with specific target group(s) rather than general population (Ball, 2019). SMS - short message service IVR - interactive voice response CATI - computer-assisted telephone interview Open in new tab Table 1 Available qualitative and quantitative remote data collection methods, their strengths, limitations and strategies to improve the data quality of different methods Method . Description . Strengths . Limitations . Strategies to improve data quality and navigate challenges . Qualitative methods  Phone interviews In-depth and semi-structured interviews can be conducted by phone. Interviews can also be via WhatsApp calls or online platforms (e.g. Skype or Zoom) Can facilitate the collection of high-quality data on personal experiences Real-time interviews allow for the interviewer to probe, check understanding and follow the direction of conversation Challenge to develop rapport and trust with the participant Inability to see visual cues reduces understanding and appropriate prompting Technological challenges with network, airtime, batteries Cost of phone technologies Disturbance by other noises and activities if participants cannot find a private space Responsibility of privacy placed on participant Training is key to developing rapport with participants over the phone, including role-play practice of interviews, especially how to set the tone at the beginning of the interview with informal conversation Phone interviews should be shorter than face-to-face interviews, to account for participant fatigue Perseverance is required to repeatedly call participants at different times of day and days of the week and call back if the interview is disturbed or cut off  Phone or online FGDs Online platforms or group phone calls can be used to facilitate group discussions remotely. These real-time discussions can be through writing, speaking or with video. Facilitates interaction to understand socially normative perceptions Can provide some peer support Moderation can be challenging, and requires a skilled facilitator Data security of online platforms need to be considered, including end-to-end encryption as some platforms (e.g. Zoom) are less secure If not participating anonymously, ensuring confidentiality is challenging If desired, participants can join anonymously, through providing a pseudonym and not using a video Group phone calls may be most appropriate in lower-income settings, where access to online is lower, but incurs additional airtime costs  Self-collection of data (including diaries, photovoice, video documenting and auto-ethnography) Participants record elements of their lived experiences themselves. Diaries or journals can be handwritten, voice memos or through online platforms or applications. Photovoice or video documenting involves participants taking photos or videos about their everyday practices and interactions that they can share with researchers. Auto-ethnography situates the researcher as the participant, documenting their own lives, experiences and perceptions. Enables participants to generate data at a time and a place that is convenient for them Facilitates tailoring data collection to participants’ personal experiences Attrition to continue data collection can be a problem, especially for longer term studies Data recorded may deviate from desired areas of enquiry or research questions Challenges in transferring self-collected data to researchers Participants using and keeping photo and video technology may not be appropriate in lower-income settings Asking participants what method of self-collection of data suits them can better tailor the method to the participant Providing relevant prompts and guiding questions can help direct participants’ documentation Keeping in touch with participants maintains participation Self-collection of data can be used as prompts for further discussion in combination with interviews and other methods  Documentary analysis Analysis of naturally occurring online data, e.g. analysing YouTube comments, or Facebook group discussions Data are already existing and available in the public domain Data are not produced for the purpose of research, and not directed by the researcher Lack of depth, or ability to probe, compared to interviews or FGDs Capturing and analysing emojis is a challenge, but important for understanding meaning Can capture and transfer text, videos and images to analytical software (e.g. Nvivo) Analytics (e.g. through YouTube) can provide data on demographics of those interacting (watching and commenting) Quantitative methods  SMS survey Individual questions sent via SMS to phone number Participants respond with a number corresponding to a response list Relative to other phone surveys, may be more representative of women (Lau et al., 2019) Appropriate for short surveys, quick to implement More inclusive for individuals who are hard of hearing May be appropriate for sensitive questions Relative to CATI, cheaper and may provide more language options Allows individuals to respond at times most convenient to them Least expensive relative to CATI and IVR (Lau et al., 2019) Low response rates compared to CATI Potential for question-level breakoff, which can increase with each subsequent question Under-represents participants with lower literacy levels Questions need to be highly specific, as there is no opportunity for participants to clarify questions and limits in character numbers Navigating keyboard can be a challenge (Leo et al., 2015), particularly for older populations (who are under-represented in SMS surveys) (Broich, 2015) Participants may have concerns regarding data charges for submitting responses Send reminders to increase response rates Combine with other mobile phone survey method to increase reach Provide an incentive to minimize concerns regarding data charges  IVR survey Automated phone survey, with individuals asked to key in or state their response to the questions Some evidence to suggest this is more inclusive of individuals with lower literacy levels (Gibson et al., 2017; Lau et al., 2019) More representative of rural populations Higher response rates than SMS survey, although not necessarily more representative overall (Lau et al., 2019) Relative to CATI, provides more language options (Lau et al., 2019) More expensive than SMS surveys, but cheaper than CATI Lacks personal touch of CATI surveys, no opportunity for rapport development Not inclusive for individuals who are hard of hearing No opportunity for individuals to clarify questions Individuals may not answer unknown phone number Send survey at different day/time combinations Combine with SMS reminder to increase response rates Combine with other mobile phone survey method to increase reach  CATI Participants are phoned by an individual and asked to complete a survey over the phone Higher response rates relative to IVR and SMS surveys (Lau et al., 2019) Higher survey completion rates, as interviewer able to build rapport and explain the purpose of the study Increased reach of populations with lower literacy Potential for less measurement error, as there is an opportunity for clarification of questions Can complete longer, more complex/sensitive surveys Costlier than IVR and SMS surveys Requires more quality control and training Potential for interviewer bias introduced (Gibson et al., 2017) Make phone calls on different days/times of day Schedule a time to interview the study participant Send SMS prior to the phone call to introduce the study  Online surveys A link to a self-completed questionnaire sent to potential study participants via email, WhatsApp or other social media or networking platform Cheaper and easier to execute than mobile phone survey methods More efficient than mobile phone surveys, reaching a high sample size in a short timeframe (particularly within a specific target population, e.g. university students or members of a professional association) Can be shared via multiple platforms (email, social media) Open-ended, more qualitative questions, can be embedded within the survey, which can provide a rich data source Under-represents individuals without an email address, smartphone or data on their phone (for survey completion) (Roy et al., 2020) If survey completion relies on snowball ‘sharing’ of the survey, the findings may be further subject to bias (Ball, 2019). Limited reach of individuals with lower literacy No opportunity for study participants to clarify questions Provide an incentive for survey completion and for inviting eligible individuals to complete the survey Conduct extensive questionnaire testing to minimize any ambiguity in eligibility criteria and questions Use with specific target group(s) rather than general population (Ball, 2019). Method . Description . Strengths . Limitations . Strategies to improve data quality and navigate challenges . Qualitative methods  Phone interviews In-depth and semi-structured interviews can be conducted by phone. Interviews can also be via WhatsApp calls or online platforms (e.g. Skype or Zoom) Can facilitate the collection of high-quality data on personal experiences Real-time interviews allow for the interviewer to probe, check understanding and follow the direction of conversation Challenge to develop rapport and trust with the participant Inability to see visual cues reduces understanding and appropriate prompting Technological challenges with network, airtime, batteries Cost of phone technologies Disturbance by other noises and activities if participants cannot find a private space Responsibility of privacy placed on participant Training is key to developing rapport with participants over the phone, including role-play practice of interviews, especially how to set the tone at the beginning of the interview with informal conversation Phone interviews should be shorter than face-to-face interviews, to account for participant fatigue Perseverance is required to repeatedly call participants at different times of day and days of the week and call back if the interview is disturbed or cut off  Phone or online FGDs Online platforms or group phone calls can be used to facilitate group discussions remotely. These real-time discussions can be through writing, speaking or with video. Facilitates interaction to understand socially normative perceptions Can provide some peer support Moderation can be challenging, and requires a skilled facilitator Data security of online platforms need to be considered, including end-to-end encryption as some platforms (e.g. Zoom) are less secure If not participating anonymously, ensuring confidentiality is challenging If desired, participants can join anonymously, through providing a pseudonym and not using a video Group phone calls may be most appropriate in lower-income settings, where access to online is lower, but incurs additional airtime costs  Self-collection of data (including diaries, photovoice, video documenting and auto-ethnography) Participants record elements of their lived experiences themselves. Diaries or journals can be handwritten, voice memos or through online platforms or applications. Photovoice or video documenting involves participants taking photos or videos about their everyday practices and interactions that they can share with researchers. Auto-ethnography situates the researcher as the participant, documenting their own lives, experiences and perceptions. Enables participants to generate data at a time and a place that is convenient for them Facilitates tailoring data collection to participants’ personal experiences Attrition to continue data collection can be a problem, especially for longer term studies Data recorded may deviate from desired areas of enquiry or research questions Challenges in transferring self-collected data to researchers Participants using and keeping photo and video technology may not be appropriate in lower-income settings Asking participants what method of self-collection of data suits them can better tailor the method to the participant Providing relevant prompts and guiding questions can help direct participants’ documentation Keeping in touch with participants maintains participation Self-collection of data can be used as prompts for further discussion in combination with interviews and other methods  Documentary analysis Analysis of naturally occurring online data, e.g. analysing YouTube comments, or Facebook group discussions Data are already existing and available in the public domain Data are not produced for the purpose of research, and not directed by the researcher Lack of depth, or ability to probe, compared to interviews or FGDs Capturing and analysing emojis is a challenge, but important for understanding meaning Can capture and transfer text, videos and images to analytical software (e.g. Nvivo) Analytics (e.g. through YouTube) can provide data on demographics of those interacting (watching and commenting) Quantitative methods  SMS survey Individual questions sent via SMS to phone number Participants respond with a number corresponding to a response list Relative to other phone surveys, may be more representative of women (Lau et al., 2019) Appropriate for short surveys, quick to implement More inclusive for individuals who are hard of hearing May be appropriate for sensitive questions Relative to CATI, cheaper and may provide more language options Allows individuals to respond at times most convenient to them Least expensive relative to CATI and IVR (Lau et al., 2019) Low response rates compared to CATI Potential for question-level breakoff, which can increase with each subsequent question Under-represents participants with lower literacy levels Questions need to be highly specific, as there is no opportunity for participants to clarify questions and limits in character numbers Navigating keyboard can be a challenge (Leo et al., 2015), particularly for older populations (who are under-represented in SMS surveys) (Broich, 2015) Participants may have concerns regarding data charges for submitting responses Send reminders to increase response rates Combine with other mobile phone survey method to increase reach Provide an incentive to minimize concerns regarding data charges  IVR survey Automated phone survey, with individuals asked to key in or state their response to the questions Some evidence to suggest this is more inclusive of individuals with lower literacy levels (Gibson et al., 2017; Lau et al., 2019) More representative of rural populations Higher response rates than SMS survey, although not necessarily more representative overall (Lau et al., 2019) Relative to CATI, provides more language options (Lau et al., 2019) More expensive than SMS surveys, but cheaper than CATI Lacks personal touch of CATI surveys, no opportunity for rapport development Not inclusive for individuals who are hard of hearing No opportunity for individuals to clarify questions Individuals may not answer unknown phone number Send survey at different day/time combinations Combine with SMS reminder to increase response rates Combine with other mobile phone survey method to increase reach  CATI Participants are phoned by an individual and asked to complete a survey over the phone Higher response rates relative to IVR and SMS surveys (Lau et al., 2019) Higher survey completion rates, as interviewer able to build rapport and explain the purpose of the study Increased reach of populations with lower literacy Potential for less measurement error, as there is an opportunity for clarification of questions Can complete longer, more complex/sensitive surveys Costlier than IVR and SMS surveys Requires more quality control and training Potential for interviewer bias introduced (Gibson et al., 2017) Make phone calls on different days/times of day Schedule a time to interview the study participant Send SMS prior to the phone call to introduce the study  Online surveys A link to a self-completed questionnaire sent to potential study participants via email, WhatsApp or other social media or networking platform Cheaper and easier to execute than mobile phone survey methods More efficient than mobile phone surveys, reaching a high sample size in a short timeframe (particularly within a specific target population, e.g. university students or members of a professional association) Can be shared via multiple platforms (email, social media) Open-ended, more qualitative questions, can be embedded within the survey, which can provide a rich data source Under-represents individuals without an email address, smartphone or data on their phone (for survey completion) (Roy et al., 2020) If survey completion relies on snowball ‘sharing’ of the survey, the findings may be further subject to bias (Ball, 2019). Limited reach of individuals with lower literacy No opportunity for study participants to clarify questions Provide an incentive for survey completion and for inviting eligible individuals to complete the survey Conduct extensive questionnaire testing to minimize any ambiguity in eligibility criteria and questions Use with specific target group(s) rather than general population (Ball, 2019). SMS - short message service IVR - interactive voice response CATI - computer-assisted telephone interview Open in new tab In the following sections, we describe the specific challenges of remote data collection throughout the design, conduct and analysis of a research study, and discuss the implications for: ethics, sampling and recruiting study participants, obtaining informed consent, maximizing response, protecting participants’ privacy and confidentiality and data analysis and interpretation. Is it ethically appropriate to conduct my research study during the COVID-19 pandemic? Individuals, communities and societies face heightened social, physical and emotional challenges during the COVID-19 pandemic. Decisions on whether to conduct research using remote methods need to consider the research burden and COVID-19-related risks to study participants. For example, remote collection of data may require greater effort on the part of the study participant, who may be required to use their own phone, their own resources to charge this phone, and to identify a private space to participate in the study. On the other hand, remote methods may be more preferable to study participants, removing the time and opportunity cost associated with travel to study sites. As with any research, potential risks need to be weighed against benefits and the ethical imperative to continue with research to generate the evidence of benefit to public health. How do I sample and recruit study participants? Key challenges in remote data collection include garnering diverse experiences (qualitative research), obtaining a sampling frame representative of the population of interest (quantitative research) and contacting ‘harder to reach’ populations (Tran et al., 2015). Whilst some of these challenges are present in face-to-face research, the limited ability to recruit participants in person, either at home, in a clinic or other venue, alongside the reliance on mobile phones for recruitment, heightens these challenges and creates the need for alternative sampling methods. For qualitative research, sampling approaches include purposive sampling, snowball and convenience sampling. Purposive sampling aims to ensure diversity according to key factors theorized to influence experience. Recruitment can be facilitated via community-based organizations and leaders, neighbourhood health committees or established networks (Sudan case study, Box 1). Snowball sampling can be effective for qualitative research, although drawing from multiple initial participants (who then recruit others from within their networks) is important to achieve diversity (Shaghaghi et al., 2011; Kirchherr and Charles, 2018). These sampling methods can also be used in quantitative research; snowball sampling may be useful for online surveys shared via email or social media platforms (Roy et al., 2020), and a convenience sample can be recruited through online social networking platforms. Box 1 A remote collaboration with youth networks for research during the COVID-19 pandemic: a case study from Sudan In April 2020, a study to explore the acceptability and feasibility of strategies to shield high-risk individuals from COVID-19 was launched in six communities in Sudan. Researchers partnered with a Sudanese network of youth volunteers, aged 20–30 years, trained in promoting health and youth participation. Volunteers were trained using social media; pre-recorded training sessions were shared via WhatsApp along with interview guides. A virtual chat meeting was held to answer questions and receive feedback on the interview guide. Volunteers identified 60 eligible study participants purposively, by calling existing community contacts, and conducted phone-based interviews. Eligible participants were any adult household member in households with a member at high risk of COVID-19 (∼38% of respondents were female). To summarize observations of emerging themes, volunteers were given a reporting template. Conference calls facilitated sharing insights from the reports and volunteers’ intimate knowledge of the data. Interview recordings and transcripts were uploaded to a secure cloud platform for further thematic analysis by researchers. Poor connectivity prevented live training, delayed uploading of interview recordings, and disrupted interviews and group discussions. With volunteers using their own phones to conduct interviews, data security concerns also emerged. The volunteer’s lack of prior research experience delayed the original study timeline, as frequent support by researchers was needed, e.g. to ensure post-interview clean-up of identifying information about study participants. Despite challenges, the partnership leveraged the expertise of researchers and the volunteers’ existing community links. The study provided an opportunity to invest in an established community-based network, with the prospect of acquiring research skills and adapting their COVID-19 prevention messaging, both of which were key motivators for the volunteers. Despite a lockdown, and without access to a sampling frame, volunteers were able to remotely identify participants and conduct interviews efficiently and with limited resources. For quantitative research, representative samples from the population of interest are either important to maximize internal validity (descriptive research) or useful to maximize external validity (aetiological/evaluation research). In countries where mobile phone ownership is high, a sampling frame of the general population can be obtained by contacting mobile phone network operators or mobile phone survey companies who maintain lists of phone numbers. A sample can then be randomly selected using these lists. Alternatively, random digit dialling could be used to generate a study sample. These methods, however, have limitations. Network operators may be unwilling to provide phone numbers and random digit dialling is unlikely to yield a representative random sample of the population. For a descriptive, population-based survey, lack of representativeness limits the validity of this approach. As with qualitative research, established relationships, e.g. with participants recruited to a cohort study (Malawi case study, Box 2), can be leveraged to facilitate continued or new research. Where the target population is a specific group, e.g. female sex workers or adolescents, respondent-driven sampling (where individuals representative of the target population are provided a fixed number of coded coupons to incentivize recruitment of their peers to the study) (Heckathorn, 1997; Johnston and Sabin, 2010), is an established method that can be implemented using mobile phones or online to, in principle, obtain a representative sample. Depending on the target population, existing lists that are representative of the population, e.g. registers of school students or email addresses/phone numbers for members of a professional association, can be leveraged. However, data protection and ethical issues around sharing personal details need to be considered; lists should be anonymized to maintain confidentiality and the owners of these lists should inform potential study participants about the research prior to recruitment. Where the target population is individuals attending particular spaces, e.g. bars and sport facilities, or indeed geographical areas, open source maps can be used to generate a sampling frame and existing social networks leveraged to initiate data collection. Box 2 Conducting telephone interviews during COVID-19: a case study from Malawi1 To document changes in COVID-19-related knowledge, attitudes and behaviours in Malawi, a cohort study of four rounds of mobile phone surveys was initiated in April 2020, with follow-ups due for completion in November 2020. Study participants were primarily adult residents of Karonga district, Northern Malawi, who had previously participated in epidemiological studies, led by MEIRU, on feasibility of measuring mortality. During these pre-COVID-19 studies (December 2019–March 2020), 1 036 individuals were asked for their phone number for recruitment and/or follow-up purposes. Among these individuals, 257 (24.8%) did not have a phone number or refused to provide one. Interviewers, working from their homes, called these phone numbers and obtained consent to participate after verifying participants’ identity. On average, three calls were required to complete an interview. Respondents received airtime credit of $1.50 upon completion of the interview. Of 779 potential respondents, 620 (79.6%; 77.8% of males and 80.9% of females) completed the first interview. Factors contributing to successful contact with participants included calling at times when they were likely to be free (late afternoon) and at times suggested by participants, making additional calls even when previous attempts were unsuccessful, and attempts at different times and days. Key challenges were that phone numbers did not exist or were disconnected from the network, and calls went unanswered throughout the study (15% overall). The median interview duration was 30 minutes, with significant variation between interviewers despite receiving the same training, practice sessions and having similar previous interviewing experiences. This variation was attributable to the time required by individual interviewers to develop rapport, obtain informed consent and navigate the survey questionnaire. Some calls lasted more than one hour due to multi-tasking on the part of study participants or calls disconnecting because of poor network and limited battery life. Despite challenges, once contacted, non-consent was low (<1%). 1This work was funded by the National Institutes of Health R01HD088516 (PI: Helleringer). In practice, a combination of approaches may be necessary to recruit study participants. However, limitations related to the diversity of experience and representativeness are likely to persist as is restricted participation of more vulnerable populations, including individuals with vision or hearing impairments, low literacy, and older populations. Where a mobile phone survey or interview is planned, one strategy to reach individuals without a phone is contacting, or even interviewing, a phone-owning friend or relative; however, this may not be appropriate for sensitive research topics. How can I obtain informed consent remotely? Oral consent (over the phone or via a voice note) or written consent (via email, WhatsApp or SMS) is being accepted by some ethics committees as written informed consent becomes challenging, or impossible. For mobile phone-based research with adolescents, which requires parental/guardian consent, additional challenges emerge in confirming the age of the participant to establish whether parental/guardian consent is needed and in ensuring consent is being provided by the parent/guardian rather than the respondent themselves, a friend or other relative. For these reasons, oral consent, which can be recorded or conducted in combination with written consent where feasible, may be preferable to written consent only. Concise and simple language is required to convey complete information remotely, whilst maintaining the rigorous ethical standards of face-to-face research. Consent should always be appropriately documented, whilst protecting patient data and confidentiality. Documentation could be in the form of a list of participants, stored on a password-protected computer, who consented to participate in different study components, which could also serve as a record for audit purposes. How do I navigate technological challenges in recruitment to maximize response rates? Researchers should anticipate higher non-response than face-to-face methods in sample size calculations. For mobile phone surveys, response rates are influenced by factors including phone ownership and autonomy to use phones. In some settings, this means rural women and elderly populations are under-represented. Even where mobile phone ownership is high, low response rates threaten study validity as how representative study participants are of the broader, target population would remain unclear. Among individuals with a phone, response rates are affected by distrust of unknown phone numbers, phone-based harassment (Lamanna et al., 2019), time required to complete the survey, poor network coverage and inadequate access to electricity to charge phones (Malawi case study, Box 2). Online surveys can achieve high participation yet they overrepresent higher-income, urban populations with higher literacy and access to smartphones and/or the Internet (Roy et al., 2020). To improve response rates to mobile phone surveys, researchers can use established relationships with participants or community-based organizations or send an SMS, prior to the phone call, to introduce the study and inform individuals that they should anticipate a call. In the absence of transport refunds, the provision of airtime to compensate for participants’ time and their own resources needed to charge their phones is important from an ethical standpoint. Airtime incentives to participate in the study and to refer friends to the study can achieve higher response (Gibson et al., 2017). However, issues of joint phone ownership need to be navigated, in which case other compensation, such as vouchers redeemable at local shops, could be considered. Perseverance (i.e. repeatedly contacting participants at different time and day combinations) is also required, which can be facilitated through protocols detailing the frequency and timing of contacts (Malawi case study, Box 2). To increase survey completion rates, questionnaires and interview guides need to be short (lasting no longer than 30 minutes) (Dabalen et al., 2016). Placing the most pertinent questions near the start of a survey is of greater importance in remote data collection, as technological challenges may occur, participants may be more likely to experience fatigue, be distracted by other activities or have their privacy compromised. How do I build rapport with participants? Intensive training of interviewers, including role play for phone-based interviews, is critical for developing strategies to build rapport. Rapport should be established in the first few minutes of a call, with informal conversations incorporated in the consent process (Zimbabwe case study, Box 3). Phone-based in-depth interviews and CATI enable researchers to develop rapport with study participants, which can improve response rates and be more appropriate for asking complex and sensitive questions (Gibson et al., 2017; Lau et al., 2019). To increase response to sensitive questions, e.g. sexual behaviours, and the validity of these data, researchers should consider combined approaches, providing individuals the opportunity to respond via SMS or IVR. This is similar to the use of audio computer‐assisted survey instruments within face-to-face surveys, which can reduce reporting bias (Langhaug et al., 2010). However, combining methods may have implications on the cost, time and technical expertise required to complete the study. Box 3 Phone interviews with healthcare providers to understand the perceptions and experiences of lockdown measures: a case study from Zimbabwe Between March and April 2020, a process evaluation nested within an existing cluster randomized trial of a community-based integrated HIV and sexual and reproductive health service for youth in Zimbabwe was adapted to explore healthcare providers’ perceptions and experiences of national lockdown measures. In the first week of the lockdown, 15 phone-based interviews were conducted. Written informed consent was obtained at a face-to-face meeting prior to the lockdown with the providers, who were purposively selected to provide diverse experiences across location, role, age and gender and whose phone numbers were already known. For participants who had existing relationships with the interviewer, rapport was easily established, although lack of visual cues obstructed the ability to probe. To work around more formal and formulaic responses, particularly, for those the interviewer had not met before, the interviewer built informal conversation into the interview, particularly during the first few minutes of discussion. Some participants were, in fact, more open over the phone: the interview offered them a rare chance to express their feelings and concerns during lockdown, knowing that they would not see the interviewer in the foreseeable future. Logistical and technological challenges were faced. Network issues interrupted interview flow, forcing the interviewer to be flexible with re-scheduling interviews. Many participants could not find a quiet and private space to participate in the interview, with children and other conversations disrupting the interview. Perseverance and flexibility were required, such as allowing participants to reschedule the interview at a time convenient to them. Despite challenges, conducting the interviews by phone circumvented the need to travel, enabling the rapid collection of data which the researchers considered to be of high quality. Importantly, participants expressed gratitude at having the opportunity to talk to someone and share the challenges they were facing as a result of the lockdown. How do I protect participants’ privacy and safety? When research is face to face, the researcher is responsible for establishing privacy and halting data collection when privacy is compromised. Remote research places this onus on the study participant. Yet, establishing privacy can be difficult where participants share homes and have limited private space or time (Zimbabwe case study, Box 3). Privacy is particularly important for studies exploring sensitive topics, such as gender-based violence, where the consequences of compromised privacy could be harmful (Peterman et al., 2020). At the start of data collection, participants should be advised of the potentially sensitive nature of the study and that they should seek a private space. To mitigate risk, strategies include using ‘code words’ or an ‘exit button’ that participants can say or press when their privacy is compromised (Peterman et al., 2020). IVR and online surveys enable participants to complete surveys at a time and place of their choosing, offering more flexibility for participants to establish privacy. These surveys could include a question on whether the respondent completed the survey in private, or in the presence of, e.g. their child, parent/guardian or friend. Data protection, including end-to-end encryption of phone calls and security of platforms used to deliver online surveys and interview transcripts, is an additional issue relevant to privacy and confidentiality that requires consideration (Eynon et al., 2017). In addition, researchers have a duty of care and need to carefully consider safeguarding issues, especially where COVID-19 has impacted the availability of support services. Information on online or phone-based services should be made available during the consent process. Specific protocols need to be developed for referrals, interviewers need to be informed if particular responses may trigger automatic referrals, and follow-up is required where safeguarding issues emerge. As a part of this protocol, researchers need to establish a system to regularly check that these services have remained operational. How do I analyse and interpret data collected remotely? Remotely collected quantitative data will likely be affected by response bias (Labrique et al., 2017). Weighting results using existing data from a census or population-based survey known to be representative of the population of interest can been used to reduce this bias (Lau et al., 2019). However, the use of weights in data analysis reduces precision and may have little effect on estimates (Lau et al., 2019). As with face-to-face data collection, transparency regarding limitations is essential, including reporting response rates and other potential sources of bias (Greenleaf et al., 2017). Data on whether the respondent was alone whilst completing a mobile phone or online survey can be used in a sensitivity analysis to assess whether having another person present compromised responses. Analysis of remote qualitative data needs to account for issues around rapport; triangulation of data from different methods can help provide depth. Findings emerging from remote methods should be interpreted in light of these limitations and the implications on generalizability discussed. What opportunities do remote data collection methods present? Remote data collection presents opportunities and challenges. The methods enable data collection in contexts where face-to-face data collection is less feasible, e.g. during violence and unrest, when travel restrictions are in place, a natural disaster and during other disease outbreaks. The methods may provide greater autonomy and privacy, e.g. through use of a pseudonym during online FGDs and surveys. Self-collected remote qualitative methods, such as audio diaries, photovoice, video documenting and auto-ethnography enable more participant-centred data collection. The engagement of members from the population of interest in the research activities demonstrates to the public the value placed on their perspectives and lived experiences and can be used to inform and strengthen activities already being implemented by communities (Sudan case study, Box 1). Remote data collection also provides an opportunity for more efficient data collection, being less expensive and time consuming than face-to-face data collection. The methods may be preferred by some study participants who may also have more time for participation, particularly during lockdowns. This efficiency, particularly with automated phone surveys, facilitates data collection from a large number of study participants over a short timeframe, providing critical information to inform the response to COVID-19 or similar crises. The benefits may be greatest for follow-up surveys among cohorts already engaged in research. Leveraging the widespread use of mobile phones among younger adult men, often under-represented in face-to-face population-based surveys, provides opportunities to reach broader cross-sections of a population (Lau et al., 2019; L’Engle et al., 2018). Concluding remarks In a COVID-19 era, remote data collection is needed to inform the response to the pandemic and other public health issues. The remote collection of data presents key ethical challenges and particular challenges related to identifying and recruiting study participants. With high and increasing ownership, remote data collection is likely to continue to rely on mobile phones, which remains easiest when building on existing relationships, where contact details are known, rapport is developed and trust established. A key challenge requiring further research and navigation is how to involve individuals who do not own mobile phones and have limited access to the Internet. Furthermore, available approaches to remote data collection are restricted in their ability to establish personal connections. Personal connections are more easily developed through face-to-face interaction and can be critical to public health research, e.g. in the case of qualitative research or to quantitative research particularly on sensitive topics. Despite limitations, remote methods can be more efficient than face-to-face data collection and provide platforms to empower individuals to engage in generating and analysing data. Lessons learnt in designing and implementing remote data collection methods in a COVID-19 era are critical to inform future execution of these methods, which are likely to become fundamental to continued research in public health. Funding BH and CMY lead this work with funding support from LSHTM. Acknowledgements We thank our colleagues at LSHTM and collaborating partners, who shared their experiences with remote data collection. We thank our colleagues at LSHTM and collaborating partners, who shared their experiences with remote data collection: Abena Amoah, Venetia Baker, Virginia Bond, Hannah Brindle, Albert Dube, Julia Fortier, Shaffa Hameed, Ada Humphrey, Meenakshi Gautham, Chris Grundy, Gareth Knight, Shelley Lees, Julia Lohmann, Fortunate Machingura, Melisa Martinez-Alvarez, Ona McCarthy, Catherine McGowan, Francis Meyo, Sandra Mounier-Jack, Matthew Quaife, Joanna Schellenberg, Nathaniel Scherer, Janet Seeley, Anisha Singh, Emma Slaymaker, Jimmy Whitworth, and Suneetha Kadiyala. Conflict of interest statement. The authors have no conflicts of interest to declare. Ethical approval. Ethical approval for this type of study is not required by our institute. References Ball HL. 2019. Conducting Online Surveys. Journal of Human Lactation 35(3):413–17. Broich C. 2015. Offline data collection in sub-Sahran Africa using SMS surveys: Lessons learned. Sample Solutions. Chu DK , Akl EA, Duda S et al. . 2020 . Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis . The Lancet 395 : 1973 – 87 . Google Scholar OpenURL Placeholder Text WorldCat Copes H , Tchoula W, Brookman F, Ragland J. 2018 . Photo-elicitation interviews with vulnerable populations: practical and ethical considerations . Deviant Behavior 39 : 475 – 94 . Google Scholar OpenURL Placeholder Text WorldCat Dabalen A , Etang A, Hoogeveen A et al. . 2016 . Mobile Phone Panel Surveys in Developing Countries: A Practical Guide for Microdata Collection. Directions in Development--Poverty. Washington, DC: World Bank. Ellis C , Bochner A. 2000 . Autoethnography, personal narrative, reflexivity: researcher as subject. In: Denzin N, Lincoln Y (eds). Handbook of Qualitative Research . 2nd edn. Thousand Oaks, CA : Sage Publications , 733 – 68 . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Eynon R , Fry J, Schroeder R. 2017 . The SAGE Handbook of Online Research Methods . London : SAGE Publications Ltd . https://methods.sagepub.com/book/the-sage-handbook-of-online-research-methods-second-edition. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Gibson DG , Pereira A, Farrenkopf BA et al. . 2017 . Mobile phone surveys for collecting population-level estimates in low- and middle-income countries: a literature review . Journal of Medical Internet Research 19 : e139 . Google Scholar OpenURL Placeholder Text WorldCat Greenleaf AR , Gibson DG, Khattar C, Labrique AB, Pariyo GW. 2017 . Building the evidence base for remote data collection in low- and middle-income countries: comparing reliability and accuracy across survey modalities . Journal of Medical Internet Research 19 : e140 . Google Scholar OpenURL Placeholder Text WorldCat Heckathorn D. 1997 . Respondent-driven sampling: a new approach to the study of hidden populations . Social Problems 44 : 174 –199, www.jstor.org/stable/3096941, accessed 15 November 2020. Google Scholar OpenURL Placeholder Text WorldCat Johnston LG , Sabin K. 2010 . Sampling hard-to-reach populations with respondent driven sampling . Methodological Innovations 5 : 38 – 48 . Google Scholar OpenURL Placeholder Text WorldCat Kirchherr J , Charles K. 2018 . Enhancing the sample diversity of snowball samples: recommendations from a research project on anti-dam movements in Southeast Asia . PLoS One 13 : e0201710 . Google Scholar OpenURL Placeholder Text WorldCat L’Engle K , Sefa E, Adimazoya EA, Yartey E et al. . 2018 . Survey research with a random digit dial national mobile phone sample in Ghana: methods and sample quality . PLoS One 13 : e0190902 . Google Scholar OpenURL Placeholder Text WorldCat Labrique A , Blynn E, Ahmed S et al. . 2017 . Health surveys using mobile phones in developing countries: automated active strata monitoring and other statistical considerations for improving precision and reducing biases . Journal of Medical Internet Research 19 : e121 . Google Scholar OpenURL Placeholder Text WorldCat Lamanna C , Hachhethu K, Chesterman S et al. . 2019 . Strengths and limitations of computer assisted telephone interviews (CATI) for nutrition data collection in rural Kenya . PLoS One 14 : e0210050 . Google Scholar OpenURL Placeholder Text WorldCat Langhaug LF , Sherr L, Cowan FM. 2010 . How to improve the validity of sexual behaviour reporting: systematic review of questionnaire delivery modes in developing countries . Tropical Medicine & International Health 15 : 362 – 81 . Google Scholar OpenURL Placeholder Text WorldCat Lau CQ , Cronberg A, Marks L, Amaya A. 2019 . In search of the optimal mode for mobile phone surveys in developing countries. A comparison of IVR, SMS, and CATI in Nigeria . Survey Research Methods 13 . https://ojs.ub.uni-konstanz.de/srm/article/view/7375, accessed 3 August 2020. Google Scholar OpenURL Placeholder Text WorldCat Leo B , Morrel R, Mellon J, Peixoto T, Davenport S. 2015 . Do mobile phone surveys work in poor countries? Center for Global Development Working Paper 398. Washington, DC. https://www.cgdev.org/publication/do-mobile-phone-surveys-work-poor-countries-working-paper-398, accessed 29 July 2020. Lupton D. 2020 . Doing Fieldwork in a Pandemic: Crowd-Sourced Document. https://docs.google.com/document/d/1clGjGABB2h2qbduTgfqribHmog9B 6P0NvMgVuiHZCl8/edit?ts=5e88ae0a#, accessed 1 July 2020. Mupambireyi Z , Bernays S. 2019 . Reflections on the use of audio diaries to access young people’s lived experiences of HIV in Zimbabwe . Qualitative Health Research 29 : 680 – 92 . Google Scholar OpenURL Placeholder Text WorldCat Peterman A , Bhatia A, Guedes A. 2020 . Remote Data Collection on Violence against Women during COVID-19: A Conversation with Experts on Ethics, Measurement & Research Priorities (Part 1). https://www.unicef-irc.org/article/1997-remote-data-collection-on-violence-against-women-during-covid-19-a-conversation-with.html, accessed 5 August 5 2020. Roy D , Tripathy S, Kar SK, Sharma N et al. . 2020 . Study of knowledge, attitude, anxiety & perceived mental healthcare need in Indian population during COVID-19 pandemic . Asian Journal of Psychiatry 51 : 102083 . doi: 10.1016/j.ajp.2020.102083 . Google Scholar OpenURL Placeholder Text WorldCat Crossref Shaghaghi A , Bhopal RS, Sheikh A. 2011 . Approaches to recruiting “hard-to-reach” populations into re-search: a review of the literature . Heal Promotion Perspectives 1 : 86 – 94 . Google Scholar OpenURL Placeholder Text WorldCat The Lancet Public Health . 2020 . COVID-19 puts societies to the test . The Lancet Public Health 5 : e235 . OpenURL Placeholder Text WorldCat Tran MC , Labrique AB, Mehra S et al. . 2015 . Analyzing the mobile “digital divide”: changing determinants of household phone ownership over time in rural Bangladesh . JMIR mHealth and uHealth 3 : e24 . Google Scholar OpenURL Placeholder Text WorldCat Author notes † B Hensen and CRS Mackworth-Young are joint first authors. ‡ C Bonell and H A Weiss are joint senior authors. © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine. All rights reserved. For permissions, please e-mail: [email protected] This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model) © The Author(s) 2021. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine. All rights reserved. For permissions, please e-mail: [email protected]