Effect of Physician-Delivered COVID-19 Public Health Messages and Messages Acknowledging Racial Inequity on Black and White Adults’ Knowledge, Beliefs, and Practices Related to COVID-19

Carlos Torres; Lucy Ogbu-Nwobodo; Marcella Alsan; Fatima Cody Stanford; Abhijit Banerjee; Emily Breza; Arun G. Chandrasekhar; Sarah Eichmeyer; Mohit Karnani; Tristan Loisel; Paul Goldsmith-Pinkham; Benjamin A. Olken; Pierre-Luc Vautrey; Erica Warner; Esther Duflo

doi:10.1001/jamanetworkopen.2021.17115

Effect of Physician-Delivered COVID-19 Public Health Messages and Messages Acknowledging Racial Inequity on Black and White Adults’ Knowledge, Beliefs, and Practices Related to COVID-19

Torres, Carlos; Ogbu-Nwobodo, Lucy; Alsan, Marcella; Stanford, Fatima Cody; Banerjee, Abhijit; Breza, Emily; Chandrasekhar, Arun G.; Eichmeyer, Sarah; Karnani, Mohit; Loisel, Tristan; Goldsmith-Pinkham, Paul; Olken, Benjamin A.; Vautrey, Pierre-Luc; Warner, Erica; Duflo, Esther 2021-07-14 00:00:00 Key Points Question Do messages delivered by IMPORTANCE Social distancing is critical to the control of COVID-19, which has disproportionately physicians increase COVID-19 affected the Black community. Physician-delivered messages may increase adherence to these knowledge and improve preventive behaviors. behaviors among White and Black individuals? OBJECTIVES To determine whether messages delivered by physicians improve COVID-19 Findings In this randomized clinical trial knowledge and preventive behaviors and to assess the differential effectiveness of messages of 18 223 White and Black adults, a tailored to the Black community. message delivered by a physician increased COVID-19 knowledge and DESIGN, SETTING, AND PARTICIPANTS This randomized clinical trial of self-identified White and shifted information-seeking and self- Black adults with less than a college education was conducted from August 7 to September 6, 2020. protective behaviors. Effects did not Of 44 743 volunteers screened, 30 174 were eligible, 5534 did not consent or failed attention checks, differ by race, and tailoring messages to and 4163 left the survey before randomization. The final sample had 20 460 individuals specific communities did not exhibit a (participation rate, 68%). Participants were randomly assigned to receive video messages on differential effect on knowledge or COVID-19 or other health topics. individual behavior. INTERVENTIONS Participants saw video messages delivered either by a Black or a White study Meaning These findings suggest that physician. In the control groups, participants saw 3 placebo videos with generic health topics. In the physician messaging campaigns may be treatment group, they saw 3 videos on COVID-19, recorded by several physicians of varied age, effective in persuading members of gender, and race. Video 1 discussed common symptoms. Video 2 highlighted case numbers; in one society from a broad range of group, the unequal burden of the disease by race was discussed. Video 3 described US Centers for backgrounds to seek information and Disease Control and Prevention social distancing guidelines. Participants in both the control and adopt preventive behaviors to combat intervention groups were also randomly assigned to see 1 of 2 American Medical Association COVID-19. statements, one on structural racism and the other on drug price transparency. Invited Commentary MAIN OUTCOMES AND MEASURES Knowledge, beliefs, and practices related to COVID-19, demand for information, willingness to pay for masks, and self-reported behavior. Supplemental content Author affiliations and article information are RESULTS Overall, 18 223 participants (9168 Black; 9055 White) completed the survey (9980 listed at the end of this article. [55.9%] women, mean [SD] age, 40.2 [17.8] years). Overall, 6303 Black participants (34.6%) and 7842 White participants (43.0%) were assigned to the intervention group, and 1576 Black participants (8.6%) and 1968 White participants (10.8%) were assigned to the control group. Compared with the control group, the intervention group had smaller gaps in COVID-19 knowledge (incidence rate ratio [IRR], 0.89 [95% CI, 0.87-0.91]) and greater demand for COVID-19 information (IRR, 1.05 [95% CI, 1.01-1.11]), willingness to pay for a mask (difference, $0.50 [95% CI, $0.15-$0.85]). (continued) Open Access. This is an open access article distributed under the terms of the CC-BY License. JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 1/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices Abstract (continued) Self-reported safety behavior improved, although the difference was not statistically significant (IRR, 0.96 [95% CI, 0.92-1.01]; P = .08). Effects did not differ by race (F = 0.0112; P > .99) or in different intervention groups (F = 0.324; P > .99). CONCLUSIONS AND RELEVANCE In this study, a physician messaging campaign was effective in increasing COVID-19 knowledge, information-seeking, and self-reported protective behaviors among diverse groups. Studies implemented at scale are needed to confirm clinical importance. TRIAL REGISTRATION ClinicalTrials.gov Identifier: NCT04502056 JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 Introduction Physical distancing and mask wearing remain essential to the control of COVID-19, yet vigilance has decreased over time. To address fatigue, health care professionals have used social media to spread public health messages. There is evidence that these messages improve knowledge, but there are less data on whether they change behavior. Black US residents have been disproportionately affected by the pandemic. This reflects the cumulative impact of systemic racism, acknowledged as a public health threat by the American Medical Association (AMA) in a June 2020 statement. This raises the question on whether the effectiveness of public health messages regarding COVID-19 would be enhanced if tailored to the Black community. The focus of this study was to identify whether messages delivered by physicians increase COVID-19 knowledge and improve preventive behaviors for White and Black individuals and to assess whether various ways of increasing the relevance of messages to the Black community (ie, physician race, AMA acknowledgments of racial injustices, or information about the disproportionate burden of COVID-19 on the Black community) affects their impact on both White and Black participants. Methods Trial Design and Oversight The trial flowchart (Figure 1; eFigure 1 in Supplement 1) describes the factorial design and the allocation of participants to each intervention arm. The design was approved by the ethical review boards of Massachusetts Institute of Technology (MIT) and Stanford, with Massachusetts General Hospital, Yale, and Harvard ceding authority to MIT. All participants provided written informed consent. The trial and the outcomes were registered on ClinicalTrials.gov (NCT04502056). Planned analyses were published on the American Economic Association trial registry (AEARCTR-0006177). The pre-analysis plan and institutional review board–approved protocol are available in Supplement 2. This study followed the Consolidated Standards of Reporting Trials (CONSORT) and American Association for Public Research (AAPOR) reporting guidelines. Participants Individuals were recruited online throughout the United States by the survey company Lucid from August 7, 2020, to September 6, 2020. Lucid recruits survey participants by advertising surveys to third-party suppliers, including double opt-in panels, publishing networks, social media, and other types of online communities. Participants aged 18 years or older, self-identifying as White or Black, and without a college degree were eligible. We focused on these 2 groups because we were interested in tailoring messages toward the Black community as well as the reaction of the White JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 2/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices community to the tailoring of those messages. Latinx individuals were included in our previous study, along with specific tailoring toward this community. Recruitment used quotas to match the 2018 population estimates by age, sex, and race issued by the US Census Bureau. Interventions Intervention vs Control Messages After reading the informed consent form (approved by the institutional review board) (eAppendix 1 in Supplement 1) and agreeing to participate, all individuals answered sociodemographic questions, saw 3 videos, and then completed the outcome survey questions. In the control groups, participants saw 3 placebo videos with generic health topics, including fitness guidelines, recommended sugar intake, and the importance of adequate sleep. In the treatment group, they saw 3 videos on COVID-19, recorded by several physicians of varied age, gender, and race. Participants in each group (placebo and intervention) saw video messages delivered either by a Black or a White study physician (including L.O.-N., M.A., F.C.S., and E.W.). Figure 1. Enrollment and Randomization of Participants 44 473 Assessed for eligibility 24 283 Excluded 14 569 After quotas were met 5534 Declined to participate or failed basic attention checks 4163 Exited survey before randomization 17 Missing race data from platform 20 460 Randomized 16 366 Assigned to intervention group 4094 Assigned to control group 8181 Received statement acknowledging 8185 Received placebo message 2051 Received statement 2051 Received placebo message systemic racism acknowledging systemic racism 4096 Assigned to Black physician, 1023 Assigned to Black physician 4090 Assigned to Black physician, with 2044 receiving message 1026 Assigned to Black physician 1020 Assigned to White physician with 2046 receiving message acknowledging increased 1025 Assigned to White physician acknowledging increased mortality for Black individuals mortality for Black individuals 4089 Assigned to White physician, 4091 Assigned to White physician, with 2046 receiving message with 2045 receiving message acknowledging increased acknowledging increased mortality for Black individuals mortality for Black individuals 676 Exited 691 Exited 162 Exited 169 Exited 7505 Included in knowledge 7494 Included in knowledge 1889 Included in knowledge 1874 Included in knowledge gaps analysis gaps analysis gaps analysis gaps analysis 213 Exited 217 Exited 63 Exited 46 Exited 7292 Included in full analysis 7277 Included in full analysis 1826 Included in full analysis 1828 Included in knowledge gaps and/or full analysis 5792 Excluded from 5755 Excluded from 1442 Excluded from 1441 Excluded from follow-up follow-up follow-up follow-up 3145 Not eligible 3132 Not eligible 800 Not eligible 792 Not eligible 2629 Did not 2610 Did not 637 Did not 644 Did not complete complete complete complete survey survey survey survey 18 Missing 13 Missing 5 Missing 5 Missing baseline baseline baseline baseline 2389 Included in follow-up analysis 2430 Included in follow-up analysis 609 Included in follow-up analysis 602 Included in follow-up analysis JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 3/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices Video 1 defined COVID-19 and discussed common symptoms associated with COVID-19 as well as asymptomatic transmission. Video 2 reminded the viewer that COVID-19 was actively circulating in the United States. Video 3 described the Centers for Disease Control and Prevention social distancing guidelines (complete scripts appear in eAppendix 1 in Supplement 1). Unequal Burden of COVID-19 Video 2 in the COVID-19 intervention had 2 randomized variants. Script 1 emphasized the number of new cases in the week of July 6, 2020. Script 2 added that, controlling for age, Black individuals were 3 times as likely to become infected as White individuals and 4 times as likely to die from it. These 2 variants of video 2 were cross-randomized with the intervention. AMA Antiracism or Placebo Statement At the beginning of the study, all participants saw a video of an actor delivering the script of a public statement by the AMA. The AMA antiracism statement, issued on June 7, 2020, “recognizes that racism in its systemic, structural, institutional, and interpersonal forms is an urgent threat to public health, the advancement of health equity, and a barrier to excellence in the delivery of medical care.” The AMA placebo intervention was an AMA statement on drug pricing. The race and gender of the person reading the statement were randomized to each recipient. Outcomes Most outcomes were measured online immediately following the intervention or the placebo. The prespecified primary outcomes were knowledge, beliefs, and practices related to COVID-19, measured immediately after the intervention; intended behavior, measured immediately after the intervention; and knowledge and behavior, measured a few days after the intervention. eAppendix 2 in Supplement 1 describes the outcome measurement in detail. Primary outcomes presented in the main text include 5 outcomes. First, knowledge gap, which measures knowledge and beliefs. Participants were asked to identify ways to prevent COVID-19 spread and identify 4 common symptoms. The knowledge gap outcome is an integer that can have values from 0 (no error) to 10 (10 errors). Second, information seeking was measured by offering participants the option of requesting additional information on COVID-19–related resources by clicking on up to 5 links that included more content. We measured information-seeking behavior as the number of links in which participants expressed interest, a count variable between 0 (lowest information-seeking behavior) and 5 (greatest information-seeking behavior). Third, self-reported safety behavior was measured a few days after the initial intervention among a subsample that was eligible for follow-up and could be tracked. Participants were asked about how often they engaged in 4 behaviors of interest: (1) whether they wore a mask indoors; (2) whether they wore a mask outdoors; (3) whether they washed their hands; and (4) whether they followed social distancing guidelines. The safety gap index had values of 0 (if a participant reported that they always practiced the 4 behaviors of interest) to 4 (participant reported that they practiced none of the behaviors). Fourth, at the end of the survey, each participant was asked the price they would be willing to pay for high-quality masks. Each participant was entered into a draw to receive either a coupon for masks or a gift card to an online retailer. When a participant was selected by the draw, a price was then randomly drawn for the coupon. If their reported willingness to pay was greater than the price, they would receive the masks; otherwise, they would receive the gift card. Therefore, it was in participants’ best interest to report their true willingness to pay for the masks. This type of procedure has been shown to lead to truthful reporting. We collected data on 3 secondary outcomes specified in our pre-analysis plan (Supplement 2). First, we asked participants to report their judgment of how well federal and state policies balanced opening the economy and limiting the health impacts of COVID-19. Second, we measured how participants prioritized COVID-19 protection vs other issues by asking the participants how they would want to allocate a donation of $1000 (which the research team would fund) between 2 JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 4/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices charities: Give a Mask or the Alzheimer Association. Third, we asked whether they would prioritize a COVID-19 relief donation to a Black-focused charity (the BET COVID-19 relief fund) or a general charity (the Give Directly Project 100+ relief program). Randomization Randomization into intervention vs control and between interventions was stratified according to sex, age (45 years), race, and self-identified partisan affiliation (Republican vs other affiliation). The flowchart (Figure 1) summarizes the randomization; a more detailed version is available as eFigure 1 in Supplement 1. Participants were first randomized into the AMA antiracism or placebo statements, with equal probability. They were then randomly assigned to intervention or control. One of 5 participants was assigned to control; the remainder were assigned to an intervention. Within each group, participants were randomized into Black or White physician groups with equal probability. Intervention participants were further randomized, with equal probability, into 1 the 2 arms for video 2: they either received the information about the unequal burden of disease or did not. Randomization was performed using the Qualtrics platform, using a randomizer block within each stratum with the option to evenly present elements. Statistical Analysis We determined that a sample of 20 000 individuals (10 000 Black and 10 000 White) would provide 85% power to detect effect sizes of 0.05 SDs for intervention relative to control and for effects of specific variations in message content. These are small effect sizes that would justify scale up of this inexpensive intervention. The analysis was performed by original assigned group, and it included all participants who completed the survey. Multivariable regression models include the stratifying variables (age × sex × race × Republican identity dummies). Effect of Any Video Message Intervention Relative to Control To analyze the effect of seeing any video message on the knowledge gap, information-seeking behavior, and safety behavior outcomes, we fit the following negative binomial regression model for the count outcome: log(μ)=β +β intervention +β stratum , i 0 1 i 2 i where μ is the estimated mean outcome value (knowledge gap count, count of demanded links, or safety behavior count), intervention is an indicator that equals 1 if the individual received the intervention videos and 0 if they received the placebo videos, and stratum is a vector of indicator variables. Similar models are estimated for binary regressions (using the logistic regression equation) and continuous variable (using ordinary least squares) (eAppendix 2 in Supplement 1). Because there were multiple outcomes, we provide P values and q values adjusted for false discovery rates. Effect of Variation in the Message Framing To analyze the impact of different arms, we fit a negative binomial regression model to the count data: log(μ)=β +β Black physician +β AMA antiracism +β intervention +(β Black i 0 1 i 2 i 3 i 4 physician × intervention)+(β AMA antracism × intervention)+(β mortality i i 5 i i 6 difference × intervention)+β stratum , i i 7 i where Black physician is an indicator that equals 1 if the physician was a Black individual and 0 otherwise; AMA antiracism is an indicator for the AMA statement featuring the antiracism message (rather than the drug pricing message); mortality difference is an indicator for video 2 mentioning JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 5/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices the excess mortality from COVID-19. Similar specifications are fitted with logistic regression for binary variables and ordinary least squares for continuous variables. To address possible bias stemming from nonrandom attrition for the follow-up survey, we weighted the follow-up data using Hainmueller entropy weights, which ensures that the observed baseline characteristics of the follow-up sample matches the original sample as closely as possible (eAppendix 2 in Supplement 1). eAppendix 3 in Supplement 1 describes further robustness checks and subgroups analysis. Analyses were performed using R version 4.0.3 (R Project for Statistical Computing). Statistical significance was set at P < .05, and all tests were 2-tailed. Results Trial Sample The trial sample was enrolled from August 7 through September 6, 2020. Of 44 743 screened, 30 174 were eligible for participation, 5534 individuals did not consent or failed 2 simple attention checks, 4163 left the survey before randomization, and 17 were excluded from the analysis due to unknown race or multiple survey completion. The final sample at randomization had 20 460 individuals, for a participation rate of 68%. After attrition, 18 223 individuals were included in the study. Summary statistics (Table 1) were computed on the sample that was randomized and that Table 1. Summary of Participant Characteristics Respondents, No. (%) Full sample Intervention group Control group All Black White All Black White All Black White Variable (N = 17 689) (n = 7879) (n = 9810) (n = 14 145) (n = 6303) (n = 7842) (n = 3544) (n = 1576) (n = 1968) Age, mean (SD), y 40.22 (17.81) 34.12 (15.48) 45.12 (18.04) 40.20 (17.83) 34.15 (15.48) 45.07 (18.09) 40.30 (17.73) 34.00 (15.47) 45.35 (17.81) Region Northeast 3024 (17.1) 1187 (15.1) 1837 (18.7) 2424 (17.1) 952 (15.1) 1472 (18.8) 600 (16.9) 235 (14.9) 365 (18.5) Midwest 3884 (22.0) 1494 (19.0) 2390 (24.4) 3114 (22.0) 1207 (19.1) 1907 (24.3) 770 (21.7) 287 (18.2) 483 (24.5) South 8046 (45.5) 4291 (54.5) 3755 (38.3) 6397 (45.2) 3406 (54.0) 2991 (38.1) 1649 (46.5) 885 (56.2) 764 (38.8) West 2735 (15.5) 907 (11.5) 1828 (18.6) 2210 (15.6) 738 (11.7) 1472 (18.8) 525 (14.8) 169 (10.7) 356 (18.1) Demographic characteristics High school 15 016 (84.9) 6125 (77.7) 8891 (90.6) 12 009 (84.9) 4899 (77.7) 7110 (90.7) 3007 (84.8) 1226 (77.8) 1781 (90.5) graduate Household income 4206 (23.8) 1657 (21.0) 2549 (26.0) 3356 (23.7) 1327 (21.1) 2029 (25.9) 850 (24.0) 330 (20.9) 520 (26.4) >$60 000 Female 9880 (55.9) 4492 (57.0) 5388 (54.9) 7907 (55.9) 3595 (57.0) 4312 (55.0) 1973 (55.7) 897 (56.9) 1076 (54.7) Male 7809 (44.1) 3387 (43.0) 4422 (45.1) 6238 (44.1) 2708 (43.0) 3530 (45.0) 1571 (44.3) 679 (43.1) 892 (45.3) Party Democrat 6977 (39.4) 4228 (53.7) 2749 (28.0) 5594 (39.5) 3385 (53.7) 2209 (28.2) 1383 (39.0) 843 (53.5) 540 (27.4) Republican 4376 (24.7) 699 (8.9) 3677 (37.5) 3494 (24.7) 553 (8.8) 2941 (37.5) 882 (24.9) 146 (9.3) 736 (37.4) Independent 6336 (35.8) 2952 (37.5) 3384 (34.5) 5057 (35.8) 2365 (37.5) 2692 (34.3) 1279 (36.1) 587 (37.2) 692 (35.2) Preventive practices Mask in (always) 12 106 (68.4) 5316 (67.5) 6790 (69.2) 9648 (68.2) 4230 (67.1) 5418 (69.1) 2458 (69.4) 1086 (68.9) 1372 (69.7) Mask out (always) 5517 (31.2) 3513 (44.6) 2004 (20.4) 4408 (31.2) 2807 (44.5) 1601 (20.4) 1109 (31.3) 706 (44.8) 403 (20.5) Wash hands 10 779 (60.9) 5084 (64.5) 5695 (58.1) 8688 (61.4) 4090 (64.9) 4598 (58.6) 2091 (59.0) 994 (63.1) 1097 (55.7) (always) Distance (always) 9461 (53.5) 4587 (58.2) 4874 (49.7) 7571 (53.5) 3681 (58.4) 3890 (49.6) 1890 (53.3) 906 (57.5) 984 (50.0) This table presents summary statistics on baseline variables for our main sample of (always) is equal to 1 if the respondent answered “always” to “Wearing a mask outside,” individuals who completed all baseline variables. otherwise it is 0. Wash hands (always) is equal to 1 if the respondent answered “always” b to “Washing your hands with soap and water right away when you come home after The preventive practices variables refer to the question: “What fraction of the time going out.” Distance (always) is equal to 1 if the respondent answered “always” to would you say that you engage in the following behaviors?” Mask in (always) is equal to “Staying at least 6 feet away from people who are not part of your household.” 1 if the respondent answered “always” to “Wearing a mask when you go inside buildings that are not your home / take public transportation,” otherwise it is 0. Mask out JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 6/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices completed all main baseline variables. Our sample included 9880 (55.9%) women. The mean (SD) age was 40.2 (17.8) years, 4206 (23.8%) reported household incomes greater than $60 000, 4228 Black participants (53.7%) and 2749 White participants (28.0%) identified as members of the Democratic party, 12 106 (68.4%) reported always wearing a mask indoors (outside the home), and 5517 (31.2%) reported always wearing a mask when outdoors. Black participants were twice as likely as White participants to report always wearing a mask outdoors (3513 [44.6%] vs 2004 [20.4%]) and were also more likely to report practicing hand hygiene (5084 [64.5%] vs 5695 [58.1%]) and physical distancing (4587 [58.2%] vs 4874 [49.7%]). Baseline covariates were balanced between intervention and control at all stages (eAppendix 5 and eTable 1A in Supplement 1). Attrition Due to the online nature of the survey, participants could exit at any point after watching the video messages without finishing the survey. To include as many participants as possible in the analysis, we included everyone who answered knowledge questions for the knowledge outcome. Overall, 18 762 participants included in the initial randomization were included at that stage (9445 Black; 9317 White). For other outcomes, except follow-up outcomes, we included 18 223 participants (9168 Black; 9055 White) who completed the survey. Attrition was similar in all groups (eTable 1B in Supplement 1). The adherence to safety behavior was collected a few days later among a smaller follow-up sample that experienced more attrition. Overall, 12 591 individuals were included in the follow-up sample to track; only people who had given permission to Lucid to be recontacted were included in this sample. Of those, we successfully contacted 6217. Attrition at this stage was 51.8% in the treatment group and 51.0% in the control group. A systematic analysis of attrition at all 3 stages (eTable 1B in Supplement 1) revealed no systematic difference between the characteristic of attritors in the treatment and comparison groups. At the realized sample size, the power was 0.055 SDs for the knowledge sample (18 762 respondents), 0.056 SDs for all other outcomes in the first survey, and 0.095 SDs for the follow up sample (6217 respondents). Effects of Any Video Message, Intervention vs Control Receiving any COVID-19 video improved knowledge of COVID-19 and adherence to preventive practices. Table 2 shows main outcomes overall and by racial group. Main Sample The knowledge gap incidence rate was 0.241 (95% CI, 0.235-0.246) in the control group and 0.214 (95% CI, 0.211-0.217) in intervention group. The intervention had a significant effect on reducing knowledge gaps relative to the control group (estimated incidence rate ratio [IRR], 0.89 [95% CI, 0.87-0.91]; P < .001; q < .001). In the control group, 315 participants (8.4%) had no gap in knowledge (Figure 2). The proportion increased to 13.0% (1915 participants) in the intervention group; 1352 participants (35.9%) in the control group had a 1-point knowledge gap compared with 6636 (44.2%) in the intervention group. The incidence rate of information seeking behavior was 0.320 in the control group and 0.338 in the intervention group (estimated IRR, 1.05 [95% CI, 1.01-1.11]; P =.03; q = .04). The willingness to pay for a mask increased from $14.07 in the control group to $14.58 in the intervention group (difference, $0.50 [95% CI, $0.15-$0.85]; P = .005; q = .013). The intervention was impactful for both Black and White participants. It was more impactful for White participants vs Black participants on knowledge (IRR, 0.80 [95% CI, 0.76-0.83] vs 0.94 [95% CI 0.91-0.97]; P for difference < .001), but equally impactful for all the other measures (eTable 2C in Supplement 1). The F statistics for the hypothesis that the coefficients across all outcomes are jointly different for both races was 0.0112 (P > .99). JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 7/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 8/13 Table 2. Impact of Any Video Message Intervention vs Control: Incidence Rate Control group Intervention group Mean incidence rate Mean incidence rate Panel Outcome (95% CI) Observations, No. (95% CI) Observations, No. IRR (CI 95%) P value q value Observations, No. All participants Knowledge gap score 0.241 (0.235 to 0.246) 3763 0.214 (0.211 to 0.217) 14 999 0.89 (0.87 to 0.91) <.001 <.001 18 762 Information seeking 0.32 (0.31 to 0.33) 3654 0.338 (0.332 to 0.344) 14 569 1.05 (1.01 to 1.11) .03 .04 18 223 behavior Safety gap score 0.47 (0.45 to 0.48) 1212 0.45 (0.44 to 0.46) 4823 0.96 (0.92 to 1.01) .08 .08 6035 Knowledge gap follow-up 0.25 (0.24 to 0.26) 1211 0.241 (0.238 to 0.244) 4819 0.956 (0.917 to 0.996) .03 .04 6030 Black participants Knowledge gap score 0.316 (0.307 to 0.324) 1892 0.297 (0.292 to 0.301) 7553 0.94 (0.91 to 0.97) <.001 <.001 9445 Information seeking 0.38 (0.36 to 0.40) 1840 0.40 (0.39 to 0.41) 7328 1.06 (1.00 to 1.12) .06 .15 9168 behavior Safety gap score 0.40 (0.36 to 0.43) 416 0.38 (0.36 to 0.40) 1683 0.95 (0.87 to 1.04) .26 .28 2099 Knowledge gap follow-up 0.27 (0.25 to 0.28) 416 0.258 (0.253 to 0.263) 1681 0.97 (0.91 to 1.03) .28 .28 2097 White participants Knowledge gap score 0.165 (0.159 to 0.170) 1871 0.131 (0.128 to 0.133) 7446 0.80 (0.76 to 0.83) <.001 <.001 9317 Information seeking 0.26 (0.25 to 0.28) 1814 0.275 (0.267 to 0.282) 7241 1.05 (0.97 to 1.13) .22 .22 9055 behavior Safety gap score 0.50 (0.48 to 0.53) 796 0.49 (0.48 to 0.50) 3140 0.96 (0.91 to 1.02) .20 .22 3936 Knowledge gap follow-up 0.24 (0.23 to 0.25) 795 0.231 (0.228 to 0.235) 3138 0.95 (0.89 to 1.00) .05 .09 3933 Panel Outcome Mean (95% CI), $ Observations, No. Mean (95% CI), $ Observations, No. Coefficient (95% CI) P value q Value Observations, No. All WTP masks 14.07 (13.76 to 14.38) 3360 14.58 (14.42 to 14.74) 13 399 0.50 (0.15 to 0.85) .005 .01 16 759 Black WTP masks 15.70 (15.22 to 16.18) 1550 16.13 (15.87 to 16.38) 6175 0.42 (−0.14 to 0.97) .14 .24 7725 White WTP masks 12.68 (12.29 to 13.07) 1810 13.26 (13.06 to 13.46) 7224 0.57 (0.12 to 1.01) .01 .01 9034 Abbreviations: IRR, incidence rate ratio; WTP, willingness to pay. fitting a negative binomial regression model with units weighted following Hainmueller entropy-based a weighting to account for imbalances due to attrition for the follow-up outcomes. q values are reported Incidence rate for knowledge gaps is the count of knowledge gaps divided by the maximum possible count (10). accounting for the different outcomes and coefficients in each panel. Incidence rate for interest in links is the count of links demanded divided by the maximum possible count (5). Incidence rate for safety gaps is the count of safety gaps divided by the maximum possible count (4). IRR (or The F statistic for a test equality of the coefficients for the Black participants and White participants (obtained by coefficients) compare the any intervention with the control group. IRRs for safety gap score were estimated by estimating all outcomes in a joint system) was 0.0112 (P > .99). JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices Follow-up Survey At the follow-up survey (Table 2), which was realized on a smaller sample with larger attrition, the safety gap index incidence rate was 0.47 (95% CI, 0.45-0.48) in the control group and 0.45 (95% CI, 0.44-0.46) in the treatment group (IRR, 0.96 [95% CI 0.92-1.01]; P = .08, q = .08). Overall, 244 participants (20.1%) and 218 participants (18.0%) in the control group and 1040 participants (21.6%) and 837 participants (17.4%) in the intervention group reported respecting all and none, respectively, of 4 safety practices (eFigure 2 in Supplement 1). Supplemental Outcomes eTable 3 in Supplement 1 reports effects on secondary outcomes. One noteworthy result is that we found significant effects of the intervention on donations both to a COVID-19 charity (vs Alzheimer) and to a COVID-19 relief charity specific to Black US residents (vs COVID-19 economic relief for everyone). Effects of the Framing of the Videos None of the efforts to make the messages more relevant to the Black community had a differential effect on knowledge or individual behavior (Table 3). An F statistic that all effects for Black physician, AMA antiracism statement, and disproportionate burden statement are jointly different from 0 was 0.324 (P > .99). The only outcome that was affected by some of these variants was the desired donation to a Black-specific COVID-19 charity (eTable 4 in Supplement 1). eTable 5 in Supplement 1 shows that the combination of the AMA antiracism statement, a Black physician, and a video acknowledging racial disparities significantly increased how White and Black participants allocated donations to a charity focused on Black communities (ordinary least squares coefficient, $30.60 [95% CI, $10.93-$50.27]; P = .002). Heterogeneity by Sex, Education, Income, and Political Affiliation Across all conditions, there were no statistically significant differences by sex or political affiliation (eTable 2A and eTable 2E in Supplement 1). The effect of intervention relative to control on knowledge gaps was more pronounced for participants with a high school education or more (eTable 2B in Supplement 1). However, there was no significant difference for the demand for links and willingness to pay for a mask. The intervention was more impactful among participants with lower incomes (ie, <$60 000). Figure 2. Distribution of the Knowledge Gap Score in the Control and Intervention Groups 0.5 Group 0.4 Control Intervention 0.3 0.2 0.1 0 1 2 3 4 5 ≥6 Knowledge gap score Whiskers indicate 95% CIs. JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 9/13 Share JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 10/13 Table 3. Impact of Tailoring Messages: Incidence Rate Ratios Intervention group Control group Black physician AMA antiracism Racial disparity Black physician AMA antiracism Intervention vs control Observations, Panel Outcome IRR (95% CI) P value IRR (95% CI) P value IRR (95% CI) P value IRR (95% CI) P value IRR (95% CI) P value IRR (95% CI) P value No. All participants Knowledge gap 1.01 (0.96 to .66 0.99 (0.94 to .71 1.01 (0.98 to .66 0.99 (0.95 to .78 0.99 (0.95 to .65 0.89 (0.85 to <.001 18 762 score 1.06) 1.04) 1.03) 1.04) 1.03) 0.93) Information 1.04 (0.95 to .42 1.03 (0.93 to .60 1.02 (0.98 to .30 0.96 (0.89 to .38 0.97 (0.89 to .42 1.01 (0.93 to .83 18 223 seeking 1.14) 1.13) 1.07) 1.05) 1.05) 1.10) Safety gap score 1.04 (0.95 to .44 1.03 (0.93 to .74 1.01 (0.97 to .73 0.97 (0.89 to .44 0.95 (0.87 to .20 0.93 (0.86 to .09 6035 1.14) 1.12) 1.05) 1.05) 1.03) 1.01) Knowledge gap 0.98 (0.90 to .65 0.98 (0.90 to .63 1.01 (0.98 to .47 1.01 (0.94 to .72 0.98 (0.91 to .66 0.97 (0.90 to .38 6030 follow up 1.07) 1.07) 1.05) 1.09) 1.06) 1.04) Black Knowledge gap 1.01 (0.94 to .89 1.01 (0.95 to .84 1.001 (0.98 to .55 1.00 (0.94 to .89 0.98 (0.93 to .48 0.93 (0.88 to .009 9445 participants score 1.07) 1.07) 1.04) 1.05) 1.04) 0.98) Information 1.04 (0.92 to .58 1.02 (0.91 to .76 1.02 (0.97 to .45 0.97 (0.84 to .60 0.98 (0.88 to .75 1.02 (0.92 to .71 9168 seeking 1.17) 1.15) 1.08) 1.08) 1.09) 1.14) Safety gap score 1.01 (0.85 to .93 1.03 (0.87 to .72 1.12 (1.04 to .005 0.98 (0.84 to .79 0.87 (0.74 to .08 0.88 (0.76 to .10 2099 1.20) 1.23) 1.21) 1.15) 1.02) 1.02) Knowledge gap 0.97 (0.86 to .66 0.99 (0.87 to .81 1.05 (0.99 to .08 1.02 (0.91 to .78 0.97 (0.87 to .63 0.96 (0.87 to .49 2097 follow up 1.10) 1.11) 1.11) 1.13) 1.09) 1.07) White Knowledge gap 1.03 (0.94 to .55 0.96 (0.88 to .32 1.00 (0.96 to .87 0.99 (0.92 to .74 1.01 (0.94 to .78 0.80 (0.75 to <.001 9317 participants score 1.12) 1.04) 1.04) 1.06) 1.09) 0.87) Information 1.05 (0.90 to .56 1.03 (0.89 to .68 1.03 (0.96 to .46 0.95 (0.83 to .48 0.95 (0.83 to .43 1.00 (0.87 to .96 9055 seeking 1.22) 1.20) 1.10) 1.09) 1.08) 1.14) Safety gap score 1.06 (0.94 to .35 1.01 (0.90 to .89 0.94 (0.89 to .02 0.96 (0.87 to .44 1,00 (0.90 to .93 0.96 (0.87 to .47 3936 1.19) 1.13) 0.99) 1.07) 1.11) 1.07) Knowledge gap 0.99 (0.88 to .81 0.98 (0.87 to .68 0.98 (0.93 to .50 1.01 (0.92 to .81 0.99 (0.90 to .87 0.97 (0.88 to .58 3933 follow up 1.11) 1.09) 1.04) 1.12) 1.10) 1.08) Panel Outcome Coefficient P value Coefficient P value Coefficient P value Coefficient P value Coefficient P value Coefficient P value Observations, (95% CI) (95% CI) (95% CI) (95% CI) (95% CI) (95% CI) No. All WTP masks −0.21 (−0.91 to .56 −0.29 (−0.99 to .42 0.07 (−0.24 to .65 0.17 (−0.45 to .59 0.32 (−0.30 to .31 0.71 (0.09 to .03 16 759 0.49) 0.41) 0.39) 0.80) 0.95) 1.34) Black WTP masks 0.04 (−1.08 to .95 −0.09 (−1.21 to .87 0.001 (−0.498 to >.99 −0.01 (−1.01 to .98 0.15 (−0.84 to .76 0.45 (−0.56 to .38 7725 1.15) 1.02) 0.501) 0.98) 1.15) 1.45) White WTP masks −0.43 (−1.31 to .35 −0.46 (−1.35 to .30 0.13 (−0.26 to .51 0.33 (−0.46 to .41 0.47 (−0.32 to .24 0.94 (0.15 to .02 9034 0.46) 0.42) 0.53) 1.12) 1.26) 1.74) Abbreviations: IRR, incidence rate ratio; WTP, willingness to pay. regression for WTP masks) following the second equation in the main text. IRRs for follow-up outcomes were a calculated from estimates obtained by fitting a negative binomial regression model with units reweighted The test statistics for the hypothesis that all the interaction coefficients are jointly 0 across equations was 0.324 following Hainmueller entropy-based reweighting to account for imbalances due to attrition. (P > .99). Estimates in each row came from a single negative binomial regression (or ordinary least squares JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices Discussion Exposure to public health video messages about COVID-19 recorded by a diverse set of physicians decreased knowledge gaps on COVID-19 symptoms, preventive behaviors, and transmission among Black and White participants with modest incomes, relative to a control condition that saw placebo videos. The effect on knowledge was substantial and clear. This replicates the results of our prior study conducted in May 2020 and extends it to White participants. New to this study, we also found a modest but statistically significant increase in the demand for more information, the willingness to pay for high quality masks, and self-reported behavior at follow-up. Despite the heightened awareness of racial justice issues during the period of this intervention and increased polarization in the political discourse in the run-up to the presidential election, effects are remarkably similar across racial and political lines. These results suggest that physicians still have the ability to inform and persuade members of society from a broad range of backgrounds. Our results also indicate that tailoring the message to specific communities did not affect its impact on behavior. Both White and Black physicians were able to effectively convey the importance of masking and social distancing to Black and White participants (unlike the previous study, in which concordance was important to change behavior ). The AMA antiracism statement did not affect participants’ attentiveness to the message delivered, for Black or White respondents. Acknowledgment of structural racism remains important, but it may not be sufficient to increase the level of trust from the Black community. Importantly, the intervention made both Black and White participants more willing to focus resources both toward COVID-19 in general and toward the Black community in particular. Highlighting health conditions that disproportionately affect the Black community is one step toward increasing public consciousness of structural racism. Limitations There are several limitations of the study. First, it was conducted online, and the participants may not be representative of the population with less than a college degree, given that they have access to the internet and are used to participating in online studies. Second, although information-seeking behavior and willingness to pay for masks were objectively measured, participants’ preventive health behaviors were not directly observed. Outcomes were self-reported. Third, outcomes might be subject to social desirability bias. Fourth, there may be bias due to attrition, particularly for the self- reported safety behavior, given that only a small fraction of the sample could be followed up a few days after the initial intervention. While the observable variables remain balanced, the unobservable may not be. Furthermore, while we found consistent effects on knowledge, information seeking, the willingness to pay for masks, and self-reported behavior, the final clinical significance of these findings is uncertain because effects on all were quantitatively small. Conclusions These results suggest physician messaging campaigns may be effective and trust in Black and White physicians is equally high. There is no evidence of preexisting bias that would have led the intervention to have a negative effect. Because it is inexpensive, it could be a promising way to encourage behavior at scale. However, future studies implemented at a large scale are needed to confirm whether these kinds of interventions can change behavior in a way that will affect clinical outcomes. In ongoing work, we will study scale up messaging by doctors using social media. JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 11/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices ARTICLE INFORMATION Accepted for Publication: May9,2021. Published: July 14, 2021. doi:10.1001/jamanetworkopen.2021.17115 Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Torres C et al. JAMA Network Open. Corresponding Author: Esther Duflo, PhD, Department of Economics, Massachusetts Institute of Technology, 77 Massachusetts Ave, Building E52-544, Cambridge, MA 02139 (eduflo@mit.edu). Author Affiliations: Harvard Medical School, Boston, Massachusetts (Torres, Ogbu-Nwobodo, Stanford, Warner); Department of Pediatrics–General Pediatrics, Massachusetts General Hospital for Children, Boston (Torres); Department of Psychiatry, Massachusetts General Hospital, Boston (Ogbu-Nwobodo, Warner); Department of Psychiatry, McLean Hospital, Belmont, Massachusetts (Ogbu-Nwobodo); Harvard Kennedy School of Government, Cambridge, Massachusetts (Alsan); Department of Medicine–Neuroendocrine Unit, Department of Pediatrics–Endocrinology, Massachusetts General Hospital, Boston (Stanford); Department of Economics, Massachusetts Institute of Technology, Cambridge (Banerjee, Karnani, Olken, Vautrey, Duflo); Department of Economics, Harvard University, Cambridge, Massachusetts (Breza); Department of Economics, Stanford University, Stanford, California (Chandrasekhar); Department of Economics, Ludwig Maximilian University of Munich, Munich, Germany (Eichmeyer); Paris School of Economics, Paris, France (Loisel); Yale School of Management, New Haven, Connecticut (Goldsmith-Pinkham); Clinical Translational Epidemiology Unit, Massachusetts General Hospital, Boston (Warner). Author Contributions: Dr Duflo had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Concept and design: Torres, Ogbu-Nwobodo, Alsan, Banerjee, Breza, Chandrasekhar, Eichmeyer, Goldsmith- Pinkham, Olken, Vautrey, Duflo. Acquisition, analysis, or interpretation of data: Ogbu-Nwobodo, Stanford, Banerjee, Breza, Chandrasekhar, Karnani, Loisel, Goldsmith-Pinkham, Olken, Vautrey, Warner, Duflo. Drafting of the manuscript: Ogbu-Nwobodo, Alsan, Stanford, Chandrasekhar, Loisel, Olken, Vautrey, Warner, Duflo. Critical revision of the manuscript for important intellectual content: Torres, Ogbu-Nwobodo, Stanford, Banerjee, Breza, Eichmeyer, Karnani, Goldsmith-Pinkham, Olken, Warner. Statistical analysis: Breza, Chandrasekhar, Karnani, Loisel, Goldsmith-Pinkham, Olken, Vautrey, Duflo. Obtained funding: Breza, Goldsmith-Pinkham, Duflo. Administrative, technical, or material support: Torres, Alsan, Stanford, Breza, Karnani, Goldsmith- Pinkham, Warner. Supervision: Torres, Ogbu-Nwobodo, Breza, Vautrey, Duflo. Conflict of Interest Disclosures: Dr Olken reported receiving ad credits from Facebook outside the submitted work. No other disclosures were reported. Funding/Support: This work was supported by grant 2029880 from the National Science Foundation to Drs Alsan and Duflo, the Physician/Scientist Development Award from the Executive Committee on Research at Massachusetts General Hospital to Dr Stanford, grant P30 DK040561from the National Institutes of Health to Dr Stanford, grant L30 DK118710 from the National Institutes of Health to Dr Stanford, and the Clinician-Teacher Development Award from the Massachusetts General Physicians Organization at Massachusetts General Hospital to Dr Torres. Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. Group Members: The members of the COVID-19 Working Group appear in Supplement 3. Disclaimer: The findings and conclusions expressed are solely those of the authors and do not represent the views of their funders. Data Sharing Statement: See Supplement 4. Additional Contributions: We thank the Center for Diversity and Inclusion, especially Sandra Pena Ordonez, BS, and Elena Olson, JD for their assistance in this project. We thank Minjeong Joyce Kim, BS (Stanford University), and Sirena Yu (Massachusetts Institute of Technology), for their research assistance. These individuals were not compensated for their time. JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 12/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices REFERENCES 1. Zhao J, Lee M, Ghader S, Younes H, Darzi A, Xiong C, Zhang L. Quarantine fatigue: first-ever decrease in social distancing measures after the COVID-19 pandemic outbreak before reopening United States. Preprint updated June 11, 2020. Accessed June 8, 2021. https://arxiv.org/abs/2006.03716 2. McDonnell Nieto del Rio G. Doctors plead with Americans to take the virus surge seriously. The New York Times. Accessed June 8, 2021. https://www.nytimes.com/live/2020/11/15/world/covid-19-coronavirus#doctors-plead- with-americans-to-take-the-virus-surge-seriously 3. Alsan MS, Stanford FC, Banerjee A, et al. Comparison of knowledge and information-seeking behavior after general COVID-19 public health messages and messages tailored for Black and Latinx communities: a randomized controlled trial. Ann Intern Med. 2021;174(4):484-492. 4. US Centers for Disease Control and Prevention. Risk for COVID-19 infection, hospitalization, and death by race/ ethnicity. Updated May 26, 2021. Accessed January 21, 2021. https://www.cdc.gov/coronavirus/2019-ncov/covid- data/investigations-discovery/hospitalization-death-by-race-ethnicity.html 5. American Medical Association. AMA Board of Trustees pledges action against racism, police brutality. June 7, 2020. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-board-trustees- pledges-action-against-racism-police-brutality 6. US Census Bureau. 2018 Population estimates by age, sex, race and Hispanic origin. June 19, 2019. Accessed June 19, 2020. https://www.census.gov/newsroom/press-kits/2019/detailed-estimates.html 7. American Medical Association. AMA applauds congressional action on prescription drug transparency. April 9, 2019. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-applauds- congressional-action-prescription-drug-transparency 8. Wertenbroch K, Skiera B. Measuring consumers’ willingness to pay at the point of purchase. J Marketing Res. 2002;39(2):228-241. doi:10.1509/jmkr.39.2.228.19086 9. Hainmueller J. Entropy balancing for causal effects: a multivariate reweighting method to produce balanced samples in observational studies. Political Analysis. 2012;20(1):25-46. doi:10.1093/pan/mpr025 SUPPLEMENT 1. eAppendix 1. Survey Design and Videos eAppendix 2. Supplementary Methods eAppendix 3. Robustness Checks and Subgroup Analysis eFigure 1. Full Study Flowchart eFigure 2. Distribution of the Safety Gap Score in the Control and Intervention Groups eTable 1. Balance and Attrition eTable 2. Outcomes by Subgroup eTable 3. Effects of Any Message Intervention: Effects on Additional Outcomes eTable 4. Effects of Tailoring Messages on Additional Outcomes eTable 5. Effects of All Black Treatments on Outcomes SUPPLEMENT 2. Pre-analysis Plan and Trial Protocol SUPPLEMENT 3. Nonauthor Contributors SUPPLEMENT 4. Data Sharing Statement JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 13/13 (Round 2) Pre-Analysis Plan COVID-19 Health Messaging to Underserved Communities This Draft: September 4, 2020 1. Introduction The aim of this study is to build on results from our ﬁrst experiment and test how acknowledging institutional racial injustice affects informational messaging. In particular, we will investigate how well study participants retain knowledge and update beliefs and behavior with respect to COVID-19. We will also test the effect of race concordance of providers with recipients, and whether highlighting the unequal burden of the disease has additional effects on knowledge, beliefs and behavior regarding COVID-19. Compared to the ﬁrst study, there are four additional changes to the experimental design of note: 1) the sample will include white respondents in addition to African American respondents; 2) the group that does not view videos about COVID-19 will watch a set of placebo videos about non-COVID health topics; 3) we are collecting follow-up behavioral data from a subset of participants; 4) this study does not include a few of the treatment variants from the ﬁrst – speciﬁcally, there is no acknowledgment by doctors in the videos discussing health behaviors, and there is no treatment arm to alter perceptions of whether mask-wearing is socially-acceptable. 2. Treatments, and Experimental Protocols 2.1 Treatments Each subject receives one AMA statement and then watches three videos pertaining to health. The AMA statement either addresses racial injustice (treatment [RI]) or drug pricing (placebo [DP]). The videos pertaining to health behavior either discuss COVID-19 and prevention (treatment) or other non- COVID-19 related issues (control). • AMA Statement: RI: Racial Injustice DP: Drug Pricing • Treatment Videos We continue to focus our sample on individuals with less than college education. We are also collecting baseline measures of these behaviors. 1 1. Video T1: – Introduction – Discussion of symptoms 2. Video T2: – Information about social distancing and hygiene – Sub-treatment T2A: Acknowledgment of racial disparities in contagion and mortality. 3. Video T3: – Information about masks • Placebo Videos 1. Video C1: – Information about ﬁtness routines 2. Video C2: – Information about sleep hygiene 3. Video C3: – Information about sugar intake We are varying ﬁve aspects of the videos across treatments: 1. AMA statement regarding racial injustice or transparency in drug pricing. 2. Racial concordance with AMA spokespeople in the videos. 3. Racial concordance with partner doctors in the videos. 4. Informative videos about COVID-19 or non-COVID-19 issues. 5. Video T2 content: • Standard message about hygiene and social distancing • Previous message, plus acknowledgment of racial disparities in contagion and mortality. All of the video scripts can be found in Appendix Section A. Figure 1 shows how these different treat- ments are incorporated in the randomization design. 2.2 Recruitment and Sampling We are using Lucid to recruit a sample and to compensate subjects for their participation. Our total target sample size is 20,000 subjects, 10,000 African American respondents, and 10,000 White American respondents. We require all participants to be age 18 or older, and we are targeting individuals who have not completed a college degree. 2 Figure 1: Treatment Design Note: There are 5 randomization instances summarized in this diagram. For the fully extended tree with all 24 ﬁnal individual cells, see Figure 2. 2.3 Experimental Protocols and Treatments Our experiment has the following structure: 1. Recruitment and Baseline survey (a) Recruit target sample via Lucid (b) Collect demographic information, political views, and mask ownership. Several demograph- ics variables are being collected directly by Lucid. 2. Randomize subjects to treatment • Treatment scripts are tailored to each racial group • Within racial group, treatment is stratiﬁed on age, gender and geographic region. • Randomization is at the individual level according to Figure 1. • There will also be a control group who will not receive COVID-19 videos, but will receive information at the very end of the survey. 3. Video delivery for all individuals. (a) After each of the three videos, respondents evaluate the video’s usefulness, and trustwor- thiness, along with the respondents intention to follow the advice in the video and share information from the video. 4. Endline survey 5. Debrief script 3 6. Follow-up survey (a) We will follow up with participants three days post-treatment. (b) The chief purpose would be to measure health-seeking and health-preserving behaviors. 3. General Hypotheses to be Tested This section lists the general hypotheses we want to test. For those, we are planning to run broadly pooled treatments (and speciﬁc interactions speciﬁed below). Main hypotheses: 1. Baseline treatment effect: Does any COVID-19 message affect knowledge/indended behavior/priorities in donations/actual behavior? (a) Compare the effectiveness of any doctor treatment video vs. any doctor placebo video. • We will test this in the full sample and in the white and African American samples, sepa- rately. 2. Concordance effect: Does the concordance of the doctor in the COVID-19 video affect the impacts of the message? (a) Within the treatment group receiving a COVID-19 doctor video, compare the effectiveness of the video if the respondent is randomized to a doctor of concordant vs. discordant race. • We will test this hypothesis separately in the white and African American samples. If the effects go in the same direction for both populations, we will also test this hypothesis in the full sample. • An alternate way to deﬁne concorance is whether the subject views a racially concordant messenger both (or either) in the COVID-19 doctor video and in an AMA statement of any kind. We will also test the effectiveness of the messaging under this alternate measure. 3. AMA acknowledgement: Does the AMA message mediate the impacts of the COVID-19 messag- ing or of the racial concordance of the messenger? (a) Compare baseline treatment effect [1] for AMA acknowledgement vs AMA placebo. (b) Compare concordance effect [2] for AMA acknowledgment vs AMA placebo. Is the concor- dance of the doctor in the COVID-19 video more or less important when there is an AMA message vs. the AMA placebo? • We will test these hypotheses separately for white and African American respondents. 4. Does the racial concordance of the AMA messenger matter? (a) Compare whether AMA acknowledgement [3a] is more effective if the AMA messenger is racially concordant. 4 (b) Compare AMA acknowledgement effect [3b] by racial concordance of AMA messenger. Is the concordance of the doctor in the COVID-19 video more or less important when there is a racially concordant AMA message vs. a non-racially concordant AMA message? • We will test these hypotheses separately for white and African American respondents. 5. Content of message: does the mention of the racial disparities (video 2RD) in disease burden affect knowledge/intended behavior/ priorities in donation/actual behavior? (a) Within the COVID video treatment group, compare videos with RD information vs. no RD information. (b) Within the COVID video treatment group, how does information about RD interact with the racial concordance of the doctor messenger? Does information about RD make racial concodance more important, especially among African American respondents? (c) Within the COVID video treatment group, how does information about RD interact with the AMA racial injustice message? Does information about RD make the AMA acknowledge- ment more effective, especially among African American respondents? (d) We are also interested in how information about RD mediates Hypotheses 3b, 4a, 4b, and 4c. • In addition to the standard outcomes, we are also particularly interested in impacts on dona- tions to a COVID fund that is speciﬁcally for African Americans vs. a general COVID fund. We also are interested in how this information changes perceptions of the policy responses of state and federal governments. • For 5a, we are interested in the white and African American groups, separately, and also the full, pooled sample. For 5b and 5c, we are mainly interested in the effects separately for white and African American respondents. • RD should have the largest impacts for individuals who have incorrect beliefs. Moreover, the impacts of the information should be different for individuals with priors about relative impacts on African Americans that are too low versus too high. (See Section 5.4, below). 4. Data Collection and Outcomes We will run our experiment beginning on August 7, 2020. We are recruiting individuals that ﬁt our screening criteria from an online survey ﬁrm. The data will be collected from respondent surveys. There is a short baseline module before individuals are exposed to the videos. Following the ﬁnal video, there is an endline outcome module that all participants complete. Respondents also evaluate each video immediately after watching it. We also plan to follow up with individuals a few days after the treatment delivery for an additional survey to measure whether the videos changed behaviors. Prior to treatment, we ask all participants whether they would be willing to answer the follow-up survey. We allow individuals to enroll in the study, independent of their follow-up availability. We plan to control for willingness to answer a follow-up in all regressions. All of the survey responses will be downloaded in a .csv ﬁle for cleaning and analysis in R. 5 4.1 Baseline Survey Variables The baseline information for all respondents includes the following demographic characteristics: • Education • Age • Location • Sex • Political Views We elicit additional health information at baseline, but participants may choose not to answer these questions: • Mask Ownership • Public Excursions • Health Behavior See Section 5, below for a summary of our key endline survey variables. 4.2 Data quality checks We have several questions in the survey that capture respondent attention. We use some of these questions to screen out low-attention survey takers before assigning treatment. We further plan to exclude those that take very little time on the survey overall and/or on the videos. 5. Empirical Analysis 5.1 Balance Checks We will conduct a series of balance tests across treatment arms to ensure that there are no chance differ- ences between subjects in the various arms. We will regress characteristics measured pre-treatment on indicators for the arms and test their individual and joint signiﬁcance. Balance tests will be conducted using all of the variables measured in the baseline survey. We will also test for balance in attrition rates (see Section 6.2, below). 5.2 Key Outcomes We intend to measure the treatment effects on the following set of outcomes: 1. The number of participants with knowledge of COVID related symptoms and transmission as assessed by a questionnaire we’ve developed speciﬁcally relevant to the intervention videos we are using for the project. Speciﬁcally, we deﬁne the following three outcomes, measured both at ﬁrst contact and followup: 6 • Knowledge of COVID symptoms: participants are asked to identify 4 symptoms of COVID from a list of 9. We will deﬁne an indicator for whether participants select the 3 most common symptoms (fever, cough, difﬁculty breathing). • Knowledge of COVID prevention: participants are asked to identify 3 COVID prevention behaviors from a list of 7. We will deﬁne an indicator for whether participants select the 3 behaviors emphasized in the videos (staying outside and 6ft from others; washing hands when going and coming from home; mask wearing). • Knowledge of asymptomatic infection: an indicator for whether participants correctly an- swer that COVID can spread asymptomatically. 2. Behavioral outcome 1: Number of participants who report behavior change related to messages provided in the intervention videos; measured via a speciﬁc questionnaire instrument we’ve de- veloped to correspond to the intervention. Speciﬁc behaviors include physical distancing, mask wearing, and hand hygiene. Since we ask about several safety behaviors, we will deﬁne a stan- dardized index to combine them into one measure, calculated at both ﬁrst contact and endline. The key outcome is the within-person change in this safety index from ﬁrst contact to endline. 3. Behavioral outcome 2: Revealed preference estimate of willingness to pay for masks. The subjects will trade-off the willingness to get a pair of masks or an amazon gift card when participating in a strategy-proof lottery. 4. Behavioral outcome 3: Number of links people click on for additional information on the COVID- 19 behaviors. Links will include testing locations, state public health hotline, and symptom tracker. 5. Behavioral outcome 4: Donations to a COVID-related charity. After providing information on the number of weekly COVID cases, we are measuring the willingness to donate to a COVID-related charity vs. a generic helath-related charity. 6. Behavioral outcome 5: Donations to African-American COVID-19 fund. After providing infor- mation on the disproportionate burden on African-American communities, we are measuring the willingness to donate to a COVID relief fund that focuses on African-American communities vs. one that generically provides relief to disadvantaged individuals. 7. Evaluation of state and local COVID policy. We are measuring how how well participants think their federal and state governments managed to balance opening the economy and limiting the health impacts of Covid-19. We believe that information about racial disparities, in particular, may change these perceptions, especially for people who believed that racial disparities did not exist or were less severe at baseline. 5.3 Regression Analysis We will perform different regression analyses to test the hypotheses listed above. Because our data contains many possible control variables, we will use a double-lasso procedure to select regression controls. We will also include a control for whether the participant’s availability to participate in a 7 follow up survey (as indicted at baseline). These control variables are denoted as X in the regression speciﬁcations below. Unless otherwise noted, we will examine treatment effects on knowledge of COVID symptoms, knowl- edge of COVID transmission, intended donation to a COVID-related charity, willingness to pay for masks, changes in reported behavior. In what follows, we present the minimal regressions to test each of the hypotheses, restricting to the smallest subset of treatments. However, we could also execute the same tests in the full sample, but with more treatment interactions. Note that the “COVID” indicator covers cases when doctors mention COVID plus racism, or COVID alone. In text, we refer to this group as the treatment group. 1. Baseline treatment effect: does any of the COVID-19 messaging affect knowledge, intended behavior, priorities in donation, or actual behavior? • Question: does COVID messaging from doctors have any effect? Samples: separate analysis for all respondents, black respondents, and white respondents. Regression: Y = β · COVID Video + X γ + ǫ 2. Does racial concordance of the doctor in the COVID-19 video change the effectiveness for messaging? • Question: Bolded, immediately above. Samples: COVID video groups. Separate analysis for black and white respondents. Regression: Y = β · DocConcord + X γ + ǫ Alternate Speciﬁcation: To allow, for a level effect of doctor concordance, we can also run the following modiﬁed regression, including both the COVID video and control groups. Y = β · DocConcord · COVID Video + α · COVID Video + δ · DocConcord + X γ + ǫ • Question: Bolded above, but using an alternate deﬁnition of concordance. We’ll require a) all messengers or b) any messenger to concord with the respondent’s race. For any outcomes where β is of the same sign for white and black respondents, we will estimate the above equation again in the full sample. 3. Does the AMA racism acknowledgment affect the impact of messaging on any outcomes? Does it change the concordance effect? In some cases, especially when including controls, these full-sample tests may be preferrable to these stripped-down regressions. 8 • Question: Does the AMA racism messaging heighten or dull COVID messaging from doc- tors? Samples: COVID video groups. Separate analysis for all respondents, black respondents, and white respondents. Regression: Y = β · AMARacism + X γ + ǫ Alternate Speciﬁcation: To allow, for a level effect of the AMA racism message, we can also run the following modiﬁed regression, including both the COVID video and control groups. Y = δ · AMARacism · COVID Video + α · COVID Video + β · AMARacism + X γ + ǫ • Question: Does the AMA racism acknowledgment heighten or dull any doctor concordance effects? Samples: COVID video groups. Separate analysis for black and white respondents. Regression: Y = δ · AMARacism · DocConcord + α · AMARacism + β · DocConcord + X γ + ǫ Alternate Speciﬁcation: Again, we can also run the following modiﬁed regression, including both the COVID video and control groups. Y = δ · DocConcord · AMARacism · COVID Video + λ · AMARacism · COVID Video +φ · DocConcord · COVID Video + ρ · AMARacism · DocConcord + α · COVID Video +β · AMARacism + ψ · DocConcord + X γ + ǫ 4. Does concordance of the AMA messenger matter? • Question: Does a race-concordant AMA messenger delivering a message about racial injus- tice make COVID messaging from doctors more or less effective, relative to a race-discordant AMA messenger? Samples: Individuals receiving an AMA message about racial injustice and a treatment mes- sage about COVID. Separate analysis for black and white respondents. Regression: Y = β · AMAConcord + X γ + ǫ Alternate Speciﬁcation: We can also run the following modiﬁed regression, the COVID video and control groups, with both types of AMA messages. Y = δ · COVID Video · AMAConcord · AMARacism + λ · AMAConcord · AMARacism +ρ · AMAConcord · COVID Video + ψ · COVID Video · AMARacism +α · AMAConcord + β · AMARacism++φ · COVID Video + X γ + ǫ Ex ante, we think that an AMA level effect in the control group is more likely than a doctor concordance level effect. However, we include all alternates for completeness. These alternate speciﬁcations including the control group are relevant in the presence of a level effect. 9 • Question: Does a race-concordant AMA messenger make COVID messaging from a race- concordant doctor more or less effective, when delivering the AMA message about racial injustice? Samples: COVID video groups receiving the AMA message about racism. Separate analysis for black and white respondents. Regression: Y = δ · AMAConcord · DocConcord + α · AMAConcord + β · DocConcord + X γ + ǫ Alternate Speciﬁcation: We can also run the following modiﬁed regression, the COVID video and control groups, with both types of AMA messages. Y = δ · AMAConcord · AMARacism · DocConcord · Covid Video +α · DocConcord · AMARacism · COVID + α · DocConcord · AMARacism · AMAConcord 1 2 +α · DocConcord · COVID · AMAConcord + α · AMARacism · COVID · AMAConcord 3 4 +β · DocConcord · AMARacism + β · DocConcord · COVID + β · DocConcord · AMAConcord 1 2 3 +β · AMARacism · COVID + β · AMARacism · AMAConcord + β · COVID · AMAConcord 4 5 6 +ρ · DocConcord + ρ · AMARacism + ρ · AMAConcord + ρ · COVID + X γ + ǫ 2 3 1 4 5. What are the effects of acknowledging racial disparities in COVID incidence? In addition to the typical set of outcomes, we will also analyze effects on allocated donations to black-speciﬁc versus race-agnostic COVID-related charities. • Question: what is the main effect of acknowledging racial disparities of COVID incidence? Samples: treated respondents only. Separate analysis for black, white, and all respondents. Regression: Y = β · Vid2RacialDisp + X γ + ǫ • Question: are race-concordant doctors more effective messengers about racial disparity? Samples: treated respondents only. Separate analysis for black, white, and all respondents. Regression: Y = δ · DocConcord · Vid2RacialDisp + α · Vid2RacialDisp + β · DocConcord + X γ + ǫ • Question: does acknowledging widespread racism alter the effectivness of later discussing racial disparities in COVID? Samples: treated respondents only. Separate analysis for black, white, and all respondents. Regression: Y = δ · AMARacism · Vid2RacialDisp + α · Vid2RacialDisp + β · AMARacism + X γ + ǫ 10 • Question: does a race-concordant AMA messenger’s prefacing statement have a different amplifying effect than a race discordant AMA messenger? That is, if doctors are going to discuss racial disparities in COVID and we’re going to preface this with the AMA racism statement, do we expect different results if the statement comes from a concordant versus discordant AMA messenger? Samples: Must have seen both a COVID video and an AMA racism video. Separate analysis for black and white respondents. Regression: Y = δ · AMAConcord · Vid2RacialDisp + α · Vid2RacialDisp + β · AMAConcord + X γ + ǫ Alternate speciﬁcation: restrict sample to those seeing a COVID video, include both types of AMA messages. Regression: Y = δ · AMARacism · AMAConcord · Vid2RacialDisp + λ · AMAConcord · Vid2RacialDisp +ρ · AMAConcord · AMARacism + φ · AMARacism · Vid2RacialDisp +α · Vid2RacialDisp + β · AMAConcord + ψ · AMARacism + X γ + ǫ 5.4 Heterogeneous Effects We plan to conduct several heterogeneity tests that we believe are of central importance: respondent race, respondent political aﬁliation (within the white sample), prior beliefs about racial disparities and COVID-19, and the timing of participation vis a vis the events of Kenosha, WI. We are very interested in studying how the impacts of our treatments vary by the race of the respondent. This is central to our research design. Speciﬁcally, • Is the impact of racial concordance different by respondent race? • Is the impact of a statement addressing racial injustice different by respondent race? • Is the impact of information about racial disparities in the COVID-19 burden different by respon- dent race? Moreover, within the white respondent population, we predict that there may be substantial hetero- geneity by the respondent’s political beliefs: • Is the impact of racial concordance different for white republicans versus white democrats? • Is the impact of a statement addressing racial injustice different for white republicans versus white democrats? • Is the impact of information on racial disparities in disease burden different for white republicans versus white democrats? We predict that the impacts of information on racial disparities in COVID burden should depend on individuals’ prior beliefs: 11 • The information about RD should cause individuals who initially believed at baseline that there were small or non-existant racial disparities to update in the opposite direction of individuals who believed that the racial disparities were larger than they actually are. • Thus, for all of our tests involving RD, we will interact the regressions with indicators for whether the priors were smaller or larger than the number we give in the videos. • Moreover, there may be bigger impacts for individuals whose priors were less accurate, so we can also interact by the size of the gap between the informaiton and the prior, separately for those with priors that were too low versus too high. This can also help us measure whether the information had an impact even for people with accurate priors, possibly through a salience effect. We intend to compare the results of our hypothesis tests separately for the time period before the Jacob Blake police shooting in Kenosha, WI on August 23, 2020, and the time period after. We propose to do this for two reasons. First, we launched the study at a time when the large-scale protests from earlier in the summer following the murder of George Floyd had somewhat ebbed. So the events in Kenosha may bring issues of racial injustice to the fore. Second, and perhaps more importantly, polarization surrounding the narrative of the protests has markedly increased following the events of Kenosha. Several speakers at the Republican National Convention explicitly discussed the violent component of the protests, for example, and both candidates for the US presidency are making trips to Kenosha. This increased polarization is likely to have the most relevance for the white respondents in our study, and may enhance any differential response we ﬁnd by political afﬁliation. We propose to split the sample into the period before August 23, 2020 and the period following August 26, 2020. We will omit surveys collected on days in the interim when Americans were only coming to learn about the events of Kenosha, WI. We are also interested in secondary analysis exploring heterogeneity on the following categories of traits/characteristics: • Age • Level of baseline knowledge and health-preserving behaviors • Place of residence (correlated with political afﬁliation, COVID-19 policies and phased reopenings, and socio-economic characteristics) Given the many ways to cut the data for this secondary analysis, we will follow the methodology of Chernozhukov et al (2019) for this latter set of potential heterogeneous treatment effects. 6. Robustness 6.1 Threats to Interpretation We would like to assume that differences across videos come from either differences in the racial iden- tity of the doctors in the video or from differences in the content of the messages, rather than from other chance differences across videos. Because we are including both white and black respondents who will be watching the same exact videos, we will be able to include video ﬁxed effects in some speciﬁcations. Chernozhukov, Victor, Mert Demirer, Esther Duﬂo, and Ivan Fernandez-Val (2019). Generic machine learning inference on heterogenous treatment effects in randomized experiments. No. w24678. National Bureau of Economic Research. 12 6.2 Attrition We have two separate endline surveys. The ﬁrst will take place immediately after treatment delivery. The type of attrition that might arise here is through dropping out of the online session before complet- ing all of the survey questions. To try to limit differential attrition, we are showing placebo videos to the control group to ﬁll approximately the same amount of time. Our second set of endline outcomes will take place a few days after the main survey. Lucid, the survey ﬁrm, will try to recontact a speciﬁed list of initial participants. We are only expecting modest recontact rates, and therefore high levels of attrition. Importantly, to try to limit differential attrition, all individ- uals will be recontacted with the exact same message, and it will not be made salient that the survey is a direct follow-up to the previous study. We plan to test for differential attrition at both endlines across our key comparison groups. 7. Funding and Human Subjects Review Funding is provided by the National Science Foundation RAPID-2029880 for Covid-19 research, and RAI Italian Broadcasting corporation (via an unrestricted gift to J-PAL that we attributed to this project). The IRB at MIT is serving as the primary institution of record and has entered into a reliance agreement with Harvard, Massachusetts General Hospital, and Yale. We have also received IRB approval from Stanford. 13 Appendix A. Scripts All respondents will receive either [Statement RI or Statement DP] + one set of [Treatment or Control] videos. A.1 Statements Each respondent is assigned to one of the following statements. All respondents will see the statement presented via video. A.1.1 Treatment Statement RI (Racial Justice): • The American Medical Association recognizes that racism in its systemic, structural, institutional, and interpersonal forms is an urgent threat to public health, the advancement of health equity, and a barrier to excellence in the delivery of medical care. • The American Medical Association opposes all forms of racism. • The American Medical Association denounces police brutality and all forms of racially-motivated violence. • The American Medical Association will actively work to dismantle racist and discriminatory poli- cies and practices across all of health care. A.1.2 Placebo Statement DP (Drug Pricing): • The American Medical Association believes in transparency in prescription drug pricing, and we are pleased the House Ways & Means Committee moved the issue forward. • Patients and their physicians want to be armed with more information, yet the current situation is opaque if not impenetrable. • The committee is rightfully determined to expose factors that lead to high drug prices, and we look forward to continuing our efforts in that regard. A.2 Treatment Videos about COVID-19 A full set of treatment videos includes T1 + T2 + [T2A or nothing] + T3 Video T1: Hello, I’m Dr [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL AFFILIATION HERE], and I?d like to tell you a little about Coronavirus or COVID-19. COVID-19 is a new virus that can infect the respiratory tract and lungs. Although many people who get sick from COVID will get better, some people who get it become very ill and some even die. 14 Although there’s no cure, there are ways medical professionals have found to protect you and your community from COVID. I hope that this message can give you information that will help you protect you or someone you love from COVID infection. First, I would like to tell you about the symptoms of COVID-19. The most common symptoms of COVID-19 are cough, fever, and trouble breathing. Another odd symptom some people have is loss of taste or smell. A large number of people who have COVID-19 actually don’t show any symptoms at all. Unfortunately, people can still spread the disease to others even with no symptoms. The next video will provide you with more information on how you can protect yourself and others. Video T2: Hello, I’m Dr [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL AFFILIATION HERE], You may be looking for ways to resume some activities as safely as possible. However, COVID-19 remains contagious and shows no signs of disappearing. In fact, during the week of July 6 there were 58,000 new COVID cases per day diagnosed in the United States. [ONLY FOR ACKNOWLEDGMENT SUB-TREATMENT T2A] Black Americans and other minority groups are three times as likely to get and, when you account for age, four times as likely to die from COVID as white Americans. Without a safe and effective vaccine or therapy, our only option is to continue taking precautionary measures to protect ourselves, our communities, and the most vulnerable among us. While there is no way to ensure zero risk of infection from COVID-19, observing these three practices will help to protect you and others. First, continue to practice social distancing whenever possible: Try to stay outdoors, and to the maxi- mum extent possible, please stay 6 feet apart. If you must be indoors, use visual reminders—like signs, chair arrangements, markings on the ﬂoor, or arrows—to help remind you to keep your distance from others, and maintain physical barriers whenever possible. Second, continue to wash your hands often for at least 20 seconds with soap and water, especially before going out, and every time you return home. Third, wear a mask when in public at all times, especially when indoors or when it is difﬁcult to stay 6 feet away. The next video will tell you a bit more about masks. Video T3: Hello, I am doctor [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL AFFILIATION HERE], and I will tell you a bit more about masks. Wearing a mask is a key way to prevent the spread of COVID-19. You are not just protecting yourself but also your grandma and your community, just in case you have COVID-19 but don’t know it. Even if wearing a mask may sometimes put you in a difﬁcult situation, it is important to protect you and the community from COVID 19 disease. As medical professionals, I am committed to delivering the best care I can to every patient. My goal is to make sure that you and everyone you love survives this COVID-19 pandemic. Thank you for listening to these messages. A.3 Control Videos about non-COVID-19 Health Behaviors A full set of control videos includes C1 + C2 + C3 15 Video C1: Most adults need to sleep between 6 and 8 hours a night. Now, there are some people who get ﬁve hours a night and they are ﬁne, so there is some variation across people. But for most adults, we need 6 to 8 hours in order to function well the next day. If you feel sleep deprived you might not be able to function as well as you would normally like. It’s important to have something called sleep hygiene which is a routine you follow at bedtime and can help you fall asleep. Things that can disrupt sleep hygiene include caffeine or alcohol too close to bedtime. Eating late at night can also cause indigestion. So keep a routine and trying to get 6-8 hours is important. Video C2: Sugar is found in many different food items. Natural sugars are those that can be found in fruits, vegetables and dairy products like milk. Sugars like these that are natural are not really problematic because they are coming alongside lots of other vitamins and minerals. There are other sugars, though, that are processed and added to a food item. These are called additive sugars. A good rule of thumb is to eat foods with less than than 5g of sugar per serving. Avoid buying products where one of the ﬁrst ﬁve products is a sugar. And it can be better to buy an unsweetened product like an unsweetened cereal or oatmeal and then add a teaspoon of sugar to it if you need the sweetness than to buy a heavily sweetened product, like a sugar cereal which can have several teaspoons of sugar per serving. Video C3: New ﬁtness guidelines can be summed up as follows: just move and anything counts. Sneaking in a few minutes of physical activity throughout the day adds up in the long run. The guidelines are trying to make it easier for individuals to be ﬁt and drop the rule that activity must be in 10 minute blocks of time. In a nutshell, activity has beneﬁts even if it’s for a short amount of time. Taking the stairs instead of the elevator, parking your car far away from the entrance to a store or walking your dog around the block can all help you be ﬁt. The guidelines still call for at least 150 minutes a week of moderately intense aerobic exercise and two weekly sessions of muscle training activity, like lifting weights or yoga. 16 17 Figure 2: Fully extended randomization tree (2( (U (B (E (P (b( (E (6 ( ( ( ( (E ( (3 (1 ( ( (1( (1( ( (N (A (B( (b (E (e (A (b (R( ( (r (P( (v( (A( ( (a( ( ( (R (d (M( (m( (A( (a( (E( (C ( ( ( ( ( (C( (p( (l( (M http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png JAMA Network Open American Medical Association http://www.deepdyve.com/lp/american-medical-association/effect-of-physician-delivered-covid-19-public-health-messages-and-HjnMbMHanN

Loading next page...

References (20)

(2021)
Open Access: This is an open access article distributed under the terms of the CC-BY License
Jun Zhao, Minha Lee, Sepehr Ghader, H. Younes, Aref Darzi, Chenfeng Xiong, Lei Zhang (2020)
Quarantine Fatigue: first-ever decrease in social distancing measures after the COVID-19 pandemic outbreak before reopening United States
(McDonnell Nieto del Rio G. Doctors plead with Americans to take the virus surge seriously. The New York Times. Accessed June 8, 2021. https://www.nytimes.com/live/2020/11/15/world/covid-19-coronavirus#doctors-plead-with-americans-to-take-the-virus-surge-seriously)
McDonnell Nieto del Rio G. Doctors plead with Americans to take the virus surge seriously. The New York Times. Accessed June 8, 2021. https://www.nytimes.com/live/2020/11/15/world/covid-19-coronavirus#doctors-plead-with-americans-to-take-the-virus-surge-seriously
McDonnell Nieto del Rio G. Doctors plead with Americans to take the virus surge seriously. The New York Times. Accessed June 8, 2021. https://www.nytimes.com/live/2020/11/15/world/covid-19-coronavirus#doctors-plead-with-americans-to-take-the-virus-surge-seriously, McDonnell Nieto del Rio G. Doctors plead with Americans to take the virus surge seriously. The New York Times. Accessed June 8, 2021. https://www.nytimes.com/live/2020/11/15/world/covid-19-coronavirus#doctors-plead-with-americans-to-take-the-virus-surge-seriously
Doctors plead with Americans to take the virus surge seriously. The New York Times
(Alsan MS, Stanford FC, Banerjee A, . Comparison of knowledge and information-seeking behavior after general COVID-19 public health messages and messages tailored for Black and Latinx communities: a randomized controlled trial. Ann Intern Med. 2021;174(4):484-492.33347320)
Alsan MS, Stanford FC, Banerjee A, . Comparison of knowledge and information-seeking behavior after general COVID-19 public health messages and messages tailored for Black and Latinx communities: a randomized controlled trial. Ann Intern Med. 2021;174(4):484-492.33347320
Alsan MS, Stanford FC, Banerjee A, . Comparison of knowledge and information-seeking behavior after general COVID-19 public health messages and messages tailored for Black and Latinx communities: a randomized controlled trial. Ann Intern Med. 2021;174(4):484-492.33347320, Alsan MS, Stanford FC, Banerjee A, . Comparison of knowledge and information-seeking behavior after general COVID-19 public health messages and messages tailored for Black and Latinx communities: a randomized controlled trial. Ann Intern Med. 2021;174(4):484-492.33347320
(American Medical Association. AMA Board of Trustees pledges action against racism, police brutality. June 7, 2020. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-board-trustees-pledges-action-against-racism-police-brutality)
American Medical Association. AMA Board of Trustees pledges action against racism, police brutality. June 7, 2020. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-board-trustees-pledges-action-against-racism-police-brutality
American Medical Association. AMA Board of Trustees pledges action against racism, police brutality. June 7, 2020. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-board-trustees-pledges-action-against-racism-police-brutality, American Medical Association. AMA Board of Trustees pledges action against racism, police brutality. June 7, 2020. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-board-trustees-pledges-action-against-racism-police-brutality
(US Census Bureau. 2018 Population estimates by age, sex, race and Hispanic origin. June 19, 2019. Accessed June 19, 2020. https://www.census.gov/newsroom/press-kits/2019/detailed-estimates.html)
US Census Bureau. 2018 Population estimates by age, sex, race and Hispanic origin. June 19, 2019. Accessed June 19, 2020. https://www.census.gov/newsroom/press-kits/2019/detailed-estimates.html
US Census Bureau. 2018 Population estimates by age, sex, race and Hispanic origin. June 19, 2019. Accessed June 19, 2020. https://www.census.gov/newsroom/press-kits/2019/detailed-estimates.html, US Census Bureau. 2018 Population estimates by age, sex, race and Hispanic origin. June 19, 2019. Accessed June 19, 2020. https://www.census.gov/newsroom/press-kits/2019/detailed-estimates.html
(Hainmueller J. Entropy balancing for causal effects: a multivariate reweighting method to produce balanced samples in observational studies. Political Analysis. 2012;20(1):25-46. doi:10.1093/pan/mpr025)
Hainmueller J. Entropy balancing for causal effects: a multivariate reweighting method to produce balanced samples in observational studies. Political Analysis. 2012;20(1):25-46. doi:10.1093/pan/mpr025
Hainmueller J. Entropy balancing for causal effects: a multivariate reweighting method to produce balanced samples in observational studies. Political Analysis. 2012;20(1):25-46. doi:10.1093/pan/mpr025, Hainmueller J. Entropy balancing for causal effects: a multivariate reweighting method to produce balanced samples in observational studies. Political Analysis. 2012;20(1):25-46. doi:10.1093/pan/mpr025
(2020)
2018 Population estimates by age, sex, race and Hispanic origin
(2021)
for Disease Control and Prevention
M. Alsan, F. Stanford, A. Banerjee, Emily Breza, Arun Chandrasekhar, Sarah Eichmeyer, Paul Goldsmith-Pinkham, L. Ogbu-Nwobodo, B. Olken, Carlos Torres, Anirudh Sankar, Pierre-Luc Vautrey, E. Duflo (2020)
Comparison of Knowledge and Information-Seeking Behavior After General COVID-19 Public Health Messages and Messages Tailored for Black and Latinx Communities
Annals of Internal Medicine
Risk for COVID-19 infection, hospitalization, and death by race/ ethnicity
(American Medical Association. AMA applauds congressional action on prescription drug transparency. April 9, 2019. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-applauds-congressional-action-prescription-drug-transparency)
American Medical Association. AMA applauds congressional action on prescription drug transparency. April 9, 2019. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-applauds-congressional-action-prescription-drug-transparency
American Medical Association. AMA applauds congressional action on prescription drug transparency. April 9, 2019. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-applauds-congressional-action-prescription-drug-transparency, American Medical Association. AMA applauds congressional action on prescription drug transparency. April 9, 2019. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-applauds-congressional-action-prescription-drug-transparency
AMA applauds congressional action on prescription drug transparency
Jens Hainmueller (2012)
Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies
Political Analysis, 20
(US Centers for Disease Control and Prevention. Risk for COVID-19 infection, hospitalization, and death by race/ethnicity. Updated May 26, 2021. Accessed January 21, 2021. https://www.cdc.gov/coronavirus/2019-ncov/covid-data/investigations-discovery/hospitalization-death-by-race-ethnicity.html)
US Centers for Disease Control and Prevention. Risk for COVID-19 infection, hospitalization, and death by race/ethnicity. Updated May 26, 2021. Accessed January 21, 2021. https://www.cdc.gov/coronavirus/2019-ncov/covid-data/investigations-discovery/hospitalization-death-by-race-ethnicity.html
US Centers for Disease Control and Prevention. Risk for COVID-19 infection, hospitalization, and death by race/ethnicity. Updated May 26, 2021. Accessed January 21, 2021. https://www.cdc.gov/coronavirus/2019-ncov/covid-data/investigations-discovery/hospitalization-death-by-race-ethnicity.html, US Centers for Disease Control and Prevention. Risk for COVID-19 infection, hospitalization, and death by race/ethnicity. Updated May 26, 2021. Accessed January 21, 2021. https://www.cdc.gov/coronavirus/2019-ncov/covid-data/investigations-discovery/hospitalization-death-by-race-ethnicity.html
K. Wertenbroch, B. Skiera (2002)
Measuring Consumers' Willingness to Pay at the Point of Purchase
Journal of Marketing Research, 39
(Wertenbroch K, Skiera B. Measuring consumers’ willingness to pay at the point of purchase. J Marketing Res. 2002;39(2):228-241. doi:10.1509/jmkr.39.2.228.19086)
Wertenbroch K, Skiera B. Measuring consumers’ willingness to pay at the point of purchase. J Marketing Res. 2002;39(2):228-241. doi:10.1509/jmkr.39.2.228.19086
Wertenbroch K, Skiera B. Measuring consumers’ willingness to pay at the point of purchase. J Marketing Res. 2002;39(2):228-241. doi:10.1509/jmkr.39.2.228.19086, Wertenbroch K, Skiera B. Measuring consumers’ willingness to pay at the point of purchase. J Marketing Res. 2002;39(2):228-241. doi:10.1509/jmkr.39.2.228.19086
(Zhao J, Lee M, Ghader S, Younes H, Darzi A, Xiong C, Zhang L. Quarantine fatigue: first-ever decrease in social distancing measures after the COVID-19 pandemic outbreak before reopening United States. Preprint updated June 11, 2020. Accessed June 8, 2021. https://arxiv.org/abs/2006.03716)
Zhao J, Lee M, Ghader S, Younes H, Darzi A, Xiong C, Zhang L. Quarantine fatigue: first-ever decrease in social distancing measures after the COVID-19 pandemic outbreak before reopening United States. Preprint updated June 11, 2020. Accessed June 8, 2021. https://arxiv.org/abs/2006.03716
Zhao J, Lee M, Ghader S, Younes H, Darzi A, Xiong C, Zhang L. Quarantine fatigue: first-ever decrease in social distancing measures after the COVID-19 pandemic outbreak before reopening United States. Preprint updated June 11, 2020. Accessed June 8, 2021. https://arxiv.org/abs/2006.03716, Zhao J, Lee M, Ghader S, Younes H, Darzi A, Xiong C, Zhang L. Quarantine fatigue: first-ever decrease in social distancing measures after the COVID-19 pandemic outbreak before reopening United States. Preprint updated June 11, 2020. Accessed June 8, 2021. https://arxiv.org/abs/2006.03716
AMA Board of Trustees pledges action against racism, police brutality

Publisher: American Medical Association
Copyright: Copyright 2021 Torres C et al. JAMA Network Open.
eISSN: 2574-3805
DOI: 10.1001/jamanetworkopen.2021.17115
Publisher site: See Article on Publisher Site

Abstract

Key Points Question Do messages delivered by IMPORTANCE Social distancing is critical to the control of COVID-19, which has disproportionately physicians increase COVID-19 affected the Black community. Physician-delivered messages may increase adherence to these knowledge and improve preventive behaviors. behaviors among White and Black individuals? OBJECTIVES To determine whether messages delivered by physicians improve COVID-19 Findings In this randomized clinical trial knowledge and preventive behaviors and to assess the differential effectiveness of messages of 18 223 White and Black adults, a tailored to the Black community. message delivered by a physician increased COVID-19 knowledge and DESIGN, SETTING, AND PARTICIPANTS This randomized clinical trial of self-identified White and shifted information-seeking and self- Black adults with less than a college education was conducted from August 7 to September 6, 2020. protective behaviors. Effects did not Of 44 743 volunteers screened, 30 174 were eligible, 5534 did not consent or failed attention checks, differ by race, and tailoring messages to and 4163 left the survey before randomization. The final sample had 20 460 individuals specific communities did not exhibit a (participation rate, 68%). Participants were randomly assigned to receive video messages on differential effect on knowledge or COVID-19 or other health topics. individual behavior. INTERVENTIONS Participants saw video messages delivered either by a Black or a White study Meaning These findings suggest that physician. In the control groups, participants saw 3 placebo videos with generic health topics. In the physician messaging campaigns may be treatment group, they saw 3 videos on COVID-19, recorded by several physicians of varied age, effective in persuading members of gender, and race. Video 1 discussed common symptoms. Video 2 highlighted case numbers; in one society from a broad range of group, the unequal burden of the disease by race was discussed. Video 3 described US Centers for backgrounds to seek information and Disease Control and Prevention social distancing guidelines. Participants in both the control and adopt preventive behaviors to combat intervention groups were also randomly assigned to see 1 of 2 American Medical Association COVID-19. statements, one on structural racism and the other on drug price transparency. Invited Commentary MAIN OUTCOMES AND MEASURES Knowledge, beliefs, and practices related to COVID-19, demand for information, willingness to pay for masks, and self-reported behavior. Supplemental content Author affiliations and article information are RESULTS Overall, 18 223 participants (9168 Black; 9055 White) completed the survey (9980 listed at the end of this article. [55.9%] women, mean [SD] age, 40.2 [17.8] years). Overall, 6303 Black participants (34.6%) and 7842 White participants (43.0%) were assigned to the intervention group, and 1576 Black participants (8.6%) and 1968 White participants (10.8%) were assigned to the control group. Compared with the control group, the intervention group had smaller gaps in COVID-19 knowledge (incidence rate ratio [IRR], 0.89 [95% CI, 0.87-0.91]) and greater demand for COVID-19 information (IRR, 1.05 [95% CI, 1.01-1.11]), willingness to pay for a mask (difference, $0.50 [95% CI, $0.15-$0.85]). (continued) Open Access. This is an open access article distributed under the terms of the CC-BY License. JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 1/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices Abstract (continued) Self-reported safety behavior improved, although the difference was not statistically significant (IRR, 0.96 [95% CI, 0.92-1.01]; P = .08). Effects did not differ by race (F = 0.0112; P > .99) or in different intervention groups (F = 0.324; P > .99). CONCLUSIONS AND RELEVANCE In this study, a physician messaging campaign was effective in increasing COVID-19 knowledge, information-seeking, and self-reported protective behaviors among diverse groups. Studies implemented at scale are needed to confirm clinical importance. TRIAL REGISTRATION ClinicalTrials.gov Identifier: NCT04502056 JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 Introduction Physical distancing and mask wearing remain essential to the control of COVID-19, yet vigilance has decreased over time. To address fatigue, health care professionals have used social media to spread public health messages. There is evidence that these messages improve knowledge, but there are less data on whether they change behavior. Black US residents have been disproportionately affected by the pandemic. This reflects the cumulative impact of systemic racism, acknowledged as a public health threat by the American Medical Association (AMA) in a June 2020 statement. This raises the question on whether the effectiveness of public health messages regarding COVID-19 would be enhanced if tailored to the Black community. The focus of this study was to identify whether messages delivered by physicians increase COVID-19 knowledge and improve preventive behaviors for White and Black individuals and to assess whether various ways of increasing the relevance of messages to the Black community (ie, physician race, AMA acknowledgments of racial injustices, or information about the disproportionate burden of COVID-19 on the Black community) affects their impact on both White and Black participants. Methods Trial Design and Oversight The trial flowchart (Figure 1; eFigure 1 in Supplement 1) describes the factorial design and the allocation of participants to each intervention arm. The design was approved by the ethical review boards of Massachusetts Institute of Technology (MIT) and Stanford, with Massachusetts General Hospital, Yale, and Harvard ceding authority to MIT. All participants provided written informed consent. The trial and the outcomes were registered on ClinicalTrials.gov (NCT04502056). Planned analyses were published on the American Economic Association trial registry (AEARCTR-0006177). The pre-analysis plan and institutional review board–approved protocol are available in Supplement 2. This study followed the Consolidated Standards of Reporting Trials (CONSORT) and American Association for Public Research (AAPOR) reporting guidelines. Participants Individuals were recruited online throughout the United States by the survey company Lucid from August 7, 2020, to September 6, 2020. Lucid recruits survey participants by advertising surveys to third-party suppliers, including double opt-in panels, publishing networks, social media, and other types of online communities. Participants aged 18 years or older, self-identifying as White or Black, and without a college degree were eligible. We focused on these 2 groups because we were interested in tailoring messages toward the Black community as well as the reaction of the White JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 2/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices community to the tailoring of those messages. Latinx individuals were included in our previous study, along with specific tailoring toward this community. Recruitment used quotas to match the 2018 population estimates by age, sex, and race issued by the US Census Bureau. Interventions Intervention vs Control Messages After reading the informed consent form (approved by the institutional review board) (eAppendix 1 in Supplement 1) and agreeing to participate, all individuals answered sociodemographic questions, saw 3 videos, and then completed the outcome survey questions. In the control groups, participants saw 3 placebo videos with generic health topics, including fitness guidelines, recommended sugar intake, and the importance of adequate sleep. In the treatment group, they saw 3 videos on COVID-19, recorded by several physicians of varied age, gender, and race. Participants in each group (placebo and intervention) saw video messages delivered either by a Black or a White study physician (including L.O.-N., M.A., F.C.S., and E.W.). Figure 1. Enrollment and Randomization of Participants 44 473 Assessed for eligibility 24 283 Excluded 14 569 After quotas were met 5534 Declined to participate or failed basic attention checks 4163 Exited survey before randomization 17 Missing race data from platform 20 460 Randomized 16 366 Assigned to intervention group 4094 Assigned to control group 8181 Received statement acknowledging 8185 Received placebo message 2051 Received statement 2051 Received placebo message systemic racism acknowledging systemic racism 4096 Assigned to Black physician, 1023 Assigned to Black physician 4090 Assigned to Black physician, with 2044 receiving message 1026 Assigned to Black physician 1020 Assigned to White physician with 2046 receiving message acknowledging increased 1025 Assigned to White physician acknowledging increased mortality for Black individuals mortality for Black individuals 4089 Assigned to White physician, 4091 Assigned to White physician, with 2046 receiving message with 2045 receiving message acknowledging increased acknowledging increased mortality for Black individuals mortality for Black individuals 676 Exited 691 Exited 162 Exited 169 Exited 7505 Included in knowledge 7494 Included in knowledge 1889 Included in knowledge 1874 Included in knowledge gaps analysis gaps analysis gaps analysis gaps analysis 213 Exited 217 Exited 63 Exited 46 Exited 7292 Included in full analysis 7277 Included in full analysis 1826 Included in full analysis 1828 Included in knowledge gaps and/or full analysis 5792 Excluded from 5755 Excluded from 1442 Excluded from 1441 Excluded from follow-up follow-up follow-up follow-up 3145 Not eligible 3132 Not eligible 800 Not eligible 792 Not eligible 2629 Did not 2610 Did not 637 Did not 644 Did not complete complete complete complete survey survey survey survey 18 Missing 13 Missing 5 Missing 5 Missing baseline baseline baseline baseline 2389 Included in follow-up analysis 2430 Included in follow-up analysis 609 Included in follow-up analysis 602 Included in follow-up analysis JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 3/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices Video 1 defined COVID-19 and discussed common symptoms associated with COVID-19 as well as asymptomatic transmission. Video 2 reminded the viewer that COVID-19 was actively circulating in the United States. Video 3 described the Centers for Disease Control and Prevention social distancing guidelines (complete scripts appear in eAppendix 1 in Supplement 1). Unequal Burden of COVID-19 Video 2 in the COVID-19 intervention had 2 randomized variants. Script 1 emphasized the number of new cases in the week of July 6, 2020. Script 2 added that, controlling for age, Black individuals were 3 times as likely to become infected as White individuals and 4 times as likely to die from it. These 2 variants of video 2 were cross-randomized with the intervention. AMA Antiracism or Placebo Statement At the beginning of the study, all participants saw a video of an actor delivering the script of a public statement by the AMA. The AMA antiracism statement, issued on June 7, 2020, “recognizes that racism in its systemic, structural, institutional, and interpersonal forms is an urgent threat to public health, the advancement of health equity, and a barrier to excellence in the delivery of medical care.” The AMA placebo intervention was an AMA statement on drug pricing. The race and gender of the person reading the statement were randomized to each recipient. Outcomes Most outcomes were measured online immediately following the intervention or the placebo. The prespecified primary outcomes were knowledge, beliefs, and practices related to COVID-19, measured immediately after the intervention; intended behavior, measured immediately after the intervention; and knowledge and behavior, measured a few days after the intervention. eAppendix 2 in Supplement 1 describes the outcome measurement in detail. Primary outcomes presented in the main text include 5 outcomes. First, knowledge gap, which measures knowledge and beliefs. Participants were asked to identify ways to prevent COVID-19 spread and identify 4 common symptoms. The knowledge gap outcome is an integer that can have values from 0 (no error) to 10 (10 errors). Second, information seeking was measured by offering participants the option of requesting additional information on COVID-19–related resources by clicking on up to 5 links that included more content. We measured information-seeking behavior as the number of links in which participants expressed interest, a count variable between 0 (lowest information-seeking behavior) and 5 (greatest information-seeking behavior). Third, self-reported safety behavior was measured a few days after the initial intervention among a subsample that was eligible for follow-up and could be tracked. Participants were asked about how often they engaged in 4 behaviors of interest: (1) whether they wore a mask indoors; (2) whether they wore a mask outdoors; (3) whether they washed their hands; and (4) whether they followed social distancing guidelines. The safety gap index had values of 0 (if a participant reported that they always practiced the 4 behaviors of interest) to 4 (participant reported that they practiced none of the behaviors). Fourth, at the end of the survey, each participant was asked the price they would be willing to pay for high-quality masks. Each participant was entered into a draw to receive either a coupon for masks or a gift card to an online retailer. When a participant was selected by the draw, a price was then randomly drawn for the coupon. If their reported willingness to pay was greater than the price, they would receive the masks; otherwise, they would receive the gift card. Therefore, it was in participants’ best interest to report their true willingness to pay for the masks. This type of procedure has been shown to lead to truthful reporting. We collected data on 3 secondary outcomes specified in our pre-analysis plan (Supplement 2). First, we asked participants to report their judgment of how well federal and state policies balanced opening the economy and limiting the health impacts of COVID-19. Second, we measured how participants prioritized COVID-19 protection vs other issues by asking the participants how they would want to allocate a donation of $1000 (which the research team would fund) between 2 JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 4/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices charities: Give a Mask or the Alzheimer Association. Third, we asked whether they would prioritize a COVID-19 relief donation to a Black-focused charity (the BET COVID-19 relief fund) or a general charity (the Give Directly Project 100+ relief program). Randomization Randomization into intervention vs control and between interventions was stratified according to sex, age (45 years), race, and self-identified partisan affiliation (Republican vs other affiliation). The flowchart (Figure 1) summarizes the randomization; a more detailed version is available as eFigure 1 in Supplement 1. Participants were first randomized into the AMA antiracism or placebo statements, with equal probability. They were then randomly assigned to intervention or control. One of 5 participants was assigned to control; the remainder were assigned to an intervention. Within each group, participants were randomized into Black or White physician groups with equal probability. Intervention participants were further randomized, with equal probability, into 1 the 2 arms for video 2: they either received the information about the unequal burden of disease or did not. Randomization was performed using the Qualtrics platform, using a randomizer block within each stratum with the option to evenly present elements. Statistical Analysis We determined that a sample of 20 000 individuals (10 000 Black and 10 000 White) would provide 85% power to detect effect sizes of 0.05 SDs for intervention relative to control and for effects of specific variations in message content. These are small effect sizes that would justify scale up of this inexpensive intervention. The analysis was performed by original assigned group, and it included all participants who completed the survey. Multivariable regression models include the stratifying variables (age × sex × race × Republican identity dummies). Effect of Any Video Message Intervention Relative to Control To analyze the effect of seeing any video message on the knowledge gap, information-seeking behavior, and safety behavior outcomes, we fit the following negative binomial regression model for the count outcome: log(μ)=β +β intervention +β stratum , i 0 1 i 2 i where μ is the estimated mean outcome value (knowledge gap count, count of demanded links, or safety behavior count), intervention is an indicator that equals 1 if the individual received the intervention videos and 0 if they received the placebo videos, and stratum is a vector of indicator variables. Similar models are estimated for binary regressions (using the logistic regression equation) and continuous variable (using ordinary least squares) (eAppendix 2 in Supplement 1). Because there were multiple outcomes, we provide P values and q values adjusted for false discovery rates. Effect of Variation in the Message Framing To analyze the impact of different arms, we fit a negative binomial regression model to the count data: log(μ)=β +β Black physician +β AMA antiracism +β intervention +(β Black i 0 1 i 2 i 3 i 4 physician × intervention)+(β AMA antracism × intervention)+(β mortality i i 5 i i 6 difference × intervention)+β stratum , i i 7 i where Black physician is an indicator that equals 1 if the physician was a Black individual and 0 otherwise; AMA antiracism is an indicator for the AMA statement featuring the antiracism message (rather than the drug pricing message); mortality difference is an indicator for video 2 mentioning JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 5/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices the excess mortality from COVID-19. Similar specifications are fitted with logistic regression for binary variables and ordinary least squares for continuous variables. To address possible bias stemming from nonrandom attrition for the follow-up survey, we weighted the follow-up data using Hainmueller entropy weights, which ensures that the observed baseline characteristics of the follow-up sample matches the original sample as closely as possible (eAppendix 2 in Supplement 1). eAppendix 3 in Supplement 1 describes further robustness checks and subgroups analysis. Analyses were performed using R version 4.0.3 (R Project for Statistical Computing). Statistical significance was set at P < .05, and all tests were 2-tailed. Results Trial Sample The trial sample was enrolled from August 7 through September 6, 2020. Of 44 743 screened, 30 174 were eligible for participation, 5534 individuals did not consent or failed 2 simple attention checks, 4163 left the survey before randomization, and 17 were excluded from the analysis due to unknown race or multiple survey completion. The final sample at randomization had 20 460 individuals, for a participation rate of 68%. After attrition, 18 223 individuals were included in the study. Summary statistics (Table 1) were computed on the sample that was randomized and that Table 1. Summary of Participant Characteristics Respondents, No. (%) Full sample Intervention group Control group All Black White All Black White All Black White Variable (N = 17 689) (n = 7879) (n = 9810) (n = 14 145) (n = 6303) (n = 7842) (n = 3544) (n = 1576) (n = 1968) Age, mean (SD), y 40.22 (17.81) 34.12 (15.48) 45.12 (18.04) 40.20 (17.83) 34.15 (15.48) 45.07 (18.09) 40.30 (17.73) 34.00 (15.47) 45.35 (17.81) Region Northeast 3024 (17.1) 1187 (15.1) 1837 (18.7) 2424 (17.1) 952 (15.1) 1472 (18.8) 600 (16.9) 235 (14.9) 365 (18.5) Midwest 3884 (22.0) 1494 (19.0) 2390 (24.4) 3114 (22.0) 1207 (19.1) 1907 (24.3) 770 (21.7) 287 (18.2) 483 (24.5) South 8046 (45.5) 4291 (54.5) 3755 (38.3) 6397 (45.2) 3406 (54.0) 2991 (38.1) 1649 (46.5) 885 (56.2) 764 (38.8) West 2735 (15.5) 907 (11.5) 1828 (18.6) 2210 (15.6) 738 (11.7) 1472 (18.8) 525 (14.8) 169 (10.7) 356 (18.1) Demographic characteristics High school 15 016 (84.9) 6125 (77.7) 8891 (90.6) 12 009 (84.9) 4899 (77.7) 7110 (90.7) 3007 (84.8) 1226 (77.8) 1781 (90.5) graduate Household income 4206 (23.8) 1657 (21.0) 2549 (26.0) 3356 (23.7) 1327 (21.1) 2029 (25.9) 850 (24.0) 330 (20.9) 520 (26.4) >$60 000 Female 9880 (55.9) 4492 (57.0) 5388 (54.9) 7907 (55.9) 3595 (57.0) 4312 (55.0) 1973 (55.7) 897 (56.9) 1076 (54.7) Male 7809 (44.1) 3387 (43.0) 4422 (45.1) 6238 (44.1) 2708 (43.0) 3530 (45.0) 1571 (44.3) 679 (43.1) 892 (45.3) Party Democrat 6977 (39.4) 4228 (53.7) 2749 (28.0) 5594 (39.5) 3385 (53.7) 2209 (28.2) 1383 (39.0) 843 (53.5) 540 (27.4) Republican 4376 (24.7) 699 (8.9) 3677 (37.5) 3494 (24.7) 553 (8.8) 2941 (37.5) 882 (24.9) 146 (9.3) 736 (37.4) Independent 6336 (35.8) 2952 (37.5) 3384 (34.5) 5057 (35.8) 2365 (37.5) 2692 (34.3) 1279 (36.1) 587 (37.2) 692 (35.2) Preventive practices Mask in (always) 12 106 (68.4) 5316 (67.5) 6790 (69.2) 9648 (68.2) 4230 (67.1) 5418 (69.1) 2458 (69.4) 1086 (68.9) 1372 (69.7) Mask out (always) 5517 (31.2) 3513 (44.6) 2004 (20.4) 4408 (31.2) 2807 (44.5) 1601 (20.4) 1109 (31.3) 706 (44.8) 403 (20.5) Wash hands 10 779 (60.9) 5084 (64.5) 5695 (58.1) 8688 (61.4) 4090 (64.9) 4598 (58.6) 2091 (59.0) 994 (63.1) 1097 (55.7) (always) Distance (always) 9461 (53.5) 4587 (58.2) 4874 (49.7) 7571 (53.5) 3681 (58.4) 3890 (49.6) 1890 (53.3) 906 (57.5) 984 (50.0) This table presents summary statistics on baseline variables for our main sample of (always) is equal to 1 if the respondent answered “always” to “Wearing a mask outside,” individuals who completed all baseline variables. otherwise it is 0. Wash hands (always) is equal to 1 if the respondent answered “always” b to “Washing your hands with soap and water right away when you come home after The preventive practices variables refer to the question: “What fraction of the time going out.” Distance (always) is equal to 1 if the respondent answered “always” to would you say that you engage in the following behaviors?” Mask in (always) is equal to “Staying at least 6 feet away from people who are not part of your household.” 1 if the respondent answered “always” to “Wearing a mask when you go inside buildings that are not your home / take public transportation,” otherwise it is 0. Mask out JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 6/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices completed all main baseline variables. Our sample included 9880 (55.9%) women. The mean (SD) age was 40.2 (17.8) years, 4206 (23.8%) reported household incomes greater than $60 000, 4228 Black participants (53.7%) and 2749 White participants (28.0%) identified as members of the Democratic party, 12 106 (68.4%) reported always wearing a mask indoors (outside the home), and 5517 (31.2%) reported always wearing a mask when outdoors. Black participants were twice as likely as White participants to report always wearing a mask outdoors (3513 [44.6%] vs 2004 [20.4%]) and were also more likely to report practicing hand hygiene (5084 [64.5%] vs 5695 [58.1%]) and physical distancing (4587 [58.2%] vs 4874 [49.7%]). Baseline covariates were balanced between intervention and control at all stages (eAppendix 5 and eTable 1A in Supplement 1). Attrition Due to the online nature of the survey, participants could exit at any point after watching the video messages without finishing the survey. To include as many participants as possible in the analysis, we included everyone who answered knowledge questions for the knowledge outcome. Overall, 18 762 participants included in the initial randomization were included at that stage (9445 Black; 9317 White). For other outcomes, except follow-up outcomes, we included 18 223 participants (9168 Black; 9055 White) who completed the survey. Attrition was similar in all groups (eTable 1B in Supplement 1). The adherence to safety behavior was collected a few days later among a smaller follow-up sample that experienced more attrition. Overall, 12 591 individuals were included in the follow-up sample to track; only people who had given permission to Lucid to be recontacted were included in this sample. Of those, we successfully contacted 6217. Attrition at this stage was 51.8% in the treatment group and 51.0% in the control group. A systematic analysis of attrition at all 3 stages (eTable 1B in Supplement 1) revealed no systematic difference between the characteristic of attritors in the treatment and comparison groups. At the realized sample size, the power was 0.055 SDs for the knowledge sample (18 762 respondents), 0.056 SDs for all other outcomes in the first survey, and 0.095 SDs for the follow up sample (6217 respondents). Effects of Any Video Message, Intervention vs Control Receiving any COVID-19 video improved knowledge of COVID-19 and adherence to preventive practices. Table 2 shows main outcomes overall and by racial group. Main Sample The knowledge gap incidence rate was 0.241 (95% CI, 0.235-0.246) in the control group and 0.214 (95% CI, 0.211-0.217) in intervention group. The intervention had a significant effect on reducing knowledge gaps relative to the control group (estimated incidence rate ratio [IRR], 0.89 [95% CI, 0.87-0.91]; P < .001; q < .001). In the control group, 315 participants (8.4%) had no gap in knowledge (Figure 2). The proportion increased to 13.0% (1915 participants) in the intervention group; 1352 participants (35.9%) in the control group had a 1-point knowledge gap compared with 6636 (44.2%) in the intervention group. The incidence rate of information seeking behavior was 0.320 in the control group and 0.338 in the intervention group (estimated IRR, 1.05 [95% CI, 1.01-1.11]; P =.03; q = .04). The willingness to pay for a mask increased from $14.07 in the control group to $14.58 in the intervention group (difference, $0.50 [95% CI, $0.15-$0.85]; P = .005; q = .013). The intervention was impactful for both Black and White participants. It was more impactful for White participants vs Black participants on knowledge (IRR, 0.80 [95% CI, 0.76-0.83] vs 0.94 [95% CI 0.91-0.97]; P for difference < .001), but equally impactful for all the other measures (eTable 2C in Supplement 1). The F statistics for the hypothesis that the coefficients across all outcomes are jointly different for both races was 0.0112 (P > .99). JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 7/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 8/13 Table 2. Impact of Any Video Message Intervention vs Control: Incidence Rate Control group Intervention group Mean incidence rate Mean incidence rate Panel Outcome (95% CI) Observations, No. (95% CI) Observations, No. IRR (CI 95%) P value q value Observations, No. All participants Knowledge gap score 0.241 (0.235 to 0.246) 3763 0.214 (0.211 to 0.217) 14 999 0.89 (0.87 to 0.91) <.001 <.001 18 762 Information seeking 0.32 (0.31 to 0.33) 3654 0.338 (0.332 to 0.344) 14 569 1.05 (1.01 to 1.11) .03 .04 18 223 behavior Safety gap score 0.47 (0.45 to 0.48) 1212 0.45 (0.44 to 0.46) 4823 0.96 (0.92 to 1.01) .08 .08 6035 Knowledge gap follow-up 0.25 (0.24 to 0.26) 1211 0.241 (0.238 to 0.244) 4819 0.956 (0.917 to 0.996) .03 .04 6030 Black participants Knowledge gap score 0.316 (0.307 to 0.324) 1892 0.297 (0.292 to 0.301) 7553 0.94 (0.91 to 0.97) <.001 <.001 9445 Information seeking 0.38 (0.36 to 0.40) 1840 0.40 (0.39 to 0.41) 7328 1.06 (1.00 to 1.12) .06 .15 9168 behavior Safety gap score 0.40 (0.36 to 0.43) 416 0.38 (0.36 to 0.40) 1683 0.95 (0.87 to 1.04) .26 .28 2099 Knowledge gap follow-up 0.27 (0.25 to 0.28) 416 0.258 (0.253 to 0.263) 1681 0.97 (0.91 to 1.03) .28 .28 2097 White participants Knowledge gap score 0.165 (0.159 to 0.170) 1871 0.131 (0.128 to 0.133) 7446 0.80 (0.76 to 0.83) <.001 <.001 9317 Information seeking 0.26 (0.25 to 0.28) 1814 0.275 (0.267 to 0.282) 7241 1.05 (0.97 to 1.13) .22 .22 9055 behavior Safety gap score 0.50 (0.48 to 0.53) 796 0.49 (0.48 to 0.50) 3140 0.96 (0.91 to 1.02) .20 .22 3936 Knowledge gap follow-up 0.24 (0.23 to 0.25) 795 0.231 (0.228 to 0.235) 3138 0.95 (0.89 to 1.00) .05 .09 3933 Panel Outcome Mean (95% CI), $ Observations, No. Mean (95% CI), $ Observations, No. Coefficient (95% CI) P value q Value Observations, No. All WTP masks 14.07 (13.76 to 14.38) 3360 14.58 (14.42 to 14.74) 13 399 0.50 (0.15 to 0.85) .005 .01 16 759 Black WTP masks 15.70 (15.22 to 16.18) 1550 16.13 (15.87 to 16.38) 6175 0.42 (−0.14 to 0.97) .14 .24 7725 White WTP masks 12.68 (12.29 to 13.07) 1810 13.26 (13.06 to 13.46) 7224 0.57 (0.12 to 1.01) .01 .01 9034 Abbreviations: IRR, incidence rate ratio; WTP, willingness to pay. fitting a negative binomial regression model with units weighted following Hainmueller entropy-based a weighting to account for imbalances due to attrition for the follow-up outcomes. q values are reported Incidence rate for knowledge gaps is the count of knowledge gaps divided by the maximum possible count (10). accounting for the different outcomes and coefficients in each panel. Incidence rate for interest in links is the count of links demanded divided by the maximum possible count (5). Incidence rate for safety gaps is the count of safety gaps divided by the maximum possible count (4). IRR (or The F statistic for a test equality of the coefficients for the Black participants and White participants (obtained by coefficients) compare the any intervention with the control group. IRRs for safety gap score were estimated by estimating all outcomes in a joint system) was 0.0112 (P > .99). JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices Follow-up Survey At the follow-up survey (Table 2), which was realized on a smaller sample with larger attrition, the safety gap index incidence rate was 0.47 (95% CI, 0.45-0.48) in the control group and 0.45 (95% CI, 0.44-0.46) in the treatment group (IRR, 0.96 [95% CI 0.92-1.01]; P = .08, q = .08). Overall, 244 participants (20.1%) and 218 participants (18.0%) in the control group and 1040 participants (21.6%) and 837 participants (17.4%) in the intervention group reported respecting all and none, respectively, of 4 safety practices (eFigure 2 in Supplement 1). Supplemental Outcomes eTable 3 in Supplement 1 reports effects on secondary outcomes. One noteworthy result is that we found significant effects of the intervention on donations both to a COVID-19 charity (vs Alzheimer) and to a COVID-19 relief charity specific to Black US residents (vs COVID-19 economic relief for everyone). Effects of the Framing of the Videos None of the efforts to make the messages more relevant to the Black community had a differential effect on knowledge or individual behavior (Table 3). An F statistic that all effects for Black physician, AMA antiracism statement, and disproportionate burden statement are jointly different from 0 was 0.324 (P > .99). The only outcome that was affected by some of these variants was the desired donation to a Black-specific COVID-19 charity (eTable 4 in Supplement 1). eTable 5 in Supplement 1 shows that the combination of the AMA antiracism statement, a Black physician, and a video acknowledging racial disparities significantly increased how White and Black participants allocated donations to a charity focused on Black communities (ordinary least squares coefficient, $30.60 [95% CI, $10.93-$50.27]; P = .002). Heterogeneity by Sex, Education, Income, and Political Affiliation Across all conditions, there were no statistically significant differences by sex or political affiliation (eTable 2A and eTable 2E in Supplement 1). The effect of intervention relative to control on knowledge gaps was more pronounced for participants with a high school education or more (eTable 2B in Supplement 1). However, there was no significant difference for the demand for links and willingness to pay for a mask. The intervention was more impactful among participants with lower incomes (ie, <$60 000). Figure 2. Distribution of the Knowledge Gap Score in the Control and Intervention Groups 0.5 Group 0.4 Control Intervention 0.3 0.2 0.1 0 1 2 3 4 5 ≥6 Knowledge gap score Whiskers indicate 95% CIs. JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 9/13 Share JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 10/13 Table 3. Impact of Tailoring Messages: Incidence Rate Ratios Intervention group Control group Black physician AMA antiracism Racial disparity Black physician AMA antiracism Intervention vs control Observations, Panel Outcome IRR (95% CI) P value IRR (95% CI) P value IRR (95% CI) P value IRR (95% CI) P value IRR (95% CI) P value IRR (95% CI) P value No. All participants Knowledge gap 1.01 (0.96 to .66 0.99 (0.94 to .71 1.01 (0.98 to .66 0.99 (0.95 to .78 0.99 (0.95 to .65 0.89 (0.85 to <.001 18 762 score 1.06) 1.04) 1.03) 1.04) 1.03) 0.93) Information 1.04 (0.95 to .42 1.03 (0.93 to .60 1.02 (0.98 to .30 0.96 (0.89 to .38 0.97 (0.89 to .42 1.01 (0.93 to .83 18 223 seeking 1.14) 1.13) 1.07) 1.05) 1.05) 1.10) Safety gap score 1.04 (0.95 to .44 1.03 (0.93 to .74 1.01 (0.97 to .73 0.97 (0.89 to .44 0.95 (0.87 to .20 0.93 (0.86 to .09 6035 1.14) 1.12) 1.05) 1.05) 1.03) 1.01) Knowledge gap 0.98 (0.90 to .65 0.98 (0.90 to .63 1.01 (0.98 to .47 1.01 (0.94 to .72 0.98 (0.91 to .66 0.97 (0.90 to .38 6030 follow up 1.07) 1.07) 1.05) 1.09) 1.06) 1.04) Black Knowledge gap 1.01 (0.94 to .89 1.01 (0.95 to .84 1.001 (0.98 to .55 1.00 (0.94 to .89 0.98 (0.93 to .48 0.93 (0.88 to .009 9445 participants score 1.07) 1.07) 1.04) 1.05) 1.04) 0.98) Information 1.04 (0.92 to .58 1.02 (0.91 to .76 1.02 (0.97 to .45 0.97 (0.84 to .60 0.98 (0.88 to .75 1.02 (0.92 to .71 9168 seeking 1.17) 1.15) 1.08) 1.08) 1.09) 1.14) Safety gap score 1.01 (0.85 to .93 1.03 (0.87 to .72 1.12 (1.04 to .005 0.98 (0.84 to .79 0.87 (0.74 to .08 0.88 (0.76 to .10 2099 1.20) 1.23) 1.21) 1.15) 1.02) 1.02) Knowledge gap 0.97 (0.86 to .66 0.99 (0.87 to .81 1.05 (0.99 to .08 1.02 (0.91 to .78 0.97 (0.87 to .63 0.96 (0.87 to .49 2097 follow up 1.10) 1.11) 1.11) 1.13) 1.09) 1.07) White Knowledge gap 1.03 (0.94 to .55 0.96 (0.88 to .32 1.00 (0.96 to .87 0.99 (0.92 to .74 1.01 (0.94 to .78 0.80 (0.75 to <.001 9317 participants score 1.12) 1.04) 1.04) 1.06) 1.09) 0.87) Information 1.05 (0.90 to .56 1.03 (0.89 to .68 1.03 (0.96 to .46 0.95 (0.83 to .48 0.95 (0.83 to .43 1.00 (0.87 to .96 9055 seeking 1.22) 1.20) 1.10) 1.09) 1.08) 1.14) Safety gap score 1.06 (0.94 to .35 1.01 (0.90 to .89 0.94 (0.89 to .02 0.96 (0.87 to .44 1,00 (0.90 to .93 0.96 (0.87 to .47 3936 1.19) 1.13) 0.99) 1.07) 1.11) 1.07) Knowledge gap 0.99 (0.88 to .81 0.98 (0.87 to .68 0.98 (0.93 to .50 1.01 (0.92 to .81 0.99 (0.90 to .87 0.97 (0.88 to .58 3933 follow up 1.11) 1.09) 1.04) 1.12) 1.10) 1.08) Panel Outcome Coefficient P value Coefficient P value Coefficient P value Coefficient P value Coefficient P value Coefficient P value Observations, (95% CI) (95% CI) (95% CI) (95% CI) (95% CI) (95% CI) No. All WTP masks −0.21 (−0.91 to .56 −0.29 (−0.99 to .42 0.07 (−0.24 to .65 0.17 (−0.45 to .59 0.32 (−0.30 to .31 0.71 (0.09 to .03 16 759 0.49) 0.41) 0.39) 0.80) 0.95) 1.34) Black WTP masks 0.04 (−1.08 to .95 −0.09 (−1.21 to .87 0.001 (−0.498 to >.99 −0.01 (−1.01 to .98 0.15 (−0.84 to .76 0.45 (−0.56 to .38 7725 1.15) 1.02) 0.501) 0.98) 1.15) 1.45) White WTP masks −0.43 (−1.31 to .35 −0.46 (−1.35 to .30 0.13 (−0.26 to .51 0.33 (−0.46 to .41 0.47 (−0.32 to .24 0.94 (0.15 to .02 9034 0.46) 0.42) 0.53) 1.12) 1.26) 1.74) Abbreviations: IRR, incidence rate ratio; WTP, willingness to pay. regression for WTP masks) following the second equation in the main text. IRRs for follow-up outcomes were a calculated from estimates obtained by fitting a negative binomial regression model with units reweighted The test statistics for the hypothesis that all the interaction coefficients are jointly 0 across equations was 0.324 following Hainmueller entropy-based reweighting to account for imbalances due to attrition. (P > .99). Estimates in each row came from a single negative binomial regression (or ordinary least squares JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices Discussion Exposure to public health video messages about COVID-19 recorded by a diverse set of physicians decreased knowledge gaps on COVID-19 symptoms, preventive behaviors, and transmission among Black and White participants with modest incomes, relative to a control condition that saw placebo videos. The effect on knowledge was substantial and clear. This replicates the results of our prior study conducted in May 2020 and extends it to White participants. New to this study, we also found a modest but statistically significant increase in the demand for more information, the willingness to pay for high quality masks, and self-reported behavior at follow-up. Despite the heightened awareness of racial justice issues during the period of this intervention and increased polarization in the political discourse in the run-up to the presidential election, effects are remarkably similar across racial and political lines. These results suggest that physicians still have the ability to inform and persuade members of society from a broad range of backgrounds. Our results also indicate that tailoring the message to specific communities did not affect its impact on behavior. Both White and Black physicians were able to effectively convey the importance of masking and social distancing to Black and White participants (unlike the previous study, in which concordance was important to change behavior ). The AMA antiracism statement did not affect participants’ attentiveness to the message delivered, for Black or White respondents. Acknowledgment of structural racism remains important, but it may not be sufficient to increase the level of trust from the Black community. Importantly, the intervention made both Black and White participants more willing to focus resources both toward COVID-19 in general and toward the Black community in particular. Highlighting health conditions that disproportionately affect the Black community is one step toward increasing public consciousness of structural racism. Limitations There are several limitations of the study. First, it was conducted online, and the participants may not be representative of the population with less than a college degree, given that they have access to the internet and are used to participating in online studies. Second, although information-seeking behavior and willingness to pay for masks were objectively measured, participants’ preventive health behaviors were not directly observed. Outcomes were self-reported. Third, outcomes might be subject to social desirability bias. Fourth, there may be bias due to attrition, particularly for the self- reported safety behavior, given that only a small fraction of the sample could be followed up a few days after the initial intervention. While the observable variables remain balanced, the unobservable may not be. Furthermore, while we found consistent effects on knowledge, information seeking, the willingness to pay for masks, and self-reported behavior, the final clinical significance of these findings is uncertain because effects on all were quantitatively small. Conclusions These results suggest physician messaging campaigns may be effective and trust in Black and White physicians is equally high. There is no evidence of preexisting bias that would have led the intervention to have a negative effect. Because it is inexpensive, it could be a promising way to encourage behavior at scale. However, future studies implemented at a large scale are needed to confirm whether these kinds of interventions can change behavior in a way that will affect clinical outcomes. In ongoing work, we will study scale up messaging by doctors using social media. JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 11/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices ARTICLE INFORMATION Accepted for Publication: May9,2021. Published: July 14, 2021. doi:10.1001/jamanetworkopen.2021.17115 Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2021 Torres C et al. JAMA Network Open. Corresponding Author: Esther Duflo, PhD, Department of Economics, Massachusetts Institute of Technology, 77 Massachusetts Ave, Building E52-544, Cambridge, MA 02139 (eduflo@mit.edu). Author Affiliations: Harvard Medical School, Boston, Massachusetts (Torres, Ogbu-Nwobodo, Stanford, Warner); Department of Pediatrics–General Pediatrics, Massachusetts General Hospital for Children, Boston (Torres); Department of Psychiatry, Massachusetts General Hospital, Boston (Ogbu-Nwobodo, Warner); Department of Psychiatry, McLean Hospital, Belmont, Massachusetts (Ogbu-Nwobodo); Harvard Kennedy School of Government, Cambridge, Massachusetts (Alsan); Department of Medicine–Neuroendocrine Unit, Department of Pediatrics–Endocrinology, Massachusetts General Hospital, Boston (Stanford); Department of Economics, Massachusetts Institute of Technology, Cambridge (Banerjee, Karnani, Olken, Vautrey, Duflo); Department of Economics, Harvard University, Cambridge, Massachusetts (Breza); Department of Economics, Stanford University, Stanford, California (Chandrasekhar); Department of Economics, Ludwig Maximilian University of Munich, Munich, Germany (Eichmeyer); Paris School of Economics, Paris, France (Loisel); Yale School of Management, New Haven, Connecticut (Goldsmith-Pinkham); Clinical Translational Epidemiology Unit, Massachusetts General Hospital, Boston (Warner). Author Contributions: Dr Duflo had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Concept and design: Torres, Ogbu-Nwobodo, Alsan, Banerjee, Breza, Chandrasekhar, Eichmeyer, Goldsmith- Pinkham, Olken, Vautrey, Duflo. Acquisition, analysis, or interpretation of data: Ogbu-Nwobodo, Stanford, Banerjee, Breza, Chandrasekhar, Karnani, Loisel, Goldsmith-Pinkham, Olken, Vautrey, Warner, Duflo. Drafting of the manuscript: Ogbu-Nwobodo, Alsan, Stanford, Chandrasekhar, Loisel, Olken, Vautrey, Warner, Duflo. Critical revision of the manuscript for important intellectual content: Torres, Ogbu-Nwobodo, Stanford, Banerjee, Breza, Eichmeyer, Karnani, Goldsmith-Pinkham, Olken, Warner. Statistical analysis: Breza, Chandrasekhar, Karnani, Loisel, Goldsmith-Pinkham, Olken, Vautrey, Duflo. Obtained funding: Breza, Goldsmith-Pinkham, Duflo. Administrative, technical, or material support: Torres, Alsan, Stanford, Breza, Karnani, Goldsmith- Pinkham, Warner. Supervision: Torres, Ogbu-Nwobodo, Breza, Vautrey, Duflo. Conflict of Interest Disclosures: Dr Olken reported receiving ad credits from Facebook outside the submitted work. No other disclosures were reported. Funding/Support: This work was supported by grant 2029880 from the National Science Foundation to Drs Alsan and Duflo, the Physician/Scientist Development Award from the Executive Committee on Research at Massachusetts General Hospital to Dr Stanford, grant P30 DK040561from the National Institutes of Health to Dr Stanford, grant L30 DK118710 from the National Institutes of Health to Dr Stanford, and the Clinician-Teacher Development Award from the Massachusetts General Physicians Organization at Massachusetts General Hospital to Dr Torres. Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. Group Members: The members of the COVID-19 Working Group appear in Supplement 3. Disclaimer: The findings and conclusions expressed are solely those of the authors and do not represent the views of their funders. Data Sharing Statement: See Supplement 4. Additional Contributions: We thank the Center for Diversity and Inclusion, especially Sandra Pena Ordonez, BS, and Elena Olson, JD for their assistance in this project. We thank Minjeong Joyce Kim, BS (Stanford University), and Sirena Yu (Massachusetts Institute of Technology), for their research assistance. These individuals were not compensated for their time. JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 12/13 JAMA Network Open | Infectious Diseases Effect of Physician-Delivered Messages on Adults’ COVID-19 Knowledge, Beliefs, and Practices REFERENCES 1. Zhao J, Lee M, Ghader S, Younes H, Darzi A, Xiong C, Zhang L. Quarantine fatigue: first-ever decrease in social distancing measures after the COVID-19 pandemic outbreak before reopening United States. Preprint updated June 11, 2020. Accessed June 8, 2021. https://arxiv.org/abs/2006.03716 2. McDonnell Nieto del Rio G. Doctors plead with Americans to take the virus surge seriously. The New York Times. Accessed June 8, 2021. https://www.nytimes.com/live/2020/11/15/world/covid-19-coronavirus#doctors-plead- with-americans-to-take-the-virus-surge-seriously 3. Alsan MS, Stanford FC, Banerjee A, et al. Comparison of knowledge and information-seeking behavior after general COVID-19 public health messages and messages tailored for Black and Latinx communities: a randomized controlled trial. Ann Intern Med. 2021;174(4):484-492. 4. US Centers for Disease Control and Prevention. Risk for COVID-19 infection, hospitalization, and death by race/ ethnicity. Updated May 26, 2021. Accessed January 21, 2021. https://www.cdc.gov/coronavirus/2019-ncov/covid- data/investigations-discovery/hospitalization-death-by-race-ethnicity.html 5. American Medical Association. AMA Board of Trustees pledges action against racism, police brutality. June 7, 2020. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-board-trustees- pledges-action-against-racism-police-brutality 6. US Census Bureau. 2018 Population estimates by age, sex, race and Hispanic origin. June 19, 2019. Accessed June 19, 2020. https://www.census.gov/newsroom/press-kits/2019/detailed-estimates.html 7. American Medical Association. AMA applauds congressional action on prescription drug transparency. April 9, 2019. Accessed June 8, 2021. https://www.ama-assn.org/press-center/ama-statements/ama-applauds- congressional-action-prescription-drug-transparency 8. Wertenbroch K, Skiera B. Measuring consumers’ willingness to pay at the point of purchase. J Marketing Res. 2002;39(2):228-241. doi:10.1509/jmkr.39.2.228.19086 9. Hainmueller J. Entropy balancing for causal effects: a multivariate reweighting method to produce balanced samples in observational studies. Political Analysis. 2012;20(1):25-46. doi:10.1093/pan/mpr025 SUPPLEMENT 1. eAppendix 1. Survey Design and Videos eAppendix 2. Supplementary Methods eAppendix 3. Robustness Checks and Subgroup Analysis eFigure 1. Full Study Flowchart eFigure 2. Distribution of the Safety Gap Score in the Control and Intervention Groups eTable 1. Balance and Attrition eTable 2. Outcomes by Subgroup eTable 3. Effects of Any Message Intervention: Effects on Additional Outcomes eTable 4. Effects of Tailoring Messages on Additional Outcomes eTable 5. Effects of All Black Treatments on Outcomes SUPPLEMENT 2. Pre-analysis Plan and Trial Protocol SUPPLEMENT 3. Nonauthor Contributors SUPPLEMENT 4. Data Sharing Statement JAMA Network Open. 2021;4(7):e2117115. doi:10.1001/jamanetworkopen.2021.17115 (Reprinted) July 14, 2021 13/13 (Round 2) Pre-Analysis Plan COVID-19 Health Messaging to Underserved Communities This Draft: September 4, 2020 1. Introduction The aim of this study is to build on results from our ﬁrst experiment and test how acknowledging institutional racial injustice affects informational messaging. In particular, we will investigate how well study participants retain knowledge and update beliefs and behavior with respect to COVID-19. We will also test the effect of race concordance of providers with recipients, and whether highlighting the unequal burden of the disease has additional effects on knowledge, beliefs and behavior regarding COVID-19. Compared to the ﬁrst study, there are four additional changes to the experimental design of note: 1) the sample will include white respondents in addition to African American respondents; 2) the group that does not view videos about COVID-19 will watch a set of placebo videos about non-COVID health topics; 3) we are collecting follow-up behavioral data from a subset of participants; 4) this study does not include a few of the treatment variants from the ﬁrst – speciﬁcally, there is no acknowledgment by doctors in the videos discussing health behaviors, and there is no treatment arm to alter perceptions of whether mask-wearing is socially-acceptable. 2. Treatments, and Experimental Protocols 2.1 Treatments Each subject receives one AMA statement and then watches three videos pertaining to health. The AMA statement either addresses racial injustice (treatment [RI]) or drug pricing (placebo [DP]). The videos pertaining to health behavior either discuss COVID-19 and prevention (treatment) or other non- COVID-19 related issues (control). • AMA Statement: RI: Racial Injustice DP: Drug Pricing • Treatment Videos We continue to focus our sample on individuals with less than college education. We are also collecting baseline measures of these behaviors. 1 1. Video T1: – Introduction – Discussion of symptoms 2. Video T2: – Information about social distancing and hygiene – Sub-treatment T2A: Acknowledgment of racial disparities in contagion and mortality. 3. Video T3: – Information about masks • Placebo Videos 1. Video C1: – Information about ﬁtness routines 2. Video C2: – Information about sleep hygiene 3. Video C3: – Information about sugar intake We are varying ﬁve aspects of the videos across treatments: 1. AMA statement regarding racial injustice or transparency in drug pricing. 2. Racial concordance with AMA spokespeople in the videos. 3. Racial concordance with partner doctors in the videos. 4. Informative videos about COVID-19 or non-COVID-19 issues. 5. Video T2 content: • Standard message about hygiene and social distancing • Previous message, plus acknowledgment of racial disparities in contagion and mortality. All of the video scripts can be found in Appendix Section A. Figure 1 shows how these different treat- ments are incorporated in the randomization design. 2.2 Recruitment and Sampling We are using Lucid to recruit a sample and to compensate subjects for their participation. Our total target sample size is 20,000 subjects, 10,000 African American respondents, and 10,000 White American respondents. We require all participants to be age 18 or older, and we are targeting individuals who have not completed a college degree. 2 Figure 1: Treatment Design Note: There are 5 randomization instances summarized in this diagram. For the fully extended tree with all 24 ﬁnal individual cells, see Figure 2. 2.3 Experimental Protocols and Treatments Our experiment has the following structure: 1. Recruitment and Baseline survey (a) Recruit target sample via Lucid (b) Collect demographic information, political views, and mask ownership. Several demograph- ics variables are being collected directly by Lucid. 2. Randomize subjects to treatment • Treatment scripts are tailored to each racial group • Within racial group, treatment is stratiﬁed on age, gender and geographic region. • Randomization is at the individual level according to Figure 1. • There will also be a control group who will not receive COVID-19 videos, but will receive information at the very end of the survey. 3. Video delivery for all individuals. (a) After each of the three videos, respondents evaluate the video’s usefulness, and trustwor- thiness, along with the respondents intention to follow the advice in the video and share information from the video. 4. Endline survey 5. Debrief script 3 6. Follow-up survey (a) We will follow up with participants three days post-treatment. (b) The chief purpose would be to measure health-seeking and health-preserving behaviors. 3. General Hypotheses to be Tested This section lists the general hypotheses we want to test. For those, we are planning to run broadly pooled treatments (and speciﬁc interactions speciﬁed below). Main hypotheses: 1. Baseline treatment effect: Does any COVID-19 message affect knowledge/indended behavior/priorities in donations/actual behavior? (a) Compare the effectiveness of any doctor treatment video vs. any doctor placebo video. • We will test this in the full sample and in the white and African American samples, sepa- rately. 2. Concordance effect: Does the concordance of the doctor in the COVID-19 video affect the impacts of the message? (a) Within the treatment group receiving a COVID-19 doctor video, compare the effectiveness of the video if the respondent is randomized to a doctor of concordant vs. discordant race. • We will test this hypothesis separately in the white and African American samples. If the effects go in the same direction for both populations, we will also test this hypothesis in the full sample. • An alternate way to deﬁne concorance is whether the subject views a racially concordant messenger both (or either) in the COVID-19 doctor video and in an AMA statement of any kind. We will also test the effectiveness of the messaging under this alternate measure. 3. AMA acknowledgement: Does the AMA message mediate the impacts of the COVID-19 messag- ing or of the racial concordance of the messenger? (a) Compare baseline treatment effect [1] for AMA acknowledgement vs AMA placebo. (b) Compare concordance effect [2] for AMA acknowledgment vs AMA placebo. Is the concor- dance of the doctor in the COVID-19 video more or less important when there is an AMA message vs. the AMA placebo? • We will test these hypotheses separately for white and African American respondents. 4. Does the racial concordance of the AMA messenger matter? (a) Compare whether AMA acknowledgement [3a] is more effective if the AMA messenger is racially concordant. 4 (b) Compare AMA acknowledgement effect [3b] by racial concordance of AMA messenger. Is the concordance of the doctor in the COVID-19 video more or less important when there is a racially concordant AMA message vs. a non-racially concordant AMA message? • We will test these hypotheses separately for white and African American respondents. 5. Content of message: does the mention of the racial disparities (video 2RD) in disease burden affect knowledge/intended behavior/ priorities in donation/actual behavior? (a) Within the COVID video treatment group, compare videos with RD information vs. no RD information. (b) Within the COVID video treatment group, how does information about RD interact with the racial concordance of the doctor messenger? Does information about RD make racial concodance more important, especially among African American respondents? (c) Within the COVID video treatment group, how does information about RD interact with the AMA racial injustice message? Does information about RD make the AMA acknowledge- ment more effective, especially among African American respondents? (d) We are also interested in how information about RD mediates Hypotheses 3b, 4a, 4b, and 4c. • In addition to the standard outcomes, we are also particularly interested in impacts on dona- tions to a COVID fund that is speciﬁcally for African Americans vs. a general COVID fund. We also are interested in how this information changes perceptions of the policy responses of state and federal governments. • For 5a, we are interested in the white and African American groups, separately, and also the full, pooled sample. For 5b and 5c, we are mainly interested in the effects separately for white and African American respondents. • RD should have the largest impacts for individuals who have incorrect beliefs. Moreover, the impacts of the information should be different for individuals with priors about relative impacts on African Americans that are too low versus too high. (See Section 5.4, below). 4. Data Collection and Outcomes We will run our experiment beginning on August 7, 2020. We are recruiting individuals that ﬁt our screening criteria from an online survey ﬁrm. The data will be collected from respondent surveys. There is a short baseline module before individuals are exposed to the videos. Following the ﬁnal video, there is an endline outcome module that all participants complete. Respondents also evaluate each video immediately after watching it. We also plan to follow up with individuals a few days after the treatment delivery for an additional survey to measure whether the videos changed behaviors. Prior to treatment, we ask all participants whether they would be willing to answer the follow-up survey. We allow individuals to enroll in the study, independent of their follow-up availability. We plan to control for willingness to answer a follow-up in all regressions. All of the survey responses will be downloaded in a .csv ﬁle for cleaning and analysis in R. 5 4.1 Baseline Survey Variables The baseline information for all respondents includes the following demographic characteristics: • Education • Age • Location • Sex • Political Views We elicit additional health information at baseline, but participants may choose not to answer these questions: • Mask Ownership • Public Excursions • Health Behavior See Section 5, below for a summary of our key endline survey variables. 4.2 Data quality checks We have several questions in the survey that capture respondent attention. We use some of these questions to screen out low-attention survey takers before assigning treatment. We further plan to exclude those that take very little time on the survey overall and/or on the videos. 5. Empirical Analysis 5.1 Balance Checks We will conduct a series of balance tests across treatment arms to ensure that there are no chance differ- ences between subjects in the various arms. We will regress characteristics measured pre-treatment on indicators for the arms and test their individual and joint signiﬁcance. Balance tests will be conducted using all of the variables measured in the baseline survey. We will also test for balance in attrition rates (see Section 6.2, below). 5.2 Key Outcomes We intend to measure the treatment effects on the following set of outcomes: 1. The number of participants with knowledge of COVID related symptoms and transmission as assessed by a questionnaire we’ve developed speciﬁcally relevant to the intervention videos we are using for the project. Speciﬁcally, we deﬁne the following three outcomes, measured both at ﬁrst contact and followup: 6 • Knowledge of COVID symptoms: participants are asked to identify 4 symptoms of COVID from a list of 9. We will deﬁne an indicator for whether participants select the 3 most common symptoms (fever, cough, difﬁculty breathing). • Knowledge of COVID prevention: participants are asked to identify 3 COVID prevention behaviors from a list of 7. We will deﬁne an indicator for whether participants select the 3 behaviors emphasized in the videos (staying outside and 6ft from others; washing hands when going and coming from home; mask wearing). • Knowledge of asymptomatic infection: an indicator for whether participants correctly an- swer that COVID can spread asymptomatically. 2. Behavioral outcome 1: Number of participants who report behavior change related to messages provided in the intervention videos; measured via a speciﬁc questionnaire instrument we’ve de- veloped to correspond to the intervention. Speciﬁc behaviors include physical distancing, mask wearing, and hand hygiene. Since we ask about several safety behaviors, we will deﬁne a stan- dardized index to combine them into one measure, calculated at both ﬁrst contact and endline. The key outcome is the within-person change in this safety index from ﬁrst contact to endline. 3. Behavioral outcome 2: Revealed preference estimate of willingness to pay for masks. The subjects will trade-off the willingness to get a pair of masks or an amazon gift card when participating in a strategy-proof lottery. 4. Behavioral outcome 3: Number of links people click on for additional information on the COVID- 19 behaviors. Links will include testing locations, state public health hotline, and symptom tracker. 5. Behavioral outcome 4: Donations to a COVID-related charity. After providing information on the number of weekly COVID cases, we are measuring the willingness to donate to a COVID-related charity vs. a generic helath-related charity. 6. Behavioral outcome 5: Donations to African-American COVID-19 fund. After providing infor- mation on the disproportionate burden on African-American communities, we are measuring the willingness to donate to a COVID relief fund that focuses on African-American communities vs. one that generically provides relief to disadvantaged individuals. 7. Evaluation of state and local COVID policy. We are measuring how how well participants think their federal and state governments managed to balance opening the economy and limiting the health impacts of Covid-19. We believe that information about racial disparities, in particular, may change these perceptions, especially for people who believed that racial disparities did not exist or were less severe at baseline. 5.3 Regression Analysis We will perform different regression analyses to test the hypotheses listed above. Because our data contains many possible control variables, we will use a double-lasso procedure to select regression controls. We will also include a control for whether the participant’s availability to participate in a 7 follow up survey (as indicted at baseline). These control variables are denoted as X in the regression speciﬁcations below. Unless otherwise noted, we will examine treatment effects on knowledge of COVID symptoms, knowl- edge of COVID transmission, intended donation to a COVID-related charity, willingness to pay for masks, changes in reported behavior. In what follows, we present the minimal regressions to test each of the hypotheses, restricting to the smallest subset of treatments. However, we could also execute the same tests in the full sample, but with more treatment interactions. Note that the “COVID” indicator covers cases when doctors mention COVID plus racism, or COVID alone. In text, we refer to this group as the treatment group. 1. Baseline treatment effect: does any of the COVID-19 messaging affect knowledge, intended behavior, priorities in donation, or actual behavior? • Question: does COVID messaging from doctors have any effect? Samples: separate analysis for all respondents, black respondents, and white respondents. Regression: Y = β · COVID Video + X γ + ǫ 2. Does racial concordance of the doctor in the COVID-19 video change the effectiveness for messaging? • Question: Bolded, immediately above. Samples: COVID video groups. Separate analysis for black and white respondents. Regression: Y = β · DocConcord + X γ + ǫ Alternate Speciﬁcation: To allow, for a level effect of doctor concordance, we can also run the following modiﬁed regression, including both the COVID video and control groups. Y = β · DocConcord · COVID Video + α · COVID Video + δ · DocConcord + X γ + ǫ • Question: Bolded above, but using an alternate deﬁnition of concordance. We’ll require a) all messengers or b) any messenger to concord with the respondent’s race. For any outcomes where β is of the same sign for white and black respondents, we will estimate the above equation again in the full sample. 3. Does the AMA racism acknowledgment affect the impact of messaging on any outcomes? Does it change the concordance effect? In some cases, especially when including controls, these full-sample tests may be preferrable to these stripped-down regressions. 8 • Question: Does the AMA racism messaging heighten or dull COVID messaging from doc- tors? Samples: COVID video groups. Separate analysis for all respondents, black respondents, and white respondents. Regression: Y = β · AMARacism + X γ + ǫ Alternate Speciﬁcation: To allow, for a level effect of the AMA racism message, we can also run the following modiﬁed regression, including both the COVID video and control groups. Y = δ · AMARacism · COVID Video + α · COVID Video + β · AMARacism + X γ + ǫ • Question: Does the AMA racism acknowledgment heighten or dull any doctor concordance effects? Samples: COVID video groups. Separate analysis for black and white respondents. Regression: Y = δ · AMARacism · DocConcord + α · AMARacism + β · DocConcord + X γ + ǫ Alternate Speciﬁcation: Again, we can also run the following modiﬁed regression, including both the COVID video and control groups. Y = δ · DocConcord · AMARacism · COVID Video + λ · AMARacism · COVID Video +φ · DocConcord · COVID Video + ρ · AMARacism · DocConcord + α · COVID Video +β · AMARacism + ψ · DocConcord + X γ + ǫ 4. Does concordance of the AMA messenger matter? • Question: Does a race-concordant AMA messenger delivering a message about racial injus- tice make COVID messaging from doctors more or less effective, relative to a race-discordant AMA messenger? Samples: Individuals receiving an AMA message about racial injustice and a treatment mes- sage about COVID. Separate analysis for black and white respondents. Regression: Y = β · AMAConcord + X γ + ǫ Alternate Speciﬁcation: We can also run the following modiﬁed regression, the COVID video and control groups, with both types of AMA messages. Y = δ · COVID Video · AMAConcord · AMARacism + λ · AMAConcord · AMARacism +ρ · AMAConcord · COVID Video + ψ · COVID Video · AMARacism +α · AMAConcord + β · AMARacism++φ · COVID Video + X γ + ǫ Ex ante, we think that an AMA level effect in the control group is more likely than a doctor concordance level effect. However, we include all alternates for completeness. These alternate speciﬁcations including the control group are relevant in the presence of a level effect. 9 • Question: Does a race-concordant AMA messenger make COVID messaging from a race- concordant doctor more or less effective, when delivering the AMA message about racial injustice? Samples: COVID video groups receiving the AMA message about racism. Separate analysis for black and white respondents. Regression: Y = δ · AMAConcord · DocConcord + α · AMAConcord + β · DocConcord + X γ + ǫ Alternate Speciﬁcation: We can also run the following modiﬁed regression, the COVID video and control groups, with both types of AMA messages. Y = δ · AMAConcord · AMARacism · DocConcord · Covid Video +α · DocConcord · AMARacism · COVID + α · DocConcord · AMARacism · AMAConcord 1 2 +α · DocConcord · COVID · AMAConcord + α · AMARacism · COVID · AMAConcord 3 4 +β · DocConcord · AMARacism + β · DocConcord · COVID + β · DocConcord · AMAConcord 1 2 3 +β · AMARacism · COVID + β · AMARacism · AMAConcord + β · COVID · AMAConcord 4 5 6 +ρ · DocConcord + ρ · AMARacism + ρ · AMAConcord + ρ · COVID + X γ + ǫ 2 3 1 4 5. What are the effects of acknowledging racial disparities in COVID incidence? In addition to the typical set of outcomes, we will also analyze effects on allocated donations to black-speciﬁc versus race-agnostic COVID-related charities. • Question: what is the main effect of acknowledging racial disparities of COVID incidence? Samples: treated respondents only. Separate analysis for black, white, and all respondents. Regression: Y = β · Vid2RacialDisp + X γ + ǫ • Question: are race-concordant doctors more effective messengers about racial disparity? Samples: treated respondents only. Separate analysis for black, white, and all respondents. Regression: Y = δ · DocConcord · Vid2RacialDisp + α · Vid2RacialDisp + β · DocConcord + X γ + ǫ • Question: does acknowledging widespread racism alter the effectivness of later discussing racial disparities in COVID? Samples: treated respondents only. Separate analysis for black, white, and all respondents. Regression: Y = δ · AMARacism · Vid2RacialDisp + α · Vid2RacialDisp + β · AMARacism + X γ + ǫ 10 • Question: does a race-concordant AMA messenger’s prefacing statement have a different amplifying effect than a race discordant AMA messenger? That is, if doctors are going to discuss racial disparities in COVID and we’re going to preface this with the AMA racism statement, do we expect different results if the statement comes from a concordant versus discordant AMA messenger? Samples: Must have seen both a COVID video and an AMA racism video. Separate analysis for black and white respondents. Regression: Y = δ · AMAConcord · Vid2RacialDisp + α · Vid2RacialDisp + β · AMAConcord + X γ + ǫ Alternate speciﬁcation: restrict sample to those seeing a COVID video, include both types of AMA messages. Regression: Y = δ · AMARacism · AMAConcord · Vid2RacialDisp + λ · AMAConcord · Vid2RacialDisp +ρ · AMAConcord · AMARacism + φ · AMARacism · Vid2RacialDisp +α · Vid2RacialDisp + β · AMAConcord + ψ · AMARacism + X γ + ǫ 5.4 Heterogeneous Effects We plan to conduct several heterogeneity tests that we believe are of central importance: respondent race, respondent political aﬁliation (within the white sample), prior beliefs about racial disparities and COVID-19, and the timing of participation vis a vis the events of Kenosha, WI. We are very interested in studying how the impacts of our treatments vary by the race of the respondent. This is central to our research design. Speciﬁcally, • Is the impact of racial concordance different by respondent race? • Is the impact of a statement addressing racial injustice different by respondent race? • Is the impact of information about racial disparities in the COVID-19 burden different by respon- dent race? Moreover, within the white respondent population, we predict that there may be substantial hetero- geneity by the respondent’s political beliefs: • Is the impact of racial concordance different for white republicans versus white democrats? • Is the impact of a statement addressing racial injustice different for white republicans versus white democrats? • Is the impact of information on racial disparities in disease burden different for white republicans versus white democrats? We predict that the impacts of information on racial disparities in COVID burden should depend on individuals’ prior beliefs: 11 • The information about RD should cause individuals who initially believed at baseline that there were small or non-existant racial disparities to update in the opposite direction of individuals who believed that the racial disparities were larger than they actually are. • Thus, for all of our tests involving RD, we will interact the regressions with indicators for whether the priors were smaller or larger than the number we give in the videos. • Moreover, there may be bigger impacts for individuals whose priors were less accurate, so we can also interact by the size of the gap between the informaiton and the prior, separately for those with priors that were too low versus too high. This can also help us measure whether the information had an impact even for people with accurate priors, possibly through a salience effect. We intend to compare the results of our hypothesis tests separately for the time period before the Jacob Blake police shooting in Kenosha, WI on August 23, 2020, and the time period after. We propose to do this for two reasons. First, we launched the study at a time when the large-scale protests from earlier in the summer following the murder of George Floyd had somewhat ebbed. So the events in Kenosha may bring issues of racial injustice to the fore. Second, and perhaps more importantly, polarization surrounding the narrative of the protests has markedly increased following the events of Kenosha. Several speakers at the Republican National Convention explicitly discussed the violent component of the protests, for example, and both candidates for the US presidency are making trips to Kenosha. This increased polarization is likely to have the most relevance for the white respondents in our study, and may enhance any differential response we ﬁnd by political afﬁliation. We propose to split the sample into the period before August 23, 2020 and the period following August 26, 2020. We will omit surveys collected on days in the interim when Americans were only coming to learn about the events of Kenosha, WI. We are also interested in secondary analysis exploring heterogeneity on the following categories of traits/characteristics: • Age • Level of baseline knowledge and health-preserving behaviors • Place of residence (correlated with political afﬁliation, COVID-19 policies and phased reopenings, and socio-economic characteristics) Given the many ways to cut the data for this secondary analysis, we will follow the methodology of Chernozhukov et al (2019) for this latter set of potential heterogeneous treatment effects. 6. Robustness 6.1 Threats to Interpretation We would like to assume that differences across videos come from either differences in the racial iden- tity of the doctors in the video or from differences in the content of the messages, rather than from other chance differences across videos. Because we are including both white and black respondents who will be watching the same exact videos, we will be able to include video ﬁxed effects in some speciﬁcations. Chernozhukov, Victor, Mert Demirer, Esther Duﬂo, and Ivan Fernandez-Val (2019). Generic machine learning inference on heterogenous treatment effects in randomized experiments. No. w24678. National Bureau of Economic Research. 12 6.2 Attrition We have two separate endline surveys. The ﬁrst will take place immediately after treatment delivery. The type of attrition that might arise here is through dropping out of the online session before complet- ing all of the survey questions. To try to limit differential attrition, we are showing placebo videos to the control group to ﬁll approximately the same amount of time. Our second set of endline outcomes will take place a few days after the main survey. Lucid, the survey ﬁrm, will try to recontact a speciﬁed list of initial participants. We are only expecting modest recontact rates, and therefore high levels of attrition. Importantly, to try to limit differential attrition, all individ- uals will be recontacted with the exact same message, and it will not be made salient that the survey is a direct follow-up to the previous study. We plan to test for differential attrition at both endlines across our key comparison groups. 7. Funding and Human Subjects Review Funding is provided by the National Science Foundation RAPID-2029880 for Covid-19 research, and RAI Italian Broadcasting corporation (via an unrestricted gift to J-PAL that we attributed to this project). The IRB at MIT is serving as the primary institution of record and has entered into a reliance agreement with Harvard, Massachusetts General Hospital, and Yale. We have also received IRB approval from Stanford. 13 Appendix A. Scripts All respondents will receive either [Statement RI or Statement DP] + one set of [Treatment or Control] videos. A.1 Statements Each respondent is assigned to one of the following statements. All respondents will see the statement presented via video. A.1.1 Treatment Statement RI (Racial Justice): • The American Medical Association recognizes that racism in its systemic, structural, institutional, and interpersonal forms is an urgent threat to public health, the advancement of health equity, and a barrier to excellence in the delivery of medical care. • The American Medical Association opposes all forms of racism. • The American Medical Association denounces police brutality and all forms of racially-motivated violence. • The American Medical Association will actively work to dismantle racist and discriminatory poli- cies and practices across all of health care. A.1.2 Placebo Statement DP (Drug Pricing): • The American Medical Association believes in transparency in prescription drug pricing, and we are pleased the House Ways & Means Committee moved the issue forward. • Patients and their physicians want to be armed with more information, yet the current situation is opaque if not impenetrable. • The committee is rightfully determined to expose factors that lead to high drug prices, and we look forward to continuing our efforts in that regard. A.2 Treatment Videos about COVID-19 A full set of treatment videos includes T1 + T2 + [T2A or nothing] + T3 Video T1: Hello, I’m Dr [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL AFFILIATION HERE], and I?d like to tell you a little about Coronavirus or COVID-19. COVID-19 is a new virus that can infect the respiratory tract and lungs. Although many people who get sick from COVID will get better, some people who get it become very ill and some even die. 14 Although there’s no cure, there are ways medical professionals have found to protect you and your community from COVID. I hope that this message can give you information that will help you protect you or someone you love from COVID infection. First, I would like to tell you about the symptoms of COVID-19. The most common symptoms of COVID-19 are cough, fever, and trouble breathing. Another odd symptom some people have is loss of taste or smell. A large number of people who have COVID-19 actually don’t show any symptoms at all. Unfortunately, people can still spread the disease to others even with no symptoms. The next video will provide you with more information on how you can protect yourself and others. Video T2: Hello, I’m Dr [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL AFFILIATION HERE], You may be looking for ways to resume some activities as safely as possible. However, COVID-19 remains contagious and shows no signs of disappearing. In fact, during the week of July 6 there were 58,000 new COVID cases per day diagnosed in the United States. [ONLY FOR ACKNOWLEDGMENT SUB-TREATMENT T2A] Black Americans and other minority groups are three times as likely to get and, when you account for age, four times as likely to die from COVID as white Americans. Without a safe and effective vaccine or therapy, our only option is to continue taking precautionary measures to protect ourselves, our communities, and the most vulnerable among us. While there is no way to ensure zero risk of infection from COVID-19, observing these three practices will help to protect you and others. First, continue to practice social distancing whenever possible: Try to stay outdoors, and to the maxi- mum extent possible, please stay 6 feet apart. If you must be indoors, use visual reminders—like signs, chair arrangements, markings on the ﬂoor, or arrows—to help remind you to keep your distance from others, and maintain physical barriers whenever possible. Second, continue to wash your hands often for at least 20 seconds with soap and water, especially before going out, and every time you return home. Third, wear a mask when in public at all times, especially when indoors or when it is difﬁcult to stay 6 feet away. The next video will tell you a bit more about masks. Video T3: Hello, I am doctor [YOUR LAST NAME HERE] from [YOUR INSTITUTIONAL AFFILIATION HERE], and I will tell you a bit more about masks. Wearing a mask is a key way to prevent the spread of COVID-19. You are not just protecting yourself but also your grandma and your community, just in case you have COVID-19 but don’t know it. Even if wearing a mask may sometimes put you in a difﬁcult situation, it is important to protect you and the community from COVID 19 disease. As medical professionals, I am committed to delivering the best care I can to every patient. My goal is to make sure that you and everyone you love survives this COVID-19 pandemic. Thank you for listening to these messages. A.3 Control Videos about non-COVID-19 Health Behaviors A full set of control videos includes C1 + C2 + C3 15 Video C1: Most adults need to sleep between 6 and 8 hours a night. Now, there are some people who get ﬁve hours a night and they are ﬁne, so there is some variation across people. But for most adults, we need 6 to 8 hours in order to function well the next day. If you feel sleep deprived you might not be able to function as well as you would normally like. It’s important to have something called sleep hygiene which is a routine you follow at bedtime and can help you fall asleep. Things that can disrupt sleep hygiene include caffeine or alcohol too close to bedtime. Eating late at night can also cause indigestion. So keep a routine and trying to get 6-8 hours is important. Video C2: Sugar is found in many different food items. Natural sugars are those that can be found in fruits, vegetables and dairy products like milk. Sugars like these that are natural are not really problematic because they are coming alongside lots of other vitamins and minerals. There are other sugars, though, that are processed and added to a food item. These are called additive sugars. A good rule of thumb is to eat foods with less than than 5g of sugar per serving. Avoid buying products where one of the ﬁrst ﬁve products is a sugar. And it can be better to buy an unsweetened product like an unsweetened cereal or oatmeal and then add a teaspoon of sugar to it if you need the sweetness than to buy a heavily sweetened product, like a sugar cereal which can have several teaspoons of sugar per serving. Video C3: New ﬁtness guidelines can be summed up as follows: just move and anything counts. Sneaking in a few minutes of physical activity throughout the day adds up in the long run. The guidelines are trying to make it easier for individuals to be ﬁt and drop the rule that activity must be in 10 minute blocks of time. In a nutshell, activity has beneﬁts even if it’s for a short amount of time. Taking the stairs instead of the elevator, parking your car far away from the entrance to a store or walking your dog around the block can all help you be ﬁt. The guidelines still call for at least 150 minutes a week of moderately intense aerobic exercise and two weekly sessions of muscle training activity, like lifting weights or yoga. 16 17 Figure 2: Fully extended randomization tree (2( (U (B (E (P (b( (E (6 ( ( ( ( (E ( (3 (1 ( ( (1( (1( ( (N (A (B( (b (E (e (A (b (R( ( (r (P( (v( (A( ( (a( ( ( (R (d (M( (m( (A( (a( (E( (C ( ( ( ( ( (C( (p( (l( (M

Journal

JAMA Network Open – American Medical Association

Published: Jul 14, 2021

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Effect of Physician-Delivered COVID-19 Public Health Messages and Messages Acknowledging Racial Inequity on Black and White Adults’ Knowledge, Beliefs, and Practices Related to COVID-19

Effect of Physician-Delivered COVID-19 Public Health Messages and Messages Acknowledging Racial Inequity on Black and White Adults’ Knowledge, Beliefs, and Practices Related to COVID-19

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Effect of Physician-Delivered COVID-19 Public Health Messages and Messages Acknowledging Racial Inequity on Black and White Adults’ Knowledge, Beliefs, and Practices Related to COVID-19

Effect of Physician-Delivered COVID-19 Public Health Messages and Messages Acknowledging Racial Inequity on Black and White Adults’ Knowledge, Beliefs, and Practices Related to COVID-19

References (20)

Abstract

Journal

Recommended Articles

There are no references for this article.

Our policy towards the use of cookies